Nothing verifies a published NWSync blob except a player's client #85

New Issue

2026-07-31T21:57:59Z

archvillainette commented

2026-07-31 21:57:59 +00:00

Problem

A published NWSync blob is verified by exactly one thing: a player's client, at download time, after the bytes have already crossed the internet. Nothing upstream of that ever checks that what is in the zone is what was meant to be there.

emit skips objects that already exist, and it decides "exists" with a 1-byte range GET (#55). Presence is treated as correctness. So an object that was written truncated — a killed CI job, a half-completed PUT, a storage hiccup — is skipped by every subsequent emit, forever. Re-running the backfill does not repair it, because the backfill also skips what is present.

The zone has no backup, on purpose, because it is reproducible (docs/runbooks/nwsync.md). That reasoning holds only if we can tell when it needs reproducing. Right now we cannot.

This surfaced during sow-tools#75: a client failed with checksum mismatch on file 017f6802…, and answering "is the zone actually corrupt?" took an ad-hoc script — parse NSYM, fetch each blob, unwrap NSYC, zstd-decode, SHA-1, compare. That script should not be ad-hoc. It is ~40 lines against code internal/nwsync already has.

In that instance the zone was clean (68/68 verified) and the failure was a transient truncated transfer. The next instance may not be, and the difference is not currently knowable without writing the script again.

What to build

crucible nwsync verify <manifest-sha1>, reading the same zone the client reads, credential-free:

fetch the merged manifest, parse NSYM v3;
for each entry, fetch data/sha1/<aa>/<bb>/<sha1>, unwrap the NSYC framing, zstd-decode, and check both that the decompressed length equals the entry's recorded size and that its SHA-1 equals the entry's SHA-1;
report per-blob failures as missing / malformed framing / size mismatch / hash mismatch, and exit non-zero if any;
--sample N for a cheap routine check, full sweep by default. A full sweep of the current manifest is 69,177 entries and ~15 GB, so sampling is what makes it runnable often.

Two properties matter:

Decompress and hash. Do not trust Content-Length. The failure mode being caught is a byte-correct-looking object whose contents are short, and a length check alone would pass a body the CDN happily serves.
Read through the pull zone, not the storage API. What matters is what a client gets, including any edge behaviour, not what the origin believes it holds.

Then

Give emit a --verify mode that hashes what it skips rather than trusting presence, for use in the backfill path.
The runbook's "When a sync fails" section currently has the operator checking the chain by hand; it should name this command.

Not this

Not a platform concern. ADR-0029 gives sow-platform exactly one NWSync responsibility — read a hash out of a catalog and render it into the server environment — and explicitly forbids it re-fetching manifests or probing the zone. This belongs where emit and assemble already live.

Refs sow-tools#54, sow-tools#55, sow-tools#75.

## Problem A published NWSync blob is verified by exactly one thing: a player's client, at download time, after the bytes have already crossed the internet. Nothing upstream of that ever checks that what is in the zone is what was meant to be there. `emit` skips objects that already exist, and it decides "exists" with a 1-byte range GET (#55). **Presence is treated as correctness.** So an object that was written truncated — a killed CI job, a half-completed PUT, a storage hiccup — is skipped by every subsequent `emit`, forever. Re-running the backfill does not repair it, because the backfill also skips what is present. The zone has no backup, on purpose, because it is reproducible (`docs/runbooks/nwsync.md`). That reasoning holds only if we can tell *when* it needs reproducing. Right now we cannot. This surfaced during sow-tools#75: a client failed with `checksum mismatch on file 017f6802…`, and answering "is the zone actually corrupt?" took an ad-hoc script — parse NSYM, fetch each blob, unwrap `NSYC`, zstd-decode, SHA-1, compare. That script should not be ad-hoc. It is ~40 lines against code `internal/nwsync` already has. In that instance the zone was clean (68/68 verified) and the failure was a transient truncated transfer. The next instance may not be, and the difference is not currently knowable without writing the script again. ## What to build `crucible nwsync verify <manifest-sha1>`, reading the same zone the client reads, credential-free: - fetch the merged manifest, parse NSYM v3; - for each entry, fetch `data/sha1/<aa>/<bb>/<sha1>`, unwrap the `NSYC` framing, zstd-decode, and check **both** that the decompressed length equals the entry's recorded size and that its SHA-1 equals the entry's SHA-1; - report per-blob failures as missing / malformed framing / size mismatch / hash mismatch, and exit non-zero if any; - `--sample N` for a cheap routine check, full sweep by default. A full sweep of the current manifest is 69,177 entries and ~15 GB, so sampling is what makes it runnable often. Two properties matter: - **Decompress and hash. Do not trust `Content-Length`.** The failure mode being caught is a byte-correct-looking object whose contents are short, and a length check alone would pass a body the CDN happily serves. - **Read through the pull zone, not the storage API.** What matters is what a client gets, including any edge behaviour, not what the origin believes it holds. ## Then - Give `emit` a `--verify` mode that hashes what it skips rather than trusting presence, for use in the backfill path. - The runbook's "When a sync fails" section currently has the operator checking the chain by hand; it should name this command. ## Not this Not a platform concern. ADR-0029 gives sow-platform exactly one NWSync responsibility — read a hash out of a catalog and render it into the server environment — and explicitly forbids it re-fetching manifests or probing the zone. This belongs where `emit` and `assemble` already live. Refs sow-tools#54, sow-tools#55, sow-tools#75.

archvillainette referenced this issue

2026-07-31 22:04:31 +00:00

Blobs under 256 bytes omit the zstd frame content size, and no client can sync past one #86

archvillainette referenced this issue

2026-07-31 22:04:45 +00:00

Acceptance gate: a real NWN client syncs from an assembled manifest on the Bunny zone #75

archvillainette referenced this issue from a commit

2026-07-31 22:27:39 +00:00

fix(nwsync): declare Frame_Content_Size, and verify what is published (#86, #85)

archvillainette referenced a pull request that will close this issue

2026-07-31 22:28:00 +00:00

fix(nwsync): declare Frame_Content_Size on every blob, and verify what is published #87