Run the verifying backfill: every blob published by emitter 1 is still unreadable by a client #88

New Issue

2026-07-31T22:28:18Z

archvillainette commented

2026-07-31 22:28:18 +00:00

#86 fixed the encoder and #85 built the repair path (PR #87). Neither has been applied. The mechanism exists; nothing has invoked it.

Every blob under 256 bytes already in the sow-sync zone was written by emitter version 1 and still omits its zstd Frame_Content_Size. emit skips an object that already exists on the strength of a 1-byte range GET, so re-running the ordinary backfill changes nothing at all — it skips exactly the blobs that need replacing. Roughly 6% of blobs are affected, and one is enough to stop a sync dead, so the live manifest is still unsyncable until this runs.

What to do

Re-emit every published artifact with crucible nwsync emit --verify. That reads each existing blob back, hashes it against its own name, asserts the frame declares a content size, and re-uploads the ones that fail. It costs a full GET per existing blob, which is why it is not the default.
Re-assemble. The old per-artifact indexes carry emitter_version: 1 and assemble now refuses to merge them, so this is not optional — and the merged manifest sha1 will change, which means the channel catalog and NWN_NWSYNCHASH move with it.
crucible nwsync verify <new-manifest-sha1> — a full sweep, not a sample, before declaring anything fixed.

Watch for

Step 2 changes the manifest hash. Whatever pins it has to follow, or the server points at a manifest that no longer exists.
The edge cache. Replacing a blob in the storage zone does not purge the pull zone, so verify may pass against the origin while clients still receive the old bytes. Tracked separately.
Do not read a passing zstd -dc as proof of anything. It streams a frame with no declared size happily; that is precisely how this defect survived an audit.

Blocks #75. Refs #85, #86, sow-platform#79.

#86 fixed the encoder and #85 built the repair path (PR #87). Neither has been *applied*. The mechanism exists; nothing has invoked it. Every blob under 256 bytes already in the `sow-sync` zone was written by emitter version 1 and still omits its zstd `Frame_Content_Size`. `emit` skips an object that already exists on the strength of a 1-byte range GET, so re-running the ordinary backfill changes nothing at all — it skips exactly the blobs that need replacing. Roughly 6% of blobs are affected, and one is enough to stop a sync dead, so the live manifest is still unsyncable until this runs. ## What to do 1. Re-emit every published artifact with `crucible nwsync emit --verify`. That reads each existing blob back, hashes it against its own name, asserts the frame declares a content size, and re-uploads the ones that fail. It costs a full GET per existing blob, which is why it is not the default. 2. Re-assemble. The old per-artifact indexes carry `emitter_version: 1` and `assemble` now refuses to merge them, so this is not optional — and the merged manifest sha1 will change, which means the channel catalog and `NWN_NWSYNCHASH` move with it. 3. `crucible nwsync verify <new-manifest-sha1>` — a full sweep, not a sample, before declaring anything fixed. ## Watch for - **Step 2 changes the manifest hash.** Whatever pins it has to follow, or the server points at a manifest that no longer exists. - **The edge cache.** Replacing a blob in the storage zone does not purge the pull zone, so verify may pass against the origin while clients still receive the old bytes. Tracked separately. - Do not read a passing `zstd -dc` as proof of anything. It streams a frame with no declared size happily; that is precisely how this defect survived an audit. Blocks #75. Refs #85, #86, sow-platform#79.

archvillainette referenced this issue

2026-07-31 22:28:36 +00:00

Replacing a blob does not purge the pull-zone cache, so verify can pass while clients still get the bad bytes #89

archvillainette added a new dependency 2026-07-31 22:29:09 +00:00

#75 Acceptance gate: a real NWN client syncs from an assembled manifest on the Bunny zone

archvillainette commented

2026-07-31 23:05:35 +00:00

Two corrections to the plan above, before anyone runs it.

Step 2 is not needed, and the manifest sha1 does not move. The merged manifest hashes NSYM bytes, and an NSYM entry is (uncompressed sha1, uncompressed size, resref, restype) — emitResources fills it from sha1.Sum(payload) before compressBlob is ever called, so blob framing appears nowhere in that chain. Re-emitting under emitter 2 rewrites every blob's bytes and leaves every index's bytes identical, so Assemble merges to the same sha1. Nothing re-pins: the channel catalog, releases/module/<tag>/nwsync.json and NWN_NWSYNCHASH all stay as they are, and a release that already has a manifest needs no re-assemble.

What the backfill does change is the per-artifact sidecars, emitter_version 1 to 2. That matters for the next release rather than this one — Assemble refuses to merge a v1 index, so until the backfill has run, no new module release can assemble at all. The stale emitter_version: 1 in the already-published merged sidecar is inert; assemble only checks the indexes it reads.

Step 3 runs after the purge, not before it. verify reads the pull zone on purpose, so between a repair and a purge it reports whatever the PoP that answered happens to hold. #89 is now decided: purge the whole zone, one call. Confirmed while there — the zone does not negative-cache. A missing key answers 404 with cache-control: no-cache and cdn-cache: MISS, still MISS on an immediate retry.

Mechanism, now open for review. An ordinary backfill would have skipped exactly the blobs this exists to replace, so both owning repos take NWSYNC_VERIFY and expose it as a verify input on backfill-nwsync:

sow-topdata#191, sow-assets-manifest#63
#90 (the #89 decision), sow-platform#138 (the procedure end to end)

Order to run, once those merge: topdata backfill with verify (minutes) -> haks backfill with verify (hours) -> purge pull zone 6046370 -> crucible nwsync verify <sha1>, full sweep, no --sample.

Two corrections to the plan above, before anyone runs it. **Step 2 is not needed, and the manifest sha1 does not move.** The merged manifest hashes NSYM bytes, and an NSYM entry is `(uncompressed sha1, uncompressed size, resref, restype)` — `emitResources` fills it from `sha1.Sum(payload)` before `compressBlob` is ever called, so blob framing appears nowhere in that chain. Re-emitting under emitter 2 rewrites every blob's *bytes* and leaves every index's bytes identical, so `Assemble` merges to the same sha1. Nothing re-pins: the channel catalog, `releases/module/<tag>/nwsync.json` and `NWN_NWSYNCHASH` all stay as they are, and a release that already has a manifest needs no re-assemble. What the backfill does change is the per-artifact sidecars, `emitter_version` 1 to 2. That matters for the *next* release rather than this one — `Assemble` refuses to merge a v1 index, so until the backfill has run, no new module release can assemble at all. The stale `emitter_version: 1` in the already-published *merged* sidecar is inert; assemble only checks the indexes it reads. **Step 3 runs after the purge, not before it.** `verify` reads the pull zone on purpose, so between a repair and a purge it reports whatever the PoP that answered happens to hold. #89 is now decided: purge the whole zone, one call. Confirmed while there — the zone does **not** negative-cache. A missing key answers 404 with `cache-control: no-cache` and `cdn-cache: MISS`, still MISS on an immediate retry. **Mechanism, now open for review.** An ordinary backfill would have skipped exactly the blobs this exists to replace, so both owning repos take `NWSYNC_VERIFY` and expose it as a `verify` input on `backfill-nwsync`: - sow-topdata#191, sow-assets-manifest#63 - #90 (the #89 decision), sow-platform#138 (the procedure end to end) Order to run, once those merge: topdata backfill with verify (minutes) -> haks backfill with verify (hours) -> purge pull zone 6046370 -> `crucible nwsync verify <sha1>`, full sweep, no `--sample`.

archvillainette referenced this issue from a commit

2026-08-01 06:50:59 +00:00

docs(nwsync): verify is what tells you which keys to purge

archvillainette referenced this issue

2026-08-01 06:51:18 +00:00

verify is what tells you which keys to purge #91

archvillainette commented

2026-08-01 07:39:35 +00:00

Run, and done

1. Re-emit with --verify. topdata first (minutes), then haks (hours). 2,603 blobs rewritten at the origin under emitter 2.

2. Re-assemble — not needed. The 01:05 correction held: the merged manifest hashes NSYM bytes, so re-emitting changed only blob bytes and the manifest sha1 did not move. The channel catalog, releases/module/<tag>/nwsync.json and NWN_NWSYNCHASH all stayed as they were. Nothing was re-pinned.

3. Purge, then verify — and the purge got much smaller. #90's whole-zone purge turned out to be unnecessary. verify reads the edge, so the run straight after the repair names the stale keys itself: 8 blobs, all of them ones the failed player sync had pulled ninety minutes earlier. The edge only caches what someone fetched, so purging the whole zone would have cooled 69,169 objects to fix 8. Purged those eight. Corrections landed in #91 here and sow-platform#139 in the runbook.

Full sweep afterwards, no --sample:

verified 69177 of 69177 blobs behind 72544 resources: 0 failures, 14887519535 bytes checked

A retail client then synced to completion and entered the module. Closing; the gate itself is #75.

## Run, and done **1. Re-emit with `--verify`.** topdata first (minutes), then haks (hours). **2,603 blobs rewritten** at the origin under emitter 2. **2. Re-assemble — not needed.** The 01:05 correction held: the merged manifest hashes NSYM bytes, so re-emitting changed only blob bytes and the manifest sha1 did not move. The channel catalog, `releases/module/<tag>/nwsync.json` and `NWN_NWSYNCHASH` all stayed as they were. Nothing was re-pinned. **3. Purge, then verify — and the purge got much smaller.** #90's whole-zone purge turned out to be unnecessary. `verify` reads the edge, so the run straight after the repair names the stale keys itself: **8 blobs**, all of them ones the failed player sync had pulled ninety minutes earlier. The edge only caches what someone fetched, so purging the whole zone would have cooled 69,169 objects to fix 8. Purged those eight. Corrections landed in #91 here and sow-platform#139 in the runbook. **Full sweep afterwards, no `--sample`:** ``` verified 69177 of 69177 blobs behind 72544 resources: 0 failures, 14887519535 bytes checked ``` A retail client then synced to completion and entered the module. Closing; the gate itself is #75.

archvillainette closed this issue

2026-08-01 07:39:36 +00:00

archvillainette referenced this issue

2026-08-01 07:39:54 +00:00

Acceptance gate: a real NWN client syncs from an assembled manifest on the Bunny zone #75

archvillainette referenced this issue

2026-08-01 07:40:10 +00:00

Map: NWSync publishing — emit at artifact birth, assemble at module release #54

archvillainette referenced this issue

2026-08-01 07:42:47 +00:00

Spot-check hak precedence in-world: the 13 duplicate resrefs and sow_top's 2DAs #92

archvillainette referenced this issue from a commit