Replacing a blob does not purge the pull-zone cache, so verify can pass while clients still get the bad bytes #89

New Issue

2026-07-31T22:28:36Z

archvillainette commented

2026-07-31 22:28:36 +00:00

emit --verify (#85, PR #87) replaces a bad blob in the sow-sync storage zone. Players read the nwsync.westgate.pw pull zone in front of it (pull zone 6046370, sow-platform ADR-0029). Nothing purges the edge after a replacement.

A blob's name is its content hash, so under normal operation an object is written once and never changes — which is exactly why caching it forever is correct, and why nobody needed a purge until now. The #86 repair breaks that assumption for the first time: the same key must now serve different bytes than it did before.

The trap this sets

crucible nwsync verify reads through the pull zone, on purpose — what matters is what a client gets. That is the right choice and it makes this failure mode confusing rather than obvious:

Verify against a warm edge PoP right after a repair: still reports the old, bad blob. Looks like the repair silently failed.
Verify against a cold PoP: passes. Two operators get opposite answers for the same zone, and the honest answer is "it depends which PoP you hit".

So the ambiguity is not cosmetic. It makes verify's verdict unreliable during precisely the window where the verdict matters most.

What to decide

Cheapest thing that resolves the ambiguity, roughly in order of laziness:

Purge by hand after the backfill and move on. The Bunny pull zone has a purge endpoint. If #86's repair is a one-off, this may be all that is ever needed — one documented step in the backfill runbook.
Purge the affected keys from emit --verify. Only the keys it actually replaced, which is normally none. It needs a pull-zone API credential that emit does not have today, which is the real cost.
Leave it and wait out the TTL, with the TTL written down somewhere so an operator knows how long "wait" is.

Whichever way it goes, verify's exit code needs to be trustworthy or documented as not-yet-trustworthy after a repair. A check that answers differently per PoP and does not say so is worse than no check.

Also worth confirming while looking: whether the pull zone is caching 404s. A blob fetched before it was published, then negative-cached, is the same class of stale answer.

Refs #85, #86, #88, #75, sow-platform ADR-0029.

`emit --verify` (#85, PR #87) replaces a bad blob in the `sow-sync` **storage** zone. Players read the `nwsync.westgate.pw` **pull** zone in front of it (pull zone `6046370`, sow-platform ADR-0029). Nothing purges the edge after a replacement. A blob's name is its content hash, so under normal operation an object is written once and never changes — which is exactly why caching it forever is correct, and why nobody needed a purge until now. The #86 repair breaks that assumption for the first time: the same key must now serve different bytes than it did before. ## The trap this sets `crucible nwsync verify` reads *through* the pull zone, on purpose — what matters is what a client gets. That is the right choice and it makes this failure mode confusing rather than obvious: - Verify against a warm edge PoP right after a repair: still reports the old, bad blob. Looks like the repair silently failed. - Verify against a cold PoP: passes. Two operators get opposite answers for the same zone, and the honest answer is "it depends which PoP you hit". So the ambiguity is not cosmetic. It makes verify's verdict unreliable during precisely the window where the verdict matters most. ## What to decide Cheapest thing that resolves the ambiguity, roughly in order of laziness: 1. **Purge by hand after the backfill and move on.** The Bunny pull zone has a purge endpoint. If #86's repair is a one-off, this may be all that is ever needed — one documented step in the backfill runbook. 2. **Purge the affected keys from `emit --verify`.** Only the keys it actually replaced, which is normally none. It needs a pull-zone API credential that `emit` does not have today, which is the real cost. 3. **Leave it and wait out the TTL**, with the TTL written down somewhere so an operator knows how long "wait" is. Whichever way it goes, verify's exit code needs to be trustworthy or documented as not-yet-trustworthy after a repair. A check that answers differently per PoP and does not say so is worse than no check. Also worth confirming while looking: whether the pull zone is caching 404s. A blob fetched before it was published, then negative-cached, is the same class of stale answer. Refs #85, #86, #88, #75, sow-platform ADR-0029.

archvillainette added a new dependency 2026-07-31 22:29:09 +00:00

#75 Acceptance gate: a real NWN client syncs from an assembled manifest on the Bunny zone

archvillainette referenced this issue from a commit

2026-07-31 23:04:34 +00:00

docs(nwsync): purge the edge after a repair, or verify answers per PoP

archvillainette referenced a pull request that will close this issue

2026-07-31 23:05:06 +00:00

Purge the edge after a repair, or verify answers per PoP (#89) #90

archvillainette referenced this issue

2026-07-31 23:05:35 +00:00

Run the verifying backfill: every blob published by emitter 1 is still unreadable by a client #88

archvillainette closed this issue