ShadowsOverWestgate/sow-tools
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 38 Activity

Acceptance gate: a real NWN client syncs from an assembled manifest on the Bunny zone #75

New Issue
Closed
opened 2026-07-30 22:01:20 +00:00 by archvillainette · 9 comments
archvillainette commented 2026-07-30 22:01:20 +00:00
Owner
Copy Link
Copy Source

Part of #54.

Split out of #53, which is otherwise complete.

Everything upstream of this gate has landed and is verified:

  • crucible nwsync emit / assemble shipped in PR #71 (a131b25), key-addressed CLI in PR #73 (f9051a2).
  • Conformance against upstream nwn_nwsync_write 2.1.2 passed in #59 — manifest byte-identical, 2,296 blobs with matching names and paths, payloads identical after decompression.

What is still unproven is the only thing conformance cannot prove: that a real
NWN:EE client can actually sync from what we publish. Upstream never runs in
production for us, and a shared misreading of the serving side — zone layout,
content types, the URL the client is handed — would pass every check we have.

Done means

  • A full corpus emit (all 11 haks plus the TLK) and one assemble run against a
    real channel group id (current=1 or testing=2), published to the Bunny zone.
  • A retail NWN:EE client pointed at that manifest completes a sync and enters
    the module with a character, with no missing or wrong assets.
  • The 13 known cross-hak duplicate resrefs (bloodstain1.tga and friends, in
    sow_core_01 and sow_over_01) and sow_top's 2DAs are spot-checked in the
    synced client — earliest-hak-wins is the rule that ships the wrong texture
    silently if it is backwards.
  • Whatever the zone needed to serve this (paths, headers, cache rules) is
    recorded, since none of it is in Crucible.

Notes

  • Needs a published zone and a game client, so it cannot be closed from the
    toolchain repo's test suite. It stays a manual run.
  • If the sync fails, the fix is likely on the serving side, not in
    internal/nwsync — the bytes are already proven against upstream.
Part of #54. Split out of #53, which is otherwise complete. Everything upstream of this gate has landed and is verified: - `crucible nwsync emit` / `assemble` shipped in PR #71 (`a131b25`), key-addressed CLI in PR #73 (`f9051a2`). - Conformance against upstream `nwn_nwsync_write` 2.1.2 passed in #59 — manifest byte-identical, 2,296 blobs with matching names and paths, payloads identical after decompression. What is still unproven is the only thing conformance cannot prove: that a real NWN:EE client can actually sync from what we publish. Upstream never runs in production for us, and a shared misreading of the *serving* side — zone layout, content types, the URL the client is handed — would pass every check we have. ## Done means - A full corpus emit (all 11 haks plus the TLK) and one `assemble` run against a real channel group id (current=1 or testing=2), published to the Bunny zone. - A retail NWN:EE client pointed at that manifest completes a sync and enters the module with a character, with no missing or wrong assets. - The 13 known cross-hak duplicate resrefs (`bloodstain1.tga` and friends, in `sow_core_01` and `sow_over_01`) and `sow_top`'s 2DAs are spot-checked in the synced client — earliest-hak-wins is the rule that ships the wrong texture silently if it is backwards. - Whatever the zone needed to serve this (paths, headers, cache rules) is recorded, since none of it is in Crucible. ## Notes - Needs a published zone and a game client, so it cannot be closed from the toolchain repo's test suite. It stays a manual run. - If the sync fails, the fix is likely on the serving side, not in `internal/nwsync` — the bytes are already proven against upstream.
archvillainette commented 2026-07-31 15:36:39 +00:00
Author
Owner
Copy Link
Copy Source

The zone side is ready; the gate is now blocked on infrastructure, not on tooling

Every hak named by a channel pointer now has its blobs and its .nsym index in the sow-sync zone, backfilled without re-releasing anything. Emit streams and runs in parallel, and the size guard that used to skip the big haks is gone (see the #54 update for the detail). So "a full corpus emit published to the Bunny zone" — the first half of Done means — is satisfied for the haks. What remains is the TLK, one assemble run against a real channel group id, and the client run itself.

Three prerequisites, none of them in a repo:

1. netcup-game must be hosting the testing server. The gate is "enters the module with a character", which needs a running server, not just a published manifest. No server, no gate.

2. A testing tag cut for every repo that has not had one. All three testing pointers are stale right now:

stream current testing
haks v0.2.0 v0.1.7
module v0.2.0 v0.1.3
topdata v0.2.2 v0.1.11

Every one of those testing values is an autoseed, not a deliberate release candidate — this is exactly the failure sow-assets-manifest#47 describes for the haks stream, and the module and topdata streams have it too. It matters specifically for this gate: Done means says the assemble run targets "a real channel group id (current=1 or testing=2)". Assembling against testing today would assemble v0.1.7 haks — content nobody chose, two releases behind, and not what the acceptance run is meant to prove. A real -rcN tag has to move each pointer first.

3. A live, joinable session. The spot-checks in Done means are visual and in-world: the 13 cross-hak duplicate resrefs shared by sow_core_01 and sow_over_01 (bloodstain1.tga and friends) and sow_top's 2DAs. Earliest-hak-wins ships the wrong texture silently if the rule is backwards, so it cannot be checked from the zone — someone has to log in and look.

Suggested order

  1. Cut real release candidates so testing names something deliberate on all three streams. sow-assets-manifest#47 covers the haks stream and names two defects a first real RC would walk into; module and topdata need the same treatment.
  2. Emit the TLK, then run assemble against channel group 2 (testing).
  3. Stand up the testing server on netcup-game against that manifest.
  4. Run the client, sync, enter with a character, spot-check the duplicates and the 2DAs.
  5. Record whatever the zone needed to serve it — paths, headers, cache rules. None of that lives in Crucible, and it is the part most likely to be the failure if the sync breaks.

Steps 1 to 3 are prerequisites owned outside this repo. Step 4 is the gate. If the sync fails, the bytes are already proven byte-identical against upstream nwn_nwsync_write 2.1.2, so look at the serving side first.

## The zone side is ready; the gate is now blocked on infrastructure, not on tooling Every hak named by a channel pointer now has its blobs and its `.nsym` index in the `sow-sync` zone, backfilled without re-releasing anything. Emit streams and runs in parallel, and the size guard that used to skip the big haks is gone (see the #54 update for the detail). So "a full corpus emit published to the Bunny zone" — the first half of **Done means** — is satisfied for the haks. What remains is the TLK, one `assemble` run against a real channel group id, and the client run itself. Three prerequisites, none of them in a repo: **1. netcup-game must be hosting the testing server.** The gate is "enters the module with a character", which needs a running server, not just a published manifest. No server, no gate. **2. A testing tag cut for every repo that has not had one.** All three testing pointers are stale right now: | stream | current | testing | |---|---|---| | haks | `v0.2.0` | `v0.1.7` | | module | `v0.2.0` | `v0.1.3` | | topdata | `v0.2.2` | `v0.1.11` | Every one of those `testing` values is an autoseed, not a deliberate release candidate — this is exactly the failure sow-assets-manifest#47 describes for the haks stream, and the module and topdata streams have it too. It matters specifically for this gate: **Done means** says the assemble run targets "a real channel group id (current=1 or testing=2)". Assembling against `testing` today would assemble `v0.1.7` haks — content nobody chose, two releases behind, and not what the acceptance run is meant to prove. A real `-rcN` tag has to move each pointer first. **3. A live, joinable session.** The spot-checks in **Done means** are visual and in-world: the 13 cross-hak duplicate resrefs shared by `sow_core_01` and `sow_over_01` (`bloodstain1.tga` and friends) and `sow_top`'s 2DAs. Earliest-hak-wins ships the wrong texture *silently* if the rule is backwards, so it cannot be checked from the zone — someone has to log in and look. ## Suggested order 1. Cut real release candidates so `testing` names something deliberate on all three streams. sow-assets-manifest#47 covers the haks stream and names two defects a first real RC would walk into; module and topdata need the same treatment. 2. Emit the TLK, then run `assemble` against channel group 2 (testing). 3. Stand up the testing server on netcup-game against that manifest. 4. Run the client, sync, enter with a character, spot-check the duplicates and the 2DAs. 5. Record whatever the zone needed to serve it — paths, headers, cache rules. None of that lives in Crucible, and it is the part most likely to be the failure if the sync breaks. Steps 1 to 3 are prerequisites owned outside this repo. Step 4 is the gate. If the sync fails, the bytes are already proven byte-identical against upstream `nwn_nwsync_write` 2.1.2, so look at the serving side first.
archvillainette commented 2026-07-31 15:37:59 +00:00
Author
Owner
Copy Link
Copy Source

Two corrections to the comment above.

The testing-channel work is already ticketed on all three streams, so prerequisite 2 needs no new issues — just the tags:

  • haks: sow-assets-manifest#47
  • module: sow-module#49
  • topdata: sow-topdata#151

All three are open and labeled ready-for-agent.

The TLK is already emitted. I listed it as remaining; it is not. sow-topdata#178 emitted blobs and a NSYM index for sow_top.hak and sow_tlk.tlk, and sow-topdata#182 backfilled the already-published topdata artifacts. Both closed.

So the corpus side of Done means is complete — every hak plus the TLK has blobs and an index in the zone. What is left on the toolchain side is only the assemble run against a real channel group id, and that wants the testing pointers moved first.

Two corrections to the comment above. **The testing-channel work is already ticketed on all three streams**, so prerequisite 2 needs no new issues — just the tags: - haks: sow-assets-manifest#47 - module: sow-module#49 - topdata: sow-topdata#151 All three are open and labeled `ready-for-agent`. **The TLK is already emitted.** I listed it as remaining; it is not. sow-topdata#178 emitted blobs and a NSYM index for `sow_top.hak` and `sow_tlk.tlk`, and sow-topdata#182 backfilled the already-published topdata artifacts. Both closed. So the corpus side of **Done means** is complete — every hak plus the TLK has blobs and an index in the zone. What is left on the toolchain side is only the `assemble` run against a real channel group id, and that wants the testing pointers moved first.
archvillainette added the wayfinder:task label 2026-07-31 16:13:10 +00:00
archvillainette self-assigned this 2026-07-31 16:13:11 +00:00
archvillainette commented 2026-07-31 16:18:39 +00:00
Author
Owner
Copy Link
Copy Source

Prerequisite 1 has moved, and this ticket now carries its blockers as native dependencies rather than as prose.

The three testing-channel PRs are merged — sow-assets-manifest#60 (ShadowsOverWestgate/sow-assets-manifest#60 (comment)), sow-module#55 (ShadowsOverWestgate/sow-module#55 (comment)), sow-topdata#187 (ShadowsOverWestgate/sow-topdata#187 (comment)) — so each stream's release-candidate path is ready. The tags themselves are still not cut, and the ordering is load-bearing: haks first (sow-assets-manifest#47, which I reopened after it was auto-closed by a Closes line although the tag is its actual deliverable), then topdata (sow-topdata#151), then module (sow-module#49).

Two blockers were not written down anywhere before today, and both are now dependencies of this issue:

  • sow-module#51 — nothing assembles a manifest. grep -rn nwsync sow-module/scripts sow-module/.gitea returns nothing; sow-tools#65 closed reading "all three repos wired" but only the two emitters landed. Without this there is no manifest to sync from at all.
  • sow-platform#128 — the serving side still points at a directory on the game host. runtime/nwn.nix:27-52 runs a Caddy file-server over ${s.stateRoot}/nwn/nwsync and hands the server NWN_NWSYNCURL = https://${s.nwsyncHost}, routed to it. On netcup-game that directory is empty. This is the failure this ticket predicted in as many words, and it contradicts ADR-0020.

So Done means now reads: corpus emitted (done), the three -rcN tags cut, sow-module#51 built, sow-platform#128 built, then assemble against group 2 and the client run. The client run is still the gate and still manual.

Prerequisite 1 has moved, and this ticket now carries its blockers as native dependencies rather than as prose. The three testing-channel PRs are merged — sow-assets-manifest#60 (https://git.westgate.pw/ShadowsOverWestgate/sow-assets-manifest/pulls/60#issuecomment-8580), sow-module#55 (https://git.westgate.pw/ShadowsOverWestgate/sow-module/pulls/55#issuecomment-8583), sow-topdata#187 (https://git.westgate.pw/ShadowsOverWestgate/sow-topdata/pulls/187#issuecomment-8591) — so each stream's release-candidate path is ready. The tags themselves are still not cut, and the ordering is load-bearing: haks first (sow-assets-manifest#47, which I reopened after it was auto-closed by a `Closes` line although the tag is its actual deliverable), then topdata (sow-topdata#151), then module (sow-module#49). Two blockers were not written down anywhere before today, and both are now dependencies of this issue: - **sow-module#51** — nothing assembles a manifest. `grep -rn nwsync sow-module/scripts sow-module/.gitea` returns nothing; sow-tools#65 closed reading "all three repos wired" but only the two emitters landed. Without this there is no manifest to sync from at all. - **sow-platform#128** — the serving side still points at a directory on the game host. `runtime/nwn.nix:27-52` runs a Caddy file-server over `${s.stateRoot}/nwn/nwsync` and hands the server `NWN_NWSYNCURL = https://${s.nwsyncHost}`, routed to it. On `netcup-game` that directory is empty. This is the failure this ticket predicted in as many words, and it contradicts ADR-0020. So **Done means** now reads: corpus emitted (done), the three `-rcN` tags cut, sow-module#51 built, sow-platform#128 built, then assemble against group 2 and the client run. The client run is still the gate and still manual.
archvillainette removed their assignment 2026-07-31 16:18:56 +00:00
archvillainette commented 2026-07-31 19:00:30 +00:00
Author
Owner
Copy Link
Copy Source

Platform's blocker is built; the serving side is now written down

sow-platform#128 (PR ShadowsOverWestgate/sow-platform#129) is the platform third of what this gate was waiting on.

"The URL the client is handed" is now real. Until this change the game server was still configured to sync from a caddy file-server over a directory on the host — a directory that has been empty since every NWSync byte moved to the sow-sync zone. That is exactly the "shared misreading of the serving side" this issue predicted: it passes every check we have and fails only in front of a tester. NWN_NWSYNCURL now names the Bunny pull zone, NWN_NWSYNCHASH carries the assembled manifest's SHA-1, and the build refuses an empty hash.

"Whatever the zone needed to serve this ... is recorded" — that bullet is answered by docs/runbooks/nwsync.md in sow-platform, written from what the zone actually returns today rather than from intent:

  • Three path shapes: data/sha1/<aa>/<bb>/<sha1> blobs, artifacts/…/<sha256>.nsym + .json per-artifact indexes beside their depot key, manifests/<sha1> + .json for the assembled manifest.
  • Content types as Bunny types them by extension: application/octet-stream for blobs, indexes and manifests; application/json for sidecars. Nothing in the path depends on them being anything else.
  • cache-control: public, max-age=2592000, accept-ranges: bytes. Safe because every name in the zone is a content hash, so nothing to purge on release — and a warning never to add a mutable pointer file there.
  • The security toggles that a naive health check trips over: root-path blocking (so / answers 403 on a healthy zone), POST blocking, Force SSL.
  • A failure walk for "a client cannot sync", and the credential-free check that the pull zone (6046370) fronts sow-sync and not the depot — a depot key 404s there, an NWSync index serves 200.

What this gate still waits on: sow-module#51, so something actually assembles a manifest, and a real -rcN tag on each of the three streams. Platform's side no longer blocks it. When the manual run happens, anything the client needed that the runbook does not say is a bug in the runbook — please add it there rather than in a comment.

## Platform's blocker is built; the serving side is now written down sow-platform#128 (PR https://git.westgate.pw/ShadowsOverWestgate/sow-platform/pulls/129) is the platform third of what this gate was waiting on. **"The URL the client is handed" is now real.** Until this change the game server was still configured to sync from a `caddy` file-server over a directory on the host — a directory that has been empty since every NWSync byte moved to the `sow-sync` zone. That is exactly the "shared misreading of the serving side" this issue predicted: it passes every check we have and fails only in front of a tester. `NWN_NWSYNCURL` now names the Bunny pull zone, `NWN_NWSYNCHASH` carries the assembled manifest's SHA-1, and the build refuses an empty hash. **"Whatever the zone needed to serve this ... is recorded"** — that bullet is answered by `docs/runbooks/nwsync.md` in sow-platform, written from what the zone actually returns today rather than from intent: - Three path shapes: `data/sha1/<aa>/<bb>/<sha1>` blobs, `artifacts/…/<sha256>.nsym` + `.json` per-artifact indexes beside their depot key, `manifests/<sha1>` + `.json` for the assembled manifest. - Content types as Bunny types them by extension: `application/octet-stream` for blobs, indexes and manifests; `application/json` for sidecars. Nothing in the path depends on them being anything else. - `cache-control: public, max-age=2592000`, `accept-ranges: bytes`. Safe because every name in the zone is a content hash, so nothing to purge on release — and a warning never to add a mutable pointer file there. - The security toggles that a naive health check trips over: root-path blocking (so `/` answers **403** on a healthy zone), POST blocking, Force SSL. - A failure walk for "a client cannot sync", and the credential-free check that the pull zone (`6046370`) fronts `sow-sync` and not the depot — a depot key 404s there, an NWSync index serves 200. **What this gate still waits on:** sow-module#51, so something actually assembles a manifest, and a real `-rcN` tag on each of the three streams. Platform's side no longer blocks it. When the manual run happens, anything the client needed that the runbook does not say is a bug in the runbook — please add it there rather than in a comment.
archvillainette commented 2026-07-31 19:39:50 +00:00
Author
Owner
Copy Link
Copy Source

One of the two blockers is gone. sow-module#51 merged (PR #57) and sow-module#49 cut v0.2.1-rc1, so a manifest assembled by sow-module release CI now exists on the zone:

manifest_sha1  061d23b96d7157b66d24993ece890d1ac6af982c
group_id       2 (testing)
72,544 resources, 15.4 GB, includes_module_contents: false

Reachable at https://nwsync.westgate.pw/manifests/061d23b9… (200) with its sidecar at …​.json (200). Sampled blobs across the sorted hash range all resolve under data/sha1/<xx>/<yy>/<sha1>. Recorded for consumers at releases/module/v0.2.1-rc1/nwsync.json on the depot.

That is the map's destination minus the client: assembled by module CI, served from Bunny, no bulk data on ovh-main.

Remaining blocker is sow-platform#79 alone — nothing sets sow.services.nwn.nwsyncManifest yet, so sow.services.nwn.enable still hard-fails the NixOS build per ADR-0029. Once #79's resolver reads the sidecar above, this gate can run.

Content pairing for the acceptance run, if it helps whoever picks it up: module v0.2.1-rc1 ← haks v0.2.1-rc1 + topdata v0.2.3-rc2, all three testing.

One of the two blockers is gone. sow-module#51 merged (PR #57) and sow-module#49 cut `v0.2.1-rc1`, so a manifest assembled by **sow-module release CI** now exists on the zone: ``` manifest_sha1 061d23b96d7157b66d24993ece890d1ac6af982c group_id 2 (testing) 72,544 resources, 15.4 GB, includes_module_contents: false ``` Reachable at `https://nwsync.westgate.pw/manifests/061d23b9…` (200) with its sidecar at `…​.json` (200). Sampled blobs across the sorted hash range all resolve under `data/sha1/<xx>/<yy>/<sha1>`. Recorded for consumers at `releases/module/v0.2.1-rc1/nwsync.json` on the depot. That is the map's destination minus the client: assembled by module CI, served from Bunny, no bulk data on ovh-main. Remaining blocker is **sow-platform#79** alone — nothing sets `sow.services.nwn.nwsyncManifest` yet, so `sow.services.nwn.enable` still hard-fails the NixOS build per ADR-0029. Once #79's resolver reads the sidecar above, this gate can run. Content pairing for the acceptance run, if it helps whoever picks it up: module `v0.2.1-rc1` ← haks `v0.2.1-rc1` + topdata `v0.2.3-rc2`, all three `testing`.
archvillainette commented 2026-07-31 21:57:27 +00:00
Author
Owner
Copy Link
Copy Source

First real client sync against the live zone — most of the way, then a transient failure

Ran this gate today from the testing target on netcup-game (sow-platform#79). Recording it here because this ticket is the gate.

What worked, end to end and unassisted:

  • The client found the server on the public list, saw it needed content, and resolved NWN_NWSYNCHASH → nwsync.westgate.pw.
  • It read the merged manifest and computed a delta: "You are missing 2.3 GB" against a 15.4 GB manifest, so the shared-repository dedupe is doing what #84 predicted.
  • It downloaded thousands of blobs before stopping with 8593 to go.

What failed:

FAILED
Download corrupted; checksum mismatch on file 017f680287c9f1617dd92d7426f51d29577ec38f
(potential compression error: ). This is likely a server-side repository issue.

It is not a repository issue. Verified that exact object by hand:

  • manifest entry: sha1 017f6802…, size 230, resref blood_fore, restype 2072
  • stored blob: NSYC v3, algorithm 2, declared size 230, 203 bytes on the wire
  • decompresses to exactly 230 bytes, and sha1(uncompressed) == the filename

Then audited the zone more broadly: 68 blobs verified end to end, 0 corrupt, 0 missing — 60 sampled at random from the manifest plus the 8 largest entries (14 MB uncompressed each, ~1.8 MB compressed). Every one decompressed to its declared size with a matching SHA-1.

Also confirmed the edge is not mangling anything: identical bytes with Accept-Encoding: gzip (no content-encoding applied), with a NWN-ish User-Agent, and a correct 206 with a proper Content-Range for a range request. No rate limiting is configured on the zone.

Conclusion: a transient truncated transfer, not stored corruption. The tell is in the screenshot — 58 KB/s, far below what the CDN serves, with many parallel fetches in flight. A stalled connection yields a short body, and a short zstd frame is exactly "potential compression error".

So the gate is nearly met: the manifest, the delta calculation, the URL handshake and the blob format all work against a real client. What is unproven is a sync running to completion. Retrying.

Filed separately: the client's checksum is currently the only thing that ever verifies a published blob, which is a thin place to find out.

## First real client sync against the live zone — most of the way, then a transient failure Ran this gate today from the `testing` target on `netcup-game` (sow-platform#79). Recording it here because this ticket is the gate. **What worked, end to end and unassisted:** - The client found the server on the public list, saw it needed content, and resolved `NWN_NWSYNCHASH` → `nwsync.westgate.pw`. - It read the merged manifest and computed a delta: *"You are missing 2.3 GB"* against a 15.4 GB manifest, so the shared-repository dedupe is doing what #84 predicted. - It downloaded thousands of blobs before stopping with 8593 to go. **What failed:** ``` FAILED Download corrupted; checksum mismatch on file 017f680287c9f1617dd92d7426f51d29577ec38f (potential compression error: ). This is likely a server-side repository issue. ``` **It is not a repository issue.** Verified that exact object by hand: - manifest entry: sha1 `017f6802…`, size 230, resref `blood_fore`, restype 2072 - stored blob: `NSYC` v3, algorithm 2, declared size 230, 203 bytes on the wire - decompresses to exactly 230 bytes, and `sha1(uncompressed)` == the filename Then audited the zone more broadly: **68 blobs verified end to end, 0 corrupt, 0 missing** — 60 sampled at random from the manifest plus the 8 largest entries (14 MB uncompressed each, ~1.8 MB compressed). Every one decompressed to its declared size with a matching SHA-1. Also confirmed the edge is not mangling anything: identical bytes with `Accept-Encoding: gzip` (no `content-encoding` applied), with a NWN-ish User-Agent, and a correct `206` with a proper `Content-Range` for a range request. No rate limiting is configured on the zone. **Conclusion: a transient truncated transfer, not stored corruption.** The tell is in the screenshot — 58 KB/s, far below what the CDN serves, with many parallel fetches in flight. A stalled connection yields a short body, and a short zstd frame is exactly "potential compression error". So the gate is **nearly** met: the manifest, the delta calculation, the URL handshake and the blob format all work against a real client. What is unproven is a sync running to completion. Retrying. Filed separately: the client's checksum is currently the **only** thing that ever verifies a published blob, which is a thin place to find out.
archvillainette commented 2026-07-31 22:04:45 +00:00
Author
Owner
Copy Link
Copy Source

Correction: not transient, and my earlier read was wrong

I said above this looked like a truncated transfer and to retry. That was wrong, and the retry disproved it: same file, 017f6802…, from 8593 files remaining to 8590. Deterministic.

It is also not the client's cache — that blob is absent from nwsyncdata_0.sqlite3 (64,715 rows present), so it is fetched fresh and rejected every time.

The cause is our zstd framing: klauspost/compress omits Frame_Content_Size for inputs under 256 bytes, reference libzstd does not, and the client needs it. Full diagnosis and reproduction in #86.

Why my audit said the zone was healthy. I verified 68 blobs by decompressing and hashing them, and they all passed — because the zstd CLI streams fine without a declared content size. I used a more capable decoder than the client's and concluded the thing it tripped on did not exist. Round-trip success is not the property that matters here; the frame header is. #85's verifier has to assert frame properties, not just that bytes come back.

So this gate stays open, and it has earned its keep: the manifest, the delta calculation, the URL handshake and ~64k blobs all work, and one 6%-of-files framing detail still stops every sync.

## Correction: not transient, and my earlier read was wrong I said above this looked like a truncated transfer and to retry. That was wrong, and the retry disproved it: same file, `017f6802…`, from 8593 files remaining to 8590. Deterministic. It is also not the client's cache — that blob is absent from `nwsyncdata_0.sqlite3` (64,715 rows present), so it is fetched fresh and rejected every time. **The cause is our zstd framing:** `klauspost/compress` omits Frame_Content_Size for inputs under 256 bytes, reference libzstd does not, and the client needs it. Full diagnosis and reproduction in #86. **Why my audit said the zone was healthy.** I verified 68 blobs by decompressing and hashing them, and they all passed — because the `zstd` CLI streams fine without a declared content size. I used a more capable decoder than the client's and concluded the thing it tripped on did not exist. Round-trip success is not the property that matters here; the frame header is. #85's verifier has to assert frame properties, not just that bytes come back. So this gate stays open, and it has earned its keep: the manifest, the delta calculation, the URL handshake and ~64k blobs all work, and one 6%-of-files framing detail still stops every sync.
archvillainette commented 2026-08-01 07:39:54 +00:00
Author
Owner
Copy Link
Copy Source

Gate met

The testing target on netcup-game is live, a retail NWN:EE client synced to completion against the published manifest, and a character is in the module.

Against Done means:

  • Full corpus emit plus one assemble against a real channel group id, published to the zone — done. sow-module release CI assembled it: module v0.2.1-rc1 against haks v0.2.1-rc1 and topdata v0.2.3-rc2, all three testing, group 2.
  • A retail client completes a sync and enters the module — done. The one thing that stopped it was ours, not the serving side: the sub-256-byte zstd framing defect in #86. Repaired, backfilled with --verify (2,603 blobs rewritten), 8 stale edge keys purged, then a full sweep — verified 69177 of 69177 blobs behind 72544 resources: 0 failures. Detail in #88.
  • Serving side recorded — ADR-0029 for where NWSync is served from, and the sow-platform NWSync runbook for the repair and purge procedure (sow-platform#138, #139).
  • The 13 cross-hak duplicate resrefs and sow_top's 2DAs, spot-checked in-world — not done. Everything machine-checkable passes and the sync is clean, but earliest-hak-wins is the rule that ships a wrong texture silently, and only eyes in the world can see that. If a wrong texture or a stale 2DA turns up, reopen here rather than suspecting internal/nwsync — the bytes are proven against upstream.

Closing on the sync and the character.

## Gate met The testing target on netcup-game is live, a retail NWN:EE client synced to completion against the published manifest, and a character is in the module. Against **Done means**: - **Full corpus emit plus one assemble against a real channel group id, published to the zone** — done. sow-module release CI assembled it: module `v0.2.1-rc1` against haks `v0.2.1-rc1` and topdata `v0.2.3-rc2`, all three testing, group 2. - **A retail client completes a sync and enters the module** — done. The one thing that stopped it was ours, not the serving side: the sub-256-byte zstd framing defect in #86. Repaired, backfilled with `--verify` (2,603 blobs rewritten), 8 stale edge keys purged, then a full sweep — `verified 69177 of 69177 blobs behind 72544 resources: 0 failures`. Detail in #88. - **Serving side recorded** — ADR-0029 for where NWSync is served from, and the sow-platform NWSync runbook for the repair and purge procedure (sow-platform#138, #139). - **The 13 cross-hak duplicate resrefs and sow_top's 2DAs, spot-checked in-world** — **not done.** Everything machine-checkable passes and the sync is clean, but earliest-hak-wins is the rule that ships a wrong texture silently, and only eyes in the world can see that. If a wrong texture or a stale 2DA turns up, reopen here rather than suspecting `internal/nwsync` — the bytes are proven against upstream. Closing on the sync and the character.
archvillainette commented 2026-08-01 07:42:53 +00:00
Author
Owner
Copy Link
Copy Source

The outstanding in-world spot-check is now #92, so it does not get lost with this ticket closed.

The outstanding in-world spot-check is now #92, so it does not get lost with this ticket closed.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
wayfinder:grilling
wayfinder:map
wayfinder:prototype
wayfinder:research
wayfinder:task
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
No labels wayfinder:task
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
Chass FallenDabus WinterZen XeroTwo Zariu archvillainette gitea-bot xtul
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Depends on
#88 Run the verifying backfill: every blob published by emitter 1 is still unreadable by a client
ShadowsOverWestgate/sow-tools
#89 Replacing a blob does not purge the pull-zone cache, so verify can pass while clients still get the bad bytes
ShadowsOverWestgate/sow-tools
You do not have permission to read 3 dependencies
Reference: ShadowsOverWestgate/sow-tools#75