Map: NWSync publishing — emit at artifact birth, assemble at module release #54

New Issue

2026-07-27T18:50:18Z

archvillainette commented

2026-07-27 18:50:18 +00:00

Destination

A real NWN client syncs a character in from a manifest that sow-module release CI assembled, served from the Bunny zone — with blobs emitted at each artifact's birth and no bulk NWSync data ever resident on ovh-main. Reaching that means crucible nwsync emit/assemble exist and are conformance-proven against upstream, and the three producer repos plus sow-module are wired to use them.

Notes

Domain: NWN NWSync repository publishing. The byte-level format spec and the reasoning behind the split live in the resolution comment of sow-platform#94 — read it before any ticket here. The build brief is sow-tools#53.
This map spans repos. Tickets may name work in sow-assets-manifest, sow-topdata, sow-module and sow-platform; the existing consumer issues are sow-assets-manifest#48 and sow-topdata#178. It sits under sow-platform#80's go-live effort.
One route ticket left, and it is in another repo. sow-module#51 (assemble at module release) and sow-platform#128 (serving side) are both closed. What stands between here and the destination is sow-platform#79 alone: nothing sets sow.services.nwn.nwsyncManifest, so sow.services.nwn.enable still hard-fails the NixOS build per ADR-0029. #75 depends on it and on nothing else.
Skills every session should consult: /grilling, /domain-modeling. Prototype tickets use /prototype, research tickets use /research.
Standing preference: no bulk data on ovh-main. The working set for blob emission is one resource (≤15 MiB), never one hak and never the corpus. A local --out tree exists only as the conformance path on a developer machine.
Upstream nwn_nwsync_write is the conformance oracle and never runs in production. Manifests compare byte for byte; blobs compare after decompression.

Decisions so far

Destination: end to end, not just merged code — the map does not close until a real client syncs from an assembled manifest; the code shipping is not the gate.
The map owns cross-repo work — tickets may drive sow-assets-manifest, sow-topdata, sow-module and sow-platform directly rather than waiting on them as external blockers.
sow-module release CI runs assemble, not sow-platform's resolve step — module release is the only point downstream of both the topdata and assets releases; platform has no control over when those cut. This corrects sow-platform#94's resolution and the assemble half of sow-platform#79.
group_id follows the tag shape — v* releases are current (group 1), v*-rc* are testing (group 2). No release is ever promoted between channels, so a manifest is born knowing its group and is never re-assembled for a different one. assemble keeps --group-id as a caller-supplied flag; deriving it is sow-module CI's job.
emit uploads directly, no local blob tree — the blob name is the SHA-1 of the uncompressed resource and no resource exceeds 15 MiB, so the working set is a single resource held in memory. Nothing bulky is ever written to the runner's disk.
Does sow-module CI know the full artifact set at release time? — yes for haks (digests are one deleted jq map away, and the order it holds is already the real Mod_HakList resolution order with sow_top first), no for topdata (two more small JSON fetches). #94's "only the resolver knows" claim is confirmed stale. sow-module CI already holds Bunny write credentials.
Bunny zone: how a blob is uploaded and how existence is checked — probe per object with a 1-byte range GET (HEAD is banned; listing costs 65,793 calls, worse than probing 71,436 objects). Reuse internal/depot's httpBackend with three sha256 assumptions made per-instance; the Checksum header is SHA-256 and will reject a SHA-1.
Restype resolution for emit — no new table needed, erf.Read already keeps the raw uint16 from the ERF key list, and all 20 relevant restypes already match upstream. emit must lowercase the resref itself and mirror upstream's table rather than Crucible's.
Where the per-artifact NSYM lives, now that topdata is not a hak — sibling of its artifact, per stream: artifacts/<stream>/sha256/<a>/<b>/<sha256>.nsym, final extension replaced. emit takes --artifact-key and derives the NSYM key itself (fail-closed digest check), so the one rule emit and assemble must share lives only in Crucible. The TLK is --as sow_tlk.tlk, one entry, one blob, no second verb. Verified from upstream source: a hak is a first-class <spec> and the .mod branch only resolves-then-readErfs, so the per-hak conformance oracle is real; module_haks is earliest-wins against upstream's last-added-wins, so comparisons feed specs reversed.
Where the nwsync verbs live in the command surface — its own group, its own binary crucible-nwsync, both verbs together: emit [--as <name>] [--out DIR] <artifact-key> and assemble --group-id <n> [--tlk-key <key>] [--out DIR] <artifact-key>.... Keys are positional in both verbs (this overrides #62's --artifact-key spelling), assemble's order is Mod_HakList order, and the TLK gets its own slot because it carries no precedence. --out DIR means write locally instead of uploading — the conformance path — so there is no --target. A new binary costs nothing: CI builds only ./cmd/crucible and the wrappers fetch one asset. Zone and credentials come from NWSYNC_STORAGE_ZONE / NWSYNC_STORAGE_PASSWORD (already on sow-module), host falling back to BUNNY_STORAGE_HOST; nwsync data is a separate zone from the assets depot.
A module release does not record which topdata it shipped against — pin a tag at release, as a record, not a binding. The build must fetch a topdata catalog for the digests anyway, so topdata_release goes beside hak_release in the module catalog (schema stays 1; the proxy ignores unknown fields). No conflict with the "deploy-pinned parity" comment on fetch-hak-manifest.sh:48-49 — sow-platform/releases/prod.yml states game-tier artifacts are never pinned there and netcup-game follows channels, so platform cannot pin and does not try. fetch-hak-manifest.sh → fetch-upstream-manifests.sh, resolving a second channel SOW_MODULE_TOPDATA_CHANNEL from the same tag-shape case, and keeping both catalogs verbatim in .cache/ so assemble fetches nothing — which is also where #58's hak digests come from. Hard fail on all three topdata failure modes. Nothing recorded about sow-codebase: it contributes no manifest resource. Four-way parity stays a human judgment at promotion and tag time; uneven versions across repos are normal and nothing compares them. No version suffixes on sow_top.hak/sow_tlk.tlk. Rename flips both sides at once, broken window accepted; implementation rides with #65.
Wire the consumer repos to the settled crucible nwsync interface — all three repos wired as issue text. emit takes two positionals, <artifact-key> <file> (#56's "exactly one" cannot hold with #62's guard; the file positional already exists in internal/nwsync/run.go:57-77). Hook points corrected against the real scripts: inside pack-haks.sh's loop after artifact_put and before the by-build record (the hak is deleted inside the script, and the record is the reuse gate), at publish-topdata-release.sh's two put_blob call sites (only the TLK takes --as), and a new sow-module step after publish+promote — sow-module#51, carrying #63's rename. Blob-already-present is skip-and-count via #55's probe, superseding "list the zone once" in both bodies. Both emitter repos need NWSYNC_STORAGE_* as new secrets; all three must also pass BUNNY_STORAGE_HOST because Crucible has no default host (internal/depot/config.go:42) while every repo's depot-lib.sh does.
Run upstream nwn_nwsync_write and capture reference output for one real hak — the format is conformance-proven. Upstream 2.1.2 runs from its prebuilt Linux release with LD_LIBRARY_PATH pointing at nixpkgs sqlite (not packaged in nixpkgs, no Nim needed). Over sow_vfxs_01.hak (521ab5de…, v0.1.9): manifest byte-identical, same 2,296 blob names and paths, blob framing headers identical byte for byte, payloads identical after decompression, sha1(uncompressed) == blob name on every blob. Only compressed size differs (+1.6%, Go zstd vs Nim libzstd), as predicted. Reference output lives at /home/vicky/nwsync-ref/ on the dev machine. Two harmless sidecar deltas (on_disk_bytes counts per entry upstream, per stored file here; module_name) and one CLI wart (flags must precede positionals) recorded there.
The emit upload path and NWCompressedBuffer round-trip — the sink is one interface with two implementations, zoneSink (production) and dirSink (--out DIR, conformance). Zone access is depot.KeyStore — ProbeKey/PutReader/GetKey bolted onto the existing httpBackend, with the sha-addressed Backend rewritten on top; #55's per-instance hash fields became "the caller passes key and checksum". Compression is paid only for blobs actually uploaded. Failure posture: a failed emit fails the producing release — no catalog entry, nothing downstream can pin it, CI re-run is the retry. Orphan blobs from a failed run stay in the zone, harmless and uncollected; assemble's missing-index check is the second line of defence, not the first. Code in PR #73, which also reshapes the CLI to #56/#62/#65's surface.
Build state (not a decision — the record): PR #71 landed emit/assemble with the local --out sink only, built from #53's original CLI rather than the surface #56/#62/#65 settled. Framing, blob paths, NSYM v3 + sidecar, shadowing, 15 MiB fail-closed and lowercased resrefs are done and tested; upload sink, CLI reshape and mid-hak failure are folded into #60. The format was read from upstream source and the tests share that reading, so nothing is conformance-proven until #59.
Correct the platform record: assemble runs in sow-module release CI — corrections posted on sow-platform#94 (closed; assemble location plus six overtaken claims) and sow-platform#79 (open; no generation unit on the host, no writer in the server image, hash not host-generated, group id owned by module CI). Platform's resolve step keeps exactly one NWSync role and it is passthrough: read the manifest SHA-1 from the module catalog it already resolves, fail closed if the pinned release has none, render it into the server env. Zero role is unreachable — platform runs the nwserver containers (nwn.nix:44-52, ADR-0025) and latest, hand-editing and host-derivation are all closed. Not redundant with emit/assemble: those are build-time on wired releases, this catches an operator pinning a pre-NWSync or backfilled release. Presence check only — never re-fetch the manifest, never probe the zone.
Bunny write credential for sow-module CI — already provisioned: NWSYNC_STORAGE_ZONE/NWSYNC_STORAGE_PASSWORD are on sow-module today, separate from the depot pair, write key only (reads go via the public CDN). Bunny has no prefix ACL — one AccessKey per zone — but the separate zone is the scope, and its contents are regenerable with no secrets or player data, a smaller grant than the depot key the same workflow already holds. Runner is nix-docker, already trusted with the depot key. Failure posture overrides #65: assemble runs before publish, not after — haks.json is the publication marker, assemble needs nothing publish produces, so the order is fetch → build → assemble → publish → promote with the manifest SHA-1 written into haks.json beside hak_release/topdata_release. A failed assemble aborts before the release is pinnable; an orphan manifest in the zone is harmless, as with orphan blobs.
Provision the NWSync write credential on the two emitter repos — done, but org-wide: NWSYNC_STORAGE_ZONE/NWSYNC_STORAGE_PASSWORD (and the BUNNY_* trio, previously duplicated per repo) are now ShadowsOverWestgate organisation Actions secrets, with the per-repo copies deprovisioned. Inherited by every repo's Actions runs, so all three emitters resolve them with no workflow change. Widens #61's grant to any repo in the org, accepted on #61's own reasoning (regenerable data, separate zone). The ticket's other two bullets are deferred into the wiring change, not done: no crucible-nwsync step exists in any repo yet, so there is nothing to pass BUNNY_STORAGE_HOST to (and it is a workflow literal, not a secret — release.yml:41, depot-lib.sh:13) and no upload to guard (pattern when written: sow-module/.gitea/workflows/release.yml:51-53). Credentials no longer block the wiring.
Emit performance (not a decision — the record): emit held ~5x the artifact size in RAM and was OOM-killed on ovh-main above ~1.4 GB (#76). Fixed by streaming (#78): erf.ReadIndex/ReadPayload read one payload on demand, so peak memory tracks the largest single resource — a 2.15 GB hak peaks at 108 MB. That removed the interim NWSYNC_MAX_HAK_BYTES ceiling (sow-assets-manifest#59), which had been a hole in the "every published artifact has NWSync data" invariant. Emit is latency-bound, not CPU-bound, so blobs now upload 16 at a time (#79/#80); manifest bytes are byte-identical at any job count, which emitterVersion promises and a test enforces. The on-disk shape is documented in #77.
Backfill complete (not a decision — the record): every hak named by a pointer in releases/haks/channels.json, plus sow_top.hak and sow_tlk.tlk, now has its blobs and its .nsym index in the sow-sync zone — sow-assets-manifest#57 and sow-topdata#182. It ran as each repo's backfill-nwsync workflow, reading the depot over the credential-free CDN and writing only the NWSync zone; no release, tag or channel was touched, and the already-published releases got their indexes without being re-released. Re-running is safe because blob names are content hashes. Credentials are org-wide Actions secrets and BUNNY_STORAGE_HOST is passed at every emit call site, backfill workflows included.
Emit wiring verified in the two emitter repos (not a decision — the record): sow-assets-manifest/scripts/pack-haks.sh:115 emits inside the pack loop after artifact_put and before the by-build record, and sow-topdata/scripts/publish-topdata-release.sh:52,54 emits at both put_blob call sites with --as sow_tlk.tlk on the TLK alone — exactly #65's hook points. sow-module has no nwsync step at all, so #65's third repo is outstanding; see the Notes bullet on open route tickets.
The first assembled manifest exists (not a decision — the record): sow-module v0.2.1-rc1 published clean (run 5025), closing sow-module#51 and sow-module#49. Module release CI assembled 061d23b96d7157b66d24993ece890d1ac6af982c — 72,544 resources, 15.4 GB, includes_module_contents: false, group 2 — and it serves from the Bunny zone with its sidecar. The hash reaches consumers as a sidecar beside the catalog, releases/module/<tag>/nwsync.json holding {manifest_sha1, group_id}, chosen over re-publishing haks.json because the catalog is the publication boundary and is already immutable by the time assemble runs; absence means the release has no manifest, never "not yet". This also settles the ordering #61 flipped: assemble runs after publish+promote, as #65 had it, because a manifest is syncable and one published before the .mod lands would let clients sync a release that then failed to ship. The catalog records fresh upstreams (hak_release v0.2.1-rc1, topdata_release v0.2.3-rc2), so all three streams' testing pointers now pair.

Not yet specified

Multi-artifact assemble against upstream. #59 proved single-hak emit; upstream's multi-spec run is still uncompared, and it is where the earliest-wins/last-added-wins reversal (#62) actually gets tested. Needs two haks that share resrefs — sow_core_01 and sow_over_01 share 13 — which is ~4 GB of download, so it is a deliberate one-off, not a per-change check.
Corpus-wide conformance. #59 fixed where a comparison runs (dev machine, /home/vicky/nwsync-ref/) and the loop is cheap enough to repeat, so what is left is only whether the remaining 10 haks are worth running and how a partial pass is tracked.
The final acceptance run. Real client, real zone, real character — #75. Depends on everything above, plus the two open route tickets in the Notes.

Out of scope

NWSync retention and pruning. "Never prune a manifest a channel pointer still names" is a real rule with no enforcer, and nwn_nwsync_prune cannot run against an unmountable zone. It belongs to sow-platform#80's fog, not to getting the first client synced.
Publishing module contents. A persistent world must never publish --with-module; the .mod resolves which haks and which TLK a manifest covers and contributes no bytes. Settled upstream, not a decision this map makes.
Five restype numbers in internal/erf disagree with upstream. Surfaced by #57 and a genuine bug in HAK writing today (.jpg is written as tml), but none of the five are in the corpus and emit mirrors upstream's table regardless. Filed as its own issue; fixing it is not on the route to a synced client.
Downloads page: resolve topdata_release into paired topdata. Surfaced by #63 — once the module catalog carries topdata_release, the proxy could resolve it into paired_topdata the way it already resolves hak_release into paired_haks. A downloads-page feature, not NWSync publishing. Filed as its own issue.
depot: replace --target local with --out DIR. Surfaced by #56 — nwsync uses --out DIR, so depot's --target local now disagrees with it. Worth doing, but it changes no NWSync behaviour. Filed as its own issue.
sow-module#52: channels.json CDN purge never runs. Surfaced by #61 while auditing sow-module's Bunny secrets — release.yml passes BUNNY_API_KEY but the secret does not exist, and depot-lib.sh skips the purge on unset, so every module promote leaves a stale channels.json on the edge. A real bug in the mutable pointer, but it touches no NWSync object and blocks no step on the route. Filed as its own issue.

## Destination A real NWN client syncs a character in from a manifest that **sow-module release CI** assembled, served from the Bunny zone — with blobs emitted at each artifact's birth and no bulk NWSync data ever resident on `ovh-main`. Reaching that means `crucible nwsync emit`/`assemble` exist and are conformance-proven against upstream, and the three producer repos plus sow-module are wired to use them. ## Notes - Domain: NWN NWSync repository publishing. The byte-level format spec and the reasoning behind the split live in the resolution comment of [sow-platform#94](https://git.westgate.pw/ShadowsOverWestgate/sow-platform/issues/94) — read it before any ticket here. The build brief is [sow-tools#53](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/53). - This map **spans repos**. Tickets may name work in `sow-assets-manifest`, `sow-topdata`, `sow-module` and `sow-platform`; the existing consumer issues are [sow-assets-manifest#48](https://git.westgate.pw/ShadowsOverWestgate/sow-assets-manifest/issues/48) and [sow-topdata#178](https://git.westgate.pw/ShadowsOverWestgate/sow-topdata/issues/178). It sits under [sow-platform#80](https://git.westgate.pw/ShadowsOverWestgate/sow-platform/issues/80)'s go-live effort. - **One route ticket left, and it is in another repo.** [sow-module#51](https://git.westgate.pw/ShadowsOverWestgate/sow-module/issues/51) (assemble at module release) and [sow-platform#128](https://git.westgate.pw/ShadowsOverWestgate/sow-platform/issues/128) (serving side) are both **closed**. What stands between here and the destination is [sow-platform#79](https://git.westgate.pw/ShadowsOverWestgate/sow-platform/issues/79) alone: nothing sets `sow.services.nwn.nwsyncManifest`, so `sow.services.nwn.enable` still hard-fails the NixOS build per ADR-0029. [#75](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/75) depends on it and on nothing else. - Skills every session should consult: `/grilling`, `/domain-modeling`. Prototype tickets use `/prototype`, research tickets use `/research`. - Standing preference: **no bulk data on `ovh-main`.** The working set for blob emission is one resource (≤15 MiB), never one hak and never the corpus. A local `--out` tree exists only as the conformance path on a developer machine. - Upstream `nwn_nwsync_write` is the conformance oracle and never runs in production. Manifests compare byte for byte; blobs compare after decompression. ## Decisions so far - [Destination: end to end, not just merged code](#) — the map does not close until a real client syncs from an assembled manifest; the code shipping is not the gate. - [The map owns cross-repo work](#) — tickets may drive sow-assets-manifest, sow-topdata, sow-module and sow-platform directly rather than waiting on them as external blockers. - [sow-module release CI runs `assemble`, not sow-platform's resolve step](#) — module release is the only point downstream of both the topdata and assets releases; platform has no control over when those cut. This **corrects** [sow-platform#94](https://git.westgate.pw/ShadowsOverWestgate/sow-platform/issues/94)'s resolution and the assemble half of [sow-platform#79](https://git.westgate.pw/ShadowsOverWestgate/sow-platform/issues/79). - [`group_id` follows the tag shape](#) — `v*` releases are `current` (group 1), `v*-rc*` are `testing` (group 2). No release is ever promoted between channels, so a manifest is born knowing its group and is never re-assembled for a different one. `assemble` keeps `--group-id` as a caller-supplied flag; deriving it is sow-module CI's job. - [`emit` uploads directly, no local blob tree](#) — the blob name is the SHA-1 of the uncompressed resource and no resource exceeds 15 MiB, so the working set is a single resource held in memory. Nothing bulky is ever written to the runner's disk. - [Does sow-module CI know the full artifact set at release time?](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/58) — yes for haks (digests are one deleted `jq` map away, and the order it holds is already the real `Mod_HakList` resolution order with `sow_top` first), no for topdata (two more small JSON fetches). #94's "only the resolver knows" claim is confirmed stale. sow-module CI already holds Bunny write credentials. - [Bunny zone: how a blob is uploaded and how existence is checked](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/55) — probe per object with a 1-byte range GET (HEAD is banned; listing costs 65,793 calls, worse than probing 71,436 objects). Reuse `internal/depot`'s `httpBackend` with three sha256 assumptions made per-instance; the `Checksum` header is SHA-256 and will reject a SHA-1. - [Restype resolution for emit](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/57) — no new table needed, `erf.Read` already keeps the raw `uint16` from the ERF key list, and all 20 relevant restypes already match upstream. `emit` must lowercase the resref itself and mirror upstream's table rather than Crucible's. - [Where the per-artifact NSYM lives, now that topdata is not a hak](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/62) — sibling of its artifact, per stream: `artifacts/<stream>/sha256/<a>/<b>/<sha256>.nsym`, final extension replaced. `emit` takes `--artifact-key` and derives the NSYM key itself (fail-closed digest check), so the one rule `emit` and `assemble` must share lives only in Crucible. The TLK is `--as sow_tlk.tlk`, one entry, one blob, no second verb. Verified from upstream source: a hak is a first-class `<spec>` and the `.mod` branch only resolves-then-`readErf`s, so the per-hak conformance oracle is real; `module_haks` is earliest-wins against upstream's last-added-wins, so comparisons feed specs reversed. - [Where the nwsync verbs live in the command surface](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/56) — its own group, its own binary `crucible-nwsync`, both verbs together: `emit [--as <name>] [--out DIR] <artifact-key>` and `assemble --group-id <n> [--tlk-key <key>] [--out DIR] <artifact-key>...`. Keys are positional in both verbs (this overrides #62's `--artifact-key` spelling), assemble's order **is** `Mod_HakList` order, and the TLK gets its own slot because it carries no precedence. `--out DIR` means write locally instead of uploading — the conformance path — so there is no `--target`. A new binary costs nothing: CI builds only `./cmd/crucible` and the wrappers fetch one asset. Zone and credentials come from `NWSYNC_STORAGE_ZONE` / `NWSYNC_STORAGE_PASSWORD` (already on sow-module), host falling back to `BUNNY_STORAGE_HOST`; nwsync data is a **separate zone** from the assets depot. - [A module release does not record which topdata it shipped against](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/63) — pin a tag at release, as a **record, not a binding**. The build must fetch a topdata catalog for the digests anyway, so `topdata_release` goes beside `hak_release` in the module catalog (`schema` stays `1`; the proxy ignores unknown fields). No conflict with the "deploy-pinned parity" comment on `fetch-hak-manifest.sh:48-49` — `sow-platform/releases/prod.yml` states game-tier artifacts are *never* pinned there and netcup-game follows channels, so platform cannot pin and does not try. `fetch-hak-manifest.sh` → `fetch-upstream-manifests.sh`, resolving a second channel `SOW_MODULE_TOPDATA_CHANNEL` from the same tag-shape `case`, and keeping both catalogs verbatim in `.cache/` so assemble fetches nothing — which is also where #58's hak digests come from. Hard fail on all three topdata failure modes. Nothing recorded about sow-codebase: it contributes no manifest resource. Four-way parity stays a **human** judgment at promotion and tag time; uneven versions across repos are normal and nothing compares them. No version suffixes on `sow_top.hak`/`sow_tlk.tlk`. Rename flips both sides at once, broken window accepted; implementation rides with #65. - [Wire the consumer repos to the settled crucible nwsync interface](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/65) — all three repos wired as issue text. `emit` takes **two** positionals, `<artifact-key> <file>` (#56's "exactly one" cannot hold with #62's guard; the file positional already exists in `internal/nwsync/run.go:57-77`). Hook points corrected against the real scripts: **inside** `pack-haks.sh`'s loop after `artifact_put` and before the by-build record (the hak is deleted inside the script, and the record is the reuse gate), at `publish-topdata-release.sh`'s two `put_blob` call sites (only the TLK takes `--as`), and a new sow-module step **after** publish+promote — [sow-module#51](https://git.westgate.pw/ShadowsOverWestgate/sow-module/issues/51), carrying #63's rename. Blob-already-present is skip-and-count via #55's probe, superseding "list the zone once" in both bodies. Both emitter repos need `NWSYNC_STORAGE_*` as new secrets; all three must also pass `BUNNY_STORAGE_HOST` because Crucible has no default host (`internal/depot/config.go:42`) while every repo's `depot-lib.sh` does. - [Run upstream `nwn_nwsync_write` and capture reference output for one real hak](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/59) — **the format is conformance-proven.** Upstream 2.1.2 runs from its prebuilt Linux release with `LD_LIBRARY_PATH` pointing at nixpkgs sqlite (not packaged in nixpkgs, no Nim needed). Over `sow_vfxs_01.hak` (`521ab5de…`, v0.1.9): manifest **byte-identical**, same 2,296 blob names and paths, blob framing headers identical byte for byte, payloads identical after decompression, `sha1(uncompressed) == blob name` on every blob. Only compressed size differs (+1.6%, Go zstd vs Nim libzstd), as predicted. Reference output lives at `/home/vicky/nwsync-ref/` on the dev machine. Two harmless sidecar deltas (`on_disk_bytes` counts per entry upstream, per stored file here; `module_name`) and one CLI wart (flags must precede positionals) recorded there. - [The emit upload path and NWCompressedBuffer round-trip](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/60) — the sink is one interface with two implementations, `zoneSink` (production) and `dirSink` (`--out DIR`, conformance). Zone access is `depot.KeyStore` — `ProbeKey`/`PutReader`/`GetKey` bolted onto the existing `httpBackend`, with the sha-addressed `Backend` rewritten on top; #55's per-instance hash fields became "the caller passes key and checksum". Compression is paid only for blobs actually uploaded. **Failure posture: a failed `emit` fails the producing release** — no catalog entry, nothing downstream can pin it, CI re-run is the retry. Orphan blobs from a failed run stay in the zone, harmless and uncollected; `assemble`'s missing-index check is the second line of defence, not the first. Code in [PR #73](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/pulls/73), which also reshapes the CLI to #56/#62/#65's surface. - **Build state (not a decision — the record):** [PR #71](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/pulls/71) landed `emit`/`assemble` with the **local `--out` sink only**, built from [#53](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/53)'s original CLI rather than the surface #56/#62/#65 settled. Framing, blob paths, NSYM v3 + sidecar, shadowing, 15 MiB fail-closed and lowercased resrefs are done and tested; upload sink, CLI reshape and mid-hak failure are folded into [#60](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/60). The format was read from upstream *source* and the tests share that reading, so nothing is conformance-proven until [#59](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/59). - [Correct the platform record: assemble runs in sow-module release CI](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/72) — corrections posted on sow-platform#94 (closed; assemble location plus six overtaken claims) and sow-platform#79 (open; no generation unit on the host, no writer in the server image, hash not host-generated, group id owned by module CI). Platform's resolve step keeps **exactly one** NWSync role and it is passthrough: read the manifest SHA-1 from the module catalog it already resolves, fail closed if the pinned release has none, render it into the server env. Zero role is unreachable — platform runs the nwserver containers (nwn.nix:44-52, ADR-0025) and `latest`, hand-editing and host-derivation are all closed. Not redundant with emit/assemble: those are build-time on wired releases, this catches an operator pinning a pre-NWSync or backfilled release. Presence check only — never re-fetch the manifest, never probe the zone. - [Bunny write credential for sow-module CI](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/61) — already provisioned: `NWSYNC_STORAGE_ZONE`/`NWSYNC_STORAGE_PASSWORD` are on sow-module today, separate from the depot pair, write key only (reads go via the public CDN). Bunny has no prefix ACL — one `AccessKey` per zone — but the separate zone *is* the scope, and its contents are regenerable with no secrets or player data, a smaller grant than the depot key the same workflow already holds. Runner is `nix-docker`, already trusted with the depot key. **Failure posture overrides #65: assemble runs *before* publish, not after** — `haks.json` is the publication marker, assemble needs nothing publish produces, so the order is fetch → build → assemble → publish → promote with the manifest SHA-1 written into `haks.json` beside `hak_release`/`topdata_release`. A failed assemble aborts before the release is pinnable; an orphan manifest in the zone is harmless, as with orphan blobs. - [Provision the NWSync write credential on the two emitter repos](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/74) — done, but **org-wide**: `NWSYNC_STORAGE_ZONE`/`NWSYNC_STORAGE_PASSWORD` (and the `BUNNY_*` trio, previously duplicated per repo) are now ShadowsOverWestgate **organisation** Actions secrets, with the per-repo copies deprovisioned. Inherited by every repo's Actions runs, so all three emitters resolve them with no workflow change. Widens #61's grant to any repo in the org, accepted on #61's own reasoning (regenerable data, separate zone). The ticket's other two bullets are **deferred into the wiring change**, not done: no `crucible-nwsync` step exists in any repo yet, so there is nothing to pass `BUNNY_STORAGE_HOST` to (and it is a workflow literal, not a secret — `release.yml:41`, `depot-lib.sh:13`) and no upload to guard (pattern when written: `sow-module/.gitea/workflows/release.yml:51-53`). Credentials no longer block the wiring. - **Emit performance (not a decision — the record):** emit held ~5x the artifact size in RAM and was OOM-killed on ovh-main above ~1.4 GB ([#76](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/76)). Fixed by streaming ([#78](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/78)): `erf.ReadIndex`/`ReadPayload` read one payload on demand, so peak memory tracks the largest single resource — a 2.15 GB hak peaks at 108 MB. That removed the interim `NWSYNC_MAX_HAK_BYTES` ceiling ([sow-assets-manifest#59](https://git.westgate.pw/ShadowsOverWestgate/sow-assets-manifest/issues/59)), which had been a hole in the "every published artifact has NWSync data" invariant. Emit is latency-bound, not CPU-bound, so blobs now upload 16 at a time ([#79](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/79)/[#80](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/80)); manifest bytes are byte-identical at any job count, which `emitterVersion` promises and a test enforces. The on-disk shape is documented in [#77](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/77). - **Backfill complete (not a decision — the record):** every hak named by a pointer in `releases/haks/channels.json`, plus `sow_top.hak` and `sow_tlk.tlk`, now has its blobs and its `.nsym` index in the `sow-sync` zone — [sow-assets-manifest#57](https://git.westgate.pw/ShadowsOverWestgate/sow-assets-manifest/issues/57) and [sow-topdata#182](https://git.westgate.pw/ShadowsOverWestgate/sow-topdata/issues/182). It ran as each repo's `backfill-nwsync` workflow, reading the depot over the credential-free CDN and writing only the NWSync zone; no release, tag or channel was touched, and the already-published releases got their indexes without being re-released. Re-running is safe because blob names are content hashes. Credentials are org-wide Actions secrets and `BUNNY_STORAGE_HOST` is passed at every emit call site, backfill workflows included. - **Emit wiring verified in the two emitter repos (not a decision — the record):** `sow-assets-manifest/scripts/pack-haks.sh:115` emits inside the pack loop after `artifact_put` and before the by-build record, and `sow-topdata/scripts/publish-topdata-release.sh:52,54` emits at both `put_blob` call sites with `--as sow_tlk.tlk` on the TLK alone — exactly #65's hook points. **sow-module has no `nwsync` step at all**, so #65's third repo is outstanding; see the Notes bullet on open route tickets. - **The first assembled manifest exists (not a decision — the record):** sow-module `v0.2.1-rc1` published clean ([run 5025](https://git.westgate.pw/ShadowsOverWestgate/sow-module/actions/runs/5025/jobs/5056)), closing [sow-module#51](https://git.westgate.pw/ShadowsOverWestgate/sow-module/issues/51) and [sow-module#49](https://git.westgate.pw/ShadowsOverWestgate/sow-module/issues/49). Module release CI assembled `061d23b96d7157b66d24993ece890d1ac6af982c` — 72,544 resources, 15.4 GB, `includes_module_contents: false`, group 2 — and it serves from the Bunny zone with its sidecar. The hash reaches consumers as a **sidecar beside the catalog**, `releases/module/<tag>/nwsync.json` holding `{manifest_sha1, group_id}`, chosen over re-publishing `haks.json` because the catalog is the publication boundary and is already immutable by the time assemble runs; absence means the release has no manifest, never "not yet". This also settles the ordering [#61](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/61) flipped: assemble runs **after** publish+promote, as [#65](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/65) had it, because a manifest is syncable and one published before the `.mod` lands would let clients sync a release that then failed to ship. The catalog records fresh upstreams (`hak_release v0.2.1-rc1`, `topdata_release v0.2.3-rc2`), so all three streams' `testing` pointers now pair. ## Not yet specified - **Multi-artifact `assemble` against upstream.** #59 proved single-hak `emit`; upstream's multi-spec run is still uncompared, and it is where the earliest-wins/last-added-wins reversal (#62) actually gets tested. Needs two haks that share resrefs — `sow_core_01` and `sow_over_01` share 13 — which is ~4 GB of download, so it is a deliberate one-off, not a per-change check. - **Corpus-wide conformance.** #59 fixed where a comparison runs (dev machine, `/home/vicky/nwsync-ref/`) and the loop is cheap enough to repeat, so what is left is only whether the remaining 10 haks are worth running and how a partial pass is tracked. - **The final acceptance run.** Real client, real zone, real character — [#75](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/75). Depends on everything above, plus the two open route tickets in the Notes. ## Out of scope - **NWSync retention and pruning.** "Never prune a manifest a channel pointer still names" is a real rule with no enforcer, and `nwn_nwsync_prune` cannot run against an unmountable zone. It belongs to [sow-platform#80](https://git.westgate.pw/ShadowsOverWestgate/sow-platform/issues/80)'s fog, not to getting the first client synced. - **Publishing module contents.** A persistent world must never publish `--with-module`; the `.mod` resolves which haks and which TLK a manifest covers and contributes no bytes. Settled upstream, not a decision this map makes. - **[Five restype numbers in internal/erf disagree with upstream](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/64).** Surfaced by #57 and a genuine bug in HAK writing today (`.jpg` is written as `tml`), but none of the five are in the corpus and `emit` mirrors upstream's table regardless. Filed as its own issue; fixing it is not on the route to a synced client. - **[Downloads page: resolve `topdata_release` into paired topdata](https://git.westgate.pw/ShadowsOverWestgate/sow-depot-proxy/issues/22).** Surfaced by #63 — once the module catalog carries `topdata_release`, the proxy could resolve it into `paired_topdata` the way it already resolves `hak_release` into `paired_haks`. A downloads-page feature, not NWSync publishing. Filed as its own issue. - **[depot: replace `--target local` with `--out DIR`](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/66).** Surfaced by #56 — `nwsync` uses `--out DIR`, so `depot`'s `--target local` now disagrees with it. Worth doing, but it changes no NWSync behaviour. Filed as its own issue. - [sow-module#52: channels.json CDN purge never runs](https://git.westgate.pw/ShadowsOverWestgate/sow-module/issues/52). Surfaced by #61 while auditing sow-module's Bunny secrets — `release.yml` passes `BUNNY_API_KEY` but the secret does not exist, and `depot-lib.sh` skips the purge on unset, so every module promote leaves a stale `channels.json` on the edge. A real bug in the mutable pointer, but it touches no NWSync object and blocks no step on the route. Filed as its own issue.

archvillainette added the wayfinder:map label 2026-07-27 18:50:18 +00:00

Map: NWSync publishing — emit at artifact birth, assemble at module release #54

Destination

Notes

Decisions so far

Not yet specified

Out of scope

Backfill is done — the "Not yet specified" section is now mostly settled

Verification pass: the map is accurate again, and two route items were missing from it

Confirmed against the code, not just the tickets

Two things stand between this map and its destination, and neither was written down

Map hygiene fixed

One open question, not resolved here

The three release-candidate PRs are merged; the tags are next

What is left, in order

Side issues filed, none of them on the route

Route ticket sow-platform#128 is built — one of the two remaining is off the board

Destination reached — closing the map