ShadowsOverWestgate/sow-tools
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 38 Activity

A module release does not record which topdata it shipped against #63

New Issue
Closed
opened 2026-07-27 19:00:31 +00:00 by archvillainette · 2 comments
archvillainette commented 2026-07-27 19:00:31 +00:00
Owner
Copy Link
Copy Source

Question

How does a module release know which topdata it shipped against?

Part of #54. #58 found that publish-module-release.sh:59 records hak_release — the paired hak tag — but there is no topdata_release field. fetch-hak-manifest.sh:48-49 prepends sow_top by name only, deliberately, so module/topdata parity stays deploy-pinned by sow-platform.

That was survivable while nothing at release time needed the topdata digest. Now sow-module CI assembles the manifest, and a manifest must cover sow_top.hak and sow_tlk.tlk with their digests. So the release has to resolve topdata somehow.

Two shapes:

  • Pin a tag at release, next to hak_release, written by the same preflight that writes .cache/hak-release.txt. Consistent with how haks are already handled, and it closes a real hole that exists today independently of NWSync — a module release currently does not record which topdata it shipped against at all.
  • Read the topdata current channel at build time and do not record it. Simpler, but the manifest then describes a content set the release itself cannot name afterwards, and two builds of the same tag could differ.

Settle which, and whether changing this conflicts with the deploy-pinned parity rule sow-platform owns — pinning a topdata tag in the module release may or may not be the same thing as resolving a topdata channel, and that distinction is what fetch-hak-manifest.sh:48-49 was protecting.

## Question How does a module release know which topdata it shipped against? Part of #54. #58 found that `publish-module-release.sh:59` records `hak_release` — the paired hak tag — but there is **no `topdata_release` field**. `fetch-hak-manifest.sh:48-49` prepends `sow_top` by name only, deliberately, so module/topdata parity stays deploy-pinned by sow-platform. That was survivable while nothing at release time needed the topdata digest. Now sow-module CI assembles the manifest, and a manifest must cover `sow_top.hak` and `sow_tlk.tlk` with their digests. So the release has to resolve topdata somehow. Two shapes: - **Pin a tag at release**, next to `hak_release`, written by the same preflight that writes `.cache/hak-release.txt`. Consistent with how haks are already handled, and it closes a real hole that exists today independently of NWSync — a module release currently does not record which topdata it shipped against at all. - **Read the topdata `current` channel at build time** and do not record it. Simpler, but the manifest then describes a content set the release itself cannot name afterwards, and two builds of the same tag could differ. Settle which, and whether changing this conflicts with the deploy-pinned parity rule sow-platform owns — pinning a topdata tag in the module release may or may not be the same thing as resolving a topdata channel, and that distinction is what `fetch-hak-manifest.sh:48-49` was protecting.
archvillainette added the wayfinder:grilling label 2026-07-27 19:00:31 +00:00
archvillainette self-assigned this 2026-07-28 07:39:29 +00:00
archvillainette commented 2026-07-28 08:09:03 +00:00
Author
Owner
Copy Link
Copy Source

Resolution

Pin a tag at release — but as a record, not a binding. The module build must fetch a topdata catalog anyway to get sow_top.hak and sow_tlk.tlk digests for the NWSync manifest; once it has, the tag falls out for free, so it gets written down.

The framing that settles it

The ticket asked whether pinning a topdata tag conflicts with the deploy-pinned parity rule fetch-hak-manifest.sh:48-49 was protecting. It does not, because that rule does not exist in the form the comment describes — and the record does not attempt it anyway.

sow-platform/releases/prod.yml says so in writing: "Game-tier artifacts (module, topdata, HAKs, and the future nwserver stack) release through Bunny channel catalogs in their owning repos and are never pinned here", and netcup-game "resolves its game-tier artifacts from Bunny channel catalogs and pins nothing here". Platform cannot pin game-tier artifacts; it follows channels (current, testing). No game-tier resolver exists in platform yet.

So topdata_release is provenance: it says "this .mod was built and manifested against topdata v0.4.7". Nothing reads it to decide what to install. Deploy still resolves topdata from its own channel and may legitimately run a different one. The comment on fetch-hak-manifest.sh:48-49 was guarding against binding module to topdata, which this is not.

This is not parity enforcement. Nothing checks whether a module built against last week's codebase is valid — no gate, no refusal, no automation. That judgment is human, made by choosing when to promote a channel and when to push a tag, and the scenarios vary (code unchanged, only haks added, and so on). The record is a receipt written after the fact, not a rule applied before it.

Version numbers across the four repos are expected to be uneven — v0.1.5 module pairing with v0.4.7 topdata is normal, because a repo with no changes does not bump. Nothing in the design compares version numbers: the channel indirects through channels.json and topdata_release is a free-form tag string copied verbatim.

Scope: topdata only, not four repos

At module build time the four repos divide like this:

repo what the module build takes channel variable
sow-assets-manifest hak list + digests SOW_MODULE_HAK_CHANNEL (exists — this is the assets-manifest channel; it is named for the content, not the repo)
sow-topdata sow_top.hak + sow_tlk.tlk digests SOW_MODULE_TOPDATA_CHANNEL (new — this ticket)
sow-codebase nothing n/a
sow-module itself the pushed tag

sow-codebase is the .NET/Anvil server code shipped as the nwserver container image (promote-nwserver.sh, a Gitea generic package whose channels.json values are 40-char commit SHAs, not version tags). It is not an input to crucible module build, contributes no resource that could appear in an NWSync manifest, and is a runtime peer the deploy resolves. Recording it would mean CI copying a SHA it never consumed — a guess, not a record. Codebase parity stays deploy-time, exactly as today.

Decisions

  1. topdata_release is a record, not a binding. Provenance only; nothing reads it to deploy.
  2. Two derived channel variables. Existing SOW_MODULE_HAK_CHANNEL plus new SOW_MODULE_TOPDATA_CHANNEL, both set from one tag-shape case arm in .gitea/workflows/release.yml:36-37 (v*-* → testing, v* → current). Pushing a tag stays the only input — no operator ever types a channel. Two variables rather than one because assets and topdata release independently and "rc haks, stable topdata" is a legitimate state to reproduce locally while debugging; CI sets both from the same arm, so a release can never diverge by accident.
  3. Topdata only. Nothing recorded about sow-codebase (see above).
  4. fetch-hak-manifest.sh → fetch-upstream-manifests.sh, extended to resolve the topdata channel as well. The script already owns "resolve upstream channels for this build" and already prepends sow_top; a second script would duplicate the same catalog shape, CDN, and fail-closed rules. Resolving at publish time was rejected outright — the build needs topdata resolved before it, or the two halves could observe different channel states.
  5. Coordinated rename flip. The script name is a hardcoded path convention, not config: sow-tools/internal/app/app.go:388 runs <root>/scripts/fetch-hak-manifest, and crucible.sh:58 resolves the latest crucible release unpinned. Both sides flip together and the broken window is accepted; the failure is loud and unmistakable (required project script is missing). Smallest gap is merging the sow-module rename and cutting the crucible release back to back. Transitional symlink and Go-side fallback were both considered and rejected as cruft.
  6. Keep the catalogs. fetch-upstream-manifests.sh writes .cache/hak-catalog.json and .cache/topdata-catalog.json verbatim, alongside .cache/hak-release.txt and .cache/topdata-release.txt. Assemble reads them locally and fetches nothing. Build and publish run in the same job (release.yml), so .cache survives between steps. This is also where the hak digests #58 found ("one deleted jq map away") come from — one change answers both sides. Re-fetching per tag would have been safe (per-tag catalogs are immutable) but needs a second fetch implementation and buys nothing.
  7. Field and schema. topdata_release, a bare tag string, beside hak_release in releases/module/<tag>/haks.json. schema stays 1: the change is additive, and sow-depot-proxy/internal/server/downloads.go:234 decodes the catalog into a struct that ignores unknown fields and never reads schema. A bump would signal a break that is not there.
  8. Hard fail on all three topdata failure modes — CDN unreachable, channel unset in channels.json, catalog present-but-malformed — matching the hak side (fetch-hak-manifest.sh:34,37,42,45), in CI and on developer machines alike. promote-topdata.sh:67-70 seeds testing to current when unset, so the realistic failure is a genuine outage or a genuinely unpublished channel, both of which should stop a build. A local-only soft fail was rejected: it produces a build that succeeds locally and fails in CI for the same tree.
  9. sow_top stays hardcoded in the module_haks prepend (fetch-hak-manifest.sh:48). Catalogs supply digests only, so module_haks output is byte-identical and tests/fetch-hak-manifest-contract.sh needs no change beyond the rename.
  10. No version suffixes on sow_top.hak / sow_tlk.tlk. Considered and rejected. Version identity already lives in two places that cannot drift — the release tag now in topdata_release, and the sha256 in the catalog; a versioned filename would be a third copy and the only one nothing verifies. It buys NWSync nothing: manifests list resources, never hak filenames, and blob names are the SHA-1 of each uncompressed resource, so client-side dedup is already per-resource. The costs are real — every topdata release would rewrite Mod_HakList, making the .mod binary itself differ by topdata tag (reintroducing exactly the lockstep this ticket keeps human); manual non-NWSync players would swap files each release and accumulate old ones; nwn-tool.yaml's hak_order and docs would need lockstep updates; and NWN's Mod_Hak / Mod_CustomTlk CExoString length budget is tight for a name plus semver suffix. If "which topdata is this, offline, from the file itself" is ever wanted, the shape is a small version resource inside sow_top.hak, not a rename — held as an idea, not adopted.

Implementation

Rides with #65 (wiring the consumer repos), not a new ticket. Touches: sow-module/scripts/fetch-hak-manifest.sh (rename + topdata resolution), publish-module-release.sh (read .cache/topdata-release.txt, emit topdata_release), sow-tools/internal/app/app.go:388, the two contract tests under sow-module/tests/, plus the stale name in .gitignore, flake.nix, nwn-tool.yaml and release.yml comments. The fetch-hak-manifest.sh:48-49 comment about parity being "deploy-pinned by sow-platform" should be corrected while there — platform pins nothing game-tier.

Follow-up filed

sow-depot-proxy#22 — resolve topdata_release into paired_topdata on the downloads page, mirroring paired_haks. Kept separate: it is a downloads-page feature, not NWSync publishing.

## Resolution **Pin a tag at release — but as a record, not a binding.** The module build must fetch a topdata catalog anyway to get `sow_top.hak` and `sow_tlk.tlk` digests for the NWSync manifest; once it has, the tag falls out for free, so it gets written down. ### The framing that settles it The ticket asked whether pinning a topdata tag conflicts with the deploy-pinned parity rule `fetch-hak-manifest.sh:48-49` was protecting. It does not, because that rule does not exist in the form the comment describes — and the record does not attempt it anyway. `sow-platform/releases/prod.yml` says so in writing: *"Game-tier artifacts (module, topdata, HAKs, and the future nwserver stack) release through Bunny channel catalogs in their owning repos and are never pinned here"*, and netcup-game "resolves its game-tier artifacts from Bunny channel catalogs and pins nothing here". Platform cannot pin game-tier artifacts; it follows channels (`current`, `testing`). No game-tier resolver exists in platform yet. So `topdata_release` is **provenance**: it says "this `.mod` was built and manifested against topdata `v0.4.7`". Nothing reads it to decide what to install. Deploy still resolves topdata from its own channel and may legitimately run a different one. The comment on `fetch-hak-manifest.sh:48-49` was guarding against *binding* module to topdata, which this is not. **This is not parity enforcement.** Nothing checks whether a module built against last week's codebase is valid — no gate, no refusal, no automation. That judgment is human, made by choosing when to promote a channel and when to push a tag, and the scenarios vary (code unchanged, only haks added, and so on). The record is a receipt written after the fact, not a rule applied before it. Version numbers across the four repos are expected to be uneven — `v0.1.5` module pairing with `v0.4.7` topdata is normal, because a repo with no changes does not bump. Nothing in the design compares version numbers: the channel indirects through `channels.json` and `topdata_release` is a free-form tag string copied verbatim. ### Scope: topdata only, not four repos At module build time the four repos divide like this: | repo | what the module build takes | channel variable | |---|---|---| | sow-assets-manifest | hak list + digests | `SOW_MODULE_HAK_CHANNEL` (exists — this **is** the assets-manifest channel; it is named for the content, not the repo) | | sow-topdata | `sow_top.hak` + `sow_tlk.tlk` digests | `SOW_MODULE_TOPDATA_CHANNEL` (new — this ticket) | | sow-codebase | nothing | n/a | | sow-module | itself | the pushed tag | sow-codebase is the .NET/Anvil server code shipped as the **nwserver container image** (`promote-nwserver.sh`, a Gitea generic package whose `channels.json` values are 40-char commit SHAs, not version tags). It is not an input to `crucible module build`, contributes no resource that could appear in an NWSync manifest, and is a runtime peer the deploy resolves. Recording it would mean CI copying a SHA it never consumed — a guess, not a record. Codebase parity stays deploy-time, exactly as today. ### Decisions 1. **`topdata_release` is a record, not a binding.** Provenance only; nothing reads it to deploy. 2. **Two derived channel variables.** Existing `SOW_MODULE_HAK_CHANNEL` plus new `SOW_MODULE_TOPDATA_CHANNEL`, both set from one tag-shape `case` arm in `.gitea/workflows/release.yml:36-37` (`v*-*` → `testing`, `v*` → `current`). Pushing a tag stays the only input — no operator ever types a channel. Two variables rather than one because assets and topdata release independently and "rc haks, stable topdata" is a legitimate state to reproduce locally while debugging; CI sets both from the same arm, so a release can never diverge by accident. 3. **Topdata only.** Nothing recorded about sow-codebase (see above). 4. **`fetch-hak-manifest.sh` → `fetch-upstream-manifests.sh`**, extended to resolve the topdata channel as well. The script already owns "resolve upstream channels for this build" and already prepends `sow_top`; a second script would duplicate the same catalog shape, CDN, and fail-closed rules. Resolving at publish time was rejected outright — the build needs topdata resolved before it, or the two halves could observe different channel states. 5. **Coordinated rename flip.** The script name is a hardcoded path convention, not config: `sow-tools/internal/app/app.go:388` runs `<root>/scripts/fetch-hak-manifest`, and `crucible.sh:58` resolves the *latest* crucible release unpinned. Both sides flip together and the broken window is accepted; the failure is loud and unmistakable (`required project script is missing`). Smallest gap is merging the sow-module rename and cutting the crucible release back to back. Transitional symlink and Go-side fallback were both considered and rejected as cruft. 6. **Keep the catalogs.** `fetch-upstream-manifests.sh` writes `.cache/hak-catalog.json` and `.cache/topdata-catalog.json` verbatim, alongside `.cache/hak-release.txt` and `.cache/topdata-release.txt`. Assemble reads them locally and fetches nothing. Build and publish run in the same job (`release.yml`), so `.cache` survives between steps. This is also where the hak digests #58 found ("one deleted `jq` map away") come from — one change answers both sides. Re-fetching per tag would have been safe (per-tag catalogs are immutable) but needs a second fetch implementation and buys nothing. 7. **Field and schema.** `topdata_release`, a bare tag string, beside `hak_release` in `releases/module/<tag>/haks.json`. `schema` stays `1`: the change is additive, and `sow-depot-proxy/internal/server/downloads.go:234` decodes the catalog into a struct that ignores unknown fields and never reads `schema`. A bump would signal a break that is not there. 8. **Hard fail on all three topdata failure modes** — CDN unreachable, channel unset in `channels.json`, catalog present-but-malformed — matching the hak side (`fetch-hak-manifest.sh:34,37,42,45`), in CI and on developer machines alike. `promote-topdata.sh:67-70` seeds `testing` to `current` when unset, so the realistic failure is a genuine outage or a genuinely unpublished channel, both of which should stop a build. A local-only soft fail was rejected: it produces a build that succeeds locally and fails in CI for the same tree. 9. **`sow_top` stays hardcoded** in the `module_haks` prepend (`fetch-hak-manifest.sh:48`). Catalogs supply digests only, so `module_haks` output is byte-identical and `tests/fetch-hak-manifest-contract.sh` needs no change beyond the rename. 10. **No version suffixes on `sow_top.hak` / `sow_tlk.tlk`.** Considered and rejected. Version identity already lives in two places that cannot drift — the release tag now in `topdata_release`, and the `sha256` in the catalog; a versioned filename would be a third copy and the only one nothing verifies. It buys NWSync nothing: manifests list **resources**, never hak filenames, and blob names are the SHA-1 of each uncompressed resource, so client-side dedup is already per-resource. The costs are real — every topdata release would rewrite `Mod_HakList`, making the `.mod` binary itself differ by topdata tag (reintroducing exactly the lockstep this ticket keeps human); manual non-NWSync players would swap files each release and accumulate old ones; `nwn-tool.yaml`'s `hak_order` and docs would need lockstep updates; and NWN's `Mod_Hak` / `Mod_CustomTlk` `CExoString` length budget is tight for a name plus semver suffix. If "which topdata is this, offline, from the file itself" is ever wanted, the shape is a small version resource *inside* `sow_top.hak`, not a rename — held as an idea, not adopted. ### Implementation Rides with #65 (wiring the consumer repos), not a new ticket. Touches: `sow-module/scripts/fetch-hak-manifest.sh` (rename + topdata resolution), `publish-module-release.sh` (read `.cache/topdata-release.txt`, emit `topdata_release`), `sow-tools/internal/app/app.go:388`, the two contract tests under `sow-module/tests/`, plus the stale name in `.gitignore`, `flake.nix`, `nwn-tool.yaml` and `release.yml` comments. The `fetch-hak-manifest.sh:48-49` comment about parity being "deploy-pinned by sow-platform" should be corrected while there — platform pins nothing game-tier. ### Follow-up filed [sow-depot-proxy#22](https://git.westgate.pw/ShadowsOverWestgate/sow-depot-proxy/issues/22) — resolve `topdata_release` into `paired_topdata` on the downloads page, mirroring `paired_haks`. Kept separate: it is a downloads-page feature, not NWSync publishing.
archvillainette commented 2026-07-31 18:58:54 +00:00
Author
Owner
Copy Link
Copy Source

Decision 5 (Coordinated rename flip), crucible half: #84 flips internal/app/app.go:388 to scripts/fetch-upstream-manifests, matching the sow-module rename in ShadowsOverWestgate/sow-module#57. Tracked as #83.

Implemented as decided — no transitional symlink, no Go-side fallback, so the broken window is real: sow-module release CI runs nix flake update sow-tools unpinned, so every crucible module build there fails from the moment sow-module#57 merges until a crucible release carrying #84 is cut. Merging and releasing back to back keeps that gap short.

Decision 5 (Coordinated rename flip), crucible half: #84 flips `internal/app/app.go:388` to `scripts/fetch-upstream-manifests`, matching the sow-module rename in ShadowsOverWestgate/sow-module#57. Tracked as #83. Implemented as decided — no transitional symlink, no Go-side fallback, so the broken window is real: sow-module release CI runs `nix flake update sow-tools` unpinned, so every crucible module build there fails from the moment sow-module#57 merges until a crucible release carrying #84 is cut. Merging and releasing back to back keeps that gap short.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
wayfinder:grilling
wayfinder:map
wayfinder:prototype
wayfinder:research
wayfinder:task
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
No labels wayfinder:grilling
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
Chass FallenDabus WinterZen XeroTwo Zariu archvillainette gitea-bot xtul
No Assignees archvillainette
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: ShadowsOverWestgate/sow-tools#63