Does sow-module CI know the full artifact set at release time? #58

New Issue

2026-07-27T18:51:20Z

archvillainette commented

2026-07-27 18:51:20 +00:00

Question

At the moment sow-module cuts a release, does its CI actually know the full set of artifacts a manifest must cover — every hak digest, sow_top.hak and sow_tlk.tlk?

Part of #54. sow-platform#94 claimed only the channel resolver knows the four-tuple, which was the stated reason for putting assemble in the resolve step. That reason is now believed stale: #87 made publish-module-release.sh emit module_haks, and fetch-hak-manifest.sh prepends sow_top by name. But it has not been checked.

Verify against the actual scripts and catalogs in sow-module, sow-assets-manifest and sow-topdata:

What fetch-hak-manifest.sh resolves, in what order, and whether the resulting module_haks list carries digests or only names. assemble needs digests, because the per-artifact NSYM lives at artifacts/haks/sha256/<a>/<b>/<sha256>.nsym.
Whether the TLK is resolvable from module CI at all, and by what name and digest. #94 makes sow_tlk.tlk a manifest entry in its own right.
Whether the hak order module CI has is the same module_haks priority order the shadowing rule depends on — earliest hak wins, sow_top prepended and therefore winning over assets.
What module CI would have to fetch that it does not fetch today. The answer should confirm it is only small index files, never bulk data.

## Question At the moment sow-module cuts a release, does its CI actually know the full set of artifacts a manifest must cover — every hak digest, `sow_top.hak` and `sow_tlk.tlk`? Part of #54. [sow-platform#94](https://git.westgate.pw/ShadowsOverWestgate/sow-platform/issues/94) claimed only the channel resolver knows the four-tuple, which was the stated reason for putting `assemble` in the resolve step. That reason is now believed stale: #87 made `publish-module-release.sh` emit `module_haks`, and `fetch-hak-manifest.sh` prepends `sow_top` by name. But it has not been checked. Verify against the actual scripts and catalogs in `sow-module`, `sow-assets-manifest` and `sow-topdata`: - What `fetch-hak-manifest.sh` resolves, in what order, and whether the resulting `module_haks` list carries digests or only names. `assemble` needs digests, because the per-artifact NSYM lives at `artifacts/haks/sha256/<a>/<b>/<sha256>.nsym`. - Whether the TLK is resolvable from module CI at all, and by what name and digest. #94 makes `sow_tlk.tlk` a manifest entry in its own right. - Whether the hak order module CI has is the same `module_haks` priority order the shadowing rule depends on — earliest hak wins, `sow_top` prepended and therefore winning over assets. - What module CI would have to fetch that it does not fetch today. The answer should confirm it is only small index files, never bulk data.

archvillainette added the wayfinder:research label 2026-07-27 18:51:20 +00:00

archvillainette added a new dependency 2026-07-27 18:51:30 +00:00

#61 Bunny write credential for sow-module CI

archvillainette self-assigned this 2026-07-27 18:53:42 +00:00

archvillainette commented

2026-07-27 18:58:30 +00:00

Resolved: yes for haks, a small closeable gap for topdata

sow-platform#94's claim that only the channel resolver knows the full artifact set is stale. Module CI already resolves the hak side end to end — tag, names, SHA-256, sizes and URLs — and throws the digests away only because nothing needed them until now. Nothing here requires bulk data.

The hak set: digests are one deleted `jq` map away

fetch-hak-manifest.sh reads releases/haks/channels.json (:34), maps channel to hak tag (:36), fetches releases/haks/<tag>/haks.json (:42), and then discards everything but the names (:50):

jq '{ module_haks: ( ["sow_top"] + ( .haks | map(.name | sub("\\.hak$"; "")) ) ) }'

The digests are already in the file it downloaded. sow-assets-manifest/scripts/pack-haks.sh:145-147 emits {name, build_key, sha256, size_bytes, url, released_at} per hak, and publish-release.sh:33-36 enforces that shape at publish time — 64-hex sha256, integer size_bytes, https:// url. The url is artifacts/haks/sha256/<a>/<b>/<sha>.hak (hak-artifact-record.sh:23-24). So the fix is to stop discarding fields: keep module_haks as-is for ApplyHAKManifest, and emit the full records alongside it.

The order is the real resolution order

Traced end to end: hak-layout.yml:7-17 hak_order -> hak-manifest.yml:3-13 order: -> source-manifest.sh:35-36,96-99 -> pack-haks.sh:139 preserving array order in the published catalog -> fetch-hak-manifest.sh:50 prepending sow_top. Visible in the built result at source/module/module.ifo.json:350-400, where Mod_HakList starts sow_top, sow_over_01, sow_core_01.

Both ends are contract-tested — tests/fetch-hak-manifest-contract.sh:35 requires sow_top at module_haks[0], and :41-42 requires the tail to equal catalog array order, explicitly guarding the "re-derive from hak_order" trap. fetch-hak-manifest.sh:12-14 says the catalog is truth and warns against re-deriving.

So sow_top beats assets, exactly as in the running game, and the shadowing rule the assembler needs is already the order module CI holds.

The TLK: known by name only, digest publicly available but unfetched

Mod_CustomTlk is the literal "sow_tlk" in source/module/module.ifo.json:64-66, hand-held, never validated or updated by the build. fetch-hak-manifest.sh:48-49 states the deliberate choice: sow_top is prepended from hak_order, not resolved from a topdata channel, so module/topdata parity stays deploy-pinned.

The digest exists and is anon-readable. sow-topdata/scripts/publish-topdata-release.sh:58-66 publishes a catalog of identical shape at releases/topdata/<tag>/haks.json, with both sow_top.hak and sow_tlk.tlk carrying sha256 and size_bytes; the channel pointer is releases/topdata/channels.json (promote-topdata.sh:19), and :35-37 guarantees a channel only ever names a tag whose catalog is already published.

One thing this changes for the emit spec: topdata artifacts live under artifacts/topdata/sha256/..., not artifacts/haks/sha256/.... sow-tools#53 assumes the haks prefix for the NSYM path. Raised as its own ticket.

What module CI has to fetch that it does not today

Two small JSON files over the anon read it already performs — releases/topdata/channels.json and releases/topdata/<tag>/haks.json — plus a zero-fetch change to stop discarding hak catalog fields.

Confirmed no bulk data anywhere: the corpus is 15.27 GB across 11 members and module CI never downloads a .hak. pack-haks.sh:143 builds its catalog "from records only (never inspect or download HAK files)" — digests already flow as metadata by design.

CI runner, and a bonus for #61

.gitea/workflows/release.yml: triggers on v* tags (:8-10), runs-on: nix-docker (:14-16), and already has BUNNY_CDN_BASE in env (:17-18) and the secrets BUNNY_STORAGE_ZONE, BUNNY_STORAGE_PASSWORD, BUNNY_API_KEY (:41-68). It publishes the .mod content-addressed, then SHA256SUMS, then the catalog last as the publication boundary (publish-module-release.sh:71-72), then moves the channel pointer.

So sow-module CI already holds Bunny write credentials. #61 is largely pre-answered — it becomes a scoping question, not a plumbing one. ci.yml:10 runs every tests/*.sh contract on PR, so a manifest-assembly change lands under an existing gate.

One decision this surfaced

A module release records hak_release but has no topdata_release field (publish-module-release.sh:59). Assembling at release time means the release must know which topdata it shipped against — pin a tag, or read the current channel at build time. That is a real hole today independent of NWSync. Raised as its own ticket.

## Resolved: yes for haks, a small closeable gap for topdata [sow-platform#94](https://git.westgate.pw/ShadowsOverWestgate/sow-platform/issues/94)'s claim that only the channel resolver knows the full artifact set is **stale**. Module CI already resolves the hak side end to end — tag, names, SHA-256, sizes and URLs — and throws the digests away only because nothing needed them until now. Nothing here requires bulk data. ### The hak set: digests are one deleted `jq` map away `fetch-hak-manifest.sh` reads `releases/haks/channels.json` (`:34`), maps channel to hak tag (`:36`), fetches `releases/haks/<tag>/haks.json` (`:42`), and then discards everything but the names (`:50`): ``` jq '{ module_haks: ( ["sow_top"] + ( .haks | map(.name | sub("\\.hak$"; "")) ) ) }' ``` The digests are already in the file it downloaded. `sow-assets-manifest/scripts/pack-haks.sh:145-147` emits `{name, build_key, sha256, size_bytes, url, released_at}` per hak, and `publish-release.sh:33-36` enforces that shape at publish time — 64-hex `sha256`, integer `size_bytes`, `https://` url. The url is `artifacts/haks/sha256/<a>/<b>/<sha>.hak` (`hak-artifact-record.sh:23-24`). So the fix is to stop discarding fields: keep `module_haks` as-is for `ApplyHAKManifest`, and emit the full records alongside it. ### The order is the real resolution order Traced end to end: `hak-layout.yml:7-17` `hak_order` -> `hak-manifest.yml:3-13` `order:` -> `source-manifest.sh:35-36,96-99` -> `pack-haks.sh:139` preserving array order in the published catalog -> `fetch-hak-manifest.sh:50` prepending `sow_top`. Visible in the built result at `source/module/module.ifo.json:350-400`, where `Mod_HakList` starts `sow_top`, `sow_over_01`, `sow_core_01`. Both ends are contract-tested — `tests/fetch-hak-manifest-contract.sh:35` requires `sow_top` at `module_haks[0]`, and `:41-42` requires the tail to equal catalog array order, explicitly guarding the "re-derive from hak_order" trap. `fetch-hak-manifest.sh:12-14` says the catalog is truth and warns against re-deriving. **So `sow_top` beats assets, exactly as in the running game, and the shadowing rule the assembler needs is already the order module CI holds.** ### The TLK: known by name only, digest publicly available but unfetched `Mod_CustomTlk` is the literal `"sow_tlk"` in `source/module/module.ifo.json:64-66`, hand-held, never validated or updated by the build. `fetch-hak-manifest.sh:48-49` states the deliberate choice: `sow_top` is prepended from `hak_order`, **not** resolved from a topdata channel, so module/topdata parity stays deploy-pinned. The digest exists and is anon-readable. `sow-topdata/scripts/publish-topdata-release.sh:58-66` publishes a catalog of identical shape at `releases/topdata/<tag>/haks.json`, with both `sow_top.hak` and `sow_tlk.tlk` carrying `sha256` and `size_bytes`; the channel pointer is `releases/topdata/channels.json` (`promote-topdata.sh:19`), and `:35-37` guarantees a channel only ever names a tag whose catalog is already published. **One thing this changes for the emit spec:** topdata artifacts live under `artifacts/topdata/sha256/...`, not `artifacts/haks/sha256/...`. sow-tools#53 assumes the `haks` prefix for the NSYM path. Raised as its own ticket. ### What module CI has to fetch that it does not today Two small JSON files over the anon read it already performs — `releases/topdata/channels.json` and `releases/topdata/<tag>/haks.json` — plus a zero-fetch change to stop discarding hak catalog fields. Confirmed no bulk data anywhere: the corpus is 15.27 GB across 11 members and module CI never downloads a `.hak`. `pack-haks.sh:143` builds its catalog "from records only (never inspect or download HAK files)" — digests already flow as metadata by design. ### CI runner, and a bonus for #61 `.gitea/workflows/release.yml`: triggers on `v*` tags (`:8-10`), `runs-on: nix-docker` (`:14-16`), and already has `BUNNY_CDN_BASE` in env (`:17-18`) and the secrets `BUNNY_STORAGE_ZONE`, `BUNNY_STORAGE_PASSWORD`, `BUNNY_API_KEY` (`:41-68`). It publishes the `.mod` content-addressed, then `SHA256SUMS`, then the catalog last as the publication boundary (`publish-module-release.sh:71-72`), then moves the channel pointer. **So sow-module CI already holds Bunny write credentials.** #61 is largely pre-answered — it becomes a scoping question, not a plumbing one. `ci.yml:10` runs every `tests/*.sh` contract on PR, so a manifest-assembly change lands under an existing gate. ### One decision this surfaced A module release records `hak_release` but has no `topdata_release` field (`publish-module-release.sh:59`). Assembling at release time means the release must know which topdata it shipped against — pin a tag, or read the `current` channel at build time. That is a real hole today independent of NWSync. Raised as its own ticket.

archvillainette closed this issue

2026-07-27 18:58:30 +00:00

archvillainette referenced this issue

2026-07-27 19:00:30 +00:00

Where the per-artifact NSYM lives, now that topdata is not a hak #62

archvillainette referenced this issue

2026-07-27 19:00:31 +00:00

A module release does not record which topdata it shipped against #63

archvillainette referenced this issue

2026-07-27 19:01:14 +00:00

Map: NWSync publishing — emit at artifact birth, assemble at module release #54

archvillainette referenced this issue