Wire the consumer repos to the settled crucible nwsync interface #65
No Assignees
archvillainette
archvillainette
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: ShadowsOverWestgate/sow-tools#65
Reference in New Issue
Block a user
Question
Part of #54.
The tool interface is settled: #62 fixed the NSYM key rule and #56 fixed the command surface —
Nothing about the verbs is still open, so the consumer issues can now say exactly what each repo has to run.
Bring the three producer repos plus sow-module onto that interface, as issue text — not as CI changes:
emitper hak at artifact birth, positional artifact key, no--out, credentials fromNWSYNC_STORAGE_ZONE/NWSYNC_STORAGE_PASSWORD.--as sow_tlk.tlkentry.assembleafter both upstream releases are known, derives--group-idfrom the tag shape (v*= 1,v*-rc*= 2), and passes hak keys inMod_HakListorder with--tlk-keyfor the TLK. #58 established it already has the digests and the order.Settle per repo: which workflow step it hangs off, which secrets that repo still needs (only sow-module currently holds
NWSYNC_*), and what the failure mode is when a blob is already present.Resolution
All three consumer repos are wired as issue text, plus one gap in the settled surface closed and one fog patch graduated. The invocations were settled against the real scripts, not against the descriptions in the existing consumer issues — both of which named a hook point that cannot work.
The surface gap:
emittakes two positionals, not one#56 says keys are positional and
emittakes exactly one. #62 needs a local file in the same call — its fail-closed guard hashes "the file it was handed" and its TLK example isemit --as sow_tlk.tlk <file>. Only one reading survives:emit [--as <name>] [--out DIR] <artifact-key> <file>— key is identity, file is bytes.The alternative (one key,
emitfetches the artifact from the depot by it) would have sow-assets-manifest re-download 15.27 GB it had in hand seconds earlier, which is the exact thing sow-assets-manifest#48 exists to prevent. It also has no code:internal/nwsync/run.go:57-77already takes a local path as its single positional, so the file positional exists today and #56 adds the key on top of it.Rider on the implementation, not a new decision: #56's usage line,
docs/command-surface.md:37-51andinternal/dispatch/dispatch.go:113-114all still show the pre-#56 surface (emit <artifact> --out DIR,assemble --order NAMES --entries DIR). They get corrected when the verbs are built.sow-assets-manifest#48 — comment
Hook point corrected: inside
scripts/pack-haks.sh's per-hak loop, not in the workflows. The issue said "afterpack-haks.sh, before the haks are discarded"; the hak never leaves the script alive (:123 rm -f "$hak", one hak on disk at a time). New step 4.5, afterartifact_put(:105) and before the by-build record (:108) — the record is this repo's publication marker and its reuse gate, so emitting before it makes "a hak with a record has an NSYM" true by construction, reusing the retry semantics already at:103-104: emit fails → no record → retry rebuilds that one hak. Emitting after the record would open a silent hole, because the reuse branch (:78-85)continues before any hak is built and would skip emission forever.No
--as(a hak is an ERF; entry names come from inside), no--out(uploading is the default). Neither workflow gets a new step — both call the same script and only gain env.sow-topdata#178 — comment
Two calls at the
put_blobcall sites (publish-topdata-release.sh:45-46), not insideput_blob, since only the TLK carries--as sow_tlk.tlk. Both files persist in.cache/(:16-17), so bytes are in hand, and this sits before the catalog build and well beforehaks.jsonuploads last as the publication boundary (:72-73) — a failed emit hard-fails before any release metadata exists.sow-module — new issue #51
A new step in
release.ymlafter "Publish to depot + promote channel" (:42-67): assemble publishes a syncable manifest under a group pointer, so running it before the.modis on the depot would let clients sync the haks of a release that then failed to ship.--group-idfrom the same tag-shapecasethe workflow already uses forSOW_MODULE_HAK_CHANNEL(:33-36). Keys positional inmodule_haksorder, not reversed — reversal is only for feeding upstream's resman when comparing. #63's rename rides with it, and the reason is now concrete:fetch-hak-manifest.sh:41-42fetches the hak catalog into amktemp -dthe trap deletes, so the digests the assembler needs are fetched and thrown away today; only.cache/hak-release.txtsurvives.Failure mode when a blob is already present: skip, not an error
Both emitters get the same rule, and it supersedes "list the zone once, diff by name" in both issue bodies. Per #55: per-object 1-byte range GET probe (
HEADis banned; listing costs 65,793 calls against 71,436 probes), present means skip and count, only a probe or upload error fails the run. sow-topdata'sput_blobalready does exactly this for the artifact (:37-42). The NSYM key is derived from the artifact's own digest, so a re-emit can only write the same index; blobs first, NSYM last, so NSYM presence stays the publication marker. Forassemblethe question does not arise — it writes no blobs.Secrets per repo
NWSYNC_STORAGE_ZONE+NWSYNC_STORAGE_PASSWORD(new).BUNNY_*here is the depot key: different zone, different credential.New finding, applies to all three: every call site must also set
BUNNY_STORAGE_HOST: storage.bunnycdn.com. Crucible's env resolution has no default host (internal/depot/config.go:42—cfg.StorageHost = getenv("BUNNY_STORAGE_HOST"), comment says "no default"), while each repo's ownscripts/depot-lib.shdefaults it. So the shell path works and the Crucible path fails with an unset host unless the step passes it. Written into all three issues.Fog graduated
"Correcting the platform record" is now specifiable — the sow-module design is firm — and became Correct the platform record: assemble runs in sow-module release CI. Takeable now; nothing blocks it.