sow-tools/docs/superpowers/specs/2026-06-21-crucible-build-parity-design.md

Crucible build parity — local == CI, zero-config reproducible

Date: 2026-06-21 Status: approved (design); implementation pending Repos touched: sow-tools (Crucible + contract doc + image notes), sow-topdata (config, flake, wrapper, CI), sow-assets-manifest (flake pin), and the consumer wrapper/flake pattern shared by every Crucible repo.

Supersedes the resolver-script approach in 2026-06-21-topdata-vfx-autogen-channel-design.md (the channel→tag→vfxs.yml resolution moves out of a bash wrapper and into Crucible).

Problem

crucible topdata build-topdata run by hand does not reproduce the CI build. CI runs scripts/build-topdata.sh, which resolves the accessory-VFX model list with scripts/resolve-accessory-vfx.sh and writes .cache/sow-accessory-vfx-manifest.json before Crucible runs; Crucible only reads that pre-staged file (nwn-tool.yaml: manifest_file:). Run Crucible directly and the file never exists, the optional consumer skips, and visualeffects.2da ships zero accessory rows — exactly the local build the operator observed (637 rows in CI, 0 locally).

Root cause: a build-output-affecting input is resolved outside Crucible, in a wrapper script. The wrapper, not Crucible, is authoritative for that input.

Goal

Crucible is the single authority for everything that lands in a build artifact. A clone of any consumer repo, on any system, builds the same bytes CI builds — with no secret beyond what cloning already required and no hand-written configuration.

Zero-config reproducibility requirements

• R1 — No extra secrets. All build-input reads are anonymous: the asset channel pointer (releases/haks/channels.json) and vfxs.yml are public CDN text. The crucible binary is fetchable anonymously — verified: the sow-tools release assets return HTTP 200 without auth, and the bootstrap tries anon before any token, so CRUCIBLE_TOKEN in consumer CI is vestigial and is removed. Git-LFS asset bytes use the same credentials the clone already used — never a separate key.
• R2 — No hand-written config. Every default needed to build is baked into committed nwn-tool.yaml and Crucible defaults: CDN base, channel selection, source paths. The user types crucible topdata build-topdata and nothing else.
• R3 — No host data assumptions, no NWN_ROOT. Builders read every input from the project tree (nwn-tool.yaml + committed data/**/base.json); no NWN install is required. NWN_ROOT is dead today — nothing in Crucible reads it (see NWN_ROOT cleanup below). The accessory build reads data/visualeffects/base.json. (Enforced by the CI parity guard, which runs in a bare checkout with no NWN_ROOT.)
• R4 — Nix is convenience, not a requirement. The flake devshell provides crucible, git-lfs, jq, yq, etc. on PATH for Nix users. Non-Nix users (incl. Windows) get the same Crucible via the crucible.sh/crucible.ps1 bootstrap and a host git-lfs; the build path through Crucible is identical.
• R5 — Same Crucible, both worlds. Local dev and CI invoke the same Crucible subcommand against the same committed config. No CI-only pre-steps that change output.

Parity contract (prevents recurrence)

Added to sow-tools/docs/consumer-contract.md. A consumer build wrapper may only:

1. validate — a pre-build gate that does not change output;
2. publish — post-build, on a separate artifact;
3. resolve the crucible binary — the bootstrap in consumer-contract.md.

A wrapper MUST NOT fetch, resolve, generate, or stage any input Crucible reads to produce the artifact. If Crucible needs an input, Crucible acquires it. This is the existing "no local-machine assumptions" + "same inputs → identical output" rule, made explicit: the wrapper is not allowed to be authoritative for build inputs.

Design

A. Accessory-VFX resolution moves into Crucible

Config (sow-topdata/nwn-tool.yaml). Drop manifest_file:; declare the source on the consumer. The 4-group + mdl filter is not re-declared — it is derived from the keys already present under accessory_visualeffects.groups:

autogen:
  consumers:
    - id: accessory_visualeffects
      producer: accessory_visualeffects
      dataset: visualeffects
      mode: accessory_visualeffects
      optional: true
      source:
        kind: cdn_channel
        cdn_base_env: BUNNY_CDN_BASE             # default https://depot.westgate.pw
        channels_path: releases/haks/channels.json
        manifest_path: releases/haks/{tag}/vfxs.yml
        release_marker_path: releases/haks/{tag}/haks.json   # the broken-release guard probe
        channel_env: SOW_TOPDATA_ASSET_CHANNEL   # else derive from git tag
        offline_override_env: SOW_VFXS_MANIFEST  # dev/air-gapped vfxs.yml path
      accessory_visualeffects:
        groups: { chest_accessories: {...}, head_accessories: {...},
                  head_decorations: {...}, head_features: {...} }
        # ... rest of the policy block unchanged

New struct field on AutogenConsumerConfig (sow-tools internal/project/project.go): Source AutogenSourceConfig with the keys above. Validation: when source.kind == cdn_channel, channels_path and manifest_path are required and manifest_path must contain {tag}.

Resolution (internal/topdata/autogen.go, resolveAutogenConsumerManifest). A new branch for source.kind == cdn_channel, reusing the existing fetchJSON HTTP client and the project YAML parser:

1. Offline override. If offline_override_env names a readable file (or a manifest-repo checkout root containing assets/vfxs.yml), parse it and skip the network — the dev/air-gapped path.
2. Channel. channel_env if set; else from the git tag / GITHUB_REF_NAME (v*-* → testing, v* → current, otherwise current).
3. CDN base. cdn_base_env value, else the baked default.
4. GET {cdn}/{channels_path} → JSON → tag = channels[channel].
5. GET {cdn}/{manifest_path with {tag}} → parse YAML assets[].
6. Filter to restype == mdl whose path is under vfxs/<g>/ for <g> in the accessory_visualeffects.groups keys. Derive each entry's source (leading vfxs/ stripped), group, subgroup, model_stem. Sort by source. Hand the entries to the existing accessory augmentor unchanged.

Fail policy (ported verbatim from the resolver — this is the v0.1.4-bug guard). Distinguish unreachable (fail open) from broken (hard fail):

• channels.json unreachable / not JSON / channel absent → fail open: the optional consumer contributes no rows and data/visualeffects/lock.json IDs are preserved.
• vfxs.yml network error, or 404 and the release_marker_path (haks.json) is absent for that tag (no published release) → fail open.
• vfxs.yml 404 and haks.json present → HARD fail ("release <tag> has haks.json but no vfxs.yml — accessory rows would silently vanish").
• vfxs.yml present but malformed / no assets array → HARD fail.

Fail-open writes nothing and lets the optional consumer skip; hard-fail aborts the build with the message above.

Deleted: sow-topdata/scripts/resolve-accessory-vfx.sh, tests/resolve-accessory-vfx-contract.sh, the manifest_file: key, and the resolver pre-step at build-topdata.sh:19. The four bash contract cases (offline override, unreachable fail-open, broken-release hard-fail, no-release fail-open) are re-expressed as Go tests in internal/topdata.

B. Git-LFS materialization moves into Crucible

So a bare clone + crucible packs real bytes, not pointer stubs (today only build-topdata.sh pulls LFS, so direct Crucible would ship stubs in CI's non-smudging checkout).

Before packing assets into a hak, Crucible: detects git-LFS pointer stubs under the asset tree; if any, runs git lfs install --local then git lfs pull in the repo; re-checks and hard-fails if stubs remain ("refusing to build from pointer stubs"). Uses the clone's own credentials (R1). Requires git + git-lfs on PATH — provided by the Nix devshell, a standard host install otherwise (a clone without git-lfs already yields stubs, so this is the existing host expectation, now enforced loudly by Crucible instead of a wrapper).

A skip control replaces TOPDATA_SKIP_LFS for the maintain-tree flow, which regenerates the data/ tree and discards the package: a Crucible flag/env (e.g. --skip-lfs / CRUCIBLE_SKIP_LFS) so that one job can opt out. Default is fail-closed (pull).

C. Wrappers thin out

scripts/build-topdata.sh (and the shared pattern in every Crucible repo) reduces to: validate (gate) → resolve crucible → crucible <build>. No input resolution, no LFS pull. sow-module/scripts/package-module.sh already matches this shape (validate → resolve → build); it stays the reference.

D. CI parity guard (catches the next regression)

A sow-tools integration test builds the topdata package via Crucible alone in a bare checkout (no wrapper, clean .cache, no NWN_ROOT, anonymous CDN) and asserts visualeffects.2da contains accessory rows. This is the test that would have caught today's bug: it fails if any build-output-affecting step ever drifts back out of Crucible into a wrapper, and it enforces R2/R3/R5 mechanically.

The contract rule in consumer-contract.md is the written half; this test is the enforced half.

E. NWN_ROOT cleanup (stale, contradicts the goal)

NWN_ROOT is unused — no os.Getenv, no --nwn-root flag, no consumer. The only traces are documentation/help-text describing a rule for a thing that does not exist: the ## NWN_ROOT rule section in consumer-contract.md, the help text in internal/dispatch/dispatch.go ("if a builder ever needs an NWN root… crucible never guesses from $HOME"), and the README.md pointer. That old rule — never guess, require an explicit env — directly contradicts the zero-config goal (R2): it would make a future builder demand a hand-set path.

• Remove the NWN_ROOT references: the consumer-contract.md section, the dispatch help-text lines, the README pointer. Leave the migration-from-nwn-tool.md bullet as historical record.
• Replace the forward rule. Should a future builder ever need NWN game data, it auto-detects it (the NWN home — containing modules/, hak/, tlk/ — and the Steam install path are reliably discoverable), with an optional explicit override for non-standard layouts. Auto-detect first, never a required hand-set env. This keeps "clone → run → Just Work" intact even for a builder that one day reads game data. None do today.

F. Toolchain convergence — one Crucible, three modes

Parity needs everyone to run the same Crucible. Today three mechanisms diverge: non-Nix uses the crucible.sh bootstrap download; sow-assets-manifest's flake pins crucible v0.2.5 via fetchurl+hash; sow-topdata's flake ships no crucible at all (falls back to the bootstrap, with a needless CRUCIBLE_TOKEN). The pin is scattered (a hardcoded flake version, none in another flake, an image sha in prod.yml, a HEAD-floating CI binary build).

Converge on one binary, one pin, three modes — all anonymous, all identical:

1. Non-Nix (local + CI fallback). crucible.sh / crucible.ps1 bootstrap downloads the release binary anonymously. CRUCIBLE_TOKEN is dropped from consumer CI (R1).
2. Nix (local + CI optimized). Every consumer flake provides crucible in its devshell from a sow-tools flake input, pinned by that repo's flake.lock. Idiomatic, reproducible, no manual hashes; bump with nix flake update. This replaces sow-assets-manifest's manual fetchurl pin and adds crucible to sow-topdata's devshell (today it has none).
3. CI runs the same crucible <build> as local, inside the nix devshell (binary-cache fast). No build container, no token, no CI-only pre-steps.

Pin source of truth = flake.lock per repo, bumped deliberately in a PR. We reject floating "latest": it would let local drift from CI between releases and break reproducibility. The non-Nix bootstrap, which resolves "latest" by default, is the convenience path; reproducible/parity-critical builds pin (nix devshell, or CRUCIBLE_* pin env for the bootstrap).

G. The crucible container is deployment-only

sow-tools builds and publishes registry.westgate.pw/deployment/crucible:<sha> on v* tags (nix build .#image → skopeo). It exists for deployment — prod.yml pins it so tools travel with the runtime host (nwn.enable), and it is a disabled placeholder until the NWN stack lands. No build/CI job consumes it, and the host-mode runners have no container runtime to run it. It is not a build tool.

• Keep the image build (cheap, v*-gated) for its deployment purpose.
• Fix consumer-contract.md: remove the crossed-out "run CI inside the pinned image" guidance and its OPERATOR NOTE; state plainly that builds use the binary (bootstrap or nix devshell) and the image is deployment-only. This kills the "why do we build a container we don't use?" confusion.

H. Autogen lock identity — model-anchored, immutable

General lock contract (hard rule, all datasets). A dataset lock maps a stable key to a row id. The build MUST: reuse the existing id for a key that still exists; allocate the first free id (lexicographic 0→1→2…) for a new key; and free an id only when its key is gone everywhere. An id, once assigned to a live key, never changes. (Already implemented for the accessory consumer via historicalLockData reuse + nextAvailableAutogenID; this section makes it the written rule and fixes the autogen-specific anchor below.)

The autogen problem. Autogen rows describe models that do not exist in sow-topdata/data/ — they live in the asset vfxs.yml manifest. Today the lock key is the config-derived presentation key: KeyFormat = {dataset}:{group}{delimiter}{category_segment}{stem}. Group token, category, delimiter, case, and prefix-stripping all come from nwn-tool.yaml, so a config-only change (key_format, delimiter, case, a group rename, category_from, strip_model_prefixes) changes the key → the lock no longer matches → a new id is allocated → every row shifts. That is a config edit silently reshuffling ids for models that never changed.

Fix — anchor the lock identity on the dataset + model, not the presentation. The lock key for an autogen row becomes {dataset}:{model_source_path} — the stable dataset namespace prefix (kept as the 2da/namespace linker developers use instead of raw ids) plus the model source path from vfxs.yml, e.g. visualeffects:head_accessories/hat/hfx_bandana.mdl (leading vfxs/ stripped — the path is the value already on entry.Source). Dropped from the key are only the config-derived parts (group token, category_segment, delimiter, case, prefix-stripping). The row's key/label columns still derive from config via KeyFormat/LabelFormat — only the lock identity is decoupled and model-anchored. Consequences:

• A config-only change keeps every id (the path is unchanged).
• A model re-export (same path, new sha256) keeps its id — a content change is not an identity change. (Audit trail comes free from git diff on the committed vfxs.yml, which carries path + sha256; the lock stays a flat string→int map — no schema change, no fork of the shared lock format. A future id+sha object form is a clean later add if observability ever needs it, without changing id semantics.)
• A renamed/moved/removed model is a genuinely different (or absent) identity.

One-time cutover remap. On the first build under {dataset}:{path} keys, for each current model compute its old config-key (with the current policy) and its new {dataset}:{path} key; if the new key is absent but the old key carries an id in the lock, adopt that id under the new key. So ids carry over with zero shift on cutover; the now-unreferenced old keys fall stale and are pruned (below).

Stale pruning. A lock key absent from the resolved entry set is dropped and its id freed for first-free reuse — but only on a successful, non-empty resolution. The fail-open paths (section A) contribute no entries and skip the augmentor entirely, so an unreachable/missing manifest can never mass-prune the lock. This realizes the general "free a fully-stale id" rule without risking the lock on a transient source outage.

Testing. Go tests in internal/topdata: config-only change (rename a group token, change key_format/delimiter/case) → ids unchanged; model re-export (same path, different sha) → id unchanged; model rename/remove → old id freed and reusable; cutover remap → existing ids preserved across the key-scheme switch; fail-open → no pruning, lock untouched.

Determinism vs. channel mutability (decision)

The committed topdata commit pins everything except the asset channel pointer. current resolves through channels.json, which is intentionally mutable, so "reproducible" here means same commit + same channel state → identical bytes, not "this commit is frozen forever." This is deliberate: the accessory rows must track the asset HAKs shipped beside sow_top.hak, which is exactly what the channel pointer expresses. A given asset release tag is immutable; the pointer moving is a real input change, and lock.json preserves accessory IDs across it (no reshuffle). Builds against an explicit tag (SOW_TOPDATA_ASSET_CHANNEL set, or a v* ref) are fully pinned.

This satisfies R1–R5: no secret, no config, deterministic for a fixed input set, on any system.

Testing

• Go unit tests (sow-tools/internal/topdata) for the cdn_channel resolution: offline-override path; unreachable channels.json → fail open; channel absent → fail open; vfxs.yml 404 + no haks.json → fail open; vfxs.yml 404 + haks.json present → hard fail (asserts message); malformed vfxs.yml → hard fail; happy path → correct filtered/sorted entries (mdl-only, 4-groups- only, vfxs/ stripped, group/subgroup/stem). Mirrors the deleted bash contract cases.
• Go config validation test for the new source block.
• CI parity guard (section D) in the topdata build workflow.
• Existing accessory augmentor / lock-preserve tests unchanged.

Rollout

1. sow-tools: add AutogenSourceConfig, the cdn_channel resolution branch, LFS materialization, the model-anchored lock identity + cutover remap + stale pruning (section H), the parity-guard test; update consumer-contract.md (add the parity contract, remove the NWN_ROOT rule per section E, replace the image guidance with deployment-only per section G).
2. Release Crucible (assets already anon-downloadable — R1).
3. Toolchain convergence (section F): expose crucible from a sow-tools flake input in each consumer flake — add it to sow-topdata's devshell, switch sow-assets-manifest off its manual fetchurl pin; drop CRUCIBLE_TOKEN from consumer CI; CI builds inside the nix devshell.
4. sow-topdata: swap manifest_file: → source: in nwn-tool.yaml; delete the resolver script + bash contract test; thin build-topdata.sh (validate → resolve crucible → build); pin Crucible via flake.lock.
5. Verify: bare clone → crucible topdata build-topdata → 637 accessory rows, no env, no key; and the same command inside nix develop.

Follow-ups (not this work)

• Parts 2da / cachedmodels reinstatement (separate spec) can reuse the same cdn_channel source type.
• Roll the thinned-wrapper pattern review across sow-module / sow-codebase.