sow-tools/docs/superpowers/specs/2026-06-21-crucible-build-parity-design.md

# Crucible build parity — local == CI, zero-config reproducible

Date: 2026-06-21
Status: approved (design); implementation pending
Repos touched: `sow-tools` (Crucible + contract doc + image notes),
`sow-topdata` (config, flake, wrapper, CI), `sow-assets-manifest` (flake pin),
and the consumer wrapper/flake pattern shared by every Crucible repo.

Supersedes the resolver-script approach in
`2026-06-21-topdata-vfx-autogen-channel-design.md` (the channel→tag→`vfxs.yml`
resolution moves out of a bash wrapper and into Crucible).

## Problem

`crucible topdata build-topdata` run by hand does **not** reproduce the CI
build. CI runs `scripts/build-topdata.sh`, which resolves the accessory-VFX
model list with `scripts/resolve-accessory-vfx.sh` and writes
`.cache/sow-accessory-vfx-manifest.json` **before** Crucible runs; Crucible only
reads that pre-staged file (`nwn-tool.yaml: manifest_file:`). Run Crucible
directly and the file never exists, the `optional` consumer skips, and
`visualeffects.2da` ships **zero accessory rows** — exactly the local build the
operator observed (637 rows in CI, 0 locally).

Root cause: a **build-output-affecting input is resolved outside Crucible**, in
a wrapper script. The wrapper, not Crucible, is authoritative for that input.

## Goal

Crucible is the single authority for everything that lands in a build artifact.
A clone of any consumer repo, on any system, builds the same bytes CI builds —
with no secret beyond what cloning already required and no hand-written
configuration.

### Zero-config reproducibility requirements

- **R1 — No extra secrets.** All build-input reads are anonymous: the asset
  channel pointer (`releases/haks/channels.json`) and `vfxs.yml` are public CDN
  text. The crucible binary is fetchable anonymously — **verified**: the
  sow-tools release assets return HTTP 200 without auth, and the bootstrap tries
  anon before any token, so `CRUCIBLE_TOKEN` in consumer CI is **vestigial and is
  removed**. Git-LFS asset bytes use the **same** credentials the clone already
  used — never a separate key.
- **R2 — No hand-written config.** Every default needed to build is baked into
  committed `nwn-tool.yaml` and Crucible defaults: CDN base, channel selection,
  source paths. The user types `crucible topdata build-topdata` and nothing else.
- **R3 — No host data assumptions, no `NWN_ROOT`.** Builders read every input
  from the project tree (`nwn-tool.yaml` + committed `data/**/base.json`); no NWN
  install is required. `NWN_ROOT` is **dead** today — nothing in Crucible reads it
  (see *NWN_ROOT cleanup* below). The accessory build reads
  `data/visualeffects/base.json`. (Enforced by the CI parity guard, which runs in
  a bare checkout with no `NWN_ROOT`.)
- **R4 — Nix is convenience, not a requirement.** The flake devshell provides
  `crucible`, `git-lfs`, `jq`, `yq`, etc. on `PATH` for Nix users. Non-Nix users
  (incl. Windows) get the same Crucible via the `crucible.sh`/`crucible.ps1`
  bootstrap and a host `git-lfs`; the build path through Crucible is identical.
- **R5 — Same Crucible, both worlds.** Local dev and CI invoke the same Crucible
  subcommand against the same committed config. No CI-only pre-steps that change
  output.

## Parity contract (prevents recurrence)

Added to `sow-tools/docs/consumer-contract.md`. A consumer build wrapper may only:

1. **validate** — a pre-build gate that does not change output;
2. **publish** — post-build, on a separate artifact;
3. **resolve the crucible binary** — the bootstrap in `consumer-contract.md`.

A wrapper MUST NOT fetch, resolve, generate, or stage any input Crucible reads
to produce the artifact. If Crucible needs an input, Crucible acquires it. This
is the existing "no local-machine assumptions" + "same inputs → identical
output" rule, made explicit: **the wrapper is not allowed to be authoritative
for build inputs.**

## Design

### A. Accessory-VFX resolution moves into Crucible

**Config (`sow-topdata/nwn-tool.yaml`).** Drop `manifest_file:`; declare the
source on the consumer. The 4-group + `mdl` filter is **not** re-declared — it
is derived from the keys already present under `accessory_visualeffects.groups`:

```yaml
autogen:
  consumers:
    - id: accessory_visualeffects
      producer: accessory_visualeffects
      dataset: visualeffects
      mode: accessory_visualeffects
      optional: true
      source:
        kind: cdn_channel
        cdn_base_env: BUNNY_CDN_BASE             # default https://depot.westgate.pw
        channels_path: releases/haks/channels.json
        manifest_path: releases/haks/{tag}/vfxs.yml
        release_marker_path: releases/haks/{tag}/haks.json   # the broken-release guard probe
        channel_env: SOW_TOPDATA_ASSET_CHANNEL   # else derive from git tag
        offline_override_env: SOW_VFXS_MANIFEST  # dev/air-gapped vfxs.yml path
      accessory_visualeffects:
        groups: { chest_accessories: {...}, head_accessories: {...},
                  head_decorations: {...}, head_features: {...} }
        # ... rest of the policy block unchanged
```

New struct field on `AutogenConsumerConfig` (sow-tools
`internal/project/project.go`): `Source AutogenSourceConfig` with the keys
above. Validation: when `source.kind == cdn_channel`, `channels_path` and
`manifest_path` are required and `manifest_path` must contain `{tag}`.

**Resolution (`internal/topdata/autogen.go`,
`resolveAutogenConsumerManifest`).** A new branch for `source.kind ==
cdn_channel`, reusing the existing `fetchJSON` HTTP client and the project YAML
parser:

1. **Offline override.** If `offline_override_env` names a readable file (or a
   manifest-repo checkout root containing `assets/vfxs.yml`), parse it and skip
   the network — the dev/air-gapped path.
2. **Channel.** `channel_env` if set; else from the git tag / `GITHUB_REF_NAME`
   (`v*-*` → `testing`, `v*` → `current`, otherwise `current`).
3. **CDN base.** `cdn_base_env` value, else the baked default.
4. `GET {cdn}/{channels_path}` → JSON → `tag = channels[channel]`.
5. `GET {cdn}/{manifest_path with {tag}}` → parse YAML `assets[]`.
6. **Filter** to `restype == mdl` whose `path` is under `vfxs/<g>/` for `<g>` in
   the `accessory_visualeffects.groups` keys. Derive each entry's `source`
   (leading `vfxs/` stripped), `group`, `subgroup`, `model_stem`. Sort by
   `source`. Hand the entries to the existing accessory augmentor unchanged.

**Fail policy (ported verbatim from the resolver — this is the v0.1.4-bug
guard).** Distinguish *unreachable* (fail open) from *broken* (hard fail):

- channels.json unreachable / not JSON / channel absent → **fail open**: the
  `optional` consumer contributes no rows and `data/visualeffects/lock.json` IDs
  are preserved.
- vfxs.yml network error, or 404 **and** the `release_marker_path` (haks.json)
  is **absent** for that tag (no published release) → **fail open**.
- vfxs.yml 404 **and** haks.json **present** → **HARD fail** ("release `<tag>`
  has haks.json but no vfxs.yml — accessory rows would silently vanish").
- vfxs.yml present but malformed / no `assets` array → **HARD fail**.

Fail-open writes nothing and lets the optional consumer skip; hard-fail aborts
the build with the message above.

**Deleted:** `sow-topdata/scripts/resolve-accessory-vfx.sh`,
`tests/resolve-accessory-vfx-contract.sh`, the `manifest_file:` key, and the
resolver pre-step at `build-topdata.sh:19`. The four bash contract cases
(offline override, unreachable fail-open, broken-release hard-fail, no-release
fail-open) are re-expressed as Go tests in `internal/topdata`.

### B. Git-LFS materialization moves into Crucible

So a bare clone + `crucible` packs real bytes, not pointer stubs (today only
`build-topdata.sh` pulls LFS, so direct Crucible would ship stubs in CI's
non-smudging checkout).

Before packing assets into a hak, Crucible: detects git-LFS pointer stubs under
the asset tree; if any, runs `git lfs install --local` then `git lfs pull` in
the repo; re-checks and **hard-fails** if stubs remain ("refusing to build from
pointer stubs"). Uses the clone's own credentials (R1). Requires `git` +
`git-lfs` on `PATH` — provided by the Nix devshell, a standard host install
otherwise (a clone without git-lfs already yields stubs, so this is the existing
host expectation, now enforced loudly by Crucible instead of a wrapper).

A skip control replaces `TOPDATA_SKIP_LFS` for the maintain-tree flow, which
regenerates the `data/` tree and **discards** the package: a Crucible flag/env
(e.g. `--skip-lfs` / `CRUCIBLE_SKIP_LFS`) so that one job can opt out. Default
is fail-closed (pull).

### C. Wrappers thin out

`scripts/build-topdata.sh` (and the shared pattern in every Crucible repo)
reduces to: validate (gate) → resolve crucible → `crucible <build>`. No input
resolution, no LFS pull. `sow-module/scripts/package-module.sh` already matches
this shape (validate → resolve → build); it stays the reference.

### D. CI parity guard (catches the next regression)

A `sow-tools` integration test builds the topdata package via Crucible **alone**
in a bare checkout (no wrapper, clean `.cache`, no `NWN_ROOT`, anonymous CDN)
and asserts `visualeffects.2da` contains accessory rows. This is the test that
would have caught today's bug: it fails if any build-output-affecting step ever
drifts back out of Crucible into a wrapper, and it enforces R2/R3/R5 mechanically.

The contract rule in `consumer-contract.md` is the written half; this test is
the enforced half.

### E. NWN_ROOT cleanup (stale, contradicts the goal)

`NWN_ROOT` is unused — no `os.Getenv`, no `--nwn-root` flag, no consumer. The
only traces are documentation/help-text describing a rule for a thing that does
not exist: the `## NWN_ROOT rule` section in `consumer-contract.md`, the help
text in `internal/dispatch/dispatch.go` ("if a builder ever needs an NWN
root… crucible never guesses from $HOME"), and the `README.md` pointer. That
old rule — *never guess, require an explicit env* — directly contradicts the
zero-config goal (R2): it would make a future builder demand a hand-set path.

- **Remove** the `NWN_ROOT` references: the `consumer-contract.md` section, the
  dispatch help-text lines, the README pointer. Leave the
  `migration-from-nwn-tool.md` bullet as historical record.
- **Replace the forward rule.** Should a future builder ever need NWN game data,
  it **auto-detects** it (the NWN home — containing `modules/`, `hak/`, `tlk/` —
  and the Steam install path are reliably discoverable), with an optional
  explicit override for non-standard layouts. Auto-detect first, never a required
  hand-set env. This keeps "clone → run → Just Work" intact even for a builder
  that one day reads game data. None do today.

### F. Toolchain convergence — one Crucible, three modes

Parity needs everyone to run the *same* Crucible. Today three mechanisms
diverge: non-Nix uses the `crucible.sh` bootstrap download; sow-assets-manifest's
flake pins `crucible v0.2.5` via `fetchurl`+hash; sow-topdata's flake ships no
crucible at all (falls back to the bootstrap, with a needless `CRUCIBLE_TOKEN`).
The pin is scattered (a hardcoded flake version, none in another flake, an image
sha in `prod.yml`, a HEAD-floating CI binary build).

Converge on **one binary, one pin, three modes — all anonymous, all identical**:

1. **Non-Nix (local + CI fallback).** `crucible.sh` / `crucible.ps1` bootstrap
   downloads the release binary anonymously. `CRUCIBLE_TOKEN` is dropped from
   consumer CI (R1).
2. **Nix (local + CI optimized).** Every consumer flake provides `crucible` in
   its devshell from a **sow-tools flake input**, pinned by that repo's
   `flake.lock`. Idiomatic, reproducible, no manual hashes; bump with
   `nix flake update`. This replaces sow-assets-manifest's manual `fetchurl`
   pin and **adds crucible to sow-topdata's devshell** (today it has none).
3. **CI** runs the *same* `crucible <build>` as local, inside the nix devshell
   (binary-cache fast). No build container, no token, no CI-only pre-steps.

**Pin source of truth = `flake.lock` per repo**, bumped deliberately in a PR. We
**reject floating "latest"**: it would let local drift from CI between releases
and break reproducibility. The non-Nix bootstrap, which resolves "latest" by
default, is the convenience path; reproducible/parity-critical builds pin (nix
devshell, or `CRUCIBLE_*` pin env for the bootstrap).

### G. The crucible container is deployment-only

`sow-tools` builds and publishes `registry.westgate.pw/deployment/crucible:<sha>`
on `v*` tags (`nix build .#image` → `skopeo`). It exists for **deployment** —
`prod.yml` pins it so tools travel with the runtime host (`nwn.enable`), and it
is a **disabled placeholder** until the NWN stack lands. No build/CI job consumes
it, and the host-mode runners have no container runtime to run it. It is **not a
build tool.**

- Keep the image build (cheap, `v*`-gated) for its deployment purpose.
- **Fix `consumer-contract.md`:** remove the crossed-out "run CI inside the
  pinned image" guidance and its OPERATOR NOTE; state plainly that builds use the
  binary (bootstrap or nix devshell) and the image is deployment-only. This kills
  the "why do we build a container we don't use?" confusion.

### H. Autogen lock identity — model-anchored, immutable

**General lock contract (hard rule, all datasets).** A dataset lock maps a stable
key to a row id. The build MUST: reuse the existing id for a key that still
exists; allocate the first free id (lexicographic 0→1→2…) for a new key; and free
an id only when its key is gone everywhere. An id, once assigned to a live key,
never changes. (Already implemented for the accessory consumer via
`historicalLockData` reuse + `nextAvailableAutogenID`; this section makes it the
written rule and fixes the autogen-specific anchor below.)

**The autogen problem.** Autogen rows describe models that do **not** exist in
`sow-topdata/data/` — they live in the asset `vfxs.yml` manifest. Today the lock
key is the *config-derived presentation key*:
`KeyFormat = {dataset}:{group}{delimiter}{category_segment}{stem}`. Group token,
category, delimiter, case, and prefix-stripping all come from `nwn-tool.yaml`, so
a **config-only** change (`key_format`, `delimiter`, `case`, a group rename,
`category_from`, `strip_model_prefixes`) changes the key → the lock no longer
matches → a new id is allocated → **every row shifts**. That is a config edit
silently reshuffling ids for models that never changed.

**Fix — anchor the lock identity on the dataset + model, not the presentation.**
The lock key for an autogen row becomes `{dataset}:{model_source_path}` — the
stable `dataset` namespace prefix (kept as the 2da/namespace linker developers use
instead of raw ids) plus the **model source path** from `vfxs.yml`, e.g.
`visualeffects:head_accessories/hat/hfx_bandana.mdl` (leading `vfxs/` stripped —
the path is the value already on `entry.Source`). Dropped from the key are only
the config-derived parts (group token, `category_segment`, `delimiter`, case,
prefix-stripping). The row's `key`/`label` *columns* still derive from config via
`KeyFormat`/`LabelFormat` — only the **lock identity** is decoupled and
model-anchored. Consequences:

- A config-only change keeps every id (the path is unchanged).
- A model re-export (same path, new `sha256`) keeps its id — a content change is
  not an identity change. (Audit trail comes free from `git diff` on the committed
  `vfxs.yml`, which carries `path` + `sha256`; the lock stays a flat `string→int`
  map — no schema change, no fork of the shared lock format. A future `id`+`sha`
  object form is a clean later add if observability ever needs it, without
  changing id semantics.)
- A renamed/moved/removed model is a genuinely different (or absent) identity.

**One-time cutover remap.** On the first build under `{dataset}:{path}` keys, for
each current model compute its *old* config-key (with the current policy) and its
new `{dataset}:{path}` key; if the new key is absent but the old key carries an id
in the lock, adopt that id under the new key. So ids carry over with zero shift on
cutover; the now-unreferenced old keys fall stale and are pruned (below).

**Stale pruning.** A lock key absent from the resolved entry set is dropped and
its id freed for first-free reuse — but **only on a successful, non-empty
resolution**. The fail-open paths (section A) contribute no entries and skip the
augmentor entirely, so an unreachable/missing manifest can never mass-prune the
lock. This realizes the general "free a fully-stale id" rule without risking the
lock on a transient source outage.

**Testing.** Go tests in `internal/topdata`: config-only change (rename a group
token, change `key_format`/`delimiter`/`case`) → ids unchanged; model re-export
(same path, different sha) → id unchanged; model rename/remove → old id freed and
reusable; cutover remap → existing ids preserved across the key-scheme switch;
fail-open → no pruning, lock untouched.

## Determinism vs. channel mutability (decision)

The committed topdata commit pins everything **except** the asset channel
pointer. `current` resolves through `channels.json`, which is intentionally
mutable, so "reproducible" here means **same commit + same channel state →
identical bytes**, not "this commit is frozen forever." This is deliberate: the
accessory rows must track the asset HAKs shipped beside `sow_top.hak`, which is
exactly what the channel pointer expresses. A given asset release **tag** is
immutable; the pointer moving is a real input change, and `lock.json` preserves
accessory IDs across it (no reshuffle). Builds against an explicit tag
(`SOW_TOPDATA_ASSET_CHANNEL` set, or a `v*` ref) are fully pinned.

This satisfies R1–R5: no secret, no config, deterministic for a fixed input
set, on any system.

## Testing

- **Go unit tests (`sow-tools/internal/topdata`)** for the `cdn_channel`
  resolution: offline-override path; unreachable channels.json → fail open;
  channel absent → fail open; vfxs.yml 404 + no haks.json → fail open; vfxs.yml
  404 + haks.json present → hard fail (asserts message); malformed vfxs.yml →
  hard fail; happy path → correct filtered/sorted entries (mdl-only, 4-groups-
  only, `vfxs/` stripped, group/subgroup/stem). Mirrors the deleted bash
  contract cases.
- **Go config validation test** for the new `source` block.
- **CI parity guard** (section D) in the topdata build workflow.
- Existing accessory augmentor / lock-preserve tests unchanged.

## Rollout

1. sow-tools: add `AutogenSourceConfig`, the `cdn_channel` resolution branch,
   LFS materialization, the model-anchored lock identity + cutover remap + stale
   pruning (section H), the parity-guard test; update `consumer-contract.md`
   (add the parity contract, remove the `NWN_ROOT` rule per section E, replace the
   image guidance with deployment-only per section G).
2. Release Crucible (assets already anon-downloadable — R1).
3. Toolchain convergence (section F): expose `crucible` from a sow-tools flake
   input in each consumer flake — add it to sow-topdata's devshell, switch
   sow-assets-manifest off its manual `fetchurl` pin; drop `CRUCIBLE_TOKEN` from
   consumer CI; CI builds inside the nix devshell.
4. sow-topdata: swap `manifest_file:` → `source:` in `nwn-tool.yaml`; delete the
   resolver script + bash contract test; thin `build-topdata.sh` (validate →
   resolve crucible → build); pin Crucible via `flake.lock`.
5. Verify: bare clone → `crucible topdata build-topdata` → 637 accessory rows,
   no env, no key; and the same command inside `nix develop`.

## Follow-ups (not this work)

- Parts `2da` / `cachedmodels` reinstatement (separate spec) can reuse the same
  `cdn_channel` source type.
- Roll the thinned-wrapper pattern review across `sow-module` / `sow-codebase`.