spec

docs/superpowers/specs/2026-06-21-crucible-build-parity-design.md

@@ -0,0 +1,357 @@
+# Crucible build parity — local == CI, zero-config reproducible
+
+Date: 2026-06-21
+Status: approved (design); implementation pending
+Repos touched: `sow-tools` (Crucible + contract doc + image notes),
+`sow-topdata` (config, flake, wrapper, CI), `sow-assets-manifest` (flake pin),
+and the consumer wrapper/flake pattern shared by every Crucible repo.
+
+Supersedes the resolver-script approach in
+`2026-06-21-topdata-vfx-autogen-channel-design.md` (the channel→tag→`vfxs.yml`
+resolution moves out of a bash wrapper and into Crucible).
+
+## Problem
+
+`crucible topdata build-topdata` run by hand does **not** reproduce the CI
+build. CI runs `scripts/build-topdata.sh`, which resolves the accessory-VFX
+model list with `scripts/resolve-accessory-vfx.sh` and writes
+`.cache/sow-accessory-vfx-manifest.json` **before** Crucible runs; Crucible only
+reads that pre-staged file (`nwn-tool.yaml: manifest_file:`). Run Crucible
+directly and the file never exists, the `optional` consumer skips, and
+`visualeffects.2da` ships **zero accessory rows** — exactly the local build the
+operator observed (637 rows in CI, 0 locally).
+
+Root cause: a **build-output-affecting input is resolved outside Crucible**, in
+a wrapper script. The wrapper, not Crucible, is authoritative for that input.
+
+## Goal
+
+Crucible is the single authority for everything that lands in a build artifact.
+A clone of any consumer repo, on any system, builds the same bytes CI builds —
+with no secret beyond what cloning already required and no hand-written
+configuration.
+
+### Zero-config reproducibility requirements
+
+- **R1 — No extra secrets.** All build-input reads are anonymous: the asset
+  channel pointer (`releases/haks/channels.json`) and `vfxs.yml` are public CDN
+  text. The crucible binary is fetchable anonymously — **verified**: the
+  sow-tools release assets return HTTP 200 without auth, and the bootstrap tries
+  anon before any token, so `CRUCIBLE_TOKEN` in consumer CI is **vestigial and is
+  removed**. Git-LFS asset bytes use the **same** credentials the clone already
+  used — never a separate key.
+- **R2 — No hand-written config.** Every default needed to build is baked into
+  committed `nwn-tool.yaml` and Crucible defaults: CDN base, channel selection,
+  source paths. The user types `crucible topdata build-topdata` and nothing else.
+- **R3 — No host data assumptions, no `NWN_ROOT`.** Builders read every input
+  from the project tree (`nwn-tool.yaml` + committed `data/**/base.json`); no NWN
+  install is required. `NWN_ROOT` is **dead** today — nothing in Crucible reads it
+  (see *NWN_ROOT cleanup* below). The accessory build reads
+  `data/visualeffects/base.json`. (Enforced by the CI parity guard, which runs in
+  a bare checkout with no `NWN_ROOT`.)
+- **R4 — Nix is convenience, not a requirement.** The flake devshell provides
+  `crucible`, `git-lfs`, `jq`, `yq`, etc. on `PATH` for Nix users. Non-Nix users
+  (incl. Windows) get the same Crucible via the `crucible.sh`/`crucible.ps1`
+  bootstrap and a host `git-lfs`; the build path through Crucible is identical.
+- **R5 — Same Crucible, both worlds.** Local dev and CI invoke the same Crucible
+  subcommand against the same committed config. No CI-only pre-steps that change
+  output.
+
+## Parity contract (prevents recurrence)
+
+Added to `sow-tools/docs/consumer-contract.md`. A consumer build wrapper may only:
+
+1. **validate** — a pre-build gate that does not change output;
+2. **publish** — post-build, on a separate artifact;
+3. **resolve the crucible binary** — the bootstrap in `consumer-contract.md`.
+
+A wrapper MUST NOT fetch, resolve, generate, or stage any input Crucible reads
+to produce the artifact. If Crucible needs an input, Crucible acquires it. This
+is the existing "no local-machine assumptions" + "same inputs → identical
+output" rule, made explicit: **the wrapper is not allowed to be authoritative
+for build inputs.**
+
+## Design
+
+### A. Accessory-VFX resolution moves into Crucible
+
+**Config (`sow-topdata/nwn-tool.yaml`).** Drop `manifest_file:`; declare the
+source on the consumer. The 4-group + `mdl` filter is **not** re-declared — it
+is derived from the keys already present under `accessory_visualeffects.groups`:
+
+```yaml
+autogen:
+  consumers:
+    - id: accessory_visualeffects
+      producer: accessory_visualeffects
+      dataset: visualeffects
+      mode: accessory_visualeffects
+      optional: true
+      source:
+        kind: cdn_channel
+        cdn_base_env: BUNNY_CDN_BASE             # default https://depot.westgate.pw
+        channels_path: releases/haks/channels.json
+        manifest_path: releases/haks/{tag}/vfxs.yml
+        release_marker_path: releases/haks/{tag}/haks.json   # the broken-release guard probe
+        channel_env: SOW_TOPDATA_ASSET_CHANNEL   # else derive from git tag
+        offline_override_env: SOW_VFXS_MANIFEST  # dev/air-gapped vfxs.yml path
+      accessory_visualeffects:
+        groups: { chest_accessories: {...}, head_accessories: {...},
+                  head_decorations: {...}, head_features: {...} }
+        # ... rest of the policy block unchanged
+```
+
+New struct field on `AutogenConsumerConfig` (sow-tools
+`internal/project/project.go`): `Source AutogenSourceConfig` with the keys
+above. Validation: when `source.kind == cdn_channel`, `channels_path` and
+`manifest_path` are required and `manifest_path` must contain `{tag}`.
+
+**Resolution (`internal/topdata/autogen.go`,
+`resolveAutogenConsumerManifest`).** A new branch for `source.kind ==
+cdn_channel`, reusing the existing `fetchJSON` HTTP client and the project YAML
+parser:
+
+1. **Offline override.** If `offline_override_env` names a readable file (or a
+   manifest-repo checkout root containing `assets/vfxs.yml`), parse it and skip
+   the network — the dev/air-gapped path.
+2. **Channel.** `channel_env` if set; else from the git tag / `GITHUB_REF_NAME`
+   (`v*-*` → `testing`, `v*` → `current`, otherwise `current`).
+3. **CDN base.** `cdn_base_env` value, else the baked default.
+4. `GET {cdn}/{channels_path}` → JSON → `tag = channels[channel]`.
+5. `GET {cdn}/{manifest_path with {tag}}` → parse YAML `assets[]`.
+6. **Filter** to `restype == mdl` whose `path` is under `vfxs/<g>/` for `<g>` in
+   the `accessory_visualeffects.groups` keys. Derive each entry's `source`
+   (leading `vfxs/` stripped), `group`, `subgroup`, `model_stem`. Sort by
+   `source`. Hand the entries to the existing accessory augmentor unchanged.
+
+**Fail policy (ported verbatim from the resolver — this is the v0.1.4-bug
+guard).** Distinguish *unreachable* (fail open) from *broken* (hard fail):
+
+- channels.json unreachable / not JSON / channel absent → **fail open**: the
+  `optional` consumer contributes no rows and `data/visualeffects/lock.json` IDs
+  are preserved.
+- vfxs.yml network error, or 404 **and** the `release_marker_path` (haks.json)
+  is **absent** for that tag (no published release) → **fail open**.
+- vfxs.yml 404 **and** haks.json **present** → **HARD fail** ("release `<tag>`
+  has haks.json but no vfxs.yml — accessory rows would silently vanish").
+- vfxs.yml present but malformed / no `assets` array → **HARD fail**.
+
+Fail-open writes nothing and lets the optional consumer skip; hard-fail aborts
+the build with the message above.
+
+**Deleted:** `sow-topdata/scripts/resolve-accessory-vfx.sh`,
+`tests/resolve-accessory-vfx-contract.sh`, the `manifest_file:` key, and the
+resolver pre-step at `build-topdata.sh:19`. The four bash contract cases
+(offline override, unreachable fail-open, broken-release hard-fail, no-release
+fail-open) are re-expressed as Go tests in `internal/topdata`.
+
+### B. Git-LFS materialization moves into Crucible
+
+So a bare clone + `crucible` packs real bytes, not pointer stubs (today only
+`build-topdata.sh` pulls LFS, so direct Crucible would ship stubs in CI's
+non-smudging checkout).
+
+Before packing assets into a hak, Crucible: detects git-LFS pointer stubs under
+the asset tree; if any, runs `git lfs install --local` then `git lfs pull` in
+the repo; re-checks and **hard-fails** if stubs remain ("refusing to build from
+pointer stubs"). Uses the clone's own credentials (R1). Requires `git` +
+`git-lfs` on `PATH` — provided by the Nix devshell, a standard host install
+otherwise (a clone without git-lfs already yields stubs, so this is the existing
+host expectation, now enforced loudly by Crucible instead of a wrapper).
+
+A skip control replaces `TOPDATA_SKIP_LFS` for the maintain-tree flow, which
+regenerates the `data/` tree and **discards** the package: a Crucible flag/env
+(e.g. `--skip-lfs` / `CRUCIBLE_SKIP_LFS`) so that one job can opt out. Default
+is fail-closed (pull).
+
+### C. Wrappers thin out
+
+`scripts/build-topdata.sh` (and the shared pattern in every Crucible repo)
+reduces to: validate (gate) → resolve crucible → `crucible <build>`. No input
+resolution, no LFS pull. `sow-module/scripts/package-module.sh` already matches
+this shape (validate → resolve → build); it stays the reference.
+
+### D. CI parity guard (catches the next regression)
+
+A `sow-tools` integration test builds the topdata package via Crucible **alone**
+in a bare checkout (no wrapper, clean `.cache`, no `NWN_ROOT`, anonymous CDN)
+and asserts `visualeffects.2da` contains accessory rows. This is the test that
+would have caught today's bug: it fails if any build-output-affecting step ever
+drifts back out of Crucible into a wrapper, and it enforces R2/R3/R5 mechanically.
+
+The contract rule in `consumer-contract.md` is the written half; this test is
+the enforced half.
+
+### E. NWN_ROOT cleanup (stale, contradicts the goal)
+
+`NWN_ROOT` is unused — no `os.Getenv`, no `--nwn-root` flag, no consumer. The
+only traces are documentation/help-text describing a rule for a thing that does
+not exist: the `## NWN_ROOT rule` section in `consumer-contract.md`, the help
+text in `internal/dispatch/dispatch.go` ("if a builder ever needs an NWN
+root… crucible never guesses from $HOME"), and the `README.md` pointer. That
+old rule — *never guess, require an explicit env* — directly contradicts the
+zero-config goal (R2): it would make a future builder demand a hand-set path.
+
+- **Remove** the `NWN_ROOT` references: the `consumer-contract.md` section, the
+  dispatch help-text lines, the README pointer. Leave the
+  `migration-from-nwn-tool.md` bullet as historical record.
+- **Replace the forward rule.** Should a future builder ever need NWN game data,
+  it **auto-detects** it (the NWN home — containing `modules/`, `hak/`, `tlk/` —
+  and the Steam install path are reliably discoverable), with an optional
+  explicit override for non-standard layouts. Auto-detect first, never a required
+  hand-set env. This keeps "clone → run → Just Work" intact even for a builder
+  that one day reads game data. None do today.
+
+### F. Toolchain convergence — one Crucible, three modes
+
+Parity needs everyone to run the *same* Crucible. Today three mechanisms
+diverge: non-Nix uses the `crucible.sh` bootstrap download; sow-assets-manifest's
+flake pins `crucible v0.2.5` via `fetchurl`+hash; sow-topdata's flake ships no
+crucible at all (falls back to the bootstrap, with a needless `CRUCIBLE_TOKEN`).
+The pin is scattered (a hardcoded flake version, none in another flake, an image
+sha in `prod.yml`, a HEAD-floating CI binary build).
+
+Converge on **one binary, one pin, three modes — all anonymous, all identical**:
+
+1. **Non-Nix (local + CI fallback).** `crucible.sh` / `crucible.ps1` bootstrap
+   downloads the release binary anonymously. `CRUCIBLE_TOKEN` is dropped from
+   consumer CI (R1).
+2. **Nix (local + CI optimized).** Every consumer flake provides `crucible` in
+   its devshell from a **sow-tools flake input**, pinned by that repo's
+   `flake.lock`. Idiomatic, reproducible, no manual hashes; bump with
+   `nix flake update`. This replaces sow-assets-manifest's manual `fetchurl`
+   pin and **adds crucible to sow-topdata's devshell** (today it has none).
+3. **CI** runs the *same* `crucible <build>` as local, inside the nix devshell
+   (binary-cache fast). No build container, no token, no CI-only pre-steps.
+
+**Pin source of truth = `flake.lock` per repo**, bumped deliberately in a PR. We
+**reject floating "latest"**: it would let local drift from CI between releases
+and break reproducibility. The non-Nix bootstrap, which resolves "latest" by
+default, is the convenience path; reproducible/parity-critical builds pin (nix
+devshell, or `CRUCIBLE_*` pin env for the bootstrap).
+
+### G. The crucible container is deployment-only
+
+`sow-tools` builds and publishes `registry.westgate.pw/deployment/crucible:<sha>`
+on `v*` tags (`nix build .#image` → `skopeo`). It exists for **deployment** —
+`prod.yml` pins it so tools travel with the runtime host (`nwn.enable`), and it
+is a **disabled placeholder** until the NWN stack lands. No build/CI job consumes
+it, and the host-mode runners have no container runtime to run it. It is **not a
+build tool.**
+
+- Keep the image build (cheap, `v*`-gated) for its deployment purpose.
+- **Fix `consumer-contract.md`:** remove the crossed-out "run CI inside the
+  pinned image" guidance and its OPERATOR NOTE; state plainly that builds use the
+  binary (bootstrap or nix devshell) and the image is deployment-only. This kills
+  the "why do we build a container we don't use?" confusion.
+
+### H. Autogen lock identity — model-anchored, immutable
+
+**General lock contract (hard rule, all datasets).** A dataset lock maps a stable
+key to a row id. The build MUST: reuse the existing id for a key that still
+exists; allocate the first free id (lexicographic 0→1→2…) for a new key; and free
+an id only when its key is gone everywhere. An id, once assigned to a live key,
+never changes. (Already implemented for the accessory consumer via
+`historicalLockData` reuse + `nextAvailableAutogenID`; this section makes it the
+written rule and fixes the autogen-specific anchor below.)
+
+**The autogen problem.** Autogen rows describe models that do **not** exist in
+`sow-topdata/data/` — they live in the asset `vfxs.yml` manifest. Today the lock
+key is the *config-derived presentation key*:
+`KeyFormat = {dataset}:{group}{delimiter}{category_segment}{stem}`. Group token,
+category, delimiter, case, and prefix-stripping all come from `nwn-tool.yaml`, so
+a **config-only** change (`key_format`, `delimiter`, `case`, a group rename,
+`category_from`, `strip_model_prefixes`) changes the key → the lock no longer
+matches → a new id is allocated → **every row shifts**. That is a config edit
+silently reshuffling ids for models that never changed.
+
+**Fix — anchor the lock identity on the dataset + model, not the presentation.**
+The lock key for an autogen row becomes `{dataset}:{model_source_path}` — the
+stable `dataset` namespace prefix (kept as the 2da/namespace linker developers use
+instead of raw ids) plus the **model source path** from `vfxs.yml`, e.g.
+`visualeffects:head_accessories/hat/hfx_bandana.mdl` (leading `vfxs/` stripped —
+the path is the value already on `entry.Source`). Dropped from the key are only
+the config-derived parts (group token, `category_segment`, `delimiter`, case,
+prefix-stripping). The row's `key`/`label` *columns* still derive from config via
+`KeyFormat`/`LabelFormat` — only the **lock identity** is decoupled and
+model-anchored. Consequences:
+
+- A config-only change keeps every id (the path is unchanged).
+- A model re-export (same path, new `sha256`) keeps its id — a content change is
+  not an identity change. (Audit trail comes free from `git diff` on the committed
+  `vfxs.yml`, which carries `path` + `sha256`; the lock stays a flat `string→int`
+  map — no schema change, no fork of the shared lock format. A future `id`+`sha`
+  object form is a clean later add if observability ever needs it, without
+  changing id semantics.)
+- A renamed/moved/removed model is a genuinely different (or absent) identity.
+
+**One-time cutover remap.** On the first build under `{dataset}:{path}` keys, for
+each current model compute its *old* config-key (with the current policy) and its
+new `{dataset}:{path}` key; if the new key is absent but the old key carries an id
+in the lock, adopt that id under the new key. So ids carry over with zero shift on
+cutover; the now-unreferenced old keys fall stale and are pruned (below).
+
+**Stale pruning.** A lock key absent from the resolved entry set is dropped and
+its id freed for first-free reuse — but **only on a successful, non-empty
+resolution**. The fail-open paths (section A) contribute no entries and skip the
+augmentor entirely, so an unreachable/missing manifest can never mass-prune the
+lock. This realizes the general "free a fully-stale id" rule without risking the
+lock on a transient source outage.
+
+**Testing.** Go tests in `internal/topdata`: config-only change (rename a group
+token, change `key_format`/`delimiter`/`case`) → ids unchanged; model re-export
+(same path, different sha) → id unchanged; model rename/remove → old id freed and
+reusable; cutover remap → existing ids preserved across the key-scheme switch;
+fail-open → no pruning, lock untouched.
+
+## Determinism vs. channel mutability (decision)
+
+The committed topdata commit pins everything **except** the asset channel
+pointer. `current` resolves through `channels.json`, which is intentionally
+mutable, so "reproducible" here means **same commit + same channel state →
+identical bytes**, not "this commit is frozen forever." This is deliberate: the
+accessory rows must track the asset HAKs shipped beside `sow_top.hak`, which is
+exactly what the channel pointer expresses. A given asset release **tag** is
+immutable; the pointer moving is a real input change, and `lock.json` preserves
+accessory IDs across it (no reshuffle). Builds against an explicit tag
+(`SOW_TOPDATA_ASSET_CHANNEL` set, or a `v*` ref) are fully pinned.
+
+This satisfies R1–R5: no secret, no config, deterministic for a fixed input
+set, on any system.
+
+## Testing
+
+- **Go unit tests (`sow-tools/internal/topdata`)** for the `cdn_channel`
+  resolution: offline-override path; unreachable channels.json → fail open;
+  channel absent → fail open; vfxs.yml 404 + no haks.json → fail open; vfxs.yml
+  404 + haks.json present → hard fail (asserts message); malformed vfxs.yml →
+  hard fail; happy path → correct filtered/sorted entries (mdl-only, 4-groups-
+  only, `vfxs/` stripped, group/subgroup/stem). Mirrors the deleted bash
+  contract cases.
+- **Go config validation test** for the new `source` block.
+- **CI parity guard** (section D) in the topdata build workflow.
+- Existing accessory augmentor / lock-preserve tests unchanged.
+
+## Rollout
+
+1. sow-tools: add `AutogenSourceConfig`, the `cdn_channel` resolution branch,
+   LFS materialization, the model-anchored lock identity + cutover remap + stale
+   pruning (section H), the parity-guard test; update `consumer-contract.md`
+   (add the parity contract, remove the `NWN_ROOT` rule per section E, replace the
+   image guidance with deployment-only per section G).
+2. Release Crucible (assets already anon-downloadable — R1).
+3. Toolchain convergence (section F): expose `crucible` from a sow-tools flake
+   input in each consumer flake — add it to sow-topdata's devshell, switch
+   sow-assets-manifest off its manual `fetchurl` pin; drop `CRUCIBLE_TOKEN` from
+   consumer CI; CI builds inside the nix devshell.
+4. sow-topdata: swap `manifest_file:` → `source:` in `nwn-tool.yaml`; delete the
+   resolver script + bash contract test; thin `build-topdata.sh` (validate →
+   resolve crucible → build); pin Crucible via `flake.lock`.
+5. Verify: bare clone → `crucible topdata build-topdata` → 637 accessory rows,
+   no env, no key; and the same command inside `nix develop`.
+
+## Follow-ups (not this work)
+
+- Parts `2da` / `cachedmodels` reinstatement (separate spec) can reuse the same
+  `cdn_channel` source type.
+- Roll the thinned-wrapper pattern review across `sow-module` / `sow-codebase`.