1732 lines
65 KiB
Markdown
1732 lines
65 KiB
Markdown
# Crucible Build Parity Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Make `crucible topdata build-topdata` reproduce the CI build byte-for-byte from a bare clone — moving accessory-VFX resolution and Git-LFS materialization out of the wrapper and into Crucible, anchoring autogen lock IDs to the model path, and converging every consumer on one anonymously-fetched, flake-pinned Crucible.
|
|
|
|
**Architecture:** Crucible (Go, `sow-tools`) becomes the single authority for every build input. A new `cdn_channel` autogen source resolves `channels.json` → tag → `vfxs.yml` over the anonymous CDN inside Go (replacing `scripts/resolve-accessory-vfx.sh`). Crucible materializes Git-LFS asset bytes before packing the hak. Autogen lock identity moves from a config-derived presentation key to a model-path key so config edits no longer reshuffle IDs. Consumer flakes pull `crucible` from a `sow-tools` flake input; the bash wrappers thin to validate→resolve→build.
|
|
|
|
**Tech Stack:** Go 1.x (stdlib + `gopkg.in/yaml.v3`), Nix flakes, Gitea Actions, bash.
|
|
|
|
## Global Constraints
|
|
|
|
- **R1 — No extra secrets.** All build-input reads anonymous; `CRUCIBLE_TOKEN` removed from consumer CI. LFS uses the clone's own credentials.
|
|
- **R2 — No hand-written config.** Every build default baked into committed `nwn-tool.yaml` + Crucible defaults. `crucible topdata build-topdata` with no flags/env must work.
|
|
- **R3 — No host data assumptions, no `NWN_ROOT`.** Inputs come from the project tree only. No NWN install required.
|
|
- **R4 — Nix is convenience, not a requirement.** Non-Nix (incl. Windows) builds via the `crucible.sh`/`crucible.ps1` bootstrap + host `git-lfs`; same build path.
|
|
- **R5 — Same Crucible, both worlds.** Local and CI run the identical `crucible topdata build-topdata` against the same committed config. No CI-only pre-steps that change output.
|
|
- **Pin source of truth = `flake.lock` per repo.** Reject floating "latest" for parity-critical builds.
|
|
- **Determinism scope:** same commit + same channel state → identical bytes. A `v*` ref / explicit `SOW_TOPDATA_ASSET_CHANNEL` is fully pinned.
|
|
- Go module path prefix: `git.westgate.pw/ShadowsOverWestgate/sow-tools`.
|
|
- CDN default base: `https://depot.westgate.pw`.
|
|
- The 4 accessory groups: `chest_accessories`, `head_accessories`, `head_decorations`, `head_features`.
|
|
|
|
---
|
|
|
|
## Phase 1 — sow-tools (Crucible) Go changes
|
|
|
|
> All Phase 1 tasks run from `sow-tools/`. Test command throughout: `go test ./internal/...`. Build check: `make build`.
|
|
|
|
### Task 1: `AutogenSourceConfig` struct, field, and validation
|
|
|
|
**Files:**
|
|
|
|
- Modify: `sow-tools/internal/project/project.go` (add struct near line 414 `AutogenDeriveConfig`; add field to `AutogenConsumerConfig` at line 355-369; add validation)
|
|
- Test: `sow-tools/internal/project/project_test.go`
|
|
|
|
**Interfaces:**
|
|
|
|
- Produces:
|
|
- `type AutogenSourceConfig struct{ Kind, CDNBase, CDNBaseEnv, ChannelsPath, ManifestPath, ReleaseMarkerPath, ChannelEnv, OfflineOverrideEnv string }`
|
|
- `AutogenConsumerConfig.Source AutogenSourceConfig` (yaml/json key `source`)
|
|
- `func validateAutogenConsumerSources(consumers []AutogenConsumerConfig) []string`
|
|
|
|
- [ ] **Step 1: Write the failing validation test**
|
|
|
|
In `sow-tools/internal/project/project_test.go` add:
|
|
|
|
```go
|
|
func TestValidateAutogenConsumerSourcesCDNChannel(t *testing.T) {
|
|
cases := []struct {
|
|
name string
|
|
source AutogenSourceConfig
|
|
wantFail bool
|
|
}{
|
|
{"valid", AutogenSourceConfig{Kind: "cdn_channel", ChannelsPath: "releases/haks/channels.json", ManifestPath: "releases/haks/{tag}/vfxs.yml"}, false},
|
|
{"missing channels_path", AutogenSourceConfig{Kind: "cdn_channel", ManifestPath: "releases/haks/{tag}/vfxs.yml"}, true},
|
|
{"missing manifest_path", AutogenSourceConfig{Kind: "cdn_channel", ChannelsPath: "releases/haks/channels.json"}, true},
|
|
{"manifest_path lacks {tag}", AutogenSourceConfig{Kind: "cdn_channel", ChannelsPath: "releases/haks/channels.json", ManifestPath: "releases/haks/vfxs.yml"}, true},
|
|
{"empty kind is ignored", AutogenSourceConfig{}, false},
|
|
}
|
|
for _, tc := range cases {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
failures := validateAutogenConsumerSources([]AutogenConsumerConfig{{ID: "x", Source: tc.source}})
|
|
if tc.wantFail && len(failures) == 0 {
|
|
t.Fatalf("expected validation failure, got none")
|
|
}
|
|
if !tc.wantFail && len(failures) != 0 {
|
|
t.Fatalf("expected no failure, got %v", failures)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 2: Run it to confirm it fails to compile**
|
|
|
|
Run: `go test ./internal/project/ -run TestValidateAutogenConsumerSourcesCDNChannel`
|
|
Expected: FAIL — `undefined: validateAutogenConsumerSources` / `AutogenSourceConfig`.
|
|
|
|
- [ ] **Step 3: Add the struct and field**
|
|
|
|
In `sow-tools/internal/project/project.go`, add to `AutogenConsumerConfig` (after the `ManifestFile` field at line 368):
|
|
|
|
```go
|
|
Source AutogenSourceConfig `json:"source,omitempty" yaml:"source,omitempty"`
|
|
```
|
|
|
|
Add the struct (place it next to `AutogenDeriveConfig`, around line 417):
|
|
|
|
```go
|
|
type AutogenSourceConfig struct {
|
|
Kind string `json:"kind,omitempty" yaml:"kind,omitempty"`
|
|
CDNBase string `json:"cdn_base,omitempty" yaml:"cdn_base,omitempty"`
|
|
CDNBaseEnv string `json:"cdn_base_env,omitempty" yaml:"cdn_base_env,omitempty"`
|
|
ChannelsPath string `json:"channels_path,omitempty" yaml:"channels_path,omitempty"`
|
|
ManifestPath string `json:"manifest_path,omitempty" yaml:"manifest_path,omitempty"`
|
|
ReleaseMarkerPath string `json:"release_marker_path,omitempty" yaml:"release_marker_path,omitempty"`
|
|
ChannelEnv string `json:"channel_env,omitempty" yaml:"channel_env,omitempty"`
|
|
OfflineOverrideEnv string `json:"offline_override_env,omitempty" yaml:"offline_override_env,omitempty"`
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Add the validation function**
|
|
|
|
In `sow-tools/internal/project/project.go`, add:
|
|
|
|
```go
|
|
func validateAutogenConsumerSources(consumers []AutogenConsumerConfig) []string {
|
|
var failures []string
|
|
for _, c := range consumers {
|
|
if strings.TrimSpace(c.Source.Kind) != "cdn_channel" {
|
|
continue
|
|
}
|
|
label := strings.TrimSpace(c.ID)
|
|
if label == "" {
|
|
label = "<unnamed>"
|
|
}
|
|
if strings.TrimSpace(c.Source.ChannelsPath) == "" {
|
|
failures = append(failures, fmt.Sprintf("autogen consumer %s: source.channels_path is required for kind cdn_channel", label))
|
|
}
|
|
manifestPath := strings.TrimSpace(c.Source.ManifestPath)
|
|
if manifestPath == "" {
|
|
failures = append(failures, fmt.Sprintf("autogen consumer %s: source.manifest_path is required for kind cdn_channel", label))
|
|
} else if !strings.Contains(manifestPath, "{tag}") {
|
|
failures = append(failures, fmt.Sprintf("autogen consumer %s: source.manifest_path must contain {tag}", label))
|
|
}
|
|
}
|
|
return failures
|
|
}
|
|
```
|
|
|
|
(`fmt` and `strings` are already imported in `project.go`.)
|
|
|
|
- [ ] **Step 5: Wire validation into `Validate`**
|
|
|
|
Find the block in `project.go` that appends topdata validation failures (around line 639, the `failures = append(failures, validateTopData...)` cluster). Add immediately after it:
|
|
|
|
```go
|
|
failures = append(failures, validateAutogenConsumerSources(effective.Autogen.Consumers)...)
|
|
```
|
|
|
|
If `effective` does not expose `Autogen`, use the same receiver the surrounding lines use for autogen (grep `effective.Autogen` / `p.Config.Autogen` in `Validate` and match the existing convention).
|
|
|
|
- [ ] **Step 6: Run the test to confirm it passes**
|
|
|
|
Run: `go test ./internal/project/ -run TestValidateAutogenConsumerSourcesCDNChannel`
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 7: Commit**
|
|
|
|
```bash
|
|
git add internal/project/project.go internal/project/project_test.go
|
|
git commit -m "feat(project): add cdn_channel AutogenSourceConfig + validation"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 2: `cdn_channel` manifest resolution
|
|
|
|
**Files:**
|
|
|
|
- Modify: `sow-tools/internal/topdata/autogen.go` (new resolution branch + helpers; imports)
|
|
- Modify: `sow-tools/internal/topdata/parts_manifest.go` (add `httpGetStatus`)
|
|
- Create: `sow-tools/internal/topdata/autogen_cdn_channel_test.go`
|
|
|
|
**Interfaces:**
|
|
|
|
- Consumes: `project.AutogenConsumerConfig.Source` (Task 1); `autogenManifest`, `autogenManifestEntry`, `unavailableAutogenManifest`, `errAutogenManifestUnavailable`, `gitOutput`, `partsManifestHTTPClient` (existing).
|
|
- Produces:
|
|
- `func resolveCDNChannelManifest(p *project.Project, consumer project.AutogenConsumerConfig, src project.AutogenSourceConfig, progress func(string)) (*autogenManifest, error)`
|
|
- `func filterCDNChannelEntries(raw []byte, consumer project.AutogenConsumerConfig) ([]autogenManifestEntry, error)`
|
|
- `func deriveAssetChannel(root string) string`
|
|
- `func httpGetStatus(target string) (int, []byte, error)` (in `parts_manifest.go`)
|
|
- `const defaultBunnyCDNBase = "https://depot.westgate.pw"`
|
|
|
|
- [ ] **Step 1: Write the failing resolution test**
|
|
|
|
Create `sow-tools/internal/topdata/autogen_cdn_channel_test.go`:
|
|
|
|
```go
|
|
package topdata
|
|
|
|
import (
|
|
"fmt"
|
|
"net/http"
|
|
"net/http/httptest"
|
|
"testing"
|
|
|
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
|
|
)
|
|
|
|
func cdnConsumer() project.AutogenConsumerConfig {
|
|
return project.AutogenConsumerConfig{
|
|
ID: "accessory_visualeffects", Producer: "accessory_visualeffects",
|
|
Dataset: "visualeffects", Mode: "accessory_visualeffects", Optional: true,
|
|
AccessoryVisualeffects: project.AccessoryVisualeffectsConfig{
|
|
Groups: map[string]project.AccessoryVisualeffectGroupConfig{
|
|
"head_accessories": {}, "chest_accessories": {},
|
|
"head_decorations": {}, "head_features": {},
|
|
},
|
|
},
|
|
Source: project.AutogenSourceConfig{
|
|
Kind: "cdn_channel",
|
|
ChannelsPath: "releases/haks/channels.json",
|
|
ManifestPath: "releases/haks/{tag}/vfxs.yml",
|
|
ReleaseMarkerPath: "releases/haks/{tag}/haks.json",
|
|
},
|
|
}
|
|
}
|
|
|
|
// cdnServer serves channels.json, vfxs.yml, and haks.json from in-memory maps.
|
|
// A key absent from the map returns 404.
|
|
func cdnServer(t *testing.T, files map[string]string) *httptest.Server {
|
|
t.Helper()
|
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
body, ok := files[r.URL.Path]
|
|
if !ok {
|
|
w.WriteHeader(http.StatusNotFound)
|
|
return
|
|
}
|
|
fmt.Fprint(w, body)
|
|
}))
|
|
t.Cleanup(srv.Close)
|
|
return srv
|
|
}
|
|
|
|
func TestResolveCDNChannelHappyPath(t *testing.T) {
|
|
root := testProjectRoot(t)
|
|
srv := cdnServer(t, map[string]string{
|
|
"/releases/haks/channels.json": `{"current":"v1.2.3"}`,
|
|
"/releases/haks/v1.2.3/vfxs.yml": `assets:
|
|
- path: vfxs/head_accessories/hat/hfx_bandana.mdl
|
|
restype: mdl
|
|
sha256: aaa
|
|
- path: vfxs/chest_accessories/cape/cfx_cloak.mdl
|
|
restype: mdl
|
|
sha256: bbb
|
|
- path: vfxs/head_accessories/hat/hfx_bandana.tga
|
|
restype: tga
|
|
sha256: ccc
|
|
- path: vfxs/weapons/sword.mdl
|
|
restype: mdl
|
|
sha256: ddd
|
|
`,
|
|
})
|
|
t.Setenv("BUNNY_CDN_BASE", srv.URL)
|
|
t.Setenv("SOW_TOPDATA_ASSET_CHANNEL", "current")
|
|
|
|
p := testProject(root)
|
|
c := cdnConsumer()
|
|
c.Source.CDNBaseEnv = "BUNNY_CDN_BASE"
|
|
c.Source.ChannelEnv = "SOW_TOPDATA_ASSET_CHANNEL"
|
|
|
|
m, err := resolveCDNChannelManifest(p, c, c.Source, nil)
|
|
if err != nil {
|
|
t.Fatalf("resolve failed: %v", err)
|
|
}
|
|
if len(m.Entries) != 2 {
|
|
t.Fatalf("want 2 mdl entries under target groups, got %d: %#v", len(m.Entries), m.Entries)
|
|
}
|
|
// Sorted by source: chest_accessories/... before head_accessories/...
|
|
if m.Entries[0].Source != "chest_accessories/cape/cfx_cloak.mdl" {
|
|
t.Fatalf("unexpected first entry: %#v", m.Entries[0])
|
|
}
|
|
if m.Entries[1].Source != "head_accessories/hat/hfx_bandana.mdl" ||
|
|
m.Entries[1].Group != "head_accessories" ||
|
|
m.Entries[1].Subgroup != "hat" ||
|
|
m.Entries[1].ModelStem != "hfx_bandana" {
|
|
t.Fatalf("unexpected derived entry: %#v", m.Entries[1])
|
|
}
|
|
if m.Ref != "v1.2.3" {
|
|
t.Fatalf("want ref v1.2.3, got %q", m.Ref)
|
|
}
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 2: Run it to confirm it fails**
|
|
|
|
Run: `go test ./internal/topdata/ -run TestResolveCDNChannelHappyPath`
|
|
Expected: FAIL — `undefined: resolveCDNChannelManifest`.
|
|
|
|
- [ ] **Step 3: Add `httpGetStatus` to `parts_manifest.go`**
|
|
|
|
In `sow-tools/internal/topdata/parts_manifest.go` (it already imports `io` and `net/http`), add:
|
|
|
|
```go
|
|
// httpGetStatus performs an anonymous GET and returns the status code plus the
|
|
// body (capped). No auth header: cdn_channel inputs (channels.json, vfxs.yml,
|
|
// haks.json) are public CDN text — R1.
|
|
func httpGetStatus(target string) (int, []byte, error) {
|
|
resp, err := partsManifestHTTPClient.Get(target)
|
|
if err != nil {
|
|
return 0, nil, err
|
|
}
|
|
defer resp.Body.Close()
|
|
body, _ := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
|
|
return resp.StatusCode, body, nil
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Add the resolution branch and helpers to `autogen.go`**
|
|
|
|
In `sow-tools/internal/topdata/autogen.go`, add `net/http` and `gopkg.in/yaml.v3` to the import block.
|
|
|
|
Add the const near the top (after line 18 `autogenManifestCacheMaxAge`):
|
|
|
|
```go
|
|
const defaultBunnyCDNBase = "https://depot.westgate.pw"
|
|
```
|
|
|
|
In `resolveAutogenConsumerManifest` (line 227), add the new branch as the first check:
|
|
|
|
```go
|
|
func resolveAutogenConsumerManifest(p *project.Project, consumer project.AutogenConsumerConfig, progress func(string)) (*autogenManifest, error) {
|
|
if strings.TrimSpace(consumer.Source.Kind) == "cdn_channel" {
|
|
return resolveCDNChannelManifest(p, consumer, consumer.Source, progress)
|
|
}
|
|
if manifestFile := strings.TrimSpace(consumer.ManifestFile); manifestFile != "" {
|
|
return readAutogenConsumerManifestFile(p, consumer, manifestFile, progress)
|
|
}
|
|
// ... rest unchanged
|
|
```
|
|
|
|
Add the helpers (anywhere in `autogen.go`, e.g. after `readAutogenConsumerManifestFile`):
|
|
|
|
```go
|
|
// resolveCDNChannelManifest resolves the accessory-VFX model list anonymously
|
|
// from the asset CDN: channel pointer -> tag -> per-tag vfxs.yml. It ports the
|
|
// fail policy of the deleted resolve-accessory-vfx.sh: unreachable inputs fail
|
|
// OPEN (errAutogenManifestUnavailable -> optional consumer preserves lock IDs);
|
|
// a published-but-broken release (vfxs.yml 404 while haks.json present) or a
|
|
// malformed vfxs.yml HARD fail.
|
|
func resolveCDNChannelManifest(p *project.Project, consumer project.AutogenConsumerConfig, src project.AutogenSourceConfig, progress func(string)) (*autogenManifest, error) {
|
|
if progress == nil {
|
|
progress = func(string) {}
|
|
}
|
|
|
|
// 1. Offline / air-gapped override: a vfxs.yml path or a manifest-repo
|
|
// checkout root containing assets/vfxs.yml. Skips the network entirely.
|
|
if envName := strings.TrimSpace(src.OfflineOverrideEnv); envName != "" {
|
|
if override := strings.TrimSpace(os.Getenv(envName)); override != "" {
|
|
vfxsPath := override
|
|
if info, err := os.Stat(override); err == nil && info.IsDir() {
|
|
vfxsPath = filepath.Join(override, "assets", "vfxs.yml")
|
|
}
|
|
raw, err := os.ReadFile(vfxsPath)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("offline vfxs override %s: %w", vfxsPath, err)
|
|
}
|
|
entries, err := filterCDNChannelEntries(raw, consumer)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
progress(fmt.Sprintf("Using offline vfxs override for %s from %s...", consumer.ID, vfxsPath))
|
|
return &autogenManifest{ID: consumer.Producer, Ref: "local", Entries: entries}, nil
|
|
}
|
|
}
|
|
|
|
// 2. Channel: explicit env, else derived from the git tag.
|
|
channel := strings.TrimSpace(os.Getenv(strings.TrimSpace(src.ChannelEnv)))
|
|
if channel == "" {
|
|
channel = deriveAssetChannel(p.Root)
|
|
}
|
|
|
|
// 3. CDN base: env override, then config literal, then baked default.
|
|
cdnBase := defaultBunnyCDNBase
|
|
if envName := strings.TrimSpace(src.CDNBaseEnv); envName != "" {
|
|
if v := strings.TrimSpace(os.Getenv(envName)); v != "" {
|
|
cdnBase = v
|
|
}
|
|
}
|
|
if v := strings.TrimSpace(src.CDNBase); v != "" {
|
|
cdnBase = v
|
|
}
|
|
cdnBase = strings.TrimRight(cdnBase, "/")
|
|
|
|
join := func(path, tag string) string {
|
|
return cdnBase + "/" + strings.TrimLeft(strings.ReplaceAll(path, "{tag}", tag), "/")
|
|
}
|
|
|
|
// 4. channels.json -> tag. Unreachable / non-JSON / channel-absent = fail open.
|
|
channelsURL := join(src.ChannelsPath, "")
|
|
status, body, err := httpGetStatus(channelsURL)
|
|
if err != nil {
|
|
return nil, unavailableAutogenManifest(fmt.Errorf("channels.json unreachable %s: %w", channelsURL, err))
|
|
}
|
|
if status != http.StatusOK {
|
|
return nil, unavailableAutogenManifest(fmt.Errorf("channels.json HTTP %d %s", status, channelsURL))
|
|
}
|
|
var channels map[string]string
|
|
if err := json.Unmarshal(body, &channels); err != nil {
|
|
return nil, unavailableAutogenManifest(fmt.Errorf("channels.json not JSON %s: %w", channelsURL, err))
|
|
}
|
|
tag := strings.TrimSpace(channels[channel])
|
|
if tag == "" {
|
|
return nil, unavailableAutogenManifest(fmt.Errorf("channel %q absent in channels.json (%s)", channel, channelsURL))
|
|
}
|
|
progress(fmt.Sprintf("Accessory VFX: channel %s -> tag %s", channel, tag))
|
|
|
|
// 5. vfxs.yml for the tag.
|
|
manifestURL := join(src.ManifestPath, tag)
|
|
vstatus, vbody, err := httpGetStatus(manifestURL)
|
|
if err != nil {
|
|
return nil, unavailableAutogenManifest(fmt.Errorf("vfxs.yml unreachable %s: %w", manifestURL, err))
|
|
}
|
|
switch vstatus {
|
|
case http.StatusOK:
|
|
// fall through to parse
|
|
case http.StatusNotFound:
|
|
// 404 + release marker present = published-but-broken release: HARD fail
|
|
// (this is the v0.1.4 silent-drop bug). 404 + marker absent = no published
|
|
// release for this tag: fail open.
|
|
if marker := strings.TrimSpace(src.ReleaseMarkerPath); marker != "" {
|
|
markerURL := join(marker, tag)
|
|
if mstatus, _, merr := httpGetStatus(markerURL); merr == nil && mstatus == http.StatusOK {
|
|
return nil, fmt.Errorf("release %s has %s but no vfxs.yml (%s) — accessory rows would silently vanish; backfill/republish vfxs.yml for %s", tag, marker, manifestURL, tag)
|
|
}
|
|
}
|
|
return nil, unavailableAutogenManifest(fmt.Errorf("vfxs.yml 404 %s (no published release for %s)", manifestURL, tag))
|
|
default:
|
|
return nil, unavailableAutogenManifest(fmt.Errorf("vfxs.yml HTTP %d %s", vstatus, manifestURL))
|
|
}
|
|
|
|
entries, err := filterCDNChannelEntries(vbody, consumer)
|
|
if err != nil {
|
|
return nil, err // malformed vfxs.yml = HARD fail
|
|
}
|
|
progress(fmt.Sprintf("Accessory VFX: resolved %d model entries from %s", len(entries), manifestURL))
|
|
return &autogenManifest{ID: consumer.Producer, Ref: tag, Entries: entries}, nil
|
|
}
|
|
|
|
// filterCDNChannelEntries parses vfxs.yml and keeps restype==mdl assets under
|
|
// vfxs/<group>/ for the consumer's 4 accessory groups, stripping the leading
|
|
// vfxs/ from each source path. A present-but-malformed manifest (unparseable, or
|
|
// no assets array) is an error; an empty assets array yields zero entries.
|
|
func filterCDNChannelEntries(raw []byte, consumer project.AutogenConsumerConfig) ([]autogenManifestEntry, error) {
|
|
var manifest struct {
|
|
Assets *[]struct {
|
|
Path string `yaml:"path"`
|
|
Restype string `yaml:"restype"`
|
|
} `yaml:"assets"`
|
|
}
|
|
if err := yaml.Unmarshal(raw, &manifest); err != nil {
|
|
return nil, fmt.Errorf("malformed vfxs.yml (cannot parse): %w", err)
|
|
}
|
|
if manifest.Assets == nil {
|
|
return nil, fmt.Errorf("malformed vfxs.yml (no assets array)")
|
|
}
|
|
groups := make(map[string]struct{}, len(consumer.AccessoryVisualeffects.Groups))
|
|
for g := range consumer.AccessoryVisualeffects.Groups {
|
|
if g = strings.TrimSpace(g); g != "" {
|
|
groups[g] = struct{}{}
|
|
}
|
|
}
|
|
var entries []autogenManifestEntry
|
|
for _, a := range *manifest.Assets {
|
|
if a.Restype != "mdl" {
|
|
continue
|
|
}
|
|
path := filepath.ToSlash(strings.TrimSpace(a.Path))
|
|
if !strings.HasPrefix(path, "vfxs/") {
|
|
continue
|
|
}
|
|
rel := strings.TrimPrefix(path, "vfxs/")
|
|
parts := strings.Split(rel, "/")
|
|
if len(parts) < 2 {
|
|
continue
|
|
}
|
|
if _, ok := groups[parts[0]]; !ok {
|
|
continue
|
|
}
|
|
entry := autogenManifestEntry{
|
|
Source: rel,
|
|
Group: parts[0],
|
|
ModelStem: strings.TrimSuffix(parts[len(parts)-1], ".mdl"),
|
|
}
|
|
if len(parts) > 2 {
|
|
entry.Subgroup = strings.Join(parts[1:len(parts)-1], "/")
|
|
}
|
|
entries = append(entries, entry)
|
|
}
|
|
slices.SortFunc(entries, func(a, b autogenManifestEntry) int {
|
|
return strings.Compare(a.Source, b.Source)
|
|
})
|
|
return entries, nil
|
|
}
|
|
|
|
// deriveAssetChannel maps the current git tag (or GITHUB_REF_NAME) to a channel:
|
|
// a prerelease tag (v*-*) -> testing; a stable tag (v*) -> current; anything
|
|
// else (branch/PR/dev) -> current. Mirrors resolve-accessory-vfx.sh.
|
|
func deriveAssetChannel(root string) string {
|
|
ref := strings.TrimSpace(os.Getenv("GITHUB_REF_NAME"))
|
|
if ref == "" {
|
|
if out, err := gitOutput(root, "describe", "--tags", "--exact-match"); err == nil {
|
|
ref = strings.TrimSpace(out)
|
|
}
|
|
}
|
|
if strings.HasPrefix(ref, "v") && strings.Contains(ref, "-") {
|
|
return "testing"
|
|
}
|
|
return "current"
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 5: Run the happy-path test**
|
|
|
|
Run: `go test ./internal/topdata/ -run TestResolveCDNChannelHappyPath`
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 6: Add the fail-policy tests**
|
|
|
|
Append to `autogen_cdn_channel_test.go`:
|
|
|
|
```go
|
|
func TestResolveCDNChannelFailOpen(t *testing.T) {
|
|
root := testProjectRoot(t)
|
|
c := cdnConsumer()
|
|
c.Source.CDNBaseEnv = "BUNNY_CDN_BASE"
|
|
c.Source.ChannelEnv = "SOW_TOPDATA_ASSET_CHANNEL"
|
|
t.Setenv("SOW_TOPDATA_ASSET_CHANNEL", "current")
|
|
|
|
t.Run("channels.json unreachable", func(t *testing.T) {
|
|
t.Setenv("BUNNY_CDN_BASE", "http://127.0.0.1:0") // unroutable
|
|
_, err := resolveCDNChannelManifest(testProject(root), c, c.Source, nil)
|
|
if err == nil || !errorIsUnavailable(err) {
|
|
t.Fatalf("want fail-open (unavailable), got %v", err)
|
|
}
|
|
})
|
|
|
|
t.Run("channel absent", func(t *testing.T) {
|
|
srv := cdnServer(t, map[string]string{"/releases/haks/channels.json": `{"testing":"v9"}`})
|
|
t.Setenv("BUNNY_CDN_BASE", srv.URL)
|
|
_, err := resolveCDNChannelManifest(testProject(root), c, c.Source, nil)
|
|
if err == nil || !errorIsUnavailable(err) {
|
|
t.Fatalf("want fail-open (unavailable), got %v", err)
|
|
}
|
|
})
|
|
|
|
t.Run("vfxs 404 no release marker", func(t *testing.T) {
|
|
srv := cdnServer(t, map[string]string{"/releases/haks/channels.json": `{"current":"v1"}`})
|
|
t.Setenv("BUNNY_CDN_BASE", srv.URL)
|
|
_, err := resolveCDNChannelManifest(testProject(root), c, c.Source, nil)
|
|
if err == nil || !errorIsUnavailable(err) {
|
|
t.Fatalf("want fail-open (unavailable), got %v", err)
|
|
}
|
|
})
|
|
}
|
|
|
|
func TestResolveCDNChannelHardFail(t *testing.T) {
|
|
root := testProjectRoot(t)
|
|
c := cdnConsumer()
|
|
c.Source.CDNBaseEnv = "BUNNY_CDN_BASE"
|
|
c.Source.ChannelEnv = "SOW_TOPDATA_ASSET_CHANNEL"
|
|
t.Setenv("SOW_TOPDATA_ASSET_CHANNEL", "current")
|
|
|
|
t.Run("broken release: vfxs 404 + haks.json present", func(t *testing.T) {
|
|
srv := cdnServer(t, map[string]string{
|
|
"/releases/haks/channels.json": `{"current":"v1"}`,
|
|
"/releases/haks/v1/haks.json": `{}`,
|
|
})
|
|
t.Setenv("BUNNY_CDN_BASE", srv.URL)
|
|
_, err := resolveCDNChannelManifest(testProject(root), c, c.Source, nil)
|
|
if err == nil || errorIsUnavailable(err) {
|
|
t.Fatalf("want HARD fail, got %v", err)
|
|
}
|
|
})
|
|
|
|
t.Run("malformed vfxs.yml: no assets array", func(t *testing.T) {
|
|
srv := cdnServer(t, map[string]string{
|
|
"/releases/haks/channels.json": `{"current":"v1"}`,
|
|
"/releases/haks/v1/vfxs.yml": "generated_at: 2026-01-01\n",
|
|
})
|
|
t.Setenv("BUNNY_CDN_BASE", srv.URL)
|
|
_, err := resolveCDNChannelManifest(testProject(root), c, c.Source, nil)
|
|
if err == nil || errorIsUnavailable(err) {
|
|
t.Fatalf("want HARD fail, got %v", err)
|
|
}
|
|
})
|
|
}
|
|
|
|
func errorIsUnavailable(err error) bool {
|
|
return errors.Is(err, errAutogenManifestUnavailable)
|
|
}
|
|
```
|
|
|
|
Add `"errors"` to the test file's import block.
|
|
|
|
- [ ] **Step 7: Run all cdn_channel tests**
|
|
|
|
Run: `go test ./internal/topdata/ -run 'TestResolveCDNChannel'`
|
|
Expected: PASS (all subtests).
|
|
|
|
- [ ] **Step 8: Add the offline-override test**
|
|
|
|
Append to `autogen_cdn_channel_test.go`:
|
|
|
|
```go
|
|
func TestResolveCDNChannelOfflineOverride(t *testing.T) {
|
|
root := testProjectRoot(t)
|
|
vfxs := filepath.Join(root, "vfxs.yml")
|
|
writeFile(t, vfxs, `assets:
|
|
- path: vfxs/head_features/scar/hfx_scar.mdl
|
|
restype: mdl
|
|
sha256: zzz
|
|
`)
|
|
t.Setenv("SOW_VFXS_MANIFEST", vfxs)
|
|
c := cdnConsumer()
|
|
c.Source.OfflineOverrideEnv = "SOW_VFXS_MANIFEST"
|
|
|
|
m, err := resolveCDNChannelManifest(testProject(root), c, c.Source, nil)
|
|
if err != nil {
|
|
t.Fatalf("offline override failed: %v", err)
|
|
}
|
|
if len(m.Entries) != 1 || m.Entries[0].Source != "head_features/scar/hfx_scar.mdl" {
|
|
t.Fatalf("unexpected offline entries: %#v", m.Entries)
|
|
}
|
|
}
|
|
```
|
|
|
|
Add `"path/filepath"` to the test imports.
|
|
|
|
- [ ] **Step 9: Run the full topdata suite**
|
|
|
|
Run: `go test ./internal/topdata/`
|
|
Expected: PASS. Then `make build` — Expected: builds clean.
|
|
|
|
- [ ] **Step 10: Commit**
|
|
|
|
```bash
|
|
git add internal/topdata/autogen.go internal/topdata/parts_manifest.go internal/topdata/autogen_cdn_channel_test.go
|
|
git commit -m "feat(topdata): resolve accessory VFX from cdn_channel inside Crucible"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 3: Model-anchored autogen lock identity + cutover remap + stale pruning
|
|
|
|
**Files:**
|
|
|
|
- Modify: `sow-tools/internal/topdata/autogen.go` (`augmentWithAutogeneratedAccessoryVisualeffects` at line 819; `autogenConsumerManagedLockKeyMatcher` at line 178; add `accessoryVisualeffectLockKey`)
|
|
- Create: `sow-tools/internal/topdata/autogen_lock_identity_test.go`
|
|
|
|
**Interfaces:**
|
|
|
|
- Consumes: `autogenManifestEntry`, `accessoryVisualeffectIdentity`, `resolveAccessoryVisualeffectsPolicy` (existing).
|
|
- Produces: `func accessoryVisualeffectLockKey(dataset string, entry autogenManifestEntry) string`. Lock key form: `{dataset}:{model_source_path}`, e.g. `visualeffects:head_accessories/hat/hfx_bandana.mdl`.
|
|
|
|
- [ ] **Step 1: Write the failing config-stability test**
|
|
|
|
Create `sow-tools/internal/topdata/autogen_lock_identity_test.go`:
|
|
|
|
```go
|
|
package topdata
|
|
|
|
import (
|
|
"testing"
|
|
|
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
|
|
)
|
|
|
|
func accConsumer() project.AutogenConsumerConfig {
|
|
return project.AutogenConsumerConfig{
|
|
ID: "accessory_visualeffects", Producer: "accessory_visualeffects",
|
|
Dataset: "visualeffects", Mode: "accessory_visualeffects", Optional: true,
|
|
AccessoryVisualeffects: project.AccessoryVisualeffectsConfig{
|
|
Groups: map[string]project.AccessoryVisualeffectGroupConfig{
|
|
"head_accessories": {ModelColumns: []string{"Imp_HeadCon_Node"}},
|
|
},
|
|
GroupTokenSource: "folder_name", CategoryFrom: "immediate_parent",
|
|
Delimiter: "/", Case: "preserve",
|
|
StripModelPrefixes: []string{"hfx_"},
|
|
KeyFormat: "{dataset}:{group}{delimiter}{category_segment}{stem}",
|
|
LabelFormat: "{group}{delimiter}{category_segment}{stem}",
|
|
ModelColumn: "Imp_HeadCon_Node",
|
|
},
|
|
}
|
|
}
|
|
|
|
func vfxEntry() autogenManifestEntry {
|
|
return autogenManifestEntry{
|
|
Source: "head_accessories/hat/hfx_bandana.mdl", Group: "head_accessories",
|
|
Subgroup: "hat", ModelStem: "hfx_bandana",
|
|
}
|
|
}
|
|
|
|
func vfxDataset(lock map[string]int) nativeCollectedDataset {
|
|
return nativeCollectedDataset{
|
|
Dataset: nativeDataset{Name: "visualeffects", Kind: nativeDatasetBase},
|
|
Columns: []string{"Label", "Imp_HeadCon_Node"},
|
|
Rows: nil,
|
|
LockData: lock,
|
|
}
|
|
}
|
|
|
|
func TestAccessoryLockKeyIsModelAnchored(t *testing.T) {
|
|
got := accessoryVisualeffectLockKey("visualeffects", vfxEntry())
|
|
want := "visualeffects:head_accessories/hat/hfx_bandana.mdl"
|
|
if got != want {
|
|
t.Fatalf("lock key = %q, want %q", got, want)
|
|
}
|
|
}
|
|
|
|
// A config-only change (here: a different key_format) must NOT move the ID,
|
|
// because identity is the model path, not the presentation key.
|
|
func TestConfigOnlyChangeKeepsLockID(t *testing.T) {
|
|
lockKey := "visualeffects:head_accessories/hat/hfx_bandana.mdl"
|
|
collected := []nativeCollectedDataset{vfxDataset(map[string]int{lockKey: 555})}
|
|
|
|
c := accConsumer()
|
|
c.AccessoryVisualeffects.KeyFormat = "{dataset}:{stem}" // config-only change
|
|
got, err := augmentWithAutogeneratedAccessoryVisualeffects(collected, []autogenManifestEntry{vfxEntry()}, c, nil)
|
|
if err != nil {
|
|
t.Fatalf("augment failed: %v", err)
|
|
}
|
|
if got[0].LockData[lockKey] != 555 {
|
|
t.Fatalf("expected ID preserved at 555, got %#v", got[0].LockData)
|
|
}
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 2: Run it to confirm it fails**
|
|
|
|
Run: `go test ./internal/topdata/ -run 'TestAccessoryLockKey|TestConfigOnlyChange'`
|
|
Expected: FAIL — `undefined: accessoryVisualeffectLockKey` and ID not preserved.
|
|
|
|
- [ ] **Step 3: Add the lock-key helper**
|
|
|
|
In `autogen.go` add:
|
|
|
|
```go
|
|
// accessoryVisualeffectLockKey is the model-anchored autogen lock identity:
|
|
// the dataset namespace prefix plus the model source path from vfxs.yml (leading
|
|
// vfxs/ already stripped on entry.Source). Config-derived presentation parts
|
|
// (group token, category, delimiter, case, prefix-stripping) are deliberately
|
|
// excluded so a config-only edit never reshuffles row IDs.
|
|
func accessoryVisualeffectLockKey(dataset string, entry autogenManifestEntry) string {
|
|
return dataset + ":" + strings.TrimPrefix(filepath.ToSlash(strings.TrimSpace(entry.Source)), "vfxs/")
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Rework the augmentor to key the lock on the model path**
|
|
|
|
In `augmentWithAutogeneratedAccessoryVisualeffects` (line 819), replace the per-entry loop (lines 873-895, from `for _, entry := range entries {` through the closing brace before the `slices.SortFunc`) with:
|
|
|
|
```go
|
|
liveLockKeys := make(map[string]struct{}, len(entries))
|
|
for _, entry := range entries {
|
|
key, label, modelStem, groupPolicy, ok := accessoryVisualeffectIdentity(dataset.Dataset.Name, entry, policy)
|
|
if !ok {
|
|
continue
|
|
}
|
|
lockKey := accessoryVisualeffectLockKey(dataset.Dataset.Name, entry)
|
|
liveLockKeys[lockKey] = struct{}{}
|
|
|
|
if existing, exists := rowByKey[key]; exists {
|
|
applyDiscoveredAccessoryVisualeffectDefaults(existing, dataset.Columns, modelStem, label, policy, groupPolicy)
|
|
if _, pinned := lockData[lockKey]; !pinned {
|
|
if rowID, ok := existing["id"].(int); ok {
|
|
lockData[lockKey] = rowID
|
|
}
|
|
}
|
|
continue
|
|
}
|
|
|
|
rowID, ok := lockData[lockKey]
|
|
if !ok {
|
|
switch {
|
|
case historicalLockData[lockKey] != 0 || hasKey(historicalLockData, lockKey):
|
|
rowID = historicalLockData[lockKey]
|
|
case hasKey(historicalLockData, key):
|
|
// One-time cutover: adopt the ID from the old config-derived key
|
|
// so IDs carry over with zero shift on the key-scheme switch. The
|
|
// stale old key is pruned below.
|
|
rowID = historicalLockData[key]
|
|
default:
|
|
rowID = allocateNextID()
|
|
}
|
|
lockData[lockKey] = rowID
|
|
}
|
|
newRow := createDefaultAccessoryVisualeffectRow(dataset.Columns, rowID, key, label, modelStem, policy, groupPolicy)
|
|
rows = append(rows, newRow)
|
|
rowByKey[key] = newRow
|
|
}
|
|
|
|
// Stale pruning: drop managed lock keys (this consumer's accessory keys,
|
|
// incl. now-superseded old config-scheme keys) that no live entry resolves
|
|
// to, freeing their IDs for first-free reuse. Safe here because the
|
|
// augmentor only runs on a successful, non-empty resolution; the fail-open
|
|
// paths skip it entirely and never reach this code.
|
|
if managed := autogenConsumerManagedLockKeyMatcher(consumer); managed != nil {
|
|
for lk := range lockData {
|
|
if _, live := liveLockKeys[lk]; live {
|
|
continue
|
|
}
|
|
if managed(lk) {
|
|
delete(lockData, lk)
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
Add the small helper (the `int`-zero ambiguity guard) to `autogen.go`:
|
|
|
|
```go
|
|
func hasKey(m map[string]int, key string) bool {
|
|
_, ok := m[key]
|
|
return ok
|
|
}
|
|
```
|
|
|
|
(Then simplify the first `case` to `case hasKey(historicalLockData, lockKey):`.)
|
|
|
|
- [ ] **Step 5: Fix the case expression**
|
|
|
|
Replace the awkward first case with the clean form:
|
|
|
|
```go
|
|
switch {
|
|
case hasKey(historicalLockData, lockKey):
|
|
rowID = historicalLockData[lockKey]
|
|
case hasKey(historicalLockData, key):
|
|
rowID = historicalLockData[key]
|
|
default:
|
|
rowID = allocateNextID()
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 6: Make the managed-key matcher model-anchored**
|
|
|
|
Replace the body of `autogenConsumerManagedLockKeyMatcher` (line 178) for the `accessory_visualeffects` case with a config-independent matcher (dataset prefix + first path segment is one of the groups). This matches both old config-scheme keys and new model-path keys, so fail-open preservation and stale pruning both stay correct:
|
|
|
|
```go
|
|
func autogenConsumerManagedLockKeyMatcher(consumer project.AutogenConsumerConfig) func(string) bool {
|
|
switch consumer.Mode {
|
|
case "accessory_visualeffects":
|
|
dataset := strings.TrimSpace(consumer.Dataset)
|
|
if dataset == "" {
|
|
dataset = "visualeffects"
|
|
}
|
|
prefix := dataset + ":"
|
|
groups := make(map[string]struct{}, len(consumer.AccessoryVisualeffects.Groups))
|
|
for group := range consumer.AccessoryVisualeffects.Groups {
|
|
if group = strings.TrimSpace(group); group != "" {
|
|
groups[group] = struct{}{}
|
|
}
|
|
}
|
|
if len(groups) == 0 {
|
|
return nil
|
|
}
|
|
return func(key string) bool {
|
|
if !strings.HasPrefix(key, prefix) {
|
|
return false
|
|
}
|
|
rest := key[len(prefix):]
|
|
idx := strings.Index(rest, "/")
|
|
if idx <= 0 {
|
|
return false
|
|
}
|
|
_, ok := groups[rest[:idx]]
|
|
return ok
|
|
}
|
|
default:
|
|
return nil
|
|
}
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 7: Run the config-stability tests**
|
|
|
|
Run: `go test ./internal/topdata/ -run 'TestAccessoryLockKey|TestConfigOnlyChange'`
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 8: Add re-export, rename/remove, cutover, and fail-open tests**
|
|
|
|
Append to `autogen_lock_identity_test.go`:
|
|
|
|
```go
|
|
// A model re-export (same path) keeps its ID — content change != identity change.
|
|
func TestModelReExportKeepsLockID(t *testing.T) {
|
|
lockKey := "visualeffects:head_accessories/hat/hfx_bandana.mdl"
|
|
collected := []nativeCollectedDataset{vfxDataset(map[string]int{lockKey: 777})}
|
|
got, err := augmentWithAutogeneratedAccessoryVisualeffects(collected, []autogenManifestEntry{vfxEntry()}, accConsumer(), nil)
|
|
if err != nil {
|
|
t.Fatalf("augment failed: %v", err)
|
|
}
|
|
if got[0].LockData[lockKey] != 777 {
|
|
t.Fatalf("expected ID 777 preserved, got %#v", got[0].LockData)
|
|
}
|
|
}
|
|
|
|
// A removed model frees its old ID; the surviving model keeps its lock entry.
|
|
func TestModelRemovalPrunesStaleLockKey(t *testing.T) {
|
|
keep := "visualeffects:head_accessories/hat/hfx_bandana.mdl"
|
|
gone := "visualeffects:head_accessories/hat/hfx_removed.mdl"
|
|
collected := []nativeCollectedDataset{vfxDataset(map[string]int{keep: 1, gone: 2})}
|
|
got, err := augmentWithAutogeneratedAccessoryVisualeffects(collected, []autogenManifestEntry{vfxEntry()}, accConsumer(), nil)
|
|
if err != nil {
|
|
t.Fatalf("augment failed: %v", err)
|
|
}
|
|
if _, present := got[0].LockData[gone]; present {
|
|
t.Fatalf("expected removed model's lock key pruned, got %#v", got[0].LockData)
|
|
}
|
|
if got[0].LockData[keep] != 1 {
|
|
t.Fatalf("expected surviving model ID 1 kept, got %#v", got[0].LockData)
|
|
}
|
|
}
|
|
|
|
// Cutover: a lock that only has the OLD config-derived key adopts that ID under
|
|
// the new model-path key, with zero shift, and prunes the old key.
|
|
func TestCutoverRemapPreservesID(t *testing.T) {
|
|
oldKey := "visualeffects:head_accessories/hat/bandana" // {dataset}:{group}/{category}/{stem-without-hfx_}
|
|
newKey := "visualeffects:head_accessories/hat/hfx_bandana.mdl"
|
|
collected := []nativeCollectedDataset{vfxDataset(map[string]int{oldKey: 4242})}
|
|
got, err := augmentWithAutogeneratedAccessoryVisualeffects(collected, []autogenManifestEntry{vfxEntry()}, accConsumer(), nil)
|
|
if err != nil {
|
|
t.Fatalf("augment failed: %v", err)
|
|
}
|
|
if got[0].LockData[newKey] != 4242 {
|
|
t.Fatalf("expected cutover to carry ID 4242 to new key, got %#v", got[0].LockData)
|
|
}
|
|
if _, present := got[0].LockData[oldKey]; present {
|
|
t.Fatalf("expected old config key pruned after cutover, got %#v", got[0].LockData)
|
|
}
|
|
}
|
|
```
|
|
|
|
> Note for the implementer: the old config key in `TestCutoverRemapPreservesID` must equal what `accessoryVisualeffectIdentity` produces for `vfxEntry()` under `accConsumer()`'s policy. If the suite reports a different computed key, set `oldKey` to the value the test prints — do not change the production code to match a guessed key.
|
|
|
|
- [ ] **Step 9: Run the full identity suite + regression suite**
|
|
|
|
Run: `go test ./internal/topdata/ -run 'TestAccessory|TestConfigOnly|TestModel|TestCutover'`
|
|
Expected: PASS.
|
|
Run: `go test ./internal/topdata/`
|
|
Expected: PASS — confirms `autogen_preserve_test.go` (fail-open preservation) still holds with the new model-anchored matcher.
|
|
|
|
- [ ] **Step 10: Commit**
|
|
|
|
```bash
|
|
git add internal/topdata/autogen.go internal/topdata/autogen_lock_identity_test.go
|
|
git commit -m "feat(topdata): anchor autogen lock identity on model path with cutover remap"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 4: Git-LFS materialization inside Crucible
|
|
|
|
**Files:**
|
|
|
|
- Modify: `sow-tools/internal/topdata/top_package.go` (`packageBuiltTopData` at line 122; add `materializeAssetLFS`, `hasLFSPointerStubs`; add `bytes` import)
|
|
- Create: `sow-tools/internal/topdata/lfs_materialize_test.go`
|
|
|
|
**Interfaces:**
|
|
|
|
- Consumes: `gitOutput` (assets_repo.go), `project.Project.Root`, `project.Project.TopDataSourceDir()`.
|
|
- Produces:
|
|
- `func materializeAssetLFS(p *project.Project, progress func(string)) error`
|
|
- `func hasLFSPointerStubs(root string) bool`
|
|
- Skip control: env `CRUCIBLE_SKIP_LFS` (read inside `materializeAssetLFS`).
|
|
|
|
- [ ] **Step 1: Write the failing stub-detection test**
|
|
|
|
Create `sow-tools/internal/topdata/lfs_materialize_test.go`:
|
|
|
|
```go
|
|
package topdata
|
|
|
|
import (
|
|
"path/filepath"
|
|
"testing"
|
|
)
|
|
|
|
func TestHasLFSPointerStubsDetectsStub(t *testing.T) {
|
|
root := t.TempDir()
|
|
stub := "version https://git-lfs.github.com/spec/v1\noid sha256:abc\nsize 12345\n"
|
|
writeFile(t, filepath.Join(root, "a", "model.mdl"), stub)
|
|
writeFile(t, filepath.Join(root, "a", "real.txt"), "not a pointer, real bytes here")
|
|
|
|
if !hasLFSPointerStubs(root) {
|
|
t.Fatal("expected stub detection to be true")
|
|
}
|
|
}
|
|
|
|
func TestHasLFSPointerStubsNoStub(t *testing.T) {
|
|
root := t.TempDir()
|
|
writeFile(t, filepath.Join(root, "real.mdl"), "binary-ish content without a pointer header")
|
|
if hasLFSPointerStubs(root) {
|
|
t.Fatal("expected no stub detected")
|
|
}
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 2: Run it to confirm it fails**
|
|
|
|
Run: `go test ./internal/topdata/ -run TestHasLFSPointerStubs`
|
|
Expected: FAIL — `undefined: hasLFSPointerStubs`.
|
|
|
|
- [ ] **Step 3: Add the materialization helpers**
|
|
|
|
In `sow-tools/internal/topdata/top_package.go` add `"bytes"` to the import block, then add:
|
|
|
|
```go
|
|
// materializeAssetLFS ensures the topdata asset tree holds real bytes, not
|
|
// Git-LFS pointer stubs, before the hak is packed. A bare CI checkout does not
|
|
// smudge LFS, so a direct Crucible build would otherwise ship ~130-byte stubs.
|
|
// It uses the clone's own credentials (R1) and hard-fails if stubs survive the
|
|
// pull. Set CRUCIBLE_SKIP_LFS=1 to opt out (maintain-tree regenerates the data/
|
|
// tree and discards the package, so it must not hard-depend on LFS).
|
|
func materializeAssetLFS(p *project.Project, progress func(string)) error {
|
|
if progress == nil {
|
|
progress = func(string) {}
|
|
}
|
|
assetsDir := filepath.Join(p.TopDataSourceDir(), "assets")
|
|
info, err := os.Stat(assetsDir)
|
|
if err != nil || !info.IsDir() {
|
|
return nil // no asset tree to materialize
|
|
}
|
|
if !hasLFSPointerStubs(assetsDir) {
|
|
return nil // already real bytes (or repo does not use LFS)
|
|
}
|
|
if isTruthyEnv(os.Getenv("CRUCIBLE_SKIP_LFS")) {
|
|
progress("CRUCIBLE_SKIP_LFS set: leaving Git-LFS pointer stubs unmaterialized (package will be discarded)")
|
|
return nil
|
|
}
|
|
if _, err := gitOutput(p.Root, "lfs", "version"); err != nil {
|
|
return fmt.Errorf("asset tree has Git-LFS pointer stubs but git-lfs is unavailable; install git-lfs or enter the nix devshell: %w", err)
|
|
}
|
|
progress("Materializing Git-LFS assets (git lfs pull)...")
|
|
// A fresh clone may lack the lfs smudge/clean filters; install them locally
|
|
// first (scoped to this repo's .git/config, no host mutation). Idempotent.
|
|
if _, err := gitOutput(p.Root, "lfs", "install", "--local"); err != nil {
|
|
return fmt.Errorf("git lfs install --local failed: %w", err)
|
|
}
|
|
if _, err := gitOutput(p.Root, "lfs", "pull"); err != nil {
|
|
return fmt.Errorf("git lfs pull failed: %w", err)
|
|
}
|
|
if hasLFSPointerStubs(assetsDir) {
|
|
return fmt.Errorf("refusing to build from pointer stubs: Git-LFS pointer stubs remain under %s after git lfs pull", assetsDir)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// hasLFSPointerStubs reports whether any small text file under root is a Git-LFS
|
|
// pointer. Real assets are larger than any pointer stub, so the size gate keeps
|
|
// this cheap on a tree of binaries.
|
|
func hasLFSPointerStubs(root string) bool {
|
|
found := false
|
|
_ = filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
|
|
if err != nil || d.IsDir() || found {
|
|
return nil
|
|
}
|
|
info, ierr := d.Info()
|
|
if ierr != nil || info.Size() == 0 || info.Size() > 1024 {
|
|
return nil
|
|
}
|
|
f, oerr := os.Open(path)
|
|
if oerr != nil {
|
|
return nil
|
|
}
|
|
defer f.Close()
|
|
buf := make([]byte, 256)
|
|
n, _ := f.Read(buf)
|
|
if bytes.Contains(buf[:n], []byte("git-lfs.github.com/spec/v1")) {
|
|
found = true
|
|
}
|
|
return nil
|
|
})
|
|
return found
|
|
}
|
|
|
|
func isTruthyEnv(v string) bool {
|
|
switch strings.TrimSpace(strings.ToLower(v)) {
|
|
case "1", "true", "yes", "on":
|
|
return true
|
|
default:
|
|
return false
|
|
}
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Call materialization before packing**
|
|
|
|
In `packageBuiltTopData` (line 122), immediately after the `if progress == nil` guard and before `outputHAK := p.TopDataPackageHAKPath()`, add:
|
|
|
|
```go
|
|
if err := materializeAssetLFS(p, progress); err != nil {
|
|
return PackageResult{}, err
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 5: Run the stub-detection tests**
|
|
|
|
Run: `go test ./internal/topdata/ -run TestHasLFSPointerStubs`
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 6: Run the full suite + build**
|
|
|
|
Run: `go test ./internal/topdata/` then `make build`
|
|
Expected: PASS, clean build.
|
|
|
|
- [ ] **Step 7: Commit**
|
|
|
|
```bash
|
|
git add internal/topdata/top_package.go internal/topdata/lfs_materialize_test.go
|
|
git commit -m "feat(topdata): materialize Git-LFS asset bytes inside Crucible before packing"
|
|
```
|
|
|
|
~~> Skipped: a `--skip-lfs` CLI flag. The `CRUCIBLE_SKIP_LFS` env covers the only opt-out consumer (maintain-tree CI); add the flag if a human ever needs it interactively.~~
|
|
|
|
> Operator redaction: Please add a `--skip-lfs` CLI flag in addition to the `CRUCIBLE_SKIP_LFS` env.
|
|
|
|
---
|
|
|
|
### Task 5: NWN_ROOT cleanup
|
|
|
|
**Files:**
|
|
|
|
- Modify: `sow-tools/internal/dispatch/dispatch.go:251-254`
|
|
- Modify: `sow-tools/docs/consumer-contract.md` (remove `## NWN_ROOT rule`, lines 39-48)
|
|
- Modify: `sow-tools/README.md:100`
|
|
- Modify: `sow-tools/internal/dispatch/dispatch_test.go` (if it asserts the old help text)
|
|
|
|
**Interfaces:** none (docs + help text only).
|
|
|
|
- [ ] **Step 1: Check whether a test pins the old help text**
|
|
|
|
Run: `grep -n "NWN_ROOT\|never guesses" internal/dispatch/dispatch_test.go`
|
|
If it asserts the old "passed explicitly via NWN_ROOT" string, update that expectation in the same step you change the help text below.
|
|
|
|
- [ ] **Step 2: Replace the dispatch help text**
|
|
|
|
In `sow-tools/internal/dispatch/dispatch.go`, replace lines 251-254 (the three `NWN_ROOT` Fprintf lines) with:
|
|
|
|
```go
|
|
fmt.Fprintf(w, "\nBuilders read all inputs from the project tree (nwn-tool.yaml + data/); no\n")
|
|
fmt.Fprintf(w, "NWN install is required. A builder that one day needs NWN game data\n")
|
|
fmt.Fprintf(w, "auto-detects the install; it never requires a hand-set path. See\n")
|
|
fmt.Fprintf(w, "docs/consumer-contract.md.\n")
|
|
```
|
|
|
|
- [ ] **Step 3: Remove the `NWN_ROOT rule` section from the contract doc**
|
|
|
|
In `sow-tools/docs/consumer-contract.md`, delete the entire `## `NWN_ROOT` rule` section (the heading through the "fall back to a home-directory default." paragraph). (The replacement auto-detect rule lands in Task 6's contract rewrite.)
|
|
|
|
- [ ] **Step 4: Fix the README pointer**
|
|
|
|
In `sow-tools/README.md:100`, change the line that reads
|
|
`How the artifact repos resolve a Crucible binary (and the `NWN_ROOT` rule) is`
|
|
to drop the `NWN_ROOT` clause:
|
|
`How the artifact repos resolve a Crucible binary is`
|
|
(Verify the surrounding sentence still reads correctly; adjust the trailing words if needed.)
|
|
|
|
- [ ] **Step 5: Verify build + dispatch test**
|
|
|
|
Run: `go test ./internal/dispatch/` then `make build`
|
|
Expected: PASS, clean build. Then `grep -rn "NWN_ROOT" internal/ README.md docs/consumer-contract.md` — Expected: no matches (the only remaining mention is `docs/migration-from-nwn-tool.md`, left as historical record).
|
|
|
|
- [ ] **Step 6: Commit**
|
|
|
|
```bash
|
|
git add internal/dispatch/dispatch.go internal/dispatch/dispatch_test.go docs/consumer-contract.md README.md
|
|
git commit -m "chore(crucible): drop dead NWN_ROOT rule; auto-detect is the forward rule"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 6: Consumer-contract parity rules + image deployment-only
|
|
|
|
**Files:**
|
|
|
|
- Modify: `sow-tools/docs/consumer-contract.md`
|
|
|
|
**Interfaces:** none (doc only). This is the written half of the parity guarantee; Task 7 is the enforced half.
|
|
|
|
- [ ] **Step 1: Add the parity contract section**
|
|
|
|
In `sow-tools/docs/consumer-contract.md`, add a new section (place it after the resolution-order section):
|
|
|
|
```markdown
|
|
## Parity contract — wrappers are not authoritative for build inputs
|
|
|
|
A consumer build wrapper may only:
|
|
|
|
1. **validate** — a pre-build gate that does not change output;
|
|
2. **publish** — a post-build step on a separate artifact;
|
|
3. **resolve the crucible binary** — the bootstrap above.
|
|
|
|
A wrapper MUST NOT fetch, resolve, generate, or stage any input Crucible reads to
|
|
produce the artifact. If Crucible needs an input, Crucible acquires it. This makes
|
|
the "no local-machine assumptions" + "same inputs → identical output" rules
|
|
explicit: local `crucible <build>` and CI `crucible <build>` against the same
|
|
committed config must produce identical bytes, with no CI-only pre-steps.
|
|
|
|
Builders read every input from the project tree. No NWN install is required. A
|
|
builder that one day needs NWN game data auto-detects the install (the NWN home
|
|
and Steam path are reliably discoverable), with an optional explicit override for
|
|
non-standard layouts — never a required hand-set env.
|
|
```
|
|
|
|
- [ ] **Step 2: Replace the CI image guidance with deployment-only**
|
|
|
|
In `consumer-contract.md`, replace the entire `## In CI (preferred)` section (the OPERATOR NOTE + struck-through `container: image:` guidance) with:
|
|
|
|
```markdown
|
|
## In CI
|
|
|
|
Builds run the Crucible **binary** — from the `crucible.sh` / `crucible.ps1`
|
|
bootstrap (anonymous download) or, for Nix users, from the flake devshell pinned
|
|
by `flake.lock`. CI runs the _same_ `crucible <build>` as local dev, inside the
|
|
nix devshell (binary-cache fast). No build container, no token, no CI-only
|
|
pre-steps.
|
|
|
|
The `registry.westgate.pw/deployment/crucible:<sha>` image is **deployment-only**:
|
|
`prod.yml` pins it so tools travel with the runtime host (`nwn.enable`), a
|
|
disabled placeholder until the NWN stack lands. It is **not** a build tool and no
|
|
build/CI job consumes it.
|
|
```
|
|
|
|
- [ ] **Step 3: Verify**
|
|
|
|
Run: `grep -n "Stop. Do not\|container:\|pinned image" docs/consumer-contract.md`
|
|
Expected: no matches.
|
|
|
|
- [ ] **Step 4: Commit**
|
|
|
|
```bash
|
|
git add docs/consumer-contract.md
|
|
git commit -m "docs(contract): add wrapper parity rules; state crucible image is deployment-only"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 7: CI parity guard (hermetic integration test)
|
|
|
|
**Files:**
|
|
|
|
- Create: `sow-tools/internal/topdata/parity_guard_test.go`
|
|
|
|
**Interfaces:**
|
|
|
|
- Consumes: `resolveCDNChannelManifest` (Task 2), `augmentWithAutogeneratedAccessoryVisualeffects` (Task 3), `applyAutogenConsumers` (existing).
|
|
|
|
This is the test that would have caught the v0.1.4 / local-zero-rows bug: with **no `manifest_file`, no `NWN_ROOT`, no token**, a `cdn_channel` consumer must contribute accessory rows. It drives the full `applyAutogenConsumers` path against an httptest CDN, mechanically enforcing R2/R3/R5 for the resolution+augmentation pipeline. (The operational, whole-build guard lands as a CI job in Task 11.)
|
|
|
|
- [ ] **Step 1: Write the parity guard test**
|
|
|
|
Create `sow-tools/internal/topdata/parity_guard_test.go`:
|
|
|
|
```go
|
|
package topdata
|
|
|
|
import (
|
|
"testing"
|
|
|
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
|
|
)
|
|
|
|
// PARITY GUARD: a bare project (no manifest_file, no NWN_ROOT, no token) with a
|
|
// cdn_channel consumer must produce accessory rows. If accessory resolution ever
|
|
// drifts back out of Crucible into a wrapper, this fails.
|
|
func TestParityGuardCDNChannelProducesAccessoryRows(t *testing.T) {
|
|
root := testProjectRoot(t)
|
|
srv := cdnServer(t, map[string]string{
|
|
"/releases/haks/channels.json": `{"current":"v1.0.0"}`,
|
|
"/releases/haks/v1.0.0/vfxs.yml": `assets:
|
|
- path: vfxs/head_accessories/hat/hfx_bandana.mdl
|
|
restype: mdl
|
|
sha256: aaa
|
|
- path: vfxs/chest_accessories/cape/cfx_cloak.mdl
|
|
restype: mdl
|
|
sha256: bbb
|
|
`,
|
|
})
|
|
t.Setenv("BUNNY_CDN_BASE", srv.URL)
|
|
t.Setenv("SOW_TOPDATA_ASSET_CHANNEL", "current")
|
|
// Prove R3: no NWN_ROOT in the environment.
|
|
t.Setenv("NWN_ROOT", "")
|
|
|
|
c := cdnConsumer()
|
|
c.Source.CDNBaseEnv = "BUNNY_CDN_BASE"
|
|
c.Source.ChannelEnv = "SOW_TOPDATA_ASSET_CHANNEL"
|
|
c.AccessoryVisualeffects.ModelColumn = "Imp_HeadCon_Node"
|
|
|
|
p := testProject(root)
|
|
p.Config.Autogen.Consumers = []project.AutogenConsumerConfig{c}
|
|
|
|
collected := []nativeCollectedDataset{{
|
|
Dataset: nativeDataset{Name: "visualeffects", Kind: nativeDatasetBase},
|
|
Columns: []string{"Label", "Imp_HeadCon_Node", "Imp_Root_S_Node"},
|
|
Rows: nil,
|
|
LockData: map[string]int{},
|
|
}}
|
|
|
|
got, err := applyAutogenConsumers(p, collected, nil)
|
|
if err != nil {
|
|
t.Fatalf("applyAutogenConsumers failed: %v", err)
|
|
}
|
|
if len(got[0].Rows) != 2 {
|
|
t.Fatalf("PARITY GUARD: expected 2 accessory rows, got %d (%#v)", len(got[0].Rows), got[0].Rows)
|
|
}
|
|
wantKey := "visualeffects:head_accessories/hat/hfx_bandana.mdl"
|
|
if _, ok := got[0].LockData[wantKey]; !ok {
|
|
t.Fatalf("PARITY GUARD: expected lock id for %s, got %#v", wantKey, got[0].LockData)
|
|
}
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 2: Run the parity guard**
|
|
|
|
Run: `go test ./internal/topdata/ -run TestParityGuard`
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 3: Run the whole sow-tools suite**
|
|
|
|
Run: `make check`
|
|
Expected: PASS (go vet + go test + shellcheck + yamllint).
|
|
|
|
- [ ] **Step 4: Commit**
|
|
|
|
```bash
|
|
git add internal/topdata/parity_guard_test.go
|
|
git commit -m "test(topdata): parity guard — cdn_channel yields accessory rows with no wrapper/env"
|
|
```
|
|
|
|
---
|
|
|
|
## Phase 2 — Release Crucible
|
|
|
|
> Operator step, no code. Flag down the operator and remind them. After Phase 1 merges, cut a Crucible `v*` release so the bootstrap and flake input resolve the parity build. Assets are already anon-downloadable (R1, verified in the design). Record the released version (e.g. `v0.3.0`) — Phase 3 pins it via `flake.lock`.
|
|
|
|
- [ ] **Step 1:** Merge Phase 1; tag/release Crucible per the repo's existing release workflow. Note the resulting version tag for Phase 3.
|
|
|
|
> ✅ Release tag: `v0.3.0`
|
|
> <https://git.westgate.pw/ShadowsOverWestgate/sow-tools/releases/tag/v0.3.0>
|
|
|
|
---
|
|
|
|
## Phase 3 — Toolchain convergence (flakes)
|
|
|
|
> One binary, one pin per repo, three modes — all anonymous, all identical. sow-tools' flake already exposes `packages.<system>.crucible`; consumers add it as a flake input.
|
|
|
|
### Task 8: sow-topdata flake — add the crucible input + devshell
|
|
|
|
**Files:**
|
|
|
|
- Modify: `sow-topdata/flake.nix`
|
|
- Regenerate: `sow-topdata/flake.lock`
|
|
|
|
**Interfaces:**
|
|
|
|
- Consumes: `sow-tools` flake output `packages.${system}.crucible` (exists in `sow-tools/flake.nix`).
|
|
|
|
- [ ] **Step 1: Add the sow-tools input and put crucible on PATH**
|
|
|
|
Edit `sow-topdata/flake.nix`. Add the input and thread it through:
|
|
|
|
```nix
|
|
{
|
|
description = "sow-topdata — 2DA/TLK/wiki validate/build dev shell";
|
|
|
|
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
|
|
inputs.sow-tools.url = "git+https://git.westgate.pw/ShadowsOverWestgate/sow-tools";
|
|
inputs.sow-tools.inputs.nixpkgs.follows = "nixpkgs";
|
|
|
|
outputs = { self, nixpkgs, sow-tools }:
|
|
let
|
|
systems = [ "x86_64-linux" "aarch64-linux" ];
|
|
forAllSystems = f: nixpkgs.lib.genAttrs systems (system: f system nixpkgs.legacyPackages.${system});
|
|
in
|
|
{
|
|
devShells = forAllSystems (system: pkgs: {
|
|
default = pkgs.mkShell {
|
|
name = "sow-topdata";
|
|
packages = with pkgs; [
|
|
bashInteractive
|
|
git
|
|
git-lfs
|
|
shellcheck
|
|
yamllint
|
|
jq
|
|
yq-go
|
|
zstd
|
|
prettier
|
|
] ++ [ sow-tools.packages.${system}.crucible ];
|
|
shellHook = ''
|
|
echo "sow-topdata devshell. crucible, git-lfs, shellcheck, yamllint, jq, yq, zstd, prettier ready."
|
|
echo " scripts/validate-topdata.sh structural checks"
|
|
echo " scripts/build-topdata.sh build package via crucible"
|
|
echo " scripts/format-tree.sh reformat tree to .editorconfig"
|
|
'';
|
|
};
|
|
});
|
|
};
|
|
}
|
|
```
|
|
|
|
> Note: `forAllSystems` now passes both `system` and `pkgs`. If the repo's existing helper signature differs, match the convention used by `sow-assets-manifest/flake.nix` (which already pins crucible per-system) rather than introducing a new pattern.
|
|
|
|
> Operator concern: The `scripts/` still exist? We should be removing wrappers for tasks Crucible controls.
|
|
|
|
- [ ] **Step 2: Lock the input**
|
|
|
|
Run: `cd sow-topdata && nix flake update sow-tools` (or `nix flake lock --update-input sow-tools`)
|
|
Expected: `flake.lock` gains a `sow-tools` node pinned to the Phase 2 release commit.
|
|
|
|
- [ ] **Step 3: Verify crucible is on PATH in the devshell**
|
|
|
|
Run: `cd sow-topdata && nix develop --command crucible version`
|
|
Expected: prints the pinned Crucible version (the Phase 2 release).
|
|
|
|
- [ ] **Step 4: Commit**
|
|
|
|
```bash
|
|
git add sow-topdata/flake.nix sow-topdata/flake.lock
|
|
git commit -m "build(topdata): provide crucible from sow-tools flake input in the devshell"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 9: sow-assets-manifest flake — switch off the manual fetchurl pin
|
|
|
|
**Files:**
|
|
|
|
- Modify: `sow-assets-manifest/flake.nix`
|
|
- Regenerate: `sow-assets-manifest/flake.lock`
|
|
|
|
**Interfaces:**
|
|
|
|
- Consumes: `sow-tools` flake output `packages.${system}.crucible`.
|
|
|
|
- [ ] **Step 1: Replace the `fetchurl`/hash crucible derivation with the flake input**
|
|
|
|
Edit `sow-assets-manifest/flake.nix`. Add the input:
|
|
|
|
```nix
|
|
inputs.sow-tools.url = "git+https://git.westgate.pw/ShadowsOverWestgate/sow-tools";
|
|
inputs.sow-tools.inputs.nixpkgs.follows = "nixpkgs";
|
|
```
|
|
|
|
Change the outputs signature to `{ self, nixpkgs, sow-tools }`, delete the entire `crucibleVersion` / `crucibleArch` / `crucibleHashes` / `crucible = pkgs.stdenvNoCC.mkDerivation { ... }` block, and replace the `++ [ crucible ]` line in `packages` with:
|
|
|
|
```nix
|
|
++ [ sow-tools.packages.${pkgs.system}.crucible ];
|
|
```
|
|
|
|
(`forAllSystems` here already passes `pkgs`; `pkgs.system` is available.)
|
|
|
|
- [ ] **Step 2: Lock the input**
|
|
|
|
Run: `cd sow-assets-manifest && nix flake lock --update-input sow-tools`
|
|
Expected: `flake.lock` gains the `sow-tools` node.
|
|
|
|
- [ ] **Step 3: Verify**
|
|
|
|
Run: `cd sow-assets-manifest && nix develop --command crucible version`
|
|
Expected: same pinned version as Task 8.
|
|
|
|
- [ ] **Step 4: Commit**
|
|
|
|
```bash
|
|
git add sow-assets-manifest/flake.nix sow-assets-manifest/flake.lock
|
|
git commit -m "build(assets-manifest): pin crucible via sow-tools flake input, drop manual fetchurl hash"
|
|
```
|
|
|
|
---
|
|
|
|
## Phase 4 — sow-topdata config, wrapper, CI
|
|
|
|
### Task 10: Swap `manifest_file:` → `source:` in nwn-tool.yaml
|
|
|
|
**Files:**
|
|
|
|
- Modify: `sow-topdata/nwn-tool.yaml:99-140`
|
|
|
|
**Interfaces:**
|
|
|
|
- Consumes: the `cdn_channel` source schema (Task 1) and resolution (Task 2).
|
|
|
|
- [ ] **Step 1: Replace the consumer source declaration**
|
|
|
|
In `sow-topdata/nwn-tool.yaml`, in the `autogen.consumers` accessory entry, delete line 106 (`manifest_file: .cache/sow-accessory-vfx-manifest.json`) and insert the `source:` block between `optional: true` (line 105) and `accessory_visualeffects:` (line 107):
|
|
|
|
```yaml
|
|
optional: true
|
|
source:
|
|
kind: cdn_channel
|
|
cdn_base_env: BUNNY_CDN_BASE
|
|
channels_path: releases/haks/channels.json
|
|
manifest_path: releases/haks/{tag}/vfxs.yml
|
|
release_marker_path: releases/haks/{tag}/haks.json
|
|
channel_env: SOW_TOPDATA_ASSET_CHANNEL
|
|
offline_override_env: SOW_VFXS_MANIFEST
|
|
accessory_visualeffects:
|
|
```
|
|
|
|
Leave the entire `accessory_visualeffects:` policy block (groups, group_token_source, delimiter, key_format, etc.) unchanged. Update the explanatory comment at lines 11-16 to describe Crucible's `cdn_channel` resolution instead of `scripts/resolve-accessory-vfx.sh`.
|
|
|
|
- [ ] **Step 2: Validate the config**
|
|
|
|
Run: `cd sow-topdata && nix develop --command crucible topdata config --validate` (use the actual validate subcommand; if unsure run `crucible topdata --help`).
|
|
Expected: no validation errors about `autogen.consumers[].source`.
|
|
|
|
- [ ] **Step 3: Commit**
|
|
|
|
```bash
|
|
git add sow-topdata/nwn-tool.yaml
|
|
git commit -m "config(topdata): resolve accessory VFX via cdn_channel source (was manifest_file)"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 11: Delete the resolver script, thin the wrapper, add the CI parity job
|
|
|
|
**Files:**
|
|
|
|
- Delete: `sow-topdata/scripts/resolve-accessory-vfx.sh`
|
|
- Delete: `sow-topdata/tests/resolve-accessory-vfx-contract.sh`
|
|
- Modify: `sow-topdata/scripts/build-topdata.sh`
|
|
- Modify: `sow-topdata/scripts/lib.sh` (remove `materialize_lfs_assets`)
|
|
- Create: `sow-topdata/.gitea/workflows/parity-guard.yml`
|
|
|
|
**Interfaces:**
|
|
|
|
- Consumes: Crucible now owns VFX resolution (Task 2/10) and LFS (Task 4).
|
|
|
|
- [ ] **Step 1: Delete the resolver script and its bash contract test**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
cd sow-topdata
|
|
git rm scripts/resolve-accessory-vfx.sh tests/resolve-accessory-vfx-contract.sh
|
|
```
|
|
|
|
(If `tests/resolve-accessory-vfx-contract.sh` is referenced by a test runner or Makefile, remove that reference too — grep first: `grep -rn "resolve-accessory-vfx" .`)
|
|
|
|
- [ ] **Step 2: Thin `build-topdata.sh`**
|
|
|
|
Replace `sow-topdata/scripts/build-topdata.sh` with the validate→resolve→build shape (no resolver pre-step, no LFS pull — Crucible owns both now):
|
|
|
|
```bash
|
|
#!/usr/bin/env bash
|
|
# Build the topdata package (.cache/sow_top.hak + .cache/sow_tlk.tlk) via Crucible.
|
|
# Crucible reads ./nwn-tool.yaml, resolves accessory VFX from the asset CDN,
|
|
# materializes Git-LFS asset bytes, compiles data/ into 2da + the custom TLK, and
|
|
# writes the package into paths.build (.cache). This wrapper only: validate (gate)
|
|
# -> resolve crucible -> build. It is NOT authoritative for any build input
|
|
# (see sow-tools/docs/consumer-contract.md). FAILS CLOSED if crucible cannot be
|
|
# resolved — it never fakes a package.
|
|
# shellcheck source=lib.sh
|
|
. "$(dirname "$0")/lib.sh"
|
|
|
|
cd "$REPO_ROOT"
|
|
|
|
"$REPO_ROOT/scripts/validate-topdata.sh"
|
|
|
|
if ! crucible="$(resolve_crucible)"; then
|
|
die "crucible not found. Install it on PATH, set \$CRUCIBLE, enter the nix \
|
|
devshell, or keep ./crucible.sh (the bootstrap wrapper) in the repo."
|
|
fi
|
|
|
|
log "building topdata package via $crucible"
|
|
mkdir -p .cache
|
|
"$crucible" topdata build-topdata
|
|
|
|
hak=".cache/sow_top.hak"
|
|
tlk=".cache/sow_tlk.tlk"
|
|
[[ -f "$hak" && -f "$tlk" ]] || die "builder did not produce the top package ($hak + $tlk)"
|
|
log "build-topdata: OK -> $hak + $tlk"
|
|
```
|
|
|
|
- [ ] **Step 3: Remove `materialize_lfs_assets` from `lib.sh`**
|
|
|
|
In `sow-topdata/scripts/lib.sh`, delete the `materialize_lfs_assets()` function (Crucible owns LFS now). Keep `log/warn/die/need/git_sha/resolve_crucible`. Run `grep -rn "materialize_lfs_assets" sow-topdata/` afterward — Expected: no remaining callers.
|
|
|
|
- [ ] **Step 4: Add the operational parity-guard CI job**
|
|
|
|
Create `sow-topdata/.gitea/workflows/parity-guard.yml` — runs Crucible **directly** (no wrapper, no token, no `NWN_ROOT`) in a bare checkout and asserts the package builds with accessory rows:
|
|
|
|
```yaml
|
|
# Operational parity guard: build the topdata package with Crucible ALONE — no
|
|
# build-topdata.sh, no CRUCIBLE_TOKEN, no NWN_ROOT, anonymous CDN — and assert
|
|
# the built visualeffects.2da carries accessory rows. This is the enforced half
|
|
# of the parity contract (sow-tools/docs/consumer-contract.md): it fails if any
|
|
# build-output-affecting step ever drifts back out of Crucible into a wrapper.
|
|
name: parity-guard
|
|
|
|
on:
|
|
pull_request:
|
|
|
|
jobs:
|
|
parity:
|
|
runs-on: nix-docker
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
lfs: false # prove Crucible materializes LFS itself
|
|
|
|
- name: Build with crucible alone (no wrapper, no env)
|
|
run: nix develop --command crucible topdata build-topdata
|
|
|
|
- name: Assert accessory rows present
|
|
run: |
|
|
nix develop --command bash -euo pipefail -c '
|
|
crucible topdata convert 2da-to-json .cache/visualeffects.2da .cache/visualeffects.json || \
|
|
crucible-hak extract .cache/sow_top.hak visualeffects.2da .cache/visualeffects.2da
|
|
grep -Eq "head_accessories|chest_accessories|head_decorations|head_features" .cache/visualeffects.json \
|
|
|| { echo "PARITY GUARD: no accessory rows in built visualeffects.2da"; exit 1; }
|
|
'
|
|
```
|
|
|
|
> Implementer note: confirm the exact path Crucible writes the compiled `visualeffects.2da` to (it lands under `paths.build` = `.cache`; run `crucible topdata build-topdata` once locally and `find .cache -name 'visualeffects.2da'`). Wire the assert step to that real path; the `convert`/`extract` fallback above is a sketch — replace it with the one command that actually reads the built 2da. The assertion (accessory group tokens present) is the load-bearing part.
|
|
|
|
> Implementer note: `visualeffects.2da` should appear at `sow-topdata/.cache/2da/visualeffects.2da` after a `build-topdata` run.
|
|
|
|
- [ ] **Step 5: Lint the workflows and scripts**
|
|
|
|
Run: `cd sow-topdata && nix develop --command bash -c 'shellcheck scripts/build-topdata.sh scripts/lib.sh && yamllint .gitea/workflows/parity-guard.yml'`
|
|
Expected: clean.
|
|
|
|
- [ ] **Step 6: Commit**
|
|
|
|
```bash
|
|
git add -A sow-topdata/scripts sow-topdata/tests sow-topdata/.gitea/workflows/parity-guard.yml
|
|
git commit -m "build(topdata): thin wrapper (Crucible owns VFX+LFS); add operational parity guard"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 12: Drop `CRUCIBLE_TOKEN`, swap the LFS skip env, build inside the devshell
|
|
|
|
**Files:**
|
|
|
|
- Modify: `sow-topdata/.gitea/workflows/build-topdata.yml`
|
|
- Modify: `sow-topdata/.gitea/workflows/maintain-tree.yml`
|
|
- Audit: every `sow-topdata/.gitea/workflows/*.yml` for `CRUCIBLE_TOKEN` / `TOPDATA_SKIP_LFS`
|
|
|
|
**Interfaces:**
|
|
|
|
- Consumes: anonymous Crucible (R1), `CRUCIBLE_SKIP_LFS` (Task 4).
|
|
|
|
- [ ] **Step 1: Find every reference**
|
|
|
|
Run: `grep -rn "CRUCIBLE_TOKEN\|TOPDATA_SKIP_LFS" sow-topdata/.gitea/workflows/`
|
|
Expected references: `build-topdata.yml` (CRUCIBLE_TOKEN) and `maintain-tree.yml` (both).
|
|
|
|
- [ ] **Step 2: Remove `CRUCIBLE_TOKEN` from build-topdata.yml**
|
|
|
|
In `sow-topdata/.gitea/workflows/build-topdata.yml`, delete the `env:` block carrying `CRUCIBLE_TOKEN` from the "Build topdata package" step (the binary now downloads anonymously / comes from the devshell). Update the leading comment that references `CRUCIBLE_TOKEN` to state the build is anonymous via the nix devshell. The step command stays `nix develop --command scripts/build-topdata.sh`.
|
|
|
|
- [ ] **Step 3: Swap the LFS skip env in maintain-tree.yml**
|
|
|
|
In `sow-topdata/.gitea/workflows/maintain-tree.yml`, in the "Regenerate lockfiles" step: delete `CRUCIBLE_TOKEN` from `env:`, and replace `TOPDATA_SKIP_LFS: "1"` with `CRUCIBLE_SKIP_LFS: "1"`. Update the adjacent comment to point at Crucible's skip control rather than the wrapper's. Command stays `nix develop --command scripts/build-topdata.sh`.
|
|
|
|
- [ ] **Step 4: Sweep the remaining workflows**
|
|
|
|
For any other workflow that set `CRUCIBLE_TOKEN` to fetch the binary (e.g. `publish-topdata.yml`, `release.yml`, `drift-check.yml` — check what grep returned), remove the token if the job builds via the nix devshell. Leave tokens that authenticate a _publish_ (release upload), which are unrelated to fetching crucible.
|
|
|
|
- [ ] **Step 5: Lint**
|
|
|
|
Run: `cd sow-topdata && nix develop --command yamllint .gitea/workflows/`
|
|
Expected: clean. Then `grep -rn "CRUCIBLE_TOKEN\|TOPDATA_SKIP_LFS" .gitea/workflows/` — Expected: no build-time CRUCIBLE_TOKEN, no TOPDATA_SKIP_LFS.
|
|
|
|
- [ ] **Step 6: Commit**
|
|
|
|
```bash
|
|
git add sow-topdata/.gitea/workflows
|
|
git commit -m "ci(topdata): drop vestigial CRUCIBLE_TOKEN; use CRUCIBLE_SKIP_LFS for maintain-tree"
|
|
```
|
|
|
|
---
|
|
|
|
## Phase 5 — Verification
|
|
|
|
### Task 13: End-to-end parity verification
|
|
|
|
**Files:** none (verification only).
|
|
|
|
- [ ] **Step 1: Bare-clone build, no env, no key**
|
|
|
|
In a fresh clone of `sow-topdata` (or a clean checkout with cleared `.cache`):
|
|
|
|
```bash
|
|
rm -rf .cache
|
|
nix develop --command crucible topdata build-topdata
|
|
```
|
|
|
|
Expected: succeeds; `.cache/sow_top.hak` + `.cache/sow_tlk.tlk` produced. Confirm accessory rows:
|
|
|
|
```bash
|
|
find .cache -name 'visualeffects.2da' # locate the built 2da
|
|
# convert/extract and count rows under the 4 accessory groups
|
|
```
|
|
|
|
Expected: accessory rows present (the design's reference figure is 637 rows for the current channel state; the exact count tracks the live channel).
|
|
|
|
- [ ] **Step 2: Same command, no wrapper, with LFS unsmudged**
|
|
|
|
```bash
|
|
git lfs uninstall --local 2>/dev/null || true
|
|
rm -rf .cache
|
|
nix develop --command crucible topdata build-topdata
|
|
```
|
|
|
|
Expected: Crucible materializes LFS itself; the hak holds real bytes (no pointer stubs), same accessory rows.
|
|
|
|
- [ ] **Step 3: Lock stability across a config-only edit**
|
|
|
|
Temporarily change a presentation-only field in `nwn-tool.yaml` (e.g. `label_format`), rebuild, and `git diff data/visualeffects/lock.json`.
|
|
Expected: **no ID changes** (only labels/columns may differ). Revert the edit.
|
|
|
|
- [ ] **Step 4: Full check suites**
|
|
|
|
```bash
|
|
cd sow-tools && make check
|
|
cd ../sow-topdata && nix develop --command scripts/validate-topdata.sh
|
|
```
|
|
|
|
Expected: both green.
|
|
|
|
- [ ] **Step 5: Confirm the parity guard catches drift (sanity)**
|
|
|
|
Run `go test ./internal/topdata/ -run TestParityGuard` in `sow-tools`.
|
|
Expected: PASS. (Optional: prove it fails if resolution is disabled, then revert.)
|
|
|
|
---
|
|
|
|
## Notes / deferred (not this plan)
|
|
|
|
- Parts `2da` / `cachedmodels` reinstatement can reuse the `cdn_channel` source type — separate spec.
|
|
- Roll the thinned-wrapper pattern review across `sow-module` / `sow-codebase`.
|
|
- A future `id`+`sha` object form for the lock is a clean later add (the lock stays a flat `string→int` map here); the audit trail comes free from `git diff` on the committed `vfxs.yml`.
|