diff --git a/CHANGELOG_AUTOMATION_CONTRACT.md b/CHANGELOG_AUTOMATION_CONTRACT.md index e34d710..9c0f75b 100644 --- a/CHANGELOG_AUTOMATION_CONTRACT.md +++ b/CHANGELOG_AUTOMATION_CONTRACT.md @@ -1,5 +1,24 @@ # Changelog Automation Contract +## Status Snapshot + +Current state as of 2026-05-13: + +- The shared implementation is present in `internal/changelog/` and is exposed + as `sow-toolkit build-changelog`. +- Consumer configs exist in both `module/scripts/changelog.json` and + `assets/scripts/changelog.json`. +- CLI parsing and generator behavior are covered by + `internal/app/app_test.go` and `internal/changelog/changelog_test.go`. +- `module/scripts/release-all.sh` already stages `build/CHANGELOG.md`. + +Remaining work: + +- confirm and document the `assets/` release workflow uses the shared command in + the same way as `module/` +- build the deferred cross-repo NodeBB aggregation bot if combined public release + posts are still desired + ## Scope This contract governs release changelog generation across: diff --git a/CONFIGURATION_HARDENING_CONTRACT.md b/CONFIGURATION_HARDENING_CONTRACT.md index 2a61aa5..9fc675c 100644 --- a/CONFIGURATION_HARDENING_CONTRACT.md +++ b/CONFIGURATION_HARDENING_CONTRACT.md @@ -1,5 +1,27 @@ Create a long-term multi-repository hardening plan focused on making YAML configuration the authoritative source of truth. +## Status Snapshot + +Current state as of 2026-05-13: + +- This hardening effort is active and partially complete. +- YAML authority, strict config loading, and effective-config inspection are + implemented. +- Major repository behavior now flows through normalized config, including HAK + layout, topdata packaging, extract behavior, autogen manifests, scripts, wiki, + and music datasets. +- The remaining gaps are mostly about reducing legacy compatibility surface, + moving more tests to YAML fixtures, and deciding which residual toolkit + defaults stay intrinsic versus become explicit schema. + +Primary outstanding work: + +- reduce reliance on `nwn-tool.json` in tests and migration fixtures +- keep pruning repository-shaped defaults that still live in code when they + should be surfaced as schema-backed settings +- maintain docs/examples for `config effective` and the normalized key space as + the schema evolves + # Goal Reduce hardcoded behavior across all repositories and toolkit code. diff --git a/CONFIGURATION_REFACTOR_CONTRACT.md b/CONFIGURATION_REFACTOR_CONTRACT.md index 8dcebbf..69c63c2 100644 --- a/CONFIGURATION_REFACTOR_CONTRACT.md +++ b/CONFIGURATION_REFACTOR_CONTRACT.md @@ -1,5 +1,82 @@ Refactor toolkit configuration from JSON to YAML only. +## Status Snapshot + +Current state as of 2026-05-13: + +- This refactor is largely implemented. +- Root config discovery now prefers `nwn-tool.yaml`, then `nwn-tool.yml`, then + legacy `nwn-tool.json`. +- YAML is canonical in the consumer repos, and the toolkit exposes + `config validate|effective|inspect|explain|sources`. +- Legacy JSON compatibility still exists by design and is still used heavily in + test fixtures. + +Remaining work: + +- continue migrating representative tests from `nwn-tool.json` fixtures to YAML + while keeping only targeted legacy coverage +- decide when to deprecate or remove legacy shorthand such as `music.prefixes` + after consumer configs are fully dataset-based +- keep documentation/examples synchronized with the normalized YAML schema as it + expands + +## Active Scope + +This contract remains active and is one of the current priorities. + +The implementation gap is no longer "load YAML at all". The real gap is +configuration authority drift: + +- some repository behavior is still encoded as toolkit defaults when it should + become explicit schema +- some tests still model old JSON-era expectations +- some topdata behavior is still driven by explicit namespace/path checks rather + than generic dataset rules + +## Current Findings + +The most important remaining configuration work is topdata-related: + +- topdata dataset discovery already derives default output names when `output` is + absent, but validation still hard-fails for + `topdata/data/masterfeats/base.json` when `output` is missing +- `internal/topdata/topdata.go` dispatches validation through explicit path + checks for `masterfeats`, `feat`, parts overrides, and racialtypes registry +- topdata defaults in `internal/project` still mix generic toolkit defaults with + behavior that may need clearer schema ownership + +This means the config-refactor track now overlaps directly with topdata +hardening. The core question is no longer file format migration; it is whether a +behavior belongs in: + +- root YAML +- dataset-authored JSON +- or a documented intrinsic toolkit invariant + +## Deferred Scope + +Defer unrelated schema expansion for now: + +- model compilation config +- broader asset-maintenance policy schema +- release-announcement config + +## Immediate Plan + +1. Continue shrinking legacy `nwn-tool.json` coverage down to deliberate + compatibility tests. +2. Audit hardcoded topdata behaviors and classify each as: + - root YAML concern + - dataset JSON concern + - toolkit invariant +3. Remove outdated namespace-specific validation assumptions where generic + dataset logic already exists. +4. Keep generated artifacts in JSON, but stop using explicit path-based logic + when the same rule can be inferred from generic dataset shape. +5. Document every remaining intentional toolkit default in `config effective` + terms. + # Context Repositories currently use `config.json` for human-authored toolkit configuration. diff --git a/LOG_REFACTOR_CONTRACT.md b/LOG_REFACTOR_CONTRACT.md index 086e914..4770a09 100644 --- a/LOG_REFACTOR_CONTRACT.md +++ b/LOG_REFACTOR_CONTRACT.md @@ -1,5 +1,71 @@ You are improving CLI/build logs for a NWN asset toolchain. +## Status Snapshot + +Current state as of 2026-05-13: + +- This work is largely implemented for the current `build-haks` path. +- The app layer now exposes `--quiet`, `--verbose`, and `--debug`, and normal + mode prints compact music summaries instead of raw per-file mappings. +- Detailed mapping output is available behind verbose logging, and app tests + cover the compact versus detailed presentation split. + +Remaining work: + +- continue applying the same presentation discipline to other command families + where raw pipeline output still leaks through +- keep summary wording and phase naming aligned as new generated-asset stages + are added + +## Active Scope + +Logging remains an active priority, but no longer for `build-haks` alone. + +The next logging phase should target: + +- `build-topdata` +- `validate-topdata` +- `compare-topdata` +- `build-wiki` +- `deploy-wiki` +- release wrapper flows that currently interleave raw shell/tool output + +## Current Findings + +The asset-side log refactor proved the presentation model. The remaining gap is +uneven command coverage: + +- topdata and wiki commands do not yet have the same polished phase/summarized + output contract as `build-haks` +- release scripts still emit long shell-oriented logs that are useful for CI + diagnosis but not shaped into the same structured presentation model +- wrapper-side status lines and toolkit-side progress lines are not yet fully + harmonized + +## Deferred Scope + +Defer cosmetic-only logging work that does not improve operator clarity. + +For now, prioritize: + +- structured summaries +- deterministic phase labels +- stderr/stdout hygiene +- parity between topdata/wiki flows and the existing build-haks presentation + +## Immediate Plan + +1. Reuse the existing verbosity model for topdata and wiki command families. +2. Define concise normal-mode summaries for topdata build/validate/compare. +3. Add deploy/build wiki progress summaries that surface: + - local pages scanned + - create/update/skip/drift counts + - manifest path +4. Keep debug mode capable of exposing raw internal progress when diagnosis is + required. +5. Extend app-layer tests to cover topdata/wiki presentation once those + commands adopt the structured output contract. + Goal: Make the logs cleaner, prettier, less repetitive, and easier to understand, without hiding important diagnostics. Current issues: diff --git a/MODEL_COMPILATION_ON_BUILDTIME_CONTRACT.md b/MODEL_COMPILATION_ON_BUILDTIME_CONTRACT.md index d475d22..a720dd7 100644 --- a/MODEL_COMPILATION_ON_BUILDTIME_CONTRACT.md +++ b/MODEL_COMPILATION_ON_BUILDTIME_CONTRACT.md @@ -1,5 +1,22 @@ # Model Compilation Build-Time Contract +## Status Snapshot + +Current state as of 2026-05-13: + +- This contract is still outstanding. +- `build-haks` does not currently compile authored ASCII `.mdl` files into + binary build artifacts. +- Toolkit validation and extraction understand `.mdl`, but there is no + config-backed build-time model compiler pipeline in `internal/pipeline/`. + +Open work remains exactly in the areas this contract describes: + +- schema and validation for `models.compile.*` +- compiler/tool resolution and staging +- generated-asset overlay integration in HAK builds +- cache/reuse semantics and regression coverage + ## Scope Expand the toolkit so authored ASCII Neverwinter Nights `.mdl` assets can be compiled into binary `.mdl` build artifacts during `sow-toolkit build-haks` without changing the authored source tree. diff --git a/MUSIC_REFACTOR_CONTRACT.md b/MUSIC_REFACTOR_CONTRACT.md index 1d8228f..d6a3832 100644 --- a/MUSIC_REFACTOR_CONTRACT.md +++ b/MUSIC_REFACTOR_CONTRACT.md @@ -1,5 +1,24 @@ Refactor the music generation / editing / credits pipeline so it exists entirely inside the shared toolkit and is no longer hardcoded or repository-specific. +## Status Snapshot + +Current state as of 2026-05-13: + +- This refactor is largely implemented. +- Music logic now lives under `internal/music/` with toolkit-owned CLI entry + points and shared use from `build-haks`. +- Dataset-based config is supported and documented; the toolkit exposes + `music list-datasets|scan|build|credits|validate|manifest|normalize`. +- Incremental credits artifact handling and compact/verbose log presentation are + present in the current implementation. + +Remaining work: + +- finish retiring old `music.prefixes`-only assumptions in docs/tests after all + consumer configs are fully dataset-native +- continue separating any remaining music-specific presentation rules from + broader `build-haks` output policy as the build graph evolves + Goal: Turn the current Shadows Over Westgate-specific music pipeline into a reusable, repository-agnostic toolkit subsystem that can operate across multiple projects and datasets with configuration-driven behavior. # Requirements diff --git a/SCRIPT_WRAPPER_CONTRACT.md b/SCRIPT_WRAPPER_CONTRACT.md index 74cceeb..3d0d213 100644 --- a/SCRIPT_WRAPPER_CONTRACT.md +++ b/SCRIPT_WRAPPER_CONTRACT.md @@ -1,5 +1,75 @@ Refactor and harden all shell wrapper scripts across the module and assets repositories. +## Status Snapshot + +Current state as of 2026-05-13: + +- Argument passthrough is largely solved for the active root wrappers. +- `module/` and `assets/` now use repo-root launchers plus + `scripts/run-nwn-tool.*` passthrough shims. +- Wrapper regression coverage already exists in `module/tests/` and + `assets/tests/`. +- The previously stale JSON-config assumption shown in this contract is no + longer current; active repos use `nwn-tool.yaml`. + +## Active Scope + +This contract is still active, but the priority has changed. + +The old priority was: + +- make `$@` / `@args` forwarding work + +The current priority is: + +- remove wrapper-side business logic that still duplicates toolkit behavior +- keep wrappers thin across shell, PowerShell, and release-entrypoint paths +- align wrapper logs and error handling with the current logging contract +- finish wrapper/tooling parity before moving deeper into wiki deployment + +## Current Findings + +The main active wrapper gap is no longer argument forwarding. The bigger gaps are +wrapper thickness and duplicated orchestration: + +- `module/build-module.sh` still shells out to `config effective --json`, parses + config locally, probes script sources, resolves manifest paths, and performs + pre-build manifest refresh. +- `module/build-module.ps1` mirrors the same logic in a second implementation. +- `module/scripts/release-all.sh` and `assets/scripts/release-haks.sh` both + parse effective config themselves and still own substantial orchestration that + is not just "launch the toolkit". + +That means the wrapper contract should now optimize for: + +- a single source of truth for preflight behavior +- thinner root wrappers +- shared repo plumbing only where toolkit ownership is not appropriate +- no duplicated config decoding logic between `.sh` and `.ps1` + +## Deferred Scope + +Defer unrelated wrapper-adjacent work for now: + +- model-compilation wrapper concerns +- release-announcement publication scripting +- broader asset-processing script consolidation outside wrapper ownership + +## Immediate Plan + +1. Audit every wrapper that does more than `cd` plus `exec`. +2. Classify wrapper logic into: + - toolkit-owned behavior + - shared repo plumbing + - release-only orchestration +3. Move duplicated config reads and command decisions out of paired `.sh` and + `.ps1` wrappers where possible. +4. Keep root wrappers as thin launchers with transparent argument forwarding. +5. Expand regression coverage from passthrough correctness into: + - wrapper-side preflight behavior + - stderr/stdout separation for JSON-producing subcommands + - shell/PowerShell parity for any remaining unavoidable wrapper logic + # Problem The current `.sh` and `.ps1` wrapper scripts do not properly forward arbitrary CLI arguments to the underlying toolkit commands. diff --git a/TOPDATA_HARDENING_CONTRACT.md b/TOPDATA_HARDENING_CONTRACT.md new file mode 100644 index 0000000..40db2ab --- /dev/null +++ b/TOPDATA_HARDENING_CONTRACT.md @@ -0,0 +1,197 @@ +# Native Topdata Hardening Contract + +## Scope + +This contract governs the current topdata hardening and bugfixing phase across: + +- `toolkit/` as implementation owner +- `module/` as the primary authored topdata consumer + +This is an active-priority contract. + +## Current Priority + +Focus now on: + +1. removing outdated explicit namespace/path hardcoding +2. fixing validation/build mismatches +3. making native topdata behavior generic where the data model is already generic +4. improving diagnostics and operator-facing output for topdata commands + +Defer for now: + +- model compilation +- release patch-note automation +- unrelated asset-processing expansion +- non-topdata feature work that does not unblock wiki deployment + +Wiki deployment itself is a follow-on phase after this hardening pass, because +`build-wiki` and `deploy-wiki` depend on topdata outputs and metadata being +stable. + +## Problem Statement + +Native topdata has reached the point where most major functionality exists, but +too much behavior is still implemented as explicit namespace knowledge instead of +generic dataset rules. + +One concrete example already observed: + +- dataset discovery in `internal/topdata/native.go` already derives a default + `output` name when `base.json` omits it +- validation in `internal/topdata/topdata.go` still hard-fails specifically for + `topdata/data/masterfeats/base.json` if `output` is missing + +That mismatch proves the current methodology is stale: + +- the runtime/build side treats `masterfeats` like a generic dataset +- the validator still treats it like a special namespace with explicit required + metadata + +This same category of problem appears in several other places. + +## Findings From Current Audit + +### 1. Validation dispatch is path-special-cased + +`validateDataObject()` currently routes behavior through explicit path checks for: + +- `masterfeats/base.json` +- `masterfeats/lock.json` +- `feat/base.json` +- `feat/lock.json` +- `feat/generated/*` +- `parts/overrides/*` +- `racialtypes/registry/races/*` + +Some of these are legitimate dataset-shape differences. Others are stale +special-cases that should be generalized. + +### 2. `masterfeats` still carries outdated explicit validation rules + +`validateMasterfeatsBaseFile()` currently requires: + +- `output == masterfeats.2da` +- specific required columns +- `masterfeats:` row keys + +The key-prefix and column requirements may still be valid dataset invariants. +The `output` requirement is not clearly justified anymore when generic dataset +discovery already computes a default output name from location. + +### 3. Migration still seeds explicit dataset exceptions + +`feat_migrate.go` explicitly writes: + +- `output = "feat.2da"` +- `compare_reference = false` + +This may still be operationally correct, but it means migration continues to +materialize special-case control fields instead of relying on a clearer generic +contract for dataset reference comparison. + +### 4. Project-level topdata defaults are still mixed + +`internal/project` still owns several topdata defaults directly: + +- build directory derivation +- compiled 2DA subdirectory +- compiled TLK name +- wiki root/pages/state defaults +- wiki deploy manifest and edit summary defaults +- package HAK/TLK fallback names + +Some of these are valid toolkit defaults. Some should be reviewed as explicit +schema-backed config. + +### 5. Wiki generation and deploy depend on stable dataset semantics + +`build-wiki` and `deploy-wiki` are downstream consumers of native topdata state. +If dataset rules remain partly explicit and partly generic, wiki generation will +keep inheriting brittle assumptions. + +## Hardening Rules + +1. If a rule can be inferred generically from dataset shape, do not hardcode it + to a namespace path. +2. If a rule is truly dataset-specific, document why it is intrinsic rather than + configuration-driven. +3. Validation and build behavior must agree. Discovery defaults and validation + requirements cannot contradict each other. +4. Dataset-authored JSON should remain the source of truth for dataset-level + semantics; root YAML should not absorb dataset internals just to avoid code + cleanup. +5. Error messages must point to the actual invariant being violated, not an + outdated historical implementation detail. + +## Immediate Work Items + +### Work Item 1: Remove outdated `masterfeats.output` hard requirement + +Investigate and likely change `validateMasterfeatsBaseFile()` so: + +- missing `output` is accepted when generic dataset discovery can derive the same + output name deterministically +- explicit `output` remains allowed +- wrong explicit `output` still fails clearly if the dataset truly requires a + fixed emitted file name + +This should be test-covered. + +### Work Item 2: Audit path-based validator routing + +Classify every explicit route in `validateDataObject()` as one of: + +- generic canonical shape handling +- dataset-family behavior that should become generic +- deliberate intrinsic special-case with justification + +Reduce the explicit route set where possible. + +### Work Item 3: Define topdata invariant boundaries + +For each current explicit topdata behavior, decide whether it belongs in: + +- root YAML config +- dataset JSON +- toolkit invariant + +Do not move dataset-authored semantics into YAML unless the behavior truly varies +at repository level rather than dataset level. + +### Work Item 4: Revisit `compare_reference` ownership + +`feat.2da` is still excluded via `compare_reference = false`. Confirm whether: + +- this remains a temporary rollout flag +- or compare coverage needs a more explicit generic contract for generated-family + datasets + +### Work Item 5: Bring topdata logging up to current CLI standards + +Coordinate with `LOG_REFACTOR_CONTRACT.md` so topdata commands emit: + +- deterministic phase summaries +- concise validation/build/compare totals +- clear file/namespace diagnostics + +## Testing Requirements + +Add or update tests for: + +- `masterfeats/base.json` without explicit `output` when the derived output would + still be `masterfeats.2da` +- explicit wrong `output` for `masterfeats` +- validator/build parity for dataset output naming +- any reduction in path-special-case validation routing +- topdata command output summaries if presentation changes land in the same phase + +## Acceptance Criteria + +- topdata validation no longer rejects data solely because of stale explicit + namespace assumptions +- build and validation agree on dataset output-name behavior +- path-special-casing is reduced or explicitly justified +- remaining dataset-specific invariants are documented as such +- topdata hardening leaves wiki generation/deployment with a cleaner and more + stable contract diff --git a/WIKI_DEPLOYMENT_CONTRACT.md b/WIKI_DEPLOYMENT_CONTRACT.md new file mode 100644 index 0000000..8936ecf --- /dev/null +++ b/WIKI_DEPLOYMENT_CONTRACT.md @@ -0,0 +1,108 @@ +# Wiki Deployment Contract + +## Scope + +This contract governs the next focused phase after wrappers, config refactor, +logging, and topdata hardening: + +- `toolkit/` owns `build-wiki` and `deploy-wiki` +- `module/` owns the consuming release workflow and operational configuration + +This work is intentionally sequenced after topdata hardening because the wiki +pipeline consumes native topdata outputs and metadata. + +## Current State + +As of 2026-05-13: + +- `build-wiki` exists +- `deploy-wiki` exists +- deployment targets NodeBB core `/api/v3` +- local deploy manifests and NodeBB topic mapping exist +- release automation in `module/scripts/release-all.sh` already runs wiki build + and deploy steps after release publication +- toolkit tests already cover dry-run behavior, topic creation, and manifest + writes + +This means wiki deployment is no longer a greenfield feature. The next phase is +hardening, integration clarity, and operator experience. + +## Deferred Until Current Priority Work Completes + +Do not prioritize the following until wrappers/config/logging/topdata work is in +better shape: + +- new NodeBB release-announcement posting flows +- broad wiki feature expansion +- unrelated formatting changes to generated page content + +## Immediate Goals For The Wiki Phase + +1. align wiki build/deploy logs with the structured logging contract +2. tighten config ownership for wiki defaults and category mappings +3. reduce residual hardcoded behavior that belongs in normalized config +4. improve release-flow clarity around dry-run, drift, create, and force modes +5. verify that topdata hardening does not silently change generated page identity + +## Current Findings + +- `deploy-wiki` already reads managed namespaces from effective config when the + caller does not specify them explicitly +- deploy manifest naming and edit-summary defaults still come from toolkit + defaults in `internal/project/effective.go` +- release orchestration already guards publication order correctly, but the + operator-facing logs are still more shell-oriented than the newer structured + `build-haks` presentation +- wiki deployment should be treated as a first-class operational command family, + not just a release-script tail step + +## Immediate Plan + +### Phase 1: Logging and UX parity + +- add concise summaries for `build-wiki` and `deploy-wiki` +- surface counts for: + - pages collected + - creates + - updates + - skips + - drifted pages +- keep raw debug output available when diagnosis is needed + +### Phase 2: Config clarity + +- audit which wiki defaults should remain toolkit defaults versus explicit repo + config +- keep namespace/category ownership visible in effective config +- ensure deploy-manifest path and edit summary remain discoverable and easy to + override + +### Phase 3: Release-flow hardening + +- verify `module/scripts/release-all.sh` remains the authoritative release entry + point +- keep dry-run before live deploy +- ensure failure and drift messages are crisp and actionable +- make sure cache restore/save behavior is clearly documented and tested + +### Phase 4: Regression verification after topdata hardening + +- confirm topdata de-hardcoding does not break page IDs or manifest mapping +- confirm generated-page identity remains stable across unchanged inputs + +## Testing Requirements + +Add or expand tests for: + +- app-layer output for `build-wiki` and `deploy-wiki` +- release-flow sequencing assumptions where wiki deploy must happen only after + release assets are published +- stable page identity and manifest mapping across repeated runs +- config-driven namespace selection and manifest path overrides + +## Acceptance Criteria + +- wiki build/deploy logs match the newer structured CLI standard +- wiki defaults and overrides are clearly represented in effective config +- release orchestration remains deterministic and easy to reason about +- topdata hardening does not regress page identity or deploy manifest behavior diff --git a/internal/app/app.go b/internal/app/app.go index 3bb1fe5..a803b49 100644 --- a/internal/app/app.go +++ b/internal/app/app.go @@ -6,7 +6,9 @@ import ( "fmt" "io" "os" + "os/exec" "path/filepath" + "runtime" "sort" "strconv" "strings" @@ -294,6 +296,9 @@ func runBuild(ctx context) error { if err != nil { return err } + if err := runBuildModulePreflight(ctx, p, nil); err != nil { + return err + } result, err := pipeline.Build(p) if err != nil { @@ -321,12 +326,19 @@ func runBuildModule(ctx context) error { return err } - result, err := pipeline.BuildModuleWithProgress(p, func(message string) { + progress := func(message string) { if ctx.logLevel == logLevelQuiet { return } spin.linebreak() fmt.Fprintf(ctx.stdout, "[build-module] %s\n", message) + } + if err := runBuildModulePreflight(ctx, p, progress); err != nil { + return err + } + + result, err := pipeline.BuildModuleWithProgress(p, func(message string) { + progress(message) }) if err != nil { return err @@ -338,6 +350,222 @@ func runBuildModule(ctx context) error { return nil } +func runBuildModulePreflight(ctx context, p *project.Project, progress func(string)) error { + if progress == nil { + progress = func(string) {} + } + if err := refreshBuildModuleManifest(ctx, p, progress); err != nil { + return err + } + if err := prepareBuildModuleCompiler(ctx, p, progress); err != nil { + return err + } + return nil +} + +func refreshBuildModuleManifest(ctx context, p *project.Project, progress func(string)) error { + manifestPath := p.HAKManifestPath() + if override := strings.TrimSpace(os.Getenv("SOW_HAK_MANIFEST_PATH")); override != "" { + manifestPath = override + if !filepath.IsAbs(manifestPath) { + manifestPath = filepath.Join(p.Root, manifestPath) + } + } + + if envEnabled("SOW_MODULE_SKIP_HAK_MANIFEST_REFRESH") { + if _, err := os.Stat(manifestPath); err != nil { + return fmt.Errorf("SOW_MODULE_SKIP_HAK_MANIFEST_REFRESH is set, but %s does not exist", manifestPath) + } + progress(fmt.Sprintf("Using existing hak manifest at %s; skipping published sow-assets refresh.", relPathFromRoot(p.Root, manifestPath))) + return nil + } + + progress("Refreshing hak list from the latest published sow-assets manifest...") + if err := runProjectScript(ctx, p, []string{"scripts", "fetch-hak-manifest"}, manifestPath); err != nil { + return err + } + if _, err := pipeline.ApplyHAKManifest(p, manifestPath); err != nil { + return err + } + return nil +} + +func prepareBuildModuleCompiler(ctx context, p *project.Project, progress func(string)) error { + if envEnabled("SOW_MODULE_SKIP_NSS_COMPILATION") { + progress("SOW_MODULE_SKIP_NSS_COMPILATION is set; skipping NWScript compiler resolution and NSS compilation for this run.") + return nil + } + if !projectHasScriptSources(p) { + return nil + } + + compilerPathEnv := p.ScriptCompilerPathEnv() + explicitCompiler := strings.TrimSpace(os.Getenv(compilerPathEnv)) + if explicitCompiler != "" { + if scriptCompilerRunnable(explicitCompiler) { + return nil + } + if fileExists(explicitCompiler) { + return disableBuildModuleCompiler(compilerPathEnv, progress, fmt.Sprintf("Configured NWScript compiler is present but cannot start: %s", explicitCompiler)) + } + } + + if _, ok := findUsableBuildModuleCompiler(p); ok { + return nil + } + + installScript := projectScriptPath(p.Root, "scripts", "install-script-compiler") + if !fileExists(installScript) { + return disableBuildModuleCompiler(compilerPathEnv, progress, "NWScript compiler not found locally and automatic installer script is unavailable.") + } + + progress("NWScript compiler not found locally. Installing it now...") + if err := runProjectScript(ctx, p, []string{"scripts", "install-script-compiler"}); err != nil { + return disableBuildModuleCompiler(compilerPathEnv, progress, "Automatic NWScript compiler install failed.") + } + if _, ok := findUsableBuildModuleCompiler(p); ok { + return nil + } + + return disableBuildModuleCompiler(compilerPathEnv, progress, "Installed NWScript compiler but it still cannot start.") +} + +func disableBuildModuleCompiler(compilerPathEnv string, progress func(string), reason string) error { + if err := os.Setenv("SOW_MODULE_SKIP_NSS_COMPILATION", "1"); err != nil { + return fmt.Errorf("set SOW_MODULE_SKIP_NSS_COMPILATION: %w", err) + } + progress("Skipping NWScript compilation for this build. " + reason) + if compilerPathEnv != "" { + _ = os.Unsetenv(compilerPathEnv) + } + return nil +} + +func projectHasScriptSources(p *project.Project) bool { + for _, rel := range p.Inventory.ScriptFiles { + if strings.EqualFold(filepath.Ext(rel), ".nss") { + return true + } + } + return false +} + +func findUsableBuildModuleCompiler(p *project.Project) (string, bool) { + candidates := make([]string, 0, 2+len(p.ScriptCompilerSearchPaths())) + if configured := p.ScriptCompilerPath(); configured != "" { + candidates = append(candidates, configured) + } + if configured := strings.TrimSpace(os.Getenv(p.ScriptCompilerPathEnv())); configured != "" { + candidates = append(candidates, configured) + } + candidates = append(candidates, p.ScriptCompilerSearchPaths()...) + + for _, candidate := range candidates { + if compiler, ok := resolveRunnableCompilerCandidate(candidate); ok { + return compiler, true + } + } + return "", false +} + +func resolveRunnableCompilerCandidate(candidate string) (string, bool) { + if candidate == "" { + return "", false + } + if binary, ok := strings.CutPrefix(candidate, "PATH:"); ok { + compiler, err := exec.LookPath(binary) + if err != nil { + return "", false + } + if scriptCompilerRunnable(compiler) { + return compiler, true + } + return "", false + } + if !fileExists(candidate) { + return "", false + } + if scriptCompilerRunnable(candidate) { + return candidate, true + } + return "", false +} + +func scriptCompilerRunnable(path string) bool { + if !fileExists(path) { + return false + } + cmd := exec.Command(path, "--help") + cmd.Stdout = io.Discard + cmd.Stderr = io.Discard + return cmd.Run() == nil +} + +func runProjectScript(ctx context, p *project.Project, scriptParts []string, args ...string) error { + if len(scriptParts) == 0 { + return errors.New("project script path is required") + } + scriptPath := projectScriptPath(p.Root, scriptParts...) + if !fileExists(scriptPath) { + return fmt.Errorf("required project script is missing: %s", scriptPath) + } + + var cmd *exec.Cmd + switch filepath.Ext(scriptPath) { + case ".ps1": + powershell := "powershell" + if runtime.GOOS != "windows" { + powershell = "pwsh" + } + commandArgs := []string{"-NoProfile", "-File", scriptPath} + commandArgs = append(commandArgs, args...) + cmd = exec.Command(powershell, commandArgs...) + default: + cmd = exec.Command(scriptPath, args...) + } + cmd.Dir = p.Root + cmd.Stdout = ctx.stderr + cmd.Stderr = ctx.stderr + return cmd.Run() +} + +func projectScriptPath(root string, pathParts ...string) string { + parts := append([]string{root}, pathParts...) + base := filepath.Join(parts...) + if runtime.GOOS == "windows" { + if strings.HasSuffix(base, ".ps1") || strings.HasSuffix(base, ".sh") { + return base + } + return base + ".ps1" + } + if strings.HasSuffix(base, ".ps1") || strings.HasSuffix(base, ".sh") { + return base + } + return base + ".sh" +} + +func envEnabled(name string) bool { + switch strings.ToLower(strings.TrimSpace(os.Getenv(name))) { + case "1", "true", "yes", "on": + return true + default: + return false + } +} + +func fileExists(path string) bool { + info, err := os.Stat(path) + return err == nil && !info.IsDir() +} + +func relPathFromRoot(root, path string) string { + rel, err := filepath.Rel(root, path) + if err != nil { + return filepath.ToSlash(path) + } + return filepath.ToSlash(rel) +} + type buildHAKOptions struct { filteredHAKs []string filteredArchives []string @@ -530,6 +758,161 @@ func (c *buildHAKConsole) relPath(path string) string { return filepath.ToSlash(rel) } +type topdataConsole struct { + stdout io.Writer + projectRoot string + projectName string + commandName string + commandLabel string + level logLevel + spinnerEnabled bool +} + +func newTopdataConsole(ctx context, p *project.Project, commandName string) *topdataConsole { + return &topdataConsole{ + stdout: ctx.stdout, + projectRoot: p.Root, + projectName: p.Config.Module.Name, + commandName: commandName, + commandLabel: topdataCommandLabel(commandName), + level: ctx.logLevel, + spinnerEnabled: isInteractiveTTY(ctx.stderr) && strings.TrimSpace(os.Getenv("CI")) == "" && ctx.logLevel == logLevelNormal, + } +} + +func topdataCommandLabel(commandName string) string { + switch commandName { + case "validate-topdata": + return "Validate Topdata" + case "build-topdata": + return "Build Topdata" + case "build-top-package": + return "Build Top Package" + case "compare-topdata": + return "Compare Topdata" + case "build-wiki": + return "Build Wiki" + case "deploy-wiki": + return "Deploy Wiki" + default: + return commandName + } +} + +func (c *topdataConsole) progress(message string) { + if phase := c.phaseLabel(message); phase != "" { + spin.update(c.commandLabel + ": " + phase) + } + if c.level != logLevelDebug { + return + } + spin.linebreak() + fmt.Fprintf(c.stdout, "[debug] %s\n", message) +} + +func (c *topdataConsole) phaseLabel(message string) string { + switch { + case strings.HasPrefix(message, "Building native topdata outputs"): + return "building native outputs" + case strings.HasPrefix(message, "Topdata outputs are current"): + return "checking cached outputs" + case strings.HasPrefix(message, "Compiled topdata outputs are current"): + return "refreshing top package" + case strings.HasPrefix(message, "Packaging compiled topdata resources into "): + return "packaging HAK" + case strings.HasPrefix(message, "Preparing compiled TLK release output "): + return "packaging TLK" + case strings.HasPrefix(message, "Building native wiki pages"): + return "building wiki pages" + case strings.HasPrefix(message, "NodeBB wiki plan: "): + return "planning wiki deploy" + case strings.HasPrefix(message, "Collecting "): + return "collecting pages" + default: + return "" + } +} + +func (c *topdataConsole) emitValidationResult(report topdata.ValidationReport, topdataRoot string) { + spin.linebreak() + fmt.Fprintln(c.stdout, "Validate Topdata ----------") + fmt.Fprintf(c.stdout, "project: %s\n", c.projectName) + fmt.Fprintf(c.stdout, "topdata root: %s\n", c.relPath(topdataRoot)) + fmt.Fprintf(c.stdout, "topdata files: %d\n", report.Files) + fmt.Fprintf(c.stdout, "data files: %d\n", report.DataFiles) + fmt.Fprintf(c.stdout, "tlk files: %d\n", report.TLKFiles) + if warnings := report.WarningCount(); warnings > 0 { + fmt.Fprintf(c.stdout, "warnings: %d\n", warnings) + } + fmt.Fprintln(c.stdout, "topdata validation: ok") +} + +func (c *topdataConsole) emitPackageResult(result topdata.PackageResult) { + spin.linebreak() + fmt.Fprintf(c.stdout, "%s ----------\n", c.commandLabel) + fmt.Fprintf(c.stdout, "project: %s\n", c.projectName) + fmt.Fprintf(c.stdout, "mode: %s\n", result.Mode) + fmt.Fprintf(c.stdout, "topdata 2da output: %s\n", c.relPath(result.Output2DADir)) + fmt.Fprintf(c.stdout, "topdata tlk output: %s\n", c.relPath(result.OutputTLKDir)) + fmt.Fprintf(c.stdout, "top package hak: %s\n", c.relPath(result.OutputHAKPath)) + fmt.Fprintf(c.stdout, "top package tlk: %s\n", c.relPath(result.OutputTLKPath)) + fmt.Fprintf(c.stdout, "2da files: %d\n", result.Files2DA) + fmt.Fprintf(c.stdout, "tlk files: %d\n", result.FilesTLK) + fmt.Fprintf(c.stdout, "top package resources: %d\n", result.HAKResources) + fmt.Fprintf(c.stdout, "top package assets: %d\n", result.AssetFiles) + if result.WikiOutputDir != "" { + fmt.Fprintf(c.stdout, "wiki output: %s\n", c.relPath(result.WikiOutputDir)) + fmt.Fprintf(c.stdout, "wiki pages: %d\n", result.WikiPages) + fmt.Fprintf(c.stdout, "wiki status: %s\n", result.WikiStatus) + } +} + +func (c *topdataConsole) emitCompareResult(result topdata.CompareResult) { + spin.linebreak() + fmt.Fprintln(c.stdout, "Compare Topdata ----------") + fmt.Fprintf(c.stdout, "project: %s\n", c.projectName) + fmt.Fprintf(c.stdout, "mode: %s\n", result.Mode) + fmt.Fprintf(c.stdout, "checked 2da files: %d\n", result.Compared2DA) + fmt.Fprintf(c.stdout, "checked tlk files: %d\n", result.ComparedTLK) + fmt.Fprintf(c.stdout, "native-pass: %d\n", result.NativePass) + fmt.Fprintf(c.stdout, "not-yet-canonical: %d\n", result.NotYetCanonical) + fmt.Fprintf(c.stdout, "native-mismatch: %d\n", result.NativeMismatch) + fmt.Fprintln(c.stdout, "topdata compare: ok") +} + +func (c *topdataConsole) emitWikiBuildResult(outputDir string, pageCount int, status string) { + spin.linebreak() + fmt.Fprintln(c.stdout, "Build Wiki ----------") + fmt.Fprintf(c.stdout, "project: %s\n", c.projectName) + fmt.Fprintf(c.stdout, "wiki output: %s\n", c.relPath(outputDir)) + fmt.Fprintf(c.stdout, "wiki pages: %d\n", pageCount) + fmt.Fprintf(c.stdout, "wiki status: %s\n", status) +} + +func (c *topdataConsole) emitWikiDeployResult(localPages, created, updated, skipped, stale, drifted int, manifest string) { + spin.linebreak() + fmt.Fprintln(c.stdout, "Deploy Wiki ----------") + fmt.Fprintf(c.stdout, "project: %s\n", c.projectName) + fmt.Fprintf(c.stdout, "local pages: %d\n", localPages) + fmt.Fprintf(c.stdout, "created: %d\n", created) + fmt.Fprintf(c.stdout, "updated: %d\n", updated) + fmt.Fprintf(c.stdout, "skipped: %d\n", skipped) + fmt.Fprintf(c.stdout, "stale: %d\n", stale) + fmt.Fprintf(c.stdout, "drifted: %d\n", drifted) + fmt.Fprintf(c.stdout, "manifest: %s\n", c.relPath(manifest)) +} + +func (c *topdataConsole) relPath(path string) string { + if path == "" { + return path + } + rel, err := filepath.Rel(c.projectRoot, path) + if err != nil { + return filepath.ToSlash(path) + } + return filepath.ToSlash(rel) +} + func formatCount(value int) string { raw := strconv.Itoa(value) if value < 1000 && value > -1000 { @@ -1145,21 +1528,14 @@ func runValidateTopData(ctx context) error { return err } + console := newTopdataConsole(ctx, p, "validate-topdata") report := topdata.ValidateProject(p) emitTopdataValidationReport(ctx.stderr, report) if report.HasErrors() { return fmt.Errorf("topdata validation failed with %d error(s)", report.ErrorCount()) } - fmt.Fprintf(ctx.stdout, "project: %s\n", p.Config.Module.Name) - fmt.Fprintf(ctx.stdout, "topdata root: %s\n", p.TopDataSourceDir()) - fmt.Fprintf(ctx.stdout, "topdata files: %d\n", report.Files) - fmt.Fprintf(ctx.stdout, "data files: %d\n", report.DataFiles) - fmt.Fprintf(ctx.stdout, "tlk files: %d\n", report.TLKFiles) - if warnings := report.WarningCount(); warnings > 0 { - fmt.Fprintf(ctx.stdout, "warnings: %d\n", warnings) - } - fmt.Fprintf(ctx.stdout, "topdata validation: ok\n") + console.emitValidationResult(report, p.TopDataSourceDir()) return nil } @@ -1174,31 +1550,16 @@ func runBuildTopData(ctx context) error { return err } - result, err := topdata.BuildAndPackageWithOptions(p, opts, func(message string) { - if ctx.logLevel == logLevelQuiet { - return - } - fmt.Fprintf(ctx.stdout, "[build-topdata] %s\n", message) - }) + console := newTopdataConsole(ctx, p, "build-topdata") + spin.configure(ctx.stderr, console.spinnerEnabled) + spin.start("Build Topdata: starting") + defer spin.stop() + result, err := topdata.BuildAndPackageWithOptions(p, opts, console.progress) if err != nil { return err } - fmt.Fprintf(ctx.stdout, "project: %s\n", p.Config.Module.Name) - fmt.Fprintf(ctx.stdout, "mode: %s\n", result.Mode) - fmt.Fprintf(ctx.stdout, "topdata 2da output: %s\n", result.Output2DADir) - fmt.Fprintf(ctx.stdout, "topdata tlk output: %s\n", result.OutputTLKDir) - fmt.Fprintf(ctx.stdout, "top package hak: %s\n", result.OutputHAKPath) - fmt.Fprintf(ctx.stdout, "top package tlk: %s\n", result.OutputTLKPath) - fmt.Fprintf(ctx.stdout, "2da files: %d\n", result.Files2DA) - fmt.Fprintf(ctx.stdout, "tlk files: %d\n", result.FilesTLK) - fmt.Fprintf(ctx.stdout, "top package resources: %d\n", result.HAKResources) - fmt.Fprintf(ctx.stdout, "top package assets: %d\n", result.AssetFiles) - if result.WikiOutputDir != "" { - fmt.Fprintf(ctx.stdout, "wiki output: %s\n", result.WikiOutputDir) - fmt.Fprintf(ctx.stdout, "wiki pages: %d\n", result.WikiPages) - fmt.Fprintf(ctx.stdout, "wiki status: %s\n", result.WikiStatus) - } + console.emitPackageResult(result) return nil } @@ -1217,31 +1578,16 @@ func runBuildTopPackage(ctx context) error { return fmt.Errorf("--wiki is not supported with build-top-package; use build-topdata when wiki generation is required") } - result, err := topdata.BuildPackage(p, func(message string) { - if ctx.logLevel == logLevelQuiet { - return - } - fmt.Fprintf(ctx.stdout, "[build-top-package] %s\n", message) - }) + console := newTopdataConsole(ctx, p, "build-top-package") + spin.configure(ctx.stderr, console.spinnerEnabled) + spin.start("Build Top Package: starting") + defer spin.stop() + result, err := topdata.BuildPackage(p, console.progress) if err != nil { return err } - fmt.Fprintf(ctx.stdout, "project: %s\n", p.Config.Module.Name) - fmt.Fprintf(ctx.stdout, "mode: %s\n", result.Mode) - fmt.Fprintf(ctx.stdout, "topdata 2da output: %s\n", result.Output2DADir) - fmt.Fprintf(ctx.stdout, "topdata tlk output: %s\n", result.OutputTLKDir) - fmt.Fprintf(ctx.stdout, "top package hak: %s\n", result.OutputHAKPath) - fmt.Fprintf(ctx.stdout, "top package tlk: %s\n", result.OutputTLKPath) - fmt.Fprintf(ctx.stdout, "2da files: %d\n", result.Files2DA) - fmt.Fprintf(ctx.stdout, "tlk files: %d\n", result.FilesTLK) - fmt.Fprintf(ctx.stdout, "top package resources: %d\n", result.HAKResources) - fmt.Fprintf(ctx.stdout, "top package assets: %d\n", result.AssetFiles) - if result.WikiOutputDir != "" { - fmt.Fprintf(ctx.stdout, "wiki output: %s\n", result.WikiOutputDir) - fmt.Fprintf(ctx.stdout, "wiki pages: %d\n", result.WikiPages) - fmt.Fprintf(ctx.stdout, "wiki status: %s\n", result.WikiStatus) - } + console.emitPackageResult(result) return nil } @@ -1268,24 +1614,16 @@ func runCompareTopData(ctx context) error { return err } - result, err := topdata.Compare(p, func(message string) { - if ctx.logLevel == logLevelQuiet { - return - } - fmt.Fprintf(ctx.stdout, "[compare-topdata] %s\n", message) - }) + console := newTopdataConsole(ctx, p, "compare-topdata") + spin.configure(ctx.stderr, console.spinnerEnabled) + spin.start("Compare Topdata: starting") + defer spin.stop() + result, err := topdata.Compare(p, console.progress) if err != nil { return err } - fmt.Fprintf(ctx.stdout, "project: %s\n", p.Config.Module.Name) - fmt.Fprintf(ctx.stdout, "mode: %s\n", result.Mode) - fmt.Fprintf(ctx.stdout, "checked 2da files: %d\n", result.Compared2DA) - fmt.Fprintf(ctx.stdout, "checked tlk files: %d\n", result.ComparedTLK) - fmt.Fprintf(ctx.stdout, "native-pass: %d\n", result.NativePass) - fmt.Fprintf(ctx.stdout, "not-yet-canonical: %d\n", result.NotYetCanonical) - fmt.Fprintf(ctx.stdout, "native-mismatch: %d\n", result.NativeMismatch) - fmt.Fprintf(ctx.stdout, "topdata compare: ok\n") + console.emitCompareResult(result) return nil } @@ -1307,20 +1645,16 @@ func runBuildWiki(ctx context) error { return err } - result, err := topdata.BuildWikiWithOptions(p, opts, func(message string) { - if ctx.logLevel == logLevelQuiet { - return - } - fmt.Fprintf(ctx.stdout, "[build-wiki] %s\n", message) - }) + console := newTopdataConsole(ctx, p, "build-wiki") + spin.configure(ctx.stderr, console.spinnerEnabled) + spin.start("Build Wiki: starting") + defer spin.stop() + result, err := topdata.BuildWikiWithOptions(p, opts, console.progress) if err != nil { return err } - fmt.Fprintf(ctx.stdout, "project: %s\n", p.Config.Module.Name) - fmt.Fprintf(ctx.stdout, "wiki output: %s\n", result.OutputDir) - fmt.Fprintf(ctx.stdout, "wiki pages: %d\n", result.PageCount) - fmt.Fprintf(ctx.stdout, "wiki status: %s\n", result.Status) + console.emitWikiBuildResult(result.OutputDir, result.PageCount, result.Status) return nil } @@ -1350,24 +1684,16 @@ func runDeployWiki(ctx context) error { return err } - result, err := topdata.DeployWikiWithOptions(p, opts, func(message string) { - if ctx.logLevel == logLevelQuiet { - return - } - fmt.Fprintf(ctx.stdout, "[deploy-wiki] %s\n", message) - }) + console := newTopdataConsole(ctx, p, "deploy-wiki") + spin.configure(ctx.stderr, console.spinnerEnabled) + spin.start("Deploy Wiki: starting") + defer spin.stop() + result, err := topdata.DeployWikiWithOptions(p, opts, console.progress) if err != nil { return err } - fmt.Fprintf(ctx.stdout, "project: %s\n", p.Config.Module.Name) - fmt.Fprintf(ctx.stdout, "local pages: %d\n", result.LocalPages) - fmt.Fprintf(ctx.stdout, "created: %d\n", result.Created) - fmt.Fprintf(ctx.stdout, "updated: %d\n", result.Updated) - fmt.Fprintf(ctx.stdout, "skipped: %d\n", result.Skipped) - fmt.Fprintf(ctx.stdout, "stale: %d\n", result.Stale) - fmt.Fprintf(ctx.stdout, "drifted: %d\n", result.Drifted) - fmt.Fprintf(ctx.stdout, "manifest: %s\n", result.Manifest) + console.emitWikiDeployResult(result.LocalPages, result.Created, result.Updated, result.Skipped, result.Stale, result.Drifted, result.Manifest) return nil } diff --git a/internal/app/app_test.go b/internal/app/app_test.go index c25caad..18d1e77 100644 --- a/internal/app/app_test.go +++ b/internal/app/app_test.go @@ -16,18 +16,16 @@ func TestRunBuildTopPackageUsesCachedCompiledOutputs(t *testing.T) { mkdirAll(t, filepath.Join(root, "topdata", "assets", "gui")) mkdirAll(t, filepath.Join(root, "topdata", "data", "repadjust")) - writeFile(t, filepath.Join(root, "nwn-tool.json"), `{ - "module": { - "name": "Test Module", - "resref": "test_module" - }, - "topdata": { - "source": "topdata", - "build": ".cache", - "package_hak": "sow_top.hak", - "package_tlk": "sow_tlk.tlk" - } -}`+"\n") + writeFile(t, filepath.Join(root, "nwn-tool.yaml"), ` +module: + name: Test Module + resref: test_module +topdata: + source: topdata + build: .cache + package_hak: sow_top.hak + package_tlk: sow_tlk.tlk +`) writeFile(t, filepath.Join(root, ".cache", "2da", "repadjust.2da"), "2DA V2.0\n\n Label\n0 TEST_LABEL\n") writeFile(t, filepath.Join(root, "build", "sow_tlk.tlk"), "compiled tlk") writeFile(t, filepath.Join(root, "topdata", "assets", "gui", "testicon.png"), "icon-data") @@ -51,8 +49,12 @@ func TestRunBuildTopPackageUsesCachedCompiledOutputs(t *testing.T) { if err := runBuildTopPackage(ctx); err != nil { t.Fatalf("runBuildTopPackage failed: %v", err) } - if !strings.Contains(stdout.String(), "top package hak:") { - t.Fatalf("expected build-top-package output, got %q", stdout.String()) + output := stdout.String() + if !strings.Contains(output, "top package hak: build/sow_top.hak") { + t.Fatalf("expected build-top-package output, got %q", output) + } + if strings.Contains(output, "[build-top-package]") { + t.Fatalf("did not expect raw progress lines in normal output, got %q", output) } if _, err := os.Stat(filepath.Join(root, "build", "sow_top.hak")); err != nil { t.Fatalf("expected packaged hak output: %v", err) @@ -65,26 +67,22 @@ func TestRunBuildHAKsEmitsCompactSummary(t *testing.T) { mkdirAll(t, filepath.Join(root, "build")) mkdirAll(t, filepath.Join(root, "src")) - writeFile(t, filepath.Join(root, "nwn-tool.json"), `{ - "module": { - "name": "Test Module", - "resref": "testmod" - }, - "paths": { - "source": "src", - "assets": "assets", - "build": "build" - }, - "haks": [ - { - "name": "envi", - "priority": 1, - "max_bytes": 1048576, - "split": false, - "include": ["envi/**"] - } - ] -}`+"\n") + writeFile(t, filepath.Join(root, "nwn-tool.yaml"), ` +module: + name: Test Module + resref: testmod +paths: + source: src + assets: assets + build: build +haks: + - name: envi + priority: 1 + max_bytes: 1048576 + split: false + include: + - envi/** +`) writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "AleandAnecdotes.mp3"), "source-mp3") writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "CREDITS.md"), "# NWN Music Pack Credits\n\n| Artist | Title | Output File | Original File | Album | Date | License / Copyright | Notes |\n|---|---|---|---|---|---|---|---|\n| Darren Curtis | Ale and Anecdotes | `mus_wg_andnc.bmu` | `AleandAnecdotes.mp3` | Darren's Commercially Free Music | | Darren Curtis Music | |\n") @@ -136,26 +134,22 @@ func TestRunBuildHAKsVerboseListsMappings(t *testing.T) { mkdirAll(t, filepath.Join(root, "build")) mkdirAll(t, filepath.Join(root, "src")) - writeFile(t, filepath.Join(root, "nwn-tool.json"), `{ - "module": { - "name": "Test Module", - "resref": "testmod" - }, - "paths": { - "source": "src", - "assets": "assets", - "build": "build" - }, - "haks": [ - { - "name": "envi", - "priority": 1, - "max_bytes": 1048576, - "split": false, - "include": ["envi/**"] - } - ] -}`+"\n") + writeFile(t, filepath.Join(root, "nwn-tool.yaml"), ` +module: + name: Test Module + resref: testmod +paths: + source: src + assets: assets + build: build +haks: + - name: envi + priority: 1 + max_bytes: 1048576 + split: false + include: + - envi/** +`) writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "AleandAnecdotes.mp3"), "source-mp3") writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "CREDITS.md"), "# NWN Music Pack Credits\n\n| Artist | Title | Output File | Original File | Album | Date | License / Copyright | Notes |\n|---|---|---|---|---|---|---|---|\n| Darren Curtis | Ale and Anecdotes | `mus_wg_andnc.bmu` | `AleandAnecdotes.mp3` | Darren's Commercially Free Music | | Darren Curtis Music | |\n") @@ -198,6 +192,204 @@ func TestRunBuildHAKsVerboseListsMappings(t *testing.T) { } } +func TestTopdataConsoleSuppressesProgressInNormalMode(t *testing.T) { + var stdout bytes.Buffer + console := &topdataConsole{ + stdout: &stdout, + projectRoot: "/workspace/project", + projectName: "Test Module", + commandName: "build-topdata", + commandLabel: "Build Topdata", + level: logLevelNormal, + } + + console.progress("Packaging compiled topdata resources into sow_top.hak...") + + if stdout.String() != "" { + t.Fatalf("expected no normal-mode progress output, got %q", stdout.String()) + } +} + +func TestTopdataConsoleDebugProgressAndRelativePaths(t *testing.T) { + var stdout bytes.Buffer + console := &topdataConsole{ + stdout: &stdout, + projectRoot: "/workspace/project", + projectName: "Test Module", + commandName: "deploy-wiki", + commandLabel: "Deploy Wiki", + level: logLevelDebug, + } + + console.progress("NodeBB wiki plan: create 1, update 2, skip 3, drift 0") + console.emitWikiDeployResult(10, 1, 2, 3, 4, 0, "/workspace/project/build/wiki/deploy-manifest.json") + + output := stdout.String() + if !strings.Contains(output, "[debug] NodeBB wiki plan: create 1, update 2, skip 3, drift 0") { + t.Fatalf("expected debug progress line, got %q", output) + } + if !strings.Contains(output, "manifest: build/wiki/deploy-manifest.json") { + t.Fatalf("expected relative deploy manifest path, got %q", output) + } +} + +func TestRunBuildModuleToolkitPreflightRefreshesManifestAndSkipsBrokenCompiler(t *testing.T) { + root := t.TempDir() + mkdirAll(t, filepath.Join(root, "src", "module")) + mkdirAll(t, filepath.Join(root, "src", "placeables")) + mkdirAll(t, filepath.Join(root, "src", "scripts")) + mkdirAll(t, filepath.Join(root, "build")) + mkdirAll(t, filepath.Join(root, "scripts")) + + writeFile(t, filepath.Join(root, "nwn-tool.yaml"), ` +module: + name: Test Module + resref: testmod +paths: + source: src + build: build +scripts: + compiler: + search: + - tools/script-compiler/nwn_script_comp +`) + writeFile(t, filepath.Join(root, "src", "module", "module.ifo.json"), `{ + "file_type": "IFO ", + "file_version": "V3.2", + "root": { + "struct_type": 0, + "fields": [ + {"label": "Mod_Name", "type": "CExoString", "value": "Test Module"} + ] + } +} +`) + writeFile(t, filepath.Join(root, "src", "placeables", "thing.utp.json"), `{ + "file_type": "UTP ", + "file_version": "V3.2", + "root": { + "struct_type": 0, + "fields": [ + {"label": "OnUsed", "type": "ResRef", "value": "use_thing"} + ] + } +} +`) + writeFile(t, filepath.Join(root, "src", "scripts", "use_thing.nss"), "void main() {}\n") + writeFile(t, filepath.Join(root, "scripts", "fetch-hak-manifest.sh"), "#!/usr/bin/env sh\nset -eu\nmkdir -p \"$(dirname \"$1\")\"\nprintf '{\"module_haks\":[\"sow_core\"],\"haks\":[]}\n' >\"$1\"\n") + if err := os.Chmod(filepath.Join(root, "scripts", "fetch-hak-manifest.sh"), 0o755); err != nil { + t.Fatalf("chmod fetch-hak-manifest.sh: %v", err) + } + + badCompiler := filepath.Join(root, "bad-compiler.sh") + writeFile(t, badCompiler, "#!/usr/bin/env sh\nexit 1\n") + if err := os.Chmod(badCompiler, 0o755); err != nil { + t.Fatalf("chmod bad compiler: %v", err) + } + t.Setenv("SOW_NWN_SCRIPT_COMPILER", badCompiler) + t.Setenv("SOW_MODULE_SKIP_NSS_COMPILATION", "") + + var stdout bytes.Buffer + var stderr bytes.Buffer + ctx := context{ + stdout: &stdout, + stderr: &stderr, + cwd: root, + args: []string{"build-module"}, + } + + if err := runBuildModule(ctx); err != nil { + t.Fatalf("runBuildModule failed: %v", err) + } + + moduleSource := mustReadFile(t, filepath.Join(root, "src", "module", "module.ifo.json")) + if !strings.Contains(moduleSource, "\"Mod_HakList\"") || !strings.Contains(moduleSource, "\"sow_core\"") { + t.Fatalf("expected manifest-applied module source, got %q", moduleSource) + } + if _, err := os.Stat(filepath.Join(root, ".cache", "ncs", "use_thing.ncs")); !os.IsNotExist(err) { + t.Fatalf("expected broken explicit compiler to skip compilation, got err=%v", err) + } + if !strings.Contains(stdout.String(), "Skipping NWScript compilation for this build.") { + t.Fatalf("expected skip-compilation progress in output, got %q", stdout.String()) + } +} + +func TestRunBuildModuleToolkitPreflightInstallsCompilerWhenNeeded(t *testing.T) { + root := t.TempDir() + mkdirAll(t, filepath.Join(root, "src", "module")) + mkdirAll(t, filepath.Join(root, "src", "placeables")) + mkdirAll(t, filepath.Join(root, "src", "scripts")) + mkdirAll(t, filepath.Join(root, "build")) + mkdirAll(t, filepath.Join(root, "scripts")) + + writeFile(t, filepath.Join(root, "nwn-tool.yaml"), ` +module: + name: Test Module + resref: testmod +paths: + source: src + build: build +scripts: + compiler: + search: + - tools/script-compiler/nwn_script_comp +`) + writeFile(t, filepath.Join(root, "src", "module", "module.ifo.json"), `{ + "file_type": "IFO ", + "file_version": "V3.2", + "root": { + "struct_type": 0, + "fields": [ + {"label": "Mod_Name", "type": "CExoString", "value": "Test Module"} + ] + } +} +`) + writeFile(t, filepath.Join(root, "src", "placeables", "thing.utp.json"), `{ + "file_type": "UTP ", + "file_version": "V3.2", + "root": { + "struct_type": 0, + "fields": [ + {"label": "OnUsed", "type": "ResRef", "value": "use_thing"} + ] + } +} +`) + writeFile(t, filepath.Join(root, "src", "scripts", "use_thing.nss"), "void main() {}\n") + writeFile(t, filepath.Join(root, "scripts", "fetch-hak-manifest.sh"), "#!/usr/bin/env sh\nset -eu\nmkdir -p \"$(dirname \"$1\")\"\nprintf '{\"module_haks\":[\"sow_core\"],\"haks\":[]}\n' >\"$1\"\n") + writeFile(t, filepath.Join(root, "scripts", "install-script-compiler.sh"), "#!/usr/bin/env sh\nset -eu\ninstall_dir=\"${SOW_SCRIPT_COMPILER_INSTALL_DIR:-./tools/script-compiler}\"\nmkdir -p \"$install_dir\"\ncat >\"$install_dir/nwn_script_comp\" <<'SH'\n#!/usr/bin/env sh\nset -eu\nif [ \"${1:-}\" = \"--help\" ]; then\n exit 0\nfi\nout=\"\"\nsrc=\"\"\nwhile [ \"$#\" -gt 0 ]; do\n case \"$1\" in\n -o)\n out=\"$2\"\n shift 2\n ;;\n --dirs)\n shift 2\n ;;\n *)\n src=\"$1\"\n shift\n ;;\n esac\ndone\nprintf 'compiled:%s\\n' \"$(basename \"$src\")\" >\"$out\"\nSH\nchmod +x \"$install_dir/nwn_script_comp\"\nprintf 'installed: %s\\n' \"$install_dir\"\n") + if err := os.Chmod(filepath.Join(root, "scripts", "fetch-hak-manifest.sh"), 0o755); err != nil { + t.Fatalf("chmod fetch-hak-manifest.sh: %v", err) + } + if err := os.Chmod(filepath.Join(root, "scripts", "install-script-compiler.sh"), 0o755); err != nil { + t.Fatalf("chmod install-script-compiler.sh: %v", err) + } + t.Setenv("SOW_NWN_SCRIPT_COMPILER", "") + t.Setenv("SOW_MODULE_SKIP_NSS_COMPILATION", "") + + var stdout bytes.Buffer + var stderr bytes.Buffer + ctx := context{ + stdout: &stdout, + stderr: &stderr, + cwd: root, + args: []string{"build-module"}, + } + + if err := runBuildModule(ctx); err != nil { + t.Fatalf("runBuildModule failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout.String(), stderr.String()) + } + + compiledScript := filepath.Join(root, ".cache", "ncs", "use_thing.ncs") + if _, err := os.Stat(compiledScript); err != nil { + t.Fatalf("expected installed compiler to compile script: %v", err) + } + if !strings.Contains(stdout.String(), "NWScript compiler not found locally. Installing it now...") { + t.Fatalf("expected install progress in output, got %q", stdout.String()) + } +} + func TestRunMusicScanUsesConfiguredDatasetsWithoutWriting(t *testing.T) { root := t.TempDir() mkdirAll(t, filepath.Join(root, "assets", "audio", "westgate")) @@ -448,3 +640,12 @@ func setFileTime(t *testing.T, path string, modTime time.Time) { t.Fatalf("set file time %s: %v", path, err) } } + +func mustReadFile(t *testing.T, path string) string { + t.Helper() + raw, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + return string(raw) +} diff --git a/internal/pipeline/build.go b/internal/pipeline/build.go index c93c3e7..e812c88 100644 --- a/internal/pipeline/build.go +++ b/internal/pipeline/build.go @@ -6,6 +6,7 @@ import ( "encoding/json" "errors" "fmt" + "io" "os" "os/exec" "path/filepath" @@ -655,14 +656,15 @@ func referencedScriptResrefs(p *project.Project) ([]string, error) { } func resolveScriptCompiler(p *project.Project) (string, error) { + candidates := make([]string, 0, 2+len(p.ScriptCompilerSearchPaths())) if configured := p.ScriptCompilerPath(); configured != "" { - return configured, nil + candidates = append(candidates, configured) } if configured := strings.TrimSpace(os.Getenv(p.ScriptCompilerPathEnv())); configured != "" { - return configured, nil + candidates = append(candidates, configured) } + candidates = append(candidates, p.ScriptCompilerSearchPaths()...) - candidates := p.ScriptCompilerSearchPaths() if runtime.GOOS == "windows" { slices.SortStableFunc(candidates, func(a, b string) int { aExe := strings.HasSuffix(strings.ToLower(a), ".exe") @@ -678,12 +680,12 @@ func resolveScriptCompiler(p *project.Project) (string, error) { } for _, candidate := range candidates { if binary, ok := strings.CutPrefix(candidate, "PATH:"); ok { - if compiler, err := exec.LookPath(binary); err == nil { + if compiler, err := exec.LookPath(binary); err == nil && scriptCompilerRunnable(compiler) { return compiler, nil } continue } - if info, err := os.Stat(candidate); err == nil && !info.IsDir() { + if info, err := os.Stat(candidate); err == nil && !info.IsDir() && scriptCompilerRunnable(candidate) { return candidate, nil } } @@ -691,6 +693,20 @@ func resolveScriptCompiler(p *project.Project) (string, error) { return "", fmt.Errorf("script compiler not found; configure scripts.compiler.path or set %s", p.ScriptCompilerPathEnv()) } +func scriptCompilerRunnable(path string) bool { + if strings.TrimSpace(path) == "" { + return false + } + info, err := os.Stat(path) + if err != nil || info.IsDir() { + return false + } + cmd := exec.Command(path, "--help") + cmd.Stdout = io.Discard + cmd.Stderr = io.Discard + return cmd.Run() == nil +} + func validatorLoadDocument(path string) (gff.Document, string, string, error) { raw, err := os.ReadFile(path) if err != nil { diff --git a/internal/pipeline/pipeline_test.go b/internal/pipeline/pipeline_test.go index bbe0b22..6495180 100644 --- a/internal/pipeline/pipeline_test.go +++ b/internal/pipeline/pipeline_test.go @@ -1031,6 +1031,9 @@ func TestBuildModuleCompilesReferencedScripts(t *testing.T) { compiler := filepath.Join(root, "fake-compiler.sh") mustWriteFile(t, compiler, `#!/bin/sh set -eu +if [ "${1:-}" = "--help" ]; then + exit 0 +fi out="" src="" while [ "$#" -gt 0 ]; do diff --git a/internal/topdata/AUTO-INCLUDE_EXISTING_PART_MODELS_IN_2DA_GENERATION_CONTRACT.md b/internal/topdata/AUTO-INCLUDE_EXISTING_PART_MODELS_IN_2DA_GENERATION_CONTRACT.md index 26d76f2..274cbe7 100644 --- a/internal/topdata/AUTO-INCLUDE_EXISTING_PART_MODELS_IN_2DA_GENERATION_CONTRACT.md +++ b/internal/topdata/AUTO-INCLUDE_EXISTING_PART_MODELS_IN_2DA_GENERATION_CONTRACT.md @@ -1,5 +1,17 @@ # AI Agent Contract: Auto-Include Existing Part Models in 2DA Generation +## Status Snapshot + +Current state as of 2026-05-13: + +- Implemented. +- The toolkit resolves released parts manifests through the configured autogen + release source and covers this behavior with `parts_manifest_test.go` plus + broader native topdata tests. +- Remaining work is operational hardening only: keep manifest contracts stable + across `assets/` releases and maintain regression coverage as autogen config + evolves. + ## Objective The native 2DA generation pipeline must automatically include existing part models already present in the **`sow-assets` repository** when building final parts tables. diff --git a/internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md b/internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md index b99656f..f045b54 100644 --- a/internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md +++ b/internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md @@ -1,5 +1,22 @@ # Global Feat Injection — Specification +## Status Snapshot + +Current state as of 2026-05-13: + +- This document is partially stale relative to the implementation in + `internal/topdata/native.go`. +- Current code injects: + - always-on menu feats: `feat:specialattacks`, `feat:throw`, + `feat:grapple`, `feat:offensivefighting`, `feat:defensivefighting`, + `feat:horsemenu` + - conditional combat feats: `feat:powerattack`, `feat:combatexpertise` + - class-skill shorthand rows for `masterfeats:skillfocus` at level `-1` and + `masterfeats:greaterskillfocus` at level `12` +- This contract should be treated as a parity note, not as an exact + implementation transcript, until it is fully rewritten around the current + behavior. + ## Objective Ensure the native class feat generation pipeline produces **functionally identical output** to the legacy system by applying deterministic global feat injections. @@ -24,6 +41,9 @@ With properties: "OnMenu": "0" } +For the current implementation, `masterfeats:greaterskillfocus` is injected with +`GrantedOnLevel = 12`, not `-1`. + --- ### R2 — Spellcasting master feats @@ -37,6 +57,16 @@ Then inject: - `masterfeats:spellfocus` - `masterfeats:greaterspellfocus` +Historical note: + +- This rule is not currently implemented as an automatic global injection in + `native.go`. +- Spell-focus expansion is supported when the class feat data explicitly + references `masterfeats:spellfocus`. + +If automatic spellcaster-driven injection is still required, it remains +outstanding work. + With properties: ```json @@ -75,6 +105,9 @@ With properties: Always inject: +- `feat:specialattacks` +- `feat:throw` +- `feat:grapple` - `feat:offensivefighting` - `feat:defensivefighting` - `feat:horsemenu` diff --git a/internal/topdata/FAMILY_EXPANSION_CONTRACT.md b/internal/topdata/FAMILY_EXPANSION_CONTRACT.md index a10dac8..3028822 100644 --- a/internal/topdata/FAMILY_EXPANSION_CONTRACT.md +++ b/internal/topdata/FAMILY_EXPANSION_CONTRACT.md @@ -1,5 +1,16 @@ # Generic Family Expansion Contract +## Status Snapshot + +Current state as of 2026-05-13: + +- Implemented and actively used by native feat generation. +- `family_expansion.go`, `native.go`, and `topdata_test.go` cover the current + underscore-family interpretation, generated family metadata, and reapplication + behavior. +- Remaining work is mostly consumer expansion: use the same primitive in more + datasets only when the authored data actually benefits from it. + Topdata now treats underscore expansion as a structural rule: - `parent_child` means `child` is an expansion of `parent` diff --git a/internal/topdata/FEAT_GENERATED_FAMILIES_CONTRACT.md b/internal/topdata/FEAT_GENERATED_FAMILIES_CONTRACT.md index 3d6a303..1b11522 100644 --- a/internal/topdata/FEAT_GENERATED_FAMILIES_CONTRACT.md +++ b/internal/topdata/FEAT_GENERATED_FAMILIES_CONTRACT.md @@ -1,5 +1,16 @@ # `feat` Generated Families Contract +## Status Snapshot + +Current state as of 2026-05-13: + +- Implemented for the current native `feat` pipeline. +- Native tests cover generated-family expansion, family-owned shared defaults, + masterfeat-backed inheritance, and lock/TLK identity preservation. +- The one explicitly deferred item in this contract is still current: + `feat.2da` remains native-built but excluded from `compare-topdata` + reference-catalog coverage through `compare_reference: false`. + Canonical authored paths: - `topdata/data/feat/base.json` diff --git a/internal/topdata/INHERITANCE_CONTRACT.md b/internal/topdata/INHERITANCE_CONTRACT.md index f4b16ec..3cf958d 100644 --- a/internal/topdata/INHERITANCE_CONTRACT.md +++ b/internal/topdata/INHERITANCE_CONTRACT.md @@ -1,5 +1,14 @@ # Topdata Inheritance Contract +## Status Snapshot + +Current state as of 2026-05-13: + +- Implemented. +- `native.go` supports both row-sugar inheritance and generic object + inheritance, including cycle detection and missing-target failures. +- Validation and build regression coverage live in `topdata_test.go`. + Native topdata now supports two inheritance forms: ## 1. Row Sugar Compatibility diff --git a/internal/topdata/MASTERFEATS_CONTRACT.md b/internal/topdata/MASTERFEATS_CONTRACT.md index 47ecd7a..35ad700 100644 --- a/internal/topdata/MASTERFEATS_CONTRACT.md +++ b/internal/topdata/MASTERFEATS_CONTRACT.md @@ -1,5 +1,15 @@ # `masterfeats` Canonical Contract +## Status Snapshot + +Current state as of 2026-05-13: + +- Implemented as a first-class native dataset. +- The repository contains `topdata/data/masterfeats/`, migration/import support, + validator checks, and native output coverage for `masterfeats.2da`. +- Remaining work is only downstream adoption and regression maintenance as other + native datasets continue to consume `masterfeats` more heavily. + `masterfeats` is a stable native canonical family consumed directly by `feat` generation, class-feat expansion, and inheritance/ref resolution.