# sow-tools Shared NWN build tooling for Shadows Over Westgate. This repository owns the Go-based `sow-toolkit` binary used by the separate module and assets repositories. ## Build Build the development binary into `tools/sow-toolkit`: ```bash ./build-tool.sh ``` On Windows: ```powershell .\build-tool.ps1 ``` Both wrappers use `.cache/go-build` for the Go build cache and skip rebuilding when the sources are older than the existing development binary. Or run directly with Go: ```bash GOCACHE="$PWD/.cache/go-build" go run ./cmd/nwn-tool --help ``` ## Repository Layout ```text cmd/nwn-tool/ CLI entrypoint internal/app/ command wiring and console UX internal/erf/ ERF/MOD/HAK reading and writing internal/gff/ canonical GFF JSON and binary conversion internal/pipeline/ build, extract, compare, and manifest workflows internal/project/ config loading and repo scanning internal/topdata/ native topdata, wiki, and autogen workflows internal/validator/validation and diagnostics tools/ built development binary output ``` ## Commands The shared CLI exposes these project utilities: ```bash sow-toolkit build sow-toolkit build-module sow-toolkit build-haks [--hak ] [--archive ] [--source-manifest ] [--plan-only] [--skip-music] [--music-dataset ] sow-toolkit extract [ ...] sow-toolkit validate sow-toolkit compare sow-toolkit apply-hak-manifest [] sow-toolkit config sow-toolkit music sow-toolkit validate-topdata sow-toolkit build-topdata [--force] [--wiki] sow-toolkit build-top-package [--force] sow-toolkit compare-topdata sow-toolkit convert-topdata <2da-to-json|2da-to-module|json-to-2da> ... sow-toolkit build-wiki [--force] sow-toolkit deploy-wiki [--source-dir ] [--endpoint ] [--token ] [--version ] [--namespace ] [--category ] [--manifest ] [--dry-run] [--create] [--force] sow-toolkit build-changelog [--config ] [--output ] [--current-tag ] [--previous-tag ] [--api-base-url ] [--token ] ``` Global logging flags `--quiet`, `--verbose`, and `--debug` may be passed after the command name. Long options that take values accept either `--flag value` or `--flag=value`; empty inline values such as `--dataset=` are rejected. ## Repository Configuration Human-authored consumer repository configuration is YAML. The toolkit discovers configuration in this order: 1. `nwn-tool.yaml` 2. `nwn-tool.yml` 3. legacy `nwn-tool.json` YAML is canonical. If both YAML and legacy JSON files are present, the YAML file wins. Legacy `nwn-tool.json` still loads for migration, but commands print a warning and should be updated to `nwn-tool.yaml`. Example: ```yaml module: name: Shadows Over Westgate resref: sow_module description: Shadows Over Westgate asset pipeline. hak_order: - sow_top - group:sow_over - group:sow_core paths: assets: content build: build music: # Legacy shorthand — automatically normalized into a dataset. prefixes: envi/music/westgate: mus_wg_ # Explicit dataset config (replaces prefixes above). tools: ffmpeg: tools/ffmpeg ffprobe: tools/ffprobe defaults: stage_root: "{paths.cache}/music" credits_root: "{paths.cache}/credits" credits_overlay: CREDITS.md max_stem_length: 16 naming_scheme: nwn_bmu output_extension: .bmu convert_extensions: [.flac, .m4a, .mp3, .ogg, .wav] datasets: westgate: source: envi/music/westgate output: envi/music/westgate prefix: mus_wg_ package_mode: hak_asset ``` Resolved configuration can be inspected without scanning or building: ```bash sow-toolkit config validate sow-toolkit config effective --json sow-toolkit config inspect paths.build sow-toolkit config explain topdata.package_hak sow-toolkit config sources ``` `config effective` prints the normalized values consumed by commands, including toolkit defaults, YAML values, legacy status, active environment overrides, and provenance. Sensitive override values such as tokens are reported as set without printing the secret. Common configurable defaults: ```yaml paths: source: src assets: assets build: build cache: .cache tools: tools outputs: module_archive: "{module.resref}.mod" hak_manifest: haks.json hak_archive: "{hak.name}.hak" build: keep_existing_haks: false inventory: source_extensions: [] # empty means built-in NWN source extensions asset_extensions: [] # empty means built-in NWN asset extensions source_json_pattern: "{resref}.{extension}.json" validation: profile: nwn_module builtin_script_prefixes: [nw_, x0_, x1_, x2_, x3_, ga_, gc_, gen_, gui_, nwg_, ta_] required_fields: ifo: [Mod_Name] scripts: source_dir: scripts cache: "{paths.cache}/ncs" compiler: path: "" env: path: SOW_NWN_SCRIPT_COMPILER nwn_root: [SOW_NWN_ROOT, NWN_ROOT] nwn_user_directory: [SOW_NWN_USER_DIRECTORY, NWN_HOME, NWN_USER_DIRECTORY] search: - "{paths.tools}/script-compiler/nwn_script_comp" - "{paths.tools}/script-compiler/nwn_script_comp.exe" - "{paths.tools}/nwn_script_comp" - "{paths.tools}/nwn_script_comp.exe" - PATH:nwn_script_comp extract: layout: nwn_canonical_json archives: - "{module.resref}.mod" cleanup_stale: true consume_archives: false ignore_extensions: [] delete_module_archive_after_success: false # deprecated compatibility alias topdata: build: "{paths.build}/topdata" compiled_2da_dir: 2da compiled_tlk: sow_tlk.tlk package_hak: sow_top.hak package_tlk: sow_tlk.tlk wiki: output_root: wiki pages_dir: pages state_file: state.json managed_namespaces: [classes, feat, itemtypes, races, skills, spells, meta] deploy_manifest: .wiki_deploy_manifest.json deploy_edit_summary: Auto-generated from native builder autogen: cache: root: "{paths.cache}" max_age: 1h refresh_env: SOW_AUTOGEN_MANIFEST_REFRESH release_source: provider: gitea server_url_env: SOW_ASSETS_SERVER_URL repo_env: SOW_ASSETS_REPO ``` `paths.source` and `paths.assets` must name real repository subtrees when they are configured. They may be omitted for repositories that do not own that class of source, but they must not resolve to the repository root (`.`). Output and cache paths must stay relative to the repository root unless a specific setting documents otherwise. Extraction is deliberately guarded because it writes source files from binary archives. `extract.archives` is a list of glob patterns relative to `paths.build`; the default is only the configured module `.mod`. HAK extraction is opt-in, for example: ```yaml extract: archives: - "{module.resref}.mod" - "sow_over*.hak" ``` Module resources require a configured `paths.source`; HAK assets require a configured `paths.assets`; both roots must resolve to safe subtrees. If a root is unset or resolves to the repository root, extraction fails before writing instead of creating extension directories like `mdl/` or `png/` at the repo top level. `.erf` files are rejected by extraction because they do not contain `module.ifo`, so they cannot safely update module-level area membership. `consume_archives: true` deletes the selected build archive files only after the entire extraction succeeds. The older `delete_module_archive_after_success` setting remains as a module-only compatibility alias when `consume_archives` is not set. Stale cleanup is scoped to roots actually touched by extracted resources, so the default module-only extraction does not prune assets. Music defaults (used when not explicitly configured): ```yaml music: defaults: stage_root: "{paths.cache}/music" credits_root: "{paths.cache}/credits" credits_overlay: CREDITS.md max_stem_length: 16 naming_scheme: nwn_bmu output_extension: .bmu convert_extensions: [.flac, .m4a, .mp3, .ogg, .wav] ``` Music datasets define individual music collections with independent sources and naming prefixes. `source` is the authored input root under `paths.assets`; `output` is the generated HAK asset root matched by HAK include globs. If `output` is omitted, it defaults to `source`. `package_mode: hak_asset` converts configured source extensions into generated HAK resources; `package_mode: none` scans and writes credits without adding generated HAK resources. The legacy `music.prefixes` shorthand is automatically normalized into datasets. `convert_extensions` is configuration-driven; the built-in default set is `.flac`, `.m4a`, `.mp3`, `.ogg`, and `.wav`. Those extensions are discovered from YAML within configured music dataset source roots, so setting them in `music.defaults.convert_extensions` or a dataset override is sufficient; you do not need a parallel `inventory.asset_extensions` override just to make music scanning see them. Music can also be run directly: ```bash sow-toolkit music list-datasets sow-toolkit music scan --dataset westgate sow-toolkit music build --dataset westgate --dry-run sow-toolkit music build --dataset westgate sow-toolkit music credits --dataset westgate --check sow-toolkit music validate --dataset westgate --json sow-toolkit music manifest --dataset westgate sow-toolkit music normalize --dataset westgate ``` `build-haks` uses the same music pipeline. Use `--plan-only` or music `--dry-run` to compute mappings without transcoding. Use `build-haks --skip-music` to package authored assets only, or `build-haks --music-dataset westgate` to limit conversion to a configured dataset. Generated and machine-readable artifacts remain JSON, including HAK manifests, topdata datasets, caches, credits inventories, build reports, and canonical GFF documents. Nested metadata such as `topdata/templates/config.yaml` remains valid when it is local workflow configuration rather than root repository configuration. ## Wiki And Changelog Utilities `build-topdata --wiki` or `build-wiki` renders wiki pages from the current native topdata state. `deploy-wiki` publishes those pages to NodeBB. It reads `NODEBB_API_ENDPOINT`, `NODEBB_API_TOKEN`, `GITHUB_REF_NAME`, and `NODEBB_WIKI_CATEGORIES` when equivalent flags are omitted. Category mappings use `namespace=cid`, for example: ```bash sow-toolkit deploy-wiki --dry-run --namespace spells --category spells=42 ``` `build-changelog` renders a release changelog from tagged pushes. Pull-request style commits keep their PR links and authors from the Gitea API; direct pushes are rendered as commit links using the commit author. It defaults to `scripts/changelog.json`, reads Gitea tokens from `GITEA_API_TOKEN`, `GITEA_TOKEN`, or `SOW_TOOLS_TOKEN`, and writes to stdout unless `--output` is provided. ## Intended Consumers - `sow-module` uses `sow-toolkit` for `build`, `build-module`, `extract`, `validate`, `compare`, and `apply-hak-manifest` - `sow-assets` uses `sow-toolkit` for `build-haks`, music processing, changelog generation, and validation - validation now fails early if duplicate asset resources would collide inside the same generated HAK - duplicate resources split across different HAK groups remain warnings because later HAK order can still override earlier content at runtime - `sow-module` now also uses `sow-toolkit` for topdata: - `validate-topdata` - `build-topdata` - `build-top-package` - `compare-topdata` - `convert-topdata` - `build-wiki` - `deploy-wiki` During development, sibling repositories can point at `../sow-tools/tools/sow-toolkit` or `../sow-tools/tools/sow-toolkit.exe`. For pinned releases, each consumer repo can instead carry its own copy in its local `tools/` directory. ## Cross-Repository Behavior The current ownership split is: - `sow-tools` - owns the shared CLI and build logic - owns module build, extract, compare, and HAK manifest application logic - owns topdata validation, build, compare, conversion, and packaging logic - `sow-module` - owns canonical module source under `src/` - owns canonical topdata authored data under `topdata/` - owns the user-facing module and topdata wrapper commands - `sow-assets` - owns canonical binary asset source under `assets/` - owns generated HAK manifests and parts manifests under `staging/` - owns the user-facing asset wrapper commands Consumer wrapper resolution is intentionally aligned: 1. Prefer a pinned local binary in `tools/` 2. Else prefer a sibling `../sow-tools` checkout for local source development 3. Else install the latest published `sow-tools` release artifact NWScript compilation behavior is also aligned and configurable under `scripts.compiler`: 1. Prefer `SOW_NWN_SCRIPT_COMPILER` when explicitly set 2. Else prefer a local compiler in `tools/script-compiler/` or `tools/` 3. Else use a system-installed `nwn_script_comp` from `PATH` When the compiler runs, the toolkit prepares `NWN_HOME` / `NWN_USER_DIRECTORY` automatically and tries to detect `NWN_ROOT` from common Steam locations, including custom Steam library folders, before an explicit override is required. ## Release Automation This repo publishes: - `sow-toolkit-linux-amd64.tar.gz` - `sow-toolkit-windows-amd64.exe` Consumer repos can fetch those binaries into their local `tools/` directory with: - `install-tool.sh` on Linux/macOS - `install-tool.ps1` on Windows ## TopData The native topdata system is implemented in this repo and authored in `sow-module`. High-level ownership split: - `sow-tools` - owns the `sow-toolkit` implementation - owns topdata validation, build, compare, conversion, and dataset-resolution logic - `sow-module` - owns canonical topdata authored data under `topdata/data/` - owns the user-facing topdata workflow scripts For a human-readable front-to-back explanation of how the native topdata builder works and how to use it, see: - [sow-module/topdata/README.md](../sow-module/topdata/README.md) Important topdata contracts still live here under `internal/topdata/` because they are code-adjacent implementation contracts, not user workflow docs.