sow-tools/README.md

sow-tools

Shared NWN build tooling for Shadows Over Westgate.

This repository owns the Go-based sow-toolkit binary used by the separate module and assets repositories.

Build

Build the development binary into tools/sow-toolkit:

./build-tool.sh

On Windows:

.\build-tool.ps1

Both wrappers use .cache/go-build for the Go build cache and skip rebuilding when the sources are older than the existing development binary.

Or run directly with Go:

GOCACHE="$PWD/.cache/go-build" go run ./cmd/nwn-tool --help

When changing native topdata id allocation, lockfile persistence, generated families, or null-and-relocate behavior, run the repeat-build determinism test:

go test ./internal/topdata -run TestBuildNativeKeepsGeneratedIDsStableAcrossRepeatedBuilds

That test builds the same topdata project ten times in one process and asserts that relocated/generated ids stay pinned across every pass.

Repository Layout

cmd/nwn-tool/      CLI entrypoint
internal/app/      command wiring and console UX
internal/erf/      ERF/MOD/HAK reading and writing
internal/gff/      canonical GFF JSON and binary conversion
internal/pipeline/ build, extract, compare, and manifest workflows
internal/project/  config loading and repo scanning
internal/topdata/  native topdata, wiki, and autogen workflows
internal/validator/validation and diagnostics
tools/             built development binary output

Commands

The shared CLI exposes these project utilities:

sow-toolkit build
sow-toolkit build-module
sow-toolkit build-haks [--hak <name>] [--archive <name>] [--source-manifest <path>] [--plan-only] [--skip-music] [--music-dataset <id>]
sow-toolkit extract [<build-relative-archive-or-glob> ...]
sow-toolkit validate
sow-toolkit compare
sow-toolkit apply-hak-manifest [<manifest-path>]
sow-toolkit config <validate|effective|inspect|explain|sources>
sow-toolkit music <list-datasets|scan|build|credits|validate|manifest|normalize>
sow-toolkit validate-topdata
sow-toolkit build-topdata [--force] [--wiki]
sow-toolkit build-top-package [--force]
sow-toolkit compare-topdata
sow-toolkit convert-topdata <2da-to-json|2da-to-module|json-to-2da> ...
sow-toolkit build-wiki [--force]
sow-toolkit deploy-wiki [--source-dir <path>] [--endpoint <url>] [--token <token>] [--version <ver>] [--namespace <ns>] [--category <namespace=cid>] [--manifest <path>] [--dry-run] [--create] [--force]
sow-toolkit build-changelog [--config <path>] [--output <path>] [--current-tag <tag>] [--previous-tag <tag>] [--api-base-url <url>] [--token <token>]

Global logging flags --quiet, --verbose, and --debug may be passed after the command name. Long options that take values accept either --flag value or --flag=value; empty inline values such as --dataset= are rejected.

Repository Configuration

Human-authored consumer repository configuration is YAML. The toolkit discovers configuration in this order:

1. nwn-tool.yaml
2. nwn-tool.yml
3. legacy nwn-tool.json

YAML is canonical. If both YAML and legacy JSON files are present, the YAML file wins. Legacy nwn-tool.json still loads for migration, but commands print a warning and should be updated to nwn-tool.yaml.

Example:

module:
  name: Shadows Over Westgate
  resref: sow_module
  description: Shadows Over Westgate asset pipeline.
  hak_order:
    - sow_top
    - group:sow_over
    - group:sow_core

paths:
  assets: content
  build: build

music:
  # Legacy shorthand — automatically normalized into a dataset.
  prefixes:
    envi/music/westgate: mus_wg_

  # Explicit dataset config (replaces prefixes above).
  tools:
    ffmpeg: tools/ffmpeg
    ffprobe: tools/ffprobe
  defaults:
    stage_root: "{paths.cache}/music"
    credits_root: "{paths.cache}/credits"
    credits_overlay: CREDITS.md
    max_stem_length: 16
    naming_scheme: nwn_bmu
    output_extension: .bmu
    convert_extensions: [.flac, .m4a, .mp3, .ogg, .wav]
  datasets:
    westgate:
      source: envi/music/westgate
      output: envi/music/westgate
      prefix: mus_wg_
      package_mode: hak_asset

Resolved configuration can be inspected without scanning or building:

sow-toolkit config validate
sow-toolkit config effective --json
sow-toolkit config inspect paths.build
sow-toolkit config explain topdata.package_hak
sow-toolkit config sources

config effective prints the normalized values consumed by commands, including toolkit defaults, YAML values, legacy status, active environment overrides, and provenance. Sensitive override values such as tokens are reported as set without printing the secret.

Common configurable defaults:

paths:
  source: src
  assets: assets
  build: build
  cache: .cache
  tools: tools

outputs:
  module_archive: "{module.resref}.mod"
  hak_manifest: haks.json
  hak_archive: "{hak.name}.hak"

build:
  keep_existing_haks: false

inventory:
  source_extensions: []      # empty means built-in NWN source extensions
  asset_extensions: []       # empty means built-in NWN asset extensions
  source_json_pattern: "{resref}.{extension}.json"

validation:
  profile: nwn_module
  builtin_script_prefixes: [nw_, x0_, x1_, x2_, x3_, ga_, gc_, gen_, gui_, nwg_, ta_]
  required_fields:
    ifo: [Mod_Name]

scripts:
  source_dir: scripts
  cache: "{paths.cache}/ncs"
  compiler:
    path: ""
    env:
      path: SOW_NWN_SCRIPT_COMPILER
      nwn_root: [SOW_NWN_ROOT, NWN_ROOT]
      nwn_user_directory: [SOW_NWN_USER_DIRECTORY, NWN_HOME, NWN_USER_DIRECTORY]
    search:
      - "{paths.tools}/script-compiler/nwn_script_comp"
      - "{paths.tools}/script-compiler/nwn_script_comp.exe"
      - "{paths.tools}/nwn_script_comp"
      - "{paths.tools}/nwn_script_comp.exe"
      - PATH:nwn_script_comp

extract:
  layout: nwn_canonical_json
  archives:
    - "{module.resref}.mod"
  cleanup_stale: true
  consume_archives: false
  ignore_extensions: []
  delete_module_archive_after_success: false # deprecated compatibility alias

topdata:
  build: "{paths.build}/topdata"
  compiled_2da_dir: 2da
  compiled_tlk: sow_tlk.tlk
  package_hak: sow_top.hak
  package_tlk: sow_tlk.tlk
  wiki:
    output_root: wiki
    pages_dir: pages
    state_file: state.json
    managed_namespaces: [classes, feat, itemtypes, races, skills, spells, meta]
    deploy_manifest: .wiki_deploy_manifest.json
    deploy_edit_summary: Auto-generated from native builder

autogen:
  cache:
    root: "{paths.cache}"
    max_age: 1h
    refresh_env: SOW_AUTOGEN_MANIFEST_REFRESH
  release_source:
    provider: gitea
    server_url_env: SOW_ASSETS_SERVER_URL
    repo_env: SOW_ASSETS_REPO

paths.source and paths.assets must name real repository subtrees when they are configured. They may be omitted for repositories that do not own that class of source, but they must not resolve to the repository root (.). Output and cache paths must stay relative to the repository root unless a specific setting documents otherwise.

Extraction is deliberately guarded because it writes source files from binary archives. extract.archives is a list of glob patterns relative to paths.build; the default is only the configured module .mod. HAK extraction is opt-in, for example:

extract:
  archives:
    - "{module.resref}.mod"
    - "sow_over*.hak"

Module resources require a configured paths.source; HAK assets require a configured paths.assets; both roots must resolve to safe subtrees. If a root is unset or resolves to the repository root, extraction fails before writing instead of creating extension directories like mdl/ or png/ at the repo top level. .erf files are rejected by extraction because they do not contain module.ifo, so they cannot safely update module-level area membership.

consume_archives: true deletes the selected build archive files only after the entire extraction succeeds. The older delete_module_archive_after_success setting remains as a module-only compatibility alias when consume_archives is not set. Stale cleanup is scoped to roots actually touched by extracted resources, so the default module-only extraction does not prune assets.

Music defaults (used when not explicitly configured):

music:
  defaults:
    stage_root: "{paths.cache}/music"
    credits_root: "{paths.cache}/credits"
    credits_overlay: CREDITS.md
    max_stem_length: 16
    naming_scheme: nwn_bmu
    output_extension: .bmu
    convert_extensions: [.flac, .m4a, .mp3, .ogg, .wav]

Music datasets define individual music collections with independent sources and naming prefixes. source is the authored input root under paths.assets; output is the generated HAK asset root matched by HAK include globs. If output is omitted, it defaults to source. package_mode: hak_asset converts configured source extensions into generated HAK resources; package_mode: none scans and writes credits without adding generated HAK resources. The legacy music.prefixes shorthand is automatically normalized into datasets. convert_extensions is configuration-driven; the built-in default set is .flac, .m4a, .mp3, .ogg, and .wav. Those extensions are discovered from YAML within configured music dataset source roots, so setting them in music.defaults.convert_extensions or a dataset override is sufficient; you do not need a parallel inventory.asset_extensions override just to make music scanning see them.

Music can also be run directly:

sow-toolkit music list-datasets
sow-toolkit music scan --dataset westgate
sow-toolkit music build --dataset westgate --dry-run
sow-toolkit music build --dataset westgate
sow-toolkit music credits --dataset westgate --check
sow-toolkit music validate --dataset westgate --json
sow-toolkit music manifest --dataset westgate
sow-toolkit music normalize --dataset westgate

build-haks uses the same music pipeline. Use --plan-only or music --dry-run to compute mappings without transcoding. Use build-haks --skip-music to package authored assets only, or build-haks --music-dataset westgate to limit conversion to a configured dataset.

Generated and machine-readable artifacts remain JSON, including HAK manifests, topdata datasets, caches, credits inventories, build reports, and canonical GFF documents. Nested metadata such as topdata/templates/config.yaml remains valid when it is local workflow configuration rather than root repository configuration.

Wiki And Changelog Utilities

build-topdata --wiki or build-wiki renders wiki pages from the current native topdata state. deploy-wiki publishes those pages to NodeBB. It reads NODEBB_API_ENDPOINT, NODEBB_API_TOKEN, GITHUB_REF_NAME, and NODEBB_WIKI_CATEGORIES when equivalent flags are omitted. Category mappings use namespace=cid, for example:

sow-toolkit deploy-wiki --dry-run --namespace spells --category spells=42

When a deploy manifest is missing but matching wiki pages already exist in NodeBB, deploy-wiki checks the Westgate Wiki namespace directory API and adopts those topic/post mappings before planning updates. --create is still required for genuinely new pages that have no existing remote topic.

build-changelog renders a release changelog from tagged pushes. Pull-request style commits keep their PR links and authors from the Gitea API; direct pushes are rendered as commit links using the commit author. It defaults to scripts/changelog.json, reads Gitea tokens from GITEA_API_TOKEN, GITEA_TOKEN, or SOW_TOOLS_TOKEN, and writes to stdout unless --output is provided.

Intended Consumers

• sow-module uses sow-toolkit for build, build-module, extract, validate, compare, and apply-hak-manifest
• sow-assets uses sow-toolkit for build-haks, music processing, changelog generation, and validation
  • validation now fails early if duplicate asset resources would collide inside the same generated HAK
  • duplicate resources split across different HAK groups remain warnings because later HAK order can still override earlier content at runtime
• sow-module now also uses sow-toolkit for topdata:
  • validate-topdata
  • build-topdata
  • build-top-package
  • compare-topdata
  • convert-topdata
  • build-wiki
  • deploy-wiki

During development, sibling repositories can point at ../sow-tools/tools/sow-toolkit or ../sow-tools/tools/sow-toolkit.exe. For pinned releases, each consumer repo can instead carry its own copy in its local tools/ directory.

Cross-Repository Behavior

The current ownership split is:

• sow-tools
  • owns the shared CLI and build logic
  • owns module build, extract, compare, and HAK manifest application logic
  • owns topdata validation, build, compare, conversion, and packaging logic
• sow-module
  • owns canonical module source under src/
  • owns canonical topdata authored data under topdata/
  • owns the user-facing module and topdata wrapper commands
• sow-assets
  • owns canonical binary asset source under assets/
  • owns generated HAK manifests and parts manifests under staging/
  • owns the user-facing asset wrapper commands

Consumer wrapper resolution is intentionally aligned:

1. Prefer a pinned local binary in tools/
2. Else prefer a sibling ../sow-tools checkout for local source development
3. Else install the latest published sow-tools release artifact

NWScript compilation behavior is also aligned and configurable under scripts.compiler:

1. Prefer SOW_NWN_SCRIPT_COMPILER when explicitly set
2. Else prefer a local compiler in tools/script-compiler/ or tools/
3. Else use a system-installed nwn_script_comp from PATH

When the compiler runs, the toolkit prepares NWN_HOME / NWN_USER_DIRECTORY automatically and tries to detect NWN_ROOT from common Steam locations, including custom Steam library folders, before an explicit override is required.

Release Automation

This repo publishes:

• sow-toolkit-linux-amd64.tar.gz
• sow-toolkit-windows-amd64.exe

Consumer repos can fetch those binaries into their local tools/ directory with:

• install-tool.sh on Linux/macOS
• install-tool.ps1 on Windows

TopData

The native topdata system is implemented in this repo and authored in sow-module.

High-level ownership split:

• sow-tools
  • owns the sow-toolkit implementation
  • owns topdata validation, build, compare, conversion, and dataset-resolution logic
• sow-module
  • owns canonical topdata authored data under topdata/data/
  • owns the user-facing topdata workflow scripts

For a human-readable front-to-back explanation of how the native topdata builder works and how to use it, see:

• sow-module/topdata/README.md

Important topdata contracts still live here under internal/topdata/ because they are code-adjacent implementation contracts, not user workflow docs.