sow-tools

Shared NWN build tooling for Shadows Over Westgate.

This repository owns the Go-based sow-toolkit binary used by the separate module and assets repositories.

Build

Build the development binary into tools/sow-toolkit:

./build-tool.sh

Or run directly with Go:

GOCACHE="$PWD/.cache/go-build" go run ./cmd/nwn-tool --help

Repository Layout

cmd/nwn-tool/      CLI entrypoint
internal/app/      command wiring and console UX
internal/erf/      ERF/MOD/HAK reading and writing
internal/gff/      canonical GFF JSON and binary conversion
internal/pipeline/ build, extract, compare, and manifest workflows
internal/project/  config loading and repo scanning
internal/validator/validation and diagnostics
tools/             built development binary output

Repository Configuration

Human-authored consumer repository configuration is YAML. The toolkit discovers configuration in this order:

1. nwn-tool.yaml
2. nwn-tool.yml
3. legacy nwn-tool.json

YAML is canonical. If both YAML and legacy JSON files are present, the YAML file wins. Legacy nwn-tool.json still loads for migration, but commands print a warning and should be updated to nwn-tool.yaml.

Example:

module:
  name: Shadows Over Westgate
  resref: sow_module
  description: Shadows Over Westgate asset pipeline.
  hak_order:
    - sow_top
    - group:sow_over
    - group:sow_core

paths:
  assets: content
  build: build

music:
  # Legacy shorthand — automatically normalized into a dataset.
  prefixes:
    envi/music/westgate: mus_wg_

  # Explicit dataset config (replaces prefixes above).
  tools:
    ffmpeg: tools/ffmpeg
    ffprobe: tools/ffprobe
  defaults:
    stage_root: "{paths.cache}/music"
    credits_root: "{paths.cache}/credits"
    credits_overlay: CREDITS.md
    max_stem_length: 16
    naming_scheme: nwn_bmu
    output_extension: .bmu
    convert_extensions: [.mp3, .ogg]
  datasets:
    westgate:
      source: envi/music/westgate
      output: envi/music/westgate
      prefix: mus_wg_
      package_mode: hak_asset

Resolved configuration can be inspected without scanning or building:

sow-toolkit config validate
sow-toolkit config effective --json
sow-toolkit config inspect paths.build
sow-toolkit config explain topdata.package_hak
sow-toolkit config sources

config effective prints the normalized values consumed by commands, including toolkit defaults, YAML values, legacy status, active environment overrides, and provenance. Sensitive override values such as tokens are reported as set without printing the secret.

Common configurable defaults:

paths:
  source: src
  assets: assets
  build: build
  cache: .cache
  tools: tools

outputs:
  module_archive: "{module.resref}.mod"
  hak_manifest: haks.json
  hak_archive: "{hak.name}.hak"

build:
  keep_existing_haks: false

inventory:
  source_extensions: []      # empty means built-in NWN source extensions
  asset_extensions: []       # empty means built-in NWN asset extensions
  source_json_pattern: "{resref}.{extension}.json"

validation:
  profile: nwn_module
  builtin_script_prefixes: [nw_, x0_, x1_, x2_, x3_, ga_, gc_, gen_, gui_, nwg_, ta_]
  required_fields:
    ifo: [Mod_Name]

scripts:
  source_dir: scripts
  cache: "{paths.cache}/ncs"
  compiler:
    path: ""
    env:
      path: SOW_NWN_SCRIPT_COMPILER
      nwn_root: [SOW_NWN_ROOT, NWN_ROOT]
      nwn_user_directory: [SOW_NWN_USER_DIRECTORY, NWN_HOME, NWN_USER_DIRECTORY]
    search:
      - "{paths.tools}/script-compiler/nwn_script_comp"
      - "{paths.tools}/script-compiler/nwn_script_comp.exe"
      - "{paths.tools}/nwn_script_comp"
      - "{paths.tools}/nwn_script_comp.exe"
      - PATH:nwn_script_comp

extract:
  layout: nwn_canonical_json
  hak_discovery: build_glob      # or configured_haks
  cleanup_stale: true
  ignore_extensions: []
  delete_module_archive_after_success: false

topdata:
  build: "{paths.build}/topdata"
  compiled_2da_dir: 2da
  compiled_tlk: sow_tlk.tlk
  package_hak: sow_top.hak
  package_tlk: sow_tlk.tlk
  wiki:
    output_root: wiki
    pages_dir: pages
    state_file: state.json
    managed_namespaces: [classes, feat, itemtypes, races, skills, spells, meta]
    deploy_manifest: .wiki_deploy_manifest.json
    deploy_edit_summary: Auto-generated from native builder

autogen:
  cache:
    root: "{paths.cache}"
    max_age: 1h
    refresh_env: SOW_AUTOGEN_MANIFEST_REFRESH
  release_source:
    provider: gitea
    server_url_env: SOW_ASSETS_SERVER_URL
    repo_env: SOW_ASSETS_REPO

Music defaults (used when not explicitly configured):

music:
  defaults:
    stage_root: "{paths.cache}/music"
    credits_root: "{paths.cache}/credits"
    credits_overlay: CREDITS.md
    max_stem_length: 16
    naming_scheme: nwn_bmu
    output_extension: .bmu
    convert_extensions: [.mp3, .ogg]

Music datasets define individual music collections with independent sources and naming prefixes. source is the authored input root under paths.assets; output is the generated HAK asset root matched by HAK include globs. If output is omitted, it defaults to source. package_mode: hak_asset converts configured source extensions into generated HAK resources; package_mode: none scans and writes credits without adding generated HAK resources. The legacy music.prefixes shorthand is automatically normalized into datasets.

Music can also be run directly:

sow-toolkit music list-datasets
sow-toolkit music scan --dataset westgate
sow-toolkit music build --dataset westgate --dry-run
sow-toolkit music build --dataset westgate
sow-toolkit music credits --dataset westgate --check
sow-toolkit music validate --dataset westgate --json

build-haks uses the same music pipeline. Use --plan-only or music --dry-run to compute mappings without transcoding. Use build-haks --skip-music to package authored assets only, or build-haks --music-dataset westgate to limit conversion to a configured dataset.

Generated and machine-readable artifacts remain JSON, including HAK manifests, topdata datasets, caches, credits inventories, build reports, and canonical GFF documents. Nested metadata such as topdata/templates/config.json is not root repository configuration and is intentionally unchanged.

Intended Consumers

• sow-module uses sow-toolkit for build-module, extract, validate, compare, and apply-hak-manifest
• sow-assets uses sow-toolkit for build-haks and validation
  • validation now fails early if duplicate asset resources would collide inside the same generated HAK
  • duplicate resources split across different HAK groups remain warnings because later HAK order can still override earlier content at runtime
• sow-module now also uses sow-toolkit for topdata:
  • validate-topdata
  • build-topdata
  • compare-topdata
  • convert-topdata

During development, sibling repositories can point at ../sow-tools/tools/sow-toolkit or ../sow-tools/tools/sow-toolkit.exe. For pinned releases, each consumer repo can instead carry its own copy in its local tools/ directory.

Cross-Repository Behavior

The current ownership split is:

• sow-tools
  • owns the shared CLI and build logic
  • owns module build, extract, compare, and HAK manifest application logic
  • owns topdata validation, build, compare, conversion, and packaging logic
• sow-module
  • owns canonical module source under src/
  • owns canonical topdata authored data under topdata/
  • owns the user-facing module and topdata wrapper commands
• sow-assets
  • owns canonical binary asset source under assets/
  • owns generated HAK manifests and parts manifests under staging/
  • owns the user-facing asset wrapper commands

Consumer wrapper resolution is intentionally aligned:

1. Prefer a pinned local binary in tools/
2. Else prefer a sibling ../sow-tools checkout for local source development
3. Else install the latest published sow-tools release artifact

NWScript compilation behavior is also aligned and configurable under scripts.compiler:

1. Prefer SOW_NWN_SCRIPT_COMPILER when explicitly set
2. Else prefer a local compiler in tools/script-compiler/ or tools/
3. Else use a system-installed nwn_script_comp from PATH

When the compiler runs, the toolkit prepares NWN_HOME / NWN_USER_DIRECTORY automatically and tries to detect NWN_ROOT from common Steam locations, including custom Steam library folders, before an explicit override is required.

Release Automation

This repo publishes:

• sow-toolkit-linux-amd64.tar.gz
• sow-toolkit-windows-amd64.exe

Consumer repos can fetch those binaries into their local tools/ directory with:

• install-tool.sh on Linux/macOS
• install-tool.ps1 on Windows

TopData

The native topdata system is implemented in this repo and authored in sow-module.

High-level ownership split:

• sow-tools
  • owns the sow-toolkit implementation
  • owns topdata validation, build, compare, conversion, and dataset-resolution logic
• sow-module
  • owns canonical topdata authored data under topdata/data/
  • owns the user-facing topdata workflow scripts

For a human-readable front-to-back explanation of how the native topdata builder works and how to use it, see:

• sow-module/topdata/README.md

Important topdata contracts still live here under internal/topdata/ because they are code-adjacent implementation contracts, not user workflow docs.