sow-tools/README.md

# sow-tools

Shared NWN build tooling for Shadows Over Westgate.

This repository owns the Go-based `sow-toolkit` binary used by the separate module and assets repositories.

## Build

Build the development binary into `tools/sow-toolkit`:

```bash
./build-tool.sh
```

On Windows:

```powershell
.\build-tool.ps1
```

Both wrappers use `.cache/go-build` for the Go build cache and skip rebuilding
when the sources are older than the existing development binary.

Or run directly with Go:

```bash
GOCACHE="$PWD/.cache/go-build" go run ./cmd/nwn-tool --help
```

When changing native topdata id allocation, lockfile persistence, generated
families, or null-and-relocate behavior, run the repeat-build determinism test:

```bash
go test ./internal/topdata -run TestBuildNativeKeepsGeneratedIDsStableAcrossRepeatedBuilds
```

That test builds the same topdata project ten times in one process and asserts
that relocated/generated ids stay pinned across every pass.

## Repository Layout

```text
cmd/nwn-tool/      CLI entrypoint
internal/app/      command wiring and console UX
internal/erf/      ERF/MOD/HAK reading and writing
internal/gff/      canonical GFF JSON and binary conversion
internal/pipeline/ build, extract, compare, and manifest workflows
internal/project/  config loading and repo scanning
internal/topdata/  native topdata, wiki, and autogen workflows
internal/validator/validation and diagnostics
tools/             built development binary output
```

## Commands

The shared CLI exposes these project utilities:

```bash
sow-toolkit build
sow-toolkit build-module
sow-toolkit build-haks [--hak <name>] [--archive <name>] [--source-manifest <path>] [--plan-only] [--skip-music] [--music-dataset <id>]
sow-toolkit extract [<build-relative-archive-or-glob> ...]
sow-toolkit validate
sow-toolkit compare
sow-toolkit apply-hak-manifest [<manifest-path>]
sow-toolkit config <validate|effective|inspect|explain|sources>
sow-toolkit music <list-datasets|scan|build|credits|validate|manifest|normalize>
sow-toolkit validate-topdata
sow-toolkit build-topdata [--force] [--wiki]
sow-toolkit build-top-package [--force]
sow-toolkit compare-topdata
sow-toolkit convert-topdata <2da-to-json|2da-to-module|json-to-2da> ...
sow-toolkit build-wiki [--force]
sow-toolkit deploy-wiki [--source-dir <path>] [--endpoint <url>] [--token <token>] [--version <ver>] [--namespace <ns>] [--category <namespace=cid>] [--manifest <path>] [--dry-run] [--create] [--force]
sow-toolkit build-changelog [--config <path>] [--output <path>] [--current-tag <tag>] [--previous-tag <tag>] [--api-base-url <url>] [--token <token>]
```

Global logging flags `--quiet`, `--verbose`, and `--debug` may be passed after
the command name. Long options that take values accept either `--flag value` or
`--flag=value`; empty inline values such as `--dataset=` are rejected.

## Repository Configuration

Human-authored consumer repository configuration is YAML. The toolkit discovers
configuration in this order:

1. `nwn-tool.yaml`
2. `nwn-tool.yml`
3. legacy `nwn-tool.json`

YAML is canonical. If both YAML and legacy JSON files are present, the YAML file
wins. Legacy `nwn-tool.json` still loads for migration, but commands print a
warning and should be updated to `nwn-tool.yaml`.

Example:

```yaml
module:
  name: Shadows Over Westgate
  resref: sow_module
  description: Shadows Over Westgate asset pipeline.
  hak_order:
    - sow_top
    - group:sow_over
    - group:sow_core

paths:
  assets: content
  build: build

music:
  # Legacy shorthand — automatically normalized into a dataset.
  prefixes:
    envi/music/westgate: mus_wg_

  # Explicit dataset config (replaces prefixes above).
  tools:
    ffmpeg: tools/ffmpeg
    ffprobe: tools/ffprobe
  defaults:
    stage_root: "{paths.cache}/music"
    credits_root: "{paths.cache}/credits"
    credits_overlay: CREDITS.md
    max_stem_length: 16
    naming_scheme: nwn_bmu
    output_extension: .bmu
    convert_extensions: [.flac, .m4a, .mp3, .ogg, .wav]
  datasets:
    westgate:
      source: envi/music/westgate
      output: envi/music/westgate
      prefix: mus_wg_
      package_mode: hak_asset
```

Resolved configuration can be inspected without scanning or building:

```bash
sow-toolkit config validate
sow-toolkit config effective --json
sow-toolkit config inspect paths.build
sow-toolkit config explain topdata.package_hak
sow-toolkit config sources
```

`config effective` prints the normalized values consumed by commands, including
toolkit defaults, YAML values, legacy status, active environment overrides, and
provenance. Sensitive override values such as tokens are reported as set without
printing the secret.

Common configurable defaults:

```yaml
paths:
  source: src
  assets: assets
  build: build
  cache: .cache
  tools: tools

outputs:
  module_archive: "{module.resref}.mod"
  hak_manifest: haks.json
  hak_archive: "{hak.name}.hak"

build:
  keep_existing_haks: false

generated_assets:
  topdata_2da:
    - id: parts
      source: topdata
      output: "{paths.cache}/generated-assets/parts-2da"
      include_datasets: [parts/**]
      package_root: part
      autogen:
        id: parts
        mode: parts_rows
        root: part
        include: ["**/*.mdl"]
        derive:
          kind: trailing_numeric_suffix
          group_from: first_path_segment
        parts_rows:
          null_undiscovered_rows: true
          row_defaults:
            COSTMODIFIER: "0"
          acbonus:
            default:
              strategy: ascending_row_id_sort_key
              divisor: 100
              format: "%.2f"
            datasets:
              chest:
                strategy: fixed
                value: "0.00"

inventory:
  source_extensions: []      # empty means built-in NWN source extensions
  asset_extensions: []       # empty means built-in NWN asset extensions
  source_json_pattern: "{resref}.{extension}.json"

validation:
  profile: nwn_module
  builtin_script_prefixes: [nw_, x0_, x1_, x2_, x3_, ga_, gc_, gen_, gui_, nwg_, ta_]
  required_fields:
    ifo: [Mod_Name]

scripts:
  source_dir: scripts
  cache: "{paths.cache}/ncs"
  compiler:
    path: ""
    env:
      path: SOW_NWN_SCRIPT_COMPILER
      nwn_root: [SOW_NWN_ROOT, NWN_ROOT]
      nwn_user_directory: [SOW_NWN_USER_DIRECTORY, NWN_HOME, NWN_USER_DIRECTORY]
    search:
      - "{paths.tools}/script-compiler/nwn_script_comp"
      - "{paths.tools}/script-compiler/nwn_script_comp.exe"
      - "{paths.tools}/nwn_script_comp"
      - "{paths.tools}/nwn_script_comp.exe"
      - PATH:nwn_script_comp

extract:
  layout: nwn_canonical_json
  archives:
    - "{module.resref}.mod"
  cleanup_stale: true
  consume_archives: false
  ignore_extensions: []
  delete_module_archive_after_success: false # deprecated compatibility alias

topdata:
  build: "{paths.build}/topdata"
  compiled_2da_dir: 2da
  compiled_tlk: sow_tlk.tlk
  package_hak: sow_top.hak
  package_tlk: sow_tlk.tlk
  wiki:
    output_root: wiki
    pages_dir: pages
    state_file: state.json
    managed_namespaces: [classes, feat, itemtypes, races, skills, spells, meta]
    deploy_manifest: .wiki_deploy_manifest.json
    deploy_edit_summary: Auto-generated from native builder
    title_prefix_min_length: 3

autogen:
  cache:
    root: "{paths.cache}"
    max_age: 1h
    refresh_env: SOW_AUTOGEN_MANIFEST_REFRESH
  release_source:
    provider: gitea
    server_url_env: SOW_ASSETS_SERVER_URL
    repo_env: SOW_ASSETS_REPO
```

`paths.source` and `paths.assets` must name real repository subtrees when they
are configured. They may be omitted for repositories that do not own that class
of source, but they must not resolve to the repository root (`.`). Output and
cache paths must stay relative to the repository root unless a specific setting
documents otherwise.

`generated_assets.topdata_2da` is for HAK repositories that need a small
native-topdata subset packaged as HAK resources. It builds only `.2da` output,
injects those generated files into `build-haks`, and rejects TLK-backed text
because this path does not produce TLK packages.

For `parts_rows` generated assets, `parts_rows.acbonus` can normalize final
ACBONUS values before module overrides are applied. The
`ascending_row_id_sort_key` strategy writes toolset ordering values from row
IDs so higher part rows can sort before lower rows in the toolset; the legacy
`descending_row_id_sort_key` strategy remains available for repositories that
already depend on it. Per-dataset policies such as `chest: fixed 0.00` can
override the default. When `null_undiscovered_rows` is true, authored part rows
without discovered model IDs are omitted from the collected row set so dense
2DA output writes them as `****` null rows. Authored source ACBONUS values are
not authoritative when this policy is configured; files under `parts/modules`
remain the final override layer.

Extraction is deliberately guarded because it writes source files from binary
archives. `extract.archives` is a list of glob patterns relative to
`paths.build`; the default is only the configured module `.mod`. HAK extraction
is opt-in, for example:

```yaml
extract:
  archives:
    - "{module.resref}.mod"
    - "sow_over*.hak"
```

Module resources require a configured `paths.source`; HAK assets require a
configured `paths.assets`; both roots must resolve to safe subtrees. If a root
is unset or resolves to the repository root, extraction fails before writing
instead of creating extension directories like `mdl/` or `png/` at the repo top
level. `.erf` files are rejected by extraction because they do not contain
`module.ifo`, so they cannot safely update module-level area membership.

`consume_archives: true` deletes the selected build archive files only after the
entire extraction succeeds. The older `delete_module_archive_after_success`
setting remains as a module-only compatibility alias when `consume_archives` is
not set. Stale cleanup is scoped to roots actually touched by extracted
resources, so the default module-only extraction does not prune assets.

Music defaults (used when not explicitly configured):

```yaml
music:
  defaults:
    stage_root: "{paths.cache}/music"
    credits_root: "{paths.cache}/credits"
    credits_overlay: CREDITS.md
    max_stem_length: 16
    naming_scheme: nwn_bmu
    output_extension: .bmu
    convert_extensions: [.flac, .m4a, .mp3, .ogg, .wav]
```

Music datasets define individual music collections with independent sources and
naming prefixes. `source` is the authored input root under `paths.assets`;
`output` is the generated HAK asset root matched by HAK include globs. If
`output` is omitted, it defaults to `source`. `package_mode: hak_asset` converts
configured source extensions into generated HAK resources; `package_mode: none`
scans and writes credits without adding generated HAK resources. The legacy
`music.prefixes` shorthand is automatically normalized into datasets.
`convert_extensions` is configuration-driven; the built-in default set is
`.flac`, `.m4a`, `.mp3`, `.ogg`, and `.wav`.
Those extensions are discovered from YAML within configured music dataset source
roots, so setting them in `music.defaults.convert_extensions` or a dataset
override is sufficient; you do not need a parallel
`inventory.asset_extensions` override just to make music scanning see them.

Music can also be run directly:

```bash
sow-toolkit music list-datasets
sow-toolkit music scan --dataset westgate
sow-toolkit music build --dataset westgate --dry-run
sow-toolkit music build --dataset westgate
sow-toolkit music credits --dataset westgate --check
sow-toolkit music validate --dataset westgate --json
sow-toolkit music manifest --dataset westgate
sow-toolkit music normalize --dataset westgate
```

`build-haks` uses the same music pipeline. Use `--plan-only` or music
`--dry-run` to compute mappings without transcoding. Use `build-haks
--skip-music` to package authored assets only, or `build-haks --music-dataset
westgate` to limit conversion to a configured dataset.

Generated and machine-readable artifacts remain JSON, including HAK manifests,
topdata datasets, caches, credits inventories, build reports, and canonical GFF
documents. Nested metadata such as `topdata/templates/config.yaml` remains
valid when it is local workflow configuration rather than root repository
configuration.

## Wiki And Changelog Utilities

`build-topdata --wiki` or `build-wiki` renders wiki pages from the current
native topdata state. `deploy-wiki` publishes those pages to NodeBB. It reads
`NODEBB_API_ENDPOINT`, `NODEBB_API_TOKEN`, `GITHUB_REF_NAME`, and
`NODEBB_WIKI_CATEGORIES` when equivalent flags are omitted. Category mappings
use `namespace=cid`, for example:

```bash
sow-toolkit deploy-wiki --dry-run --namespace spells --category spells=42
```

When a deploy manifest is missing but matching wiki pages already exist in
NodeBB, `deploy-wiki` checks the Westgate Wiki namespace directory API and
adopts those topic/post mappings before planning updates. `--create` is still
required for genuinely new pages that have no existing remote topic. If NodeBB
rejects a create because the clean wiki URL already exists, `deploy-wiki`
re-queries the namespace by page leaf, adopts the matching topic, and updates it
instead. Updates and stale-page archives acquire a Westgate Wiki edit lock
before saving, then send the returned `wikiEditLockToken` with the NodeBB post
edit request.

For newly created NodeBB topics, `topdata.wiki.title_prefix_min_length`
controls when the deployer prefixes the namespace in the topic title. The
default is `3`, so one- and two-character page titles are created as
`Namespace: Title`, while titles with three or more characters keep the page
title alone. Existing bot-managed pages with older prefixed titles are renamed
on later deploy passes when their current title no longer matches this policy.

`build-changelog` renders a release changelog from tagged pushes. Pull-request
style commits keep their PR links and authors from the Gitea API; direct pushes
are rendered as commit links using the commit author. It defaults to
`scripts/changelog.json`, reads Gitea tokens from `GITEA_API_TOKEN`,
`GITEA_TOKEN`, or `SOW_TOOLS_TOKEN`, and writes to stdout unless `--output` is
provided.

## Intended Consumers

- `sow-module` uses `sow-toolkit` for `build`, `build-module`, `extract`, `validate`, `compare`, and `apply-hak-manifest`
- `sow-assets` uses `sow-toolkit` for `build-haks`, music processing, changelog generation, and validation
  - validation now fails early if duplicate asset resources would collide inside the same generated HAK
  - duplicate resources split across different HAK groups remain warnings because later HAK order can still override earlier content at runtime
- `sow-module` now also uses `sow-toolkit` for topdata:
  - `validate-topdata`
  - `build-topdata`
  - `build-top-package`
  - `compare-topdata`
  - `convert-topdata`
  - `build-wiki`
  - `deploy-wiki`

During development, sibling repositories can point at `../sow-tools/tools/sow-toolkit`
or `../sow-tools/tools/sow-toolkit.exe`. For pinned releases, each consumer repo can
instead carry its own copy in its local `tools/` directory.

## Cross-Repository Behavior

The current ownership split is:

- `sow-tools`
  - owns the shared CLI and build logic
  - owns module build, extract, compare, and HAK manifest application logic
  - owns topdata validation, build, compare, conversion, and packaging logic
- `sow-module`
  - owns canonical module source under `src/`
  - owns canonical topdata authored data under `topdata/`
  - owns the user-facing module and topdata wrapper commands
- `sow-assets`
  - owns canonical binary asset source under `assets/`
  - owns generated HAK manifests and HAK-side parts 2DA assets
  - owns the user-facing asset wrapper commands

Consumer wrapper resolution is intentionally aligned:

1. Prefer a pinned local binary in `tools/`
2. Else prefer a sibling `../sow-tools` checkout for local source development
3. Else install the latest published `sow-tools` release artifact

NWScript compilation behavior is also aligned and configurable under
`scripts.compiler`:

1. Prefer `SOW_NWN_SCRIPT_COMPILER` when explicitly set
2. Else prefer a local compiler in `tools/script-compiler/` or `tools/`
3. Else use a system-installed `nwn_script_comp` from `PATH`

When the compiler runs, the toolkit prepares `NWN_HOME` / `NWN_USER_DIRECTORY`
automatically and tries to detect `NWN_ROOT` from common Steam locations, including
custom Steam library folders, before an explicit override is required.


## Release Automation

This repo publishes:

- `sow-toolkit-linux-amd64.tar.gz`
- `sow-toolkit-windows-amd64.exe`

Consumer repos can fetch those binaries into their local `tools/` directory with:

- `install-tool.sh` on Linux/macOS
- `install-tool.ps1` on Windows

## TopData

The native topdata system is implemented in this repo and authored in `sow-module`.

High-level ownership split:

- `sow-tools`
  - owns the `sow-toolkit` implementation
  - owns topdata validation, build, compare, conversion, and dataset-resolution logic
- `sow-module`
  - owns canonical topdata authored data under `topdata/data/`
  - owns the user-facing topdata workflow scripts

For a human-readable front-to-back explanation of how the native topdata builder works
and how to use it, see:

- [sow-module/topdata/README.md](../sow-module/topdata/README.md)

Important topdata contracts still live here under `internal/topdata/` because they are
code-adjacent implementation contracts, not user workflow docs.