sow-tools/README.md

# sow-tools

Shared NWN build tooling for Shadows Over Westgate.

This repository owns the Go-based `sow-toolkit` binary used by the separate module and assets repositories.

## Build

Build the development binary into `tools/sow-toolkit`:

```bash
./build-tool.sh
```

On Windows:

```powershell
.\build-tool.ps1
```

Both wrappers use `.cache/go-build` for the Go build cache and skip rebuilding
when the sources are older than the existing development binary.

Or run directly with Go:

```bash
GOCACHE="$PWD/.cache/go-build" go run ./cmd/nwn-tool --help
```

When changing native topdata id allocation, lockfile persistence, generated
families, or null-and-relocate behavior, run the repeat-build determinism test:

```bash
go test ./internal/topdata -run TestBuildNativeKeepsGeneratedIDsStableAcrossRepeatedBuilds
```

That test builds the same topdata project ten times in one process and asserts
that relocated/generated ids stay pinned across every pass.

## Repository Layout

```text
cmd/nwn-tool/      CLI entrypoint
internal/app/      command wiring and console UX
internal/erf/      ERF/MOD/HAK reading and writing
internal/gff/      canonical GFF JSON and binary conversion
internal/pipeline/ build, extract, compare, and manifest workflows
internal/project/  config loading and repo scanning
internal/topdata/  native topdata, wiki, and autogen workflows
internal/validator/validation and diagnostics
tools/             built development binary output
```

## Commands

The shared CLI exposes these project utilities:

```bash
sow-toolkit build
sow-toolkit build-module
sow-toolkit build-haks [--hak <name>] [--archive <name>] [--source-manifest <path>] [--plan-only] [--skip-music] [--music-dataset <id>]
sow-toolkit extract [<build-relative-archive-or-glob> ...]
sow-toolkit validate
sow-toolkit compare
sow-toolkit apply-hak-manifest [<manifest-path>]
sow-toolkit config <validate|effective|inspect|explain|sources>
sow-toolkit music <list-datasets|scan|build|credits|validate|manifest|normalize>
sow-toolkit validate-topdata
sow-toolkit build-topdata [--force] [--wiki]
sow-toolkit build-top-package [--force]
sow-toolkit compare-topdata
sow-toolkit convert-topdata <2da-to-json|2da-to-module|json-to-2da> ...
sow-toolkit build-wiki [--force]
sow-toolkit deploy-wiki [--source-dir <path>] [--endpoint <url>] [--token <token>] [--version <ver>] [--namespace <ns>] [--category <namespace=cid>] [--manifest <path>] [--stale-policy <report|archive|purge>] [--dry-run] [--create] [--force] [--reset-managed-namespaces]
sow-toolkit build-changelog [--config <path>] [--output <path>] [--current-tag <tag>] [--previous-tag <tag>] [--api-base-url <url>] [--token <token>]
```

Global logging flags `--quiet`, `--verbose`, and `--debug` may be passed after
the command name. Long options that take values accept either `--flag value` or
`--flag=value`; empty inline values such as `--dataset=` are rejected.

## Repository Configuration

Human-authored consumer repository configuration is YAML. The toolkit discovers
configuration in this order:

1. `nwn-tool.yaml`
2. `nwn-tool.yml`
3. legacy `nwn-tool.json`

YAML is canonical. If both YAML and legacy JSON files are present, the YAML file
wins. Legacy `nwn-tool.json` still loads for migration, but commands print a
warning and should be updated to `nwn-tool.yaml`.

Example:

```yaml
module:
  name: Shadows Over Westgate
  resref: sow_module
  description: Shadows Over Westgate asset pipeline.
  hak_order:
    - sow_top
    - group:sow_over
    - group:sow_core

paths:
  assets: content
  build: build

music:
  # Legacy shorthand — automatically normalized into a dataset.
  prefixes:
    envi/music/westgate: mus_wg_

  # Explicit dataset config (replaces prefixes above).
  tools:
    ffmpeg: tools/ffmpeg
    ffprobe: tools/ffprobe
  defaults:
    stage_root: "{paths.cache}/music"
    credits_root: "{paths.cache}/credits"
    credits_overlay: CREDITS.md
    max_stem_length: 16
    naming_scheme: nwn_bmu
    output_extension: .bmu
    convert_extensions: [.flac, .m4a, .mp3, .ogg, .wav]
  datasets:
    westgate:
      source: envi/music/westgate
      output: envi/music/westgate
      prefix: mus_wg_
      package_mode: hak_asset
```

Resolved configuration can be inspected without scanning or building:

```bash
sow-toolkit config validate
sow-toolkit config effective --json
sow-toolkit config inspect paths.build
sow-toolkit config explain topdata.package_hak
sow-toolkit config sources
```

`config effective` prints the normalized values consumed by commands, including
toolkit defaults, YAML values, legacy status, active environment overrides, and
provenance. Sensitive override values such as tokens are reported as set without
printing the secret.

Common configurable defaults:

```yaml
paths:
  source: src
  assets: assets
  build: build
  cache: .cache
  tools: tools

outputs:
  module_archive: "{module.resref}.mod"
  hak_manifest: haks.json
  hak_archive: "{hak.name}.hak"

build:
  keep_existing_haks: false

generated_assets:
  topdata_2da:
    - id: parts
      source: topdata
      output: "{paths.cache}/generated-assets/parts-2da"
      include_datasets: [parts/**]
      package_root: part
      autogen:
        id: parts
        mode: parts_rows
        root: part
        include: ["**/*.mdl"]
        derive:
          kind: trailing_numeric_suffix
          group_from: first_path_segment
        parts_rows:
          row_defaults:
            COSTMODIFIER: "0"
          acbonus:
            default:
              strategy: ascending_row_id_sort_key
              divisor: 100
              format: "%.2f"
            datasets:
              chest:
                strategy: fixed
                value: "0.00"

inventory:
  source_extensions: []      # empty means built-in NWN source extensions
  asset_extensions: []       # empty means built-in NWN asset extensions
  source_json_pattern: "{resref}.{extension}.json"

validation:
  profile: nwn_module
  builtin_script_prefixes: [nw_, x0_, x1_, x2_, x3_, ga_, gc_, gen_, gui_, nwg_, ta_]
  required_fields:
    ifo: [Mod_Name]

scripts:
  source_dir: scripts
  cache: "{paths.cache}/ncs"
  compiler:
    path: ""
    env:
      path: SOW_NWN_SCRIPT_COMPILER
      nwn_root: [SOW_NWN_ROOT, NWN_ROOT]
      nwn_user_directory: [SOW_NWN_USER_DIRECTORY, NWN_HOME, NWN_USER_DIRECTORY]
    search:
      - "{paths.tools}/script-compiler/nwn_script_comp"
      - "{paths.tools}/script-compiler/nwn_script_comp.exe"
      - "{paths.tools}/nwn_script_comp"
      - "{paths.tools}/nwn_script_comp.exe"
      - PATH:nwn_script_comp

extract:
  layout: nwn_canonical_json
  archives:
    - "{module.resref}.mod"
  cleanup_stale: true
  consume_archives: false
  ignore_extensions: []
  delete_module_archive_after_success: false # deprecated compatibility alias

topdata:
  build: "{paths.build}/topdata"
  compiled_2da_dir: 2da
  compiled_tlk: sow_tlk.tlk
  package_hak: sow_top.hak
  package_tlk: sow_tlk.tlk
  value_encodings:
    - dataset: racialtypes/core
      column: AvailableSkinColors
      mode: packed_hex_list
      min: 0
      max: 175
      hex_width: 2
  wiki:
    output_root: wiki
    pages_dir: pages
    state_file: state.json
    managed_namespaces: [classes, feat, itemtypes, races, skills, spells, meta]
    deploy_manifest: .wiki_deploy_manifest.json
    deploy_edit_summary: Auto-generated from native builder
    visibility_file: visibility.yaml
    title_prefix_min_length: 3
    status_listing_scope: all

autogen:
  cache:
    root: "{paths.cache}"
    max_age: 1h
    refresh_env: SOW_AUTOGEN_MANIFEST_REFRESH
  release_source:
    provider: gitea
    server_url_env: SOW_ASSETS_SERVER_URL
    repo_env: SOW_ASSETS_REPO
```

`topdata.value_encodings` enables configured authoring sugar for compact 2DA
cell values. The `packed_hex_list` mode accepts authored values such as
`{"list": [0, {"range": [10, 12]}]}` and emits a single `0x...` string with
fixed-width hex chunks. The `alignment_hex_list` mode accepts two-letter
alignment codes such as `{"list": ["le", "CG"]}` and emits the corresponding
fixed-width hex chunks.

`topdata.value_defaults` applies configured fallback values to missing or
null-like cells before 2DA emission:

```yaml
topdata:
  value_defaults:
    - dataset: racialtypes/core
      column: ECL
      value: 0
```

`paths.source` and `paths.assets` must name real repository subtrees when they
are configured. They may be omitted for repositories that do not own that class
of source, but they must not resolve to the repository root (`.`). Output and
cache paths must stay relative to the repository root unless a specific setting
documents otherwise.

`generated_assets.topdata_2da` is for HAK repositories that need a small
native-topdata subset packaged as HAK resources. It builds only `.2da` output,
injects those generated files into `build-haks`, and rejects TLK-backed text
because this path does not produce TLK packages. Static generated groups can
omit `autogen.mode`; in that case the selected datasets are compiled directly,
and `package_root` plus the repository's HAK include patterns determine where
the resources are packaged.

For `cachedmodels_rows` generated assets, the configured autogen scan appends
discovered model stems to the selected `cachedmodels` dataset without `.mdl`
extensions and skips models already present in authored rows.

For `parts_rows` generated assets, `parts_rows.acbonus` can normalize final
ACBONUS values before module overrides are applied. The
`ascending_row_id_sort_key` strategy writes toolset ordering values from row
IDs so higher part rows can sort before lower rows in the toolset; the legacy
`descending_row_id_sort_key` strategy remains available for repositories that
already depend on it. Per-dataset policies such as `chest: fixed 0.00` can
override the default. Authored source ACBONUS values are not authoritative when
this policy is configured, except that authored `****` values remain null
unless model discovery activates the row. Dense gaps without authored rows or
discovered part models are emitted as `****` null rows. Files under
`parts/modules` remain the final override layer.

Extraction is deliberately guarded because it writes source files from binary
archives. `extract.archives` is a list of glob patterns relative to
`paths.build`; the default is only the configured module `.mod`. HAK extraction
is opt-in, for example:

```yaml
extract:
  archives:
    - "{module.resref}.mod"
    - "sow_over*.hak"
```

Module resources require a configured `paths.source`; HAK assets require a
configured `paths.assets`; both roots must resolve to safe subtrees. If a root
is unset or resolves to the repository root, extraction fails before writing
instead of creating extension directories like `mdl/` or `png/` at the repo top
level. `.erf` files are rejected by extraction because they do not contain
`module.ifo`, so they cannot safely update module-level area membership.

`consume_archives: true` deletes the selected build archive files only after the
entire extraction succeeds. The older `delete_module_archive_after_success`
setting remains as a module-only compatibility alias when `consume_archives` is
not set. Stale cleanup is scoped to roots actually touched by extracted
resources, so the default module-only extraction does not prune assets.

`topdata.wiki.status_listing_scope` controls the generated wiki-status detail
pages. The default `all` writes both per-namespace listings and aggregate
`meta:wikistatus:<status>` listings. Use `namespace` when aggregate status
lists would be too large for the target NodeBB post limit. Set
`topdata.wiki.status_pages: false` to omit status pages entirely; existing
tracked status pages then become stale deploy-manifest entries and can be
removed with `deploy-wiki --stale-policy purge`.

GFF JSON extraction can merge selected fields and lists from the existing source
file instead of replacing the whole extracted document. Rules are matched by
`target` relative to `paths.source`; files without a matching rule keep the
default overwrite behavior. For example:

```yaml
extract:
  merge:
    gff_json:
      - target: module/module.ifo.json
        preserve_fields:
          - Mod_Entry_Area
          - Mod_Entry_X
          - Mod_Entry_Y
          - Mod_Entry_Z
          - Mod_Entry_Dir_X
          - Mod_Entry_Dir_Y
        merge_lists:
          - field: Mod_Area_list
            key_field: Area_Name
            strategy: preserve_existing_order_append_new
```

`preserve_fields` copies matching root fields from the existing source document
into the extracted document before writing. The
`preserve_existing_order_append_new` list strategy keeps existing keyed list
order for entries still present in the extracted archive, uses extracted entries
as the value source, appends archive-only entries in archive order, and removes
existing entries that are no longer present in the archive.

Music defaults (used when not explicitly configured):

```yaml
music:
  defaults:
    stage_root: "{paths.cache}/music"
    credits_root: "{paths.cache}/credits"
    credits_overlay: CREDITS.md
    max_stem_length: 16
    naming_scheme: nwn_bmu
    output_extension: .bmu
    convert_extensions: [.flac, .m4a, .mp3, .ogg, .wav]
```

Music datasets define individual music collections with independent sources and
naming prefixes. `source` is the authored input root under `paths.assets`;
`output` is the generated HAK asset root matched by HAK include globs. If
`output` is omitted, it defaults to `source`. `package_mode: hak_asset` converts
configured source extensions into generated HAK resources; `package_mode: none`
scans and writes credits without adding generated HAK resources. The legacy
`music.prefixes` shorthand is automatically normalized into datasets.
`convert_extensions` is configuration-driven; the built-in default set is
`.flac`, `.m4a`, `.mp3`, `.ogg`, and `.wav`.
Those extensions are discovered from YAML within configured music dataset source
roots, so setting them in `music.defaults.convert_extensions` or a dataset
override is sufficient; you do not need a parallel
`inventory.asset_extensions` override just to make music scanning see them.

Music can also be run directly:

```bash
sow-toolkit music list-datasets
sow-toolkit music scan --dataset westgate
sow-toolkit music build --dataset westgate --dry-run
sow-toolkit music build --dataset westgate
sow-toolkit music credits --dataset westgate --check
sow-toolkit music validate --dataset westgate --json
sow-toolkit music manifest --dataset westgate
sow-toolkit music normalize --dataset westgate
```

`build-haks` uses the same music pipeline. Use `--plan-only` or music
`--dry-run` to compute mappings without transcoding. Use `build-haks
--skip-music` to package authored assets only, or `build-haks --music-dataset
westgate` to limit conversion to a configured dataset.

Generated and machine-readable artifacts remain JSON, including HAK manifests,
topdata datasets, caches, credits inventories, build reports, and canonical GFF
documents. Nested metadata such as `topdata/templates/config.yaml` remains
valid when it is local workflow configuration rather than root repository
configuration.

## Wiki And Changelog Utilities

`build-topdata --wiki` or `build-wiki` renders wiki pages from the current
native topdata state as Tiptap-compatible NodeBB HTML. Page templates are HTML
fragments; they do not need to include a visible title heading because deploy
uses generated `page-index.json` metadata for page titles and public paths.
`deploy-wiki` publishes those pages to NodeBB. It reads
`NODEBB_API_ENDPOINT`, `NODEBB_API_TOKEN`, `GITHUB_REF_NAME`, and
`NODEBB_WIKI_CATEGORIES` when equivalent flags are omitted. Category mappings
use `namespace=cid`, for example:

```bash
sow-toolkit deploy-wiki --dry-run --namespace spells --category spells=42
```

Namespace declarations in `topdata/wiki/namespaces.yaml` may set
`canonical_path` when a generated namespace lives below a fuller NodeBB category
path, for example `canonical_path: Wiki/Feats`. The `title` remains the display
label, while `canonical_path` drives generated `public_path` metadata, wiki
links, page-index entries, and deploy adoption matching.

When a deploy manifest is missing but matching wiki pages already exist in
NodeBB, `deploy-wiki` checks the Westgate Wiki namespace directory API and
adopts those topic/post mappings before planning updates. It does the same when
a cached manifest mapping points to a post NodeBB no longer has. Canonical
`wikiPath`/`canonicalPath` data is preferred during adoption. Nested generated
paths require that exact canonical path data; leaf/title compatibility matching
is only used where it cannot cross a canonical parent path. `--create` is still
required for genuinely new pages that have no existing remote topic. If NodeBB
rejects a create because the clean wiki URL already exists, `deploy-wiki`
re-queries the namespace by page leaf, adopts the matching topic, and updates it
instead. Pass `--debug` when deploying large wiki changes to emit planning and
live mutation progress. Updates and stale-page archives acquire a Westgate Wiki
edit lock before saving, then send the returned
`wikiEditLockToken` with the NodeBB post edit request. Creates, updates, and
archives write the same managed HTML to both NodeBB `content` and
`sourceContent`; the deploy manifest records `source_content_synced` so older
manifests receive a one-time body repair even when the generated hash is
unchanged. Stale generated pages are reported by default.
`--stale-policy archive` rewrites stale generated pages in place as archived
managed content. `--stale-policy purge` permanently removes only stale generated
topics already tracked in the wiki deploy manifest. Dry-run that policy before
the live run; it never discovers manual wiki pages from NodeBB for purge.
`--reset-managed-namespaces` is a stronger destructive mode for slug or path
standard migrations: it permanently removes every topic currently listed by the
configured managed namespace categories, ignores the previous deploy manifest,
and recreates the current generated pages. It requires `--create` when local
generated pages are present.

For newly created NodeBB topics, `topdata.wiki.title_prefix_min_length`
controls when the deployer prefixes the namespace in the topic title. The
default is `3`, so one- and two-character page titles are created as
`Namespace: Title`, while titles with three or more characters keep the page
title alone. Existing bot-managed pages with older prefixed titles are renamed
on later deploy passes when their current title no longer matches this policy.

`build-changelog` renders a release changelog from tagged pushes. Pull-request
style commits keep their PR links and authors from the Gitea API; direct pushes
are rendered as commit links using the commit author. It defaults to
`scripts/changelog.json`, reads Gitea tokens from `GITEA_API_TOKEN`,
`GITEA_TOKEN`, or `SOW_TOOLS_TOKEN`, and writes to stdout unless `--output` is
provided.

## Intended Consumers

- `sow-module` uses `sow-toolkit` for `build`, `build-module`, `extract`, `validate`, `compare`, and `apply-hak-manifest`
- `sow-assets` uses `sow-toolkit` for `build-haks`, music processing, changelog generation, and validation
  - validation now fails early if duplicate asset resources would collide inside the same generated HAK
  - duplicate resources split across different HAK groups remain warnings because later HAK order can still override earlier content at runtime
- `sow-module` now also uses `sow-toolkit` for topdata:
  - `validate-topdata`
  - `build-topdata`
  - `build-top-package`
  - `compare-topdata`
  - `convert-topdata`
  - `build-wiki`
  - `deploy-wiki`

During development, sibling repositories can point at `../sow-tools/tools/sow-toolkit`
or `../sow-tools/tools/sow-toolkit.exe`. For pinned releases, each consumer repo can
instead carry its own copy in its local `tools/` directory.

## Cross-Repository Behavior

The current ownership split is:

- `sow-tools`
  - owns the shared CLI and build logic
  - owns module build, extract, compare, and HAK manifest application logic
  - owns topdata validation, build, compare, conversion, and packaging logic
- `sow-module`
  - owns canonical module source under `src/`
  - owns canonical topdata authored data under `topdata/`
  - owns the user-facing module and topdata wrapper commands
- `sow-assets`
  - owns canonical binary asset source under `assets/`
  - owns generated HAK manifests and HAK-side parts 2DA assets
  - owns the user-facing asset wrapper commands

Consumer wrapper resolution is intentionally aligned:

1. Prefer a pinned local binary in `tools/`
2. Else prefer a sibling `../sow-tools` checkout for local source development
3. Else install the latest published `sow-tools` release artifact

NWScript compilation behavior is also aligned and configurable under
`scripts.compiler`:

1. Prefer `SOW_NWN_SCRIPT_COMPILER` when explicitly set
2. Else prefer a local compiler in `tools/script-compiler/` or `tools/`
3. Else use a system-installed `nwn_script_comp` from `PATH`

When the compiler runs, the toolkit prepares `NWN_HOME` / `NWN_USER_DIRECTORY`
automatically and tries to detect `NWN_ROOT` from common Steam locations, including
custom Steam library folders, before an explicit override is required.


## Release Automation

This repo publishes:

- `sow-toolkit-linux-amd64.tar.gz`
- `sow-toolkit-windows-amd64.exe`

Consumer repos can fetch those binaries into their local `tools/` directory with:

- `install-tool.sh` on Linux/macOS
- `install-tool.ps1` on Windows

## TopData

The native topdata system is implemented in this repo and authored in `sow-module`.

High-level ownership split:

- `sow-tools`
  - owns the `sow-toolkit` implementation
  - owns topdata validation, build, compare, conversion, and dataset-resolution logic
- `sow-module`
  - owns canonical topdata authored data under `topdata/data/`
  - owns the user-facing topdata workflow scripts

For a human-readable front-to-back explanation of how the native topdata builder works
and how to use it, see:

- [sow-module/topdata/README.md](../sow-module/topdata/README.md)

Important topdata contracts still live here under `internal/topdata/` because they are
code-adjacent implementation contracts, not user workflow docs.