Refactor the music generation / editing / credits pipeline so it exists entirely inside the shared toolkit and is no longer hardcoded or repository-specific.

## Status Snapshot

Current state as of 2026-05-13:

- This refactor is largely implemented.
- Music logic now lives under `internal/music/` with toolkit-owned CLI entry
  points and shared use from `build-haks`.
- Dataset-based config is supported and documented; the toolkit exposes
  `music list-datasets|scan|build|credits|validate|manifest|normalize`.
- Incremental credits artifact handling and compact/verbose log presentation are
  present in the current implementation.

Remaining work:

- finish retiring old `music.prefixes`-only assumptions in docs/tests after all
  consumer configs are fully dataset-native
- continue separating any remaining music-specific presentation rules from
  broader `build-haks` output policy as the build graph evolves

Goal: Turn the current Shadows Over Westgate-specific music pipeline into a reusable, repository-agnostic toolkit subsystem that can operate across multiple projects and datasets with configuration-driven behavior.

# Requirements

## Architecture

- Move all music pipeline logic into the toolkit itself.
- Eliminate repository-specific assumptions, paths, naming conventions, or hardcoded behavior.
- Repositories should only provide configuration and assets.
- The toolkit should expose reusable commands/services for:
  - music scanning
  - metadata extraction
  - credits generation
  - track normalization
  - BMU generation/mapping
  - music editing/transcoding
  - manifest generation
  - cache generation
  - validation

## Configuration

Support configuration at multiple levels:

1. Global toolkit defaults
2. Repository-level configuration
3. Dataset-level overrides

Example hierarchy:

- toolkit defaults
- repo config
- dataset config
- per-task overrides

The system should merge configuration predictably.

## Datasets

A "dataset" should represent an isolated music collection/pipeline target.

Examples:

- envi/music/westgate
- modules/oc/music
- premium/music/bardpack

Each dataset should support:

- independent input/output paths
- naming rules
- normalization settings
- credits settings
- encoding/transcoding options
- cache behavior
- manifest generation
- validation rules

# Desired Features

## Reusable pipeline stages

Design the pipeline as composable stages.

Example:

- discover
- scan metadata
- normalize
- transcode
- generate identifiers
- build mappings
- generate credits
- write manifests
- validate outputs

Stages should:

- be independently callable
- support dry-run
- support incremental rebuilds
- support verbose/debug logging
- expose machine-readable results

## Incremental behavior

Avoid unnecessary rebuilds.

The pipeline should:

- detect unchanged inputs
- reuse generated artifacts
- avoid duplicate rescans
- avoid rewriting unchanged outputs
- report concise summaries

## Repository independence

Do not assume:

- NWN-specific paths
- westgate naming
- BMU naming formats
- specific cache directories
- specific manifest names

All of these should be configurable.

# CLI design

Design clean toolkit commands.

Example direction:

- toolkit music scan
- toolkit music build
- toolkit music credits
- toolkit music normalize
- toolkit music validate

Or:

- toolkit dataset build <dataset>

The exact CLI structure may change if a cleaner abstraction exists.

# Configuration examples

Support something like:

```yaml
music:
  defaults:
    normalization:
      enabled: true

  datasets:
    westgate:
      source: envi/music/westgate
      output: build/music/westgate

      naming:
        scheme: nwn_bmu

      credits:
        enabled: true
        output: .cache/credits/westgate

    premium_bardpack:
      source: premium/music/bardpack

      normalization:
        loudness: -14

      transcoding:
        format: ogg
```

# Implementation expectations

- Separate orchestration from implementation details.
- Separate scanning, transformation, and output stages cleanly.
- Introduce typed dataset definitions/config models.
- Centralize logging and cache management.
- Prefer declarative configuration over imperative repository scripts.
- Keep generated outputs deterministic.
- Preserve compatibility where possible through adapters/shims.
- Minimize repository glue code.

# Migration

- Migrate existing Shadows Over Westgate behavior onto the new system.
- Preserve existing outputs unless intentionally improved.
- Add compatibility wrappers if needed.
- Ensure old workflows continue functioning during transition.

# Deliverables

1. Proposed architecture
2. Configuration schema
3. CLI design
4. Migration strategy
5. Refactor plan
6. Example configs
7. Before/after repository responsibilities
8. Suggested module/package layout
9. Identification of reusable abstractions
10. Example logs/output

Focus on maintainability, reuse, composability, and repository independence.

# Initial project-needs analysis

## Current state

- Music processing is currently implemented in `internal/pipeline/music.go`.
- The music pipeline is not independently callable. It is invoked as part of:
  - `BuildHAKs` / `planOrBuildHAKs`
  - `plannedModuleHAKOrder`
- The current behavior assumes the repository has an asset tree and that music lives under `envi/music`.
- Convertible source formats are currently hardcoded to:
  - `.mp3`
  - `.ogg`
- Passthrough audio/resource formats are currently hardcoded to:
  - `.bmu`
  - `.wav`
- Generated BMU-like resources are staged under `.cache/music`, loaded into HAK resources, and then the staging root is deleted.
- Generated credits are written under `.cache/credits`.
- Credits inventory is written to `.cache/credits/credits.json`.
- Authored credit overlays are discovered by walking the whole configured assets directory for `CREDITS.md`.
- A per-directory `CREDITS.md` can override probed metadata and manually pin the generated output filename.
- FFmpeg and ffprobe are resolved from:
  - `SOW_FFMPEG`
  - `SOW_FFPROBE`
  - `PATH`
- The ffmpeg command currently strips metadata and writes MP3-compatible audio bytes to a file named `.bmu`.
- Output stems are constrained to 16 characters, lower-case ASCII letters, digits, and underscores.
- Generated output names are reserved against every existing asset stem, not just music files.
- Current generated credits writes are already incremental: desired files are rendered in memory, unchanged files are not rewritten, and stale generated artifacts are removed.
- App-layer output for music is currently coupled to `build-haks` summary output in `internal/app/app.go`.

## Existing behavior to preserve during migration

- `build-haks` must continue to convert configured music sources into HAK resources without requiring repository scripts.
- Existing Shadows Over Westgate outputs should remain stable by default:
  - source root: `envi/music`
  - westgate prefix mapping: `envi/music/westgate -> mus_wg_`
  - generated output extension: `.bmu`
  - generated cache inventory: `.cache/credits/credits.json`
  - generated credits markdown shape and columns
  - manual `CREDITS.md` overlay semantics
  - duplicate output-name collision behavior
  - 16-character NWN resource stem limit
- Existing concise/verbose `build-haks` log behavior should keep working.
- Existing environment variable overrides for ffmpeg/ffprobe should continue to work as compatibility aliases.
- Machine-readable generated artifacts should remain JSON unless there is a separate explicit migration contract.

## Current code areas affected

- `internal/project/project.go`
  - `Config.Music` currently only supports `prefixes`.
  - `ValidateLayout` calls `validateMusicConfig`.
  - `AssetExtensions` currently defines which music files appear in project inventory.
  - `Scan` only scans the configured assets directory, so datasets outside `paths.assets` need a deliberate design.
- `internal/pipeline/music.go`
  - all scanning, naming, metadata, conversion, credits, cache, and cleanup logic is currently one private pipeline helper.
  - hardcoded assumptions include `envi/music`, `.cache/music`, `.cache/credits`, `CREDITS.md`, BMU output, ffmpeg args, metadata tags, and NWN stem rules.
- `internal/pipeline/build.go`
  - HAK planning/building depends on `prepareMusicAssets`.
  - plan-only HAK workflows can still invoke music probing/conversion today through HAK planning paths.
- `internal/app/app.go`
  - `build-haks` owns the only user-facing music summary.
  - New music commands should reuse the same logging/verbosity model instead of inventing unrelated output.
- `internal/pipeline/pipeline_test.go`
  - currently has regression coverage for conversion, generated credits, manual overlay, inventory inclusion, staging cleanup, and unchanged second pass.
- `internal/app/app_test.go`
  - currently checks normal versus verbose mapping output.
- `README.md`
  - currently documents only `music.prefixes`.

# Design constraints and decisions

## Dataset roots

Datasets should explicitly state where they read from and where generated artifacts go.

Important distinction:

- `source` is the authored input root for a dataset.
- `output` is the generated publish/staging root for transformed audio resources.
- `credits.output` is where generated human-readable credits go.
- `manifest.output` is where generated machine-readable dataset state goes.
- `cache.root` is where reusable implementation caches live.

Do not assume dataset sources are under `paths.assets`. Support both:

- asset-relative dataset roots for existing HAK packaging behavior
- project-relative dataset roots for future non-HAK or external datasets

For HAK builds, the pipeline must know whether generated outputs are meant to be injected into the project asset inventory and under which resource-relative path.

## NWN-specific behavior

The toolkit can support NWN conventions, but they must be selected by configuration instead of implicit global behavior.

Suggested built-in profiles:

- `nwn_bmu`
  - output extension: `.bmu`
  - max stem length: 16
  - allowed stem characters: `[a-z0-9_]`
  - lower-case output names
  - default ffmpeg encoder compatible with current behavior
- `passthrough`
  - no renaming unless configured
  - no transcoding unless configured
- `audio_export`
  - configurable format, stem length, and codec rules without NWN resource constraints

Do not make `nwn_bmu` the only naming implementation.

## Configuration merge semantics

Merge configuration in this order:

1. toolkit defaults
2. repository `music.defaults`
3. dataset definition
4. command-line/per-task overrides

Merge rules should be explicit:

- scalar fields replace inherited values
- maps merge by key
- lists replace by default unless a field is documented as appendable
- empty strings mean "not set", not "clear this inherited field"
- booleans that need tri-state inheritance should use pointer/nullable semantics internally
- command-line overrides should be represented in result metadata so logs can show effective behavior

Reject unknown fields via strict YAML decode, matching the rest of the project config philosophy.

## Compatibility config shape

The current config:

```yaml
music:
  prefixes:
    envi/music/westgate: mus_wg_
```

should be treated as a compatibility shorthand equivalent to:

```yaml
music:
  datasets:
    envi_music:
      source: envi/music
      package:
        mode: hak_asset
        output_root: envi/music
      naming:
        scheme: nwn_bmu
        prefixes:
          westgate: mus_wg_
```

Exact normalized dataset ids may differ, but the migration must define how old `music.prefixes` maps into the new dataset model.

## Proposed expanded schema

Example target:

```yaml
music:
  tools:
    ffmpeg: ""   # optional; fallback to SOW_FFMPEG then PATH
    ffprobe: ""  # optional; fallback to SOW_FFPROBE then PATH

  defaults:
    discover:
      include:
        - "**/*.mp3"
        - "**/*.ogg"
      passthrough:
        - "**/*.bmu"
        - "**/*.wav"
      exclude: []
      credits_file: CREDITS.md

    naming:
      scheme: nwn_bmu
      max_stem_length: 16
      prefix: ""
      prefixes: {}
      drop_words: []
      reserved_stems_from_assets: true

    metadata:
      enabled: true
      prefer_overlay: true
      probe_tags:
        title:
          - title
        artist:
          - artist
          - album_artist
          - composer
        rights:
          - license
          - license_url
          - copyright

    transcoding:
      enabled: true
      output_extension: .bmu
      codec: libmp3lame
      format: mp3
      sample_rate: 44100
      channels: 2
      bitrate: 192k
      strip_metadata: true
      ffmpeg_args: []

    credits:
      enabled: true
      output: .cache/credits
      inventory: .cache/credits/credits.json
      format: nwn_markdown_table

    cache:
      root: .cache/music
      mode: content_hash
      keep_staging: false

    manifest:
      enabled: true
      output: .cache/music/manifest.json
      format: json

    validation:
      require_credits_for_generated: false
      reject_unknown_overlay_rows: true
      reject_output_collisions: true

  datasets:
    westgate:
      source: envi/music/westgate
      output: envi/music/westgate
      package:
        mode: hak_asset
      naming:
        prefix: mus_wg_
```

Schema names can change during implementation, but the final shape should preserve these concepts.

# Pipeline architecture

## Suggested package layout

Prefer a dedicated internal subsystem instead of expanding `internal/pipeline/music.go`.

Suggested layout:

```text
internal/music/
  config.go        typed config, defaults, merge, validation
  dataset.go       resolved dataset model and effective settings
  discover.go      source discovery and file classification
  metadata.go      ffprobe integration and metadata normalization
  credits.go       markdown parsing/rendering and inventory generation
  naming.go        naming schemes, collision detection, compatibility prefix logic
  transcode.go     ffmpeg integration and output writing
  cache.go         content hashing and incremental state
  manifest.go      machine-readable dataset/stage manifests
  validate.go      dataset validation diagnostics
  pipeline.go      stage orchestration
  result.go        machine-readable stage/build result types
```

Then keep `internal/pipeline` responsible for HAK/module orchestration and call `internal/music` through a narrow adapter.

## Stage model

Each stage should have:

- typed input
- typed output
- deterministic ordering
- dry-run support where it can avoid writes
- changed/unchanged reporting
- structured diagnostics
- stable machine-readable result data

Suggested stages:

1. Resolve datasets and effective config.
2. Discover candidate files.
3. Load authored overlays.
4. Probe source metadata.
5. Plan output names and detect collisions.
6. Plan transformations.
7. Reuse cached outputs or transcode changed files.
8. Render generated credits.
9. Write dataset manifest/inventory.
10. Return generated package resources to the HAK pipeline.
11. Validate outputs and report diagnostics.

## Result model

Music command results should be structured enough for both CLI output and tests.

Include:

- dataset id
- source root
- output root
- files discovered
- files converted
- files reused from cache
- passthrough files
- skipped files
- generated resources
- credit artifacts
- manifest paths
- changed files
- warnings/errors
- source-to-output mappings
- effective naming/transcoding profile

Do not require the app layer to parse free-form progress strings.

# Incremental and cache behavior

## Content identity

Incremental rebuilds should be based on content and effective configuration, not only mtimes.

Cache keys should include:

- source content hash
- source relative path
- effective dataset id
- effective naming settings
- effective transcoding settings
- ffmpeg command/profile version
- toolkit music pipeline version
- manual overlay data that affects output metadata/name

Changing an output prefix, codec, bitrate, overlay output filename, or metadata rendering rules must invalidate the relevant planned output.

## Output writes

- Avoid rewriting unchanged credits, manifests, and transformed outputs.
- Write through temporary files and atomic rename where practical.
- Preserve deterministic JSON indentation and key ordering.
- Remove stale generated artifacts inside owned generated roots.
- Never remove authored source files.
- Never remove files outside configured owned cache/output roots.

## Plan-only behavior

Current HAK planning can trigger conversion work. The refactor should define stricter behavior:

- `music scan` and HAK `--plan-only` should not transcode by default.
- They may probe metadata only if needed and allowed.
- They should be able to produce planned mappings without writing transformed audio.
- If actual resource size is needed for HAK chunk planning, use cached transformed size when available and report when exact planning requires a build.

# CLI design expansion

Add first-class commands while preserving `build-haks` compatibility.

Suggested commands:

```text
sow-toolkit music list-datasets
sow-toolkit music scan [--dataset <id>] [--json]
sow-toolkit music build [--dataset <id>] [--dry-run] [--force]
sow-toolkit music credits [--dataset <id>] [--check]
sow-toolkit music validate [--dataset <id>] [--json]
sow-toolkit music manifest [--dataset <id>] [--check]
```

Integration commands:

```text
sow-toolkit build-haks
sow-toolkit build-haks --skip-music
sow-toolkit build-haks --music-dataset <id>
```

Command output should follow the existing log refactor direction:

- normal mode: phase summaries and counts
- verbose mode: mappings and per-file actions
- debug mode: resolved config, tool paths, ffmpeg args, cache keys
- quiet mode: final summary only
- `--json`: machine-readable result on stdout and human diagnostics on stderr

# Validation requirements

Validate repository config:

- duplicate dataset ids
- invalid dataset source/output paths
- absolute paths unless explicitly allowed
- path traversal outside project root for owned outputs/caches
- duplicate or ambiguous prefix mappings
- prefix too long for selected naming scheme
- unsupported output extension for selected package mode
- invalid codec/profile fields
- impossible inherited config combinations
- `package.mode: hak_asset` without an asset-relative output root
- unknown naming/transcoding/credits/manifest schemes

Validate discovered music:

- source files with unsupported extensions
- duplicate case-insensitive output stems
- output name collision with existing assets and generated outputs
- manual overlay rows referencing unknown originals or outputs
- manual overlay output filenames violating selected naming scheme
- metadata probe failures
- source paths differing only by case on case-insensitive targets
- generated resource names that exceed NWN constraints
- generated output extension/resource type compatibility

Validation severity should distinguish:

- errors that block deterministic build output
- warnings for incomplete credits or missing optional metadata
- info for skipped files and unchanged caches

# Fringe considerations

## Credits and licensing

- Preserve authored `CREDITS.md` files as source overlays.
- Generated credits should clearly identify generated versus authored sources in inventory.
- Missing artist/title/license can be warning-level or error-level per dataset.
- Overlay parsing should keep accepting backtick-wrapped filenames.
- Markdown parsing should continue to support escaped pipes and `<br>` line breaks.
- Generated credits should be deterministic across platforms.
- Consider whether aggregate credits inventory should include datasets disabled for build but enabled for credits.

## Filename and path edge cases

- Normalize separators to forward slashes in config, manifests, and logs.
- Treat paths case-insensitively for resource collision checks.
- Reject NUL bytes and unsafe relative paths.
- Define behavior for spaces, Unicode, apostrophes, brackets, years, and punctuation in stems.
- Keep existing slug/drop-word behavior for `nwn_bmu` compatibility, but make drop words configurable.
- Avoid Windows-reserved names and problematic trailing dots/spaces if outputs may be checked out or built on Windows.
- Detect collisions after truncation and suffixing.
- Validate suffixing still respects max stem length.

## Audio tooling

- Keep `SOW_FFMPEG` and `SOW_FFPROBE` as compatibility fallbacks.
- Add config-level tool paths for repositories that pin local toolchain binaries.
- Expose ffmpeg/ffprobe version in debug output and manifest metadata.
- Capture ffmpeg failures with enough context to diagnose source file, dataset, and command profile.
- Do not print enormous ffmpeg logs in normal mode unless the command fails.
- Decide whether `.ogg -> .bmu` should continue transcoding or become configurable passthrough for datasets that want native OGG output.

## HAK/resource integration

- Generated music resources must enter collision detection alongside authored resources.
- Source `.mp3`/`.ogg` files that are converted must be omitted from HAK packaging unless config says otherwise.
- Passthrough `.bmu` and `.wav` handling should remain distinct from generated outputs.
- Resource sorting should stay deterministic and reuse existing HAK resource ordering rules.
- Generated music should be represented in build manifests with enough source provenance to debug.
- HAK include globs should match generated output paths, not source paths, unless explicitly configured otherwise.

## Cross-repository behavior

- A repository with no `music` config should not pay a heavy ffmpeg/ffprobe cost.
- A repository with authored `.bmu` files but no generated music config should preserve current passthrough packaging.
- Multiple datasets should be able to target different output roots without collisions.
- Multiple repositories should be able to use different naming schemes without code changes.
- Dataset config should be portable between Linux and Windows path conventions.
- The toolkit should not assume Shadows Over Westgate names, HAK names, prefixes, or source layout.

# Migration strategy

## Phase 1: Extract without behavior changes

- Move existing private music functions from `internal/pipeline/music.go` into `internal/music`.
- Keep the current `prepareMusicAssets` adapter in `internal/pipeline` to avoid broad HAK changes.
- Preserve every existing test expectation.
- Add focused unit tests around moved naming, overlay, metadata, and credits functions.

## Phase 2: Introduce dataset config

- Extend `project.MusicConfig` with typed defaults and datasets.
- Keep `music.prefixes` as deprecated compatibility shorthand.
- Implement config normalization from legacy prefixes into a default dataset.
- Add strict validation for dataset ids, paths, naming, and prefix constraints.
- Update README examples.

## Phase 3: Add first-class music commands

- Wire `music scan`, `music build`, `music credits`, and `music validate` into `internal/app`.
- Reuse existing verbosity levels and summary formatting.
- Add JSON output support if the command result model is ready.
- Ensure `build-haks` calls the same service API as the music commands.

## Phase 4: Incremental generated output cache

- Replace temporary-only `.cache/music` staging with a content-addressed cache when configured.
- Keep cleanup behavior compatible for existing builds unless `cache.keep_staging` is enabled.
- Avoid conversion during dry-run/plan-only unless exact output artifacts are explicitly requested.
- Add stale generated artifact cleanup scoped only to owned roots.

## Phase 5: Repository migration

- Convert Shadows Over Westgate config from `music.prefixes` to explicit dataset config.
- Keep compatibility warning for one release window.
- Update consumer wrapper scripts only if command names change.
- Verify existing generated HAK manifests and credits inventories remain stable or document intentional diffs.

# Test plan

Add or preserve tests for:

- legacy `music.prefixes` compatibility
- explicit dataset config equivalent to current Westgate behavior
- dataset source outside `envi/music`
- multiple datasets with independent prefixes and outputs
- no music config with authored `.bmu` passthrough assets
- manual overlay output filename validation
- manual overlay references unknown source/output
- duplicate generated output collisions
- collision with existing authored asset stem
- max stem length and suffix collision behavior
- unchanged second build does not rewrite credits or manifests
- dry-run/plan-only does not transcode
- missing ffmpeg/ffprobe failure messages
- configured ffmpeg/ffprobe paths
- normal, verbose, quiet, debug, and JSON CLI output
- Windows-style path separators in config normalize deterministically
- generated credits and inventory are deterministic

# Acceptance criteria

- Music functionality is reusable through a dedicated toolkit subsystem.
- No repository-specific path or prefix is hardcoded in music implementation.
- Existing `build-haks` behavior remains compatible for Shadows Over Westgate.
- First-class music commands can scan, build, generate credits, and validate a configured dataset.
- Dataset configuration can express current Westgate behavior and at least one non-Westgate layout.
- Generated outputs are deterministic and unchanged files are not rewritten.
- Machine-readable artifacts remain JSON.
- Logs are concise by default and detailed under verbose/debug.
- Tests cover the compatibility path and the new dataset-driven path.
