ShadowsOverWestgate/sow-tools
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 38 Activity

Configuration Hardening

Browse Source 
Key changes:

  - Added internal/project/effective.go with normalized effective config, defaults, provenance, and
    JSON output.
  - Added config subcommands:
      - config validate
      - config effective [--json|--yaml]
      - config inspect [<key>]
      - config explain <key>
      - config sources
  - Added YAML schema fields for configurable outputs, cache roots, script cache, inventory extensions,
    topdata output/wiki paths, and autogen cache root.
  - Routed existing hardcoded paths through effective config wrappers for module archives, HAK
    manifests/archives, script cache, music cache/credits, topdata outputs, and autogen caches.
  - Added validation for unsafe generated output/cache paths.
  - Updated command descriptions to avoid fixed build/, .cache, sow_top, etc.
  - Added regression tests for defaults, provenance, deterministic effective config, YAML-controlled
    derivation, config commands, and invalid paths.
This commit is contained in:
archvillainette
2026-05-07 14:09:20 +02:00
parent 000d31ad24
commit 60d2de9f2d
12 changed files with 2481 additions and 62 deletions
Download Patch File Download Diff File Expand all files Collapse all files
MUSIC_REFACTOR_CONTRACT.md
+738
View File
@@ -0,0 +1,738 @@
Refactor the music generation / editing / credits pipeline so it exists entirely inside the shared toolkit and is no longer hardcoded or repository-specific.
Goal: Turn the current Shadows Over Westgate-specific music pipeline into a reusable, repository-agnostic toolkit subsystem that can operate across multiple projects and datasets with configuration-driven behavior.
# Requirements
## Architecture
- Move all music pipeline logic into the toolkit itself.
- Eliminate repository-specific assumptions, paths, naming conventions, or hardcoded behavior.
- Repositories should only provide configuration and assets.
- The toolkit should expose reusable commands/services for:
- music scanning
- metadata extraction
- credits generation
- track normalization
- BMU generation/mapping
- music editing/transcoding
- manifest generation
- cache generation
- validation
## Configuration
Support configuration at multiple levels:
1. Global toolkit defaults
2. Repository-level configuration
3. Dataset-level overrides
Example hierarchy:
- toolkit defaults
- repo config
- dataset config
- per-task overrides
The system should merge configuration predictably.
## Datasets
A "dataset" should represent an isolated music collection/pipeline target.
Examples:
- envi/music/westgate
- modules/oc/music
- premium/music/bardpack
Each dataset should support:
- independent input/output paths
- naming rules
- normalization settings
- credits settings
- encoding/transcoding options
- cache behavior
- manifest generation
- validation rules
# Desired Features
## Reusable pipeline stages
Design the pipeline as composable stages.
Example:
- discover
- scan metadata
- normalize
- transcode
- generate identifiers
- build mappings
- generate credits
- write manifests
- validate outputs
Stages should:
- be independently callable
- support dry-run
- support incremental rebuilds
- support verbose/debug logging
- expose machine-readable results
## Incremental behavior
Avoid unnecessary rebuilds.
The pipeline should:
- detect unchanged inputs
- reuse generated artifacts
- avoid duplicate rescans
- avoid rewriting unchanged outputs
- report concise summaries
## Repository independence
Do not assume:
- NWN-specific paths
- westgate naming
- BMU naming formats
- specific cache directories
- specific manifest names
All of these should be configurable.
# CLI design
Design clean toolkit commands.
Example direction:
- toolkit music scan
- toolkit music build
- toolkit music credits
- toolkit music normalize
- toolkit music validate
Or:
- toolkit dataset build <dataset>
The exact CLI structure may change if a cleaner abstraction exists.
# Configuration examples
Support something like:
```yaml
music:
defaults:
normalization:
enabled: true
datasets:
westgate:
source: envi/music/westgate
output: build/music/westgate
naming:
scheme: nwn_bmu
credits:
enabled: true
output: .cache/credits/westgate
premium_bardpack:
source: premium/music/bardpack
normalization:
loudness: -14
transcoding:
format: ogg
```
# Implementation expectations
- Separate orchestration from implementation details.
- Separate scanning, transformation, and output stages cleanly.
- Introduce typed dataset definitions/config models.
- Centralize logging and cache management.
- Prefer declarative configuration over imperative repository scripts.
- Keep generated outputs deterministic.
- Preserve compatibility where possible through adapters/shims.
- Minimize repository glue code.
# Migration
- Migrate existing Shadows Over Westgate behavior onto the new system.
- Preserve existing outputs unless intentionally improved.
- Add compatibility wrappers if needed.
- Ensure old workflows continue functioning during transition.
# Deliverables
1. Proposed architecture
2. Configuration schema
3. CLI design
4. Migration strategy
5. Refactor plan
6. Example configs
7. Before/after repository responsibilities
8. Suggested module/package layout
9. Identification of reusable abstractions
10. Example logs/output
Focus on maintainability, reuse, composability, and repository independence.
# Initial project-needs analysis
## Current state
- Music processing is currently implemented in `internal/pipeline/music.go`.
- The music pipeline is not independently callable. It is invoked as part of:
- `BuildHAKs` / `planOrBuildHAKs`
- `plannedModuleHAKOrder`
- The current behavior assumes the repository has an asset tree and that music lives under `envi/music`.
- Convertible source formats are currently hardcoded to:
- `.mp3`
- `.ogg`
- Passthrough audio/resource formats are currently hardcoded to:
- `.bmu`
- `.wav`
- Generated BMU-like resources are staged under `.cache/music`, loaded into HAK resources, and then the staging root is deleted.
- Generated credits are written under `.cache/credits`.
- Credits inventory is written to `.cache/credits/credits.json`.
- Authored credit overlays are discovered by walking the whole configured assets directory for `CREDITS.md`.
- A per-directory `CREDITS.md` can override probed metadata and manually pin the generated output filename.
- FFmpeg and ffprobe are resolved from:
- `SOW_FFMPEG`
- `SOW_FFPROBE`
- `PATH`
- The ffmpeg command currently strips metadata and writes MP3-compatible audio bytes to a file named `.bmu`.
- Output stems are constrained to 16 characters, lower-case ASCII letters, digits, and underscores.
- Generated output names are reserved against every existing asset stem, not just music files.
- Current generated credits writes are already incremental: desired files are rendered in memory, unchanged files are not rewritten, and stale generated artifacts are removed.
- App-layer output for music is currently coupled to `build-haks` summary output in `internal/app/app.go`.
## Existing behavior to preserve during migration
- `build-haks` must continue to convert configured music sources into HAK resources without requiring repository scripts.
- Existing Shadows Over Westgate outputs should remain stable by default:
- source root: `envi/music`
- westgate prefix mapping: `envi/music/westgate -> mus_wg_`
- generated output extension: `.bmu`
- generated cache inventory: `.cache/credits/credits.json`
- generated credits markdown shape and columns
- manual `CREDITS.md` overlay semantics
- duplicate output-name collision behavior
- 16-character NWN resource stem limit
- Existing concise/verbose `build-haks` log behavior should keep working.
- Existing environment variable overrides for ffmpeg/ffprobe should continue to work as compatibility aliases.
- Machine-readable generated artifacts should remain JSON unless there is a separate explicit migration contract.
## Current code areas affected
- `internal/project/project.go`
- `Config.Music` currently only supports `prefixes`.
- `ValidateLayout` calls `validateMusicConfig`.
- `AssetExtensions` currently defines which music files appear in project inventory.
- `Scan` only scans the configured assets directory, so datasets outside `paths.assets` need a deliberate design.
- `internal/pipeline/music.go`
- all scanning, naming, metadata, conversion, credits, cache, and cleanup logic is currently one private pipeline helper.
- hardcoded assumptions include `envi/music`, `.cache/music`, `.cache/credits`, `CREDITS.md`, BMU output, ffmpeg args, metadata tags, and NWN stem rules.
- `internal/pipeline/build.go`
- HAK planning/building depends on `prepareMusicAssets`.
- plan-only HAK workflows can still invoke music probing/conversion today through HAK planning paths.
- `internal/app/app.go`
- `build-haks` owns the only user-facing music summary.
- New music commands should reuse the same logging/verbosity model instead of inventing unrelated output.
- `internal/pipeline/pipeline_test.go`
- currently has regression coverage for conversion, generated credits, manual overlay, inventory inclusion, staging cleanup, and unchanged second pass.
- `internal/app/app_test.go`
- currently checks normal versus verbose mapping output.
- `README.md`
- currently documents only `music.prefixes`.
# Design constraints and decisions
## Dataset roots
Datasets should explicitly state where they read from and where generated artifacts go.
Important distinction:
- `source` is the authored input root for a dataset.
- `output` is the generated publish/staging root for transformed audio resources.
- `credits.output` is where generated human-readable credits go.
- `manifest.output` is where generated machine-readable dataset state goes.
- `cache.root` is where reusable implementation caches live.
Do not assume dataset sources are under `paths.assets`. Support both:
- asset-relative dataset roots for existing HAK packaging behavior
- project-relative dataset roots for future non-HAK or external datasets
For HAK builds, the pipeline must know whether generated outputs are meant to be injected into the project asset inventory and under which resource-relative path.
## NWN-specific behavior
The toolkit can support NWN conventions, but they must be selected by configuration instead of implicit global behavior.
Suggested built-in profiles:
- `nwn_bmu`
- output extension: `.bmu`
- max stem length: 16
- allowed stem characters: `[a-z0-9_]`
- lower-case output names
- default ffmpeg encoder compatible with current behavior
- `passthrough`
- no renaming unless configured
- no transcoding unless configured
- `audio_export`
- configurable format, stem length, and codec rules without NWN resource constraints
Do not make `nwn_bmu` the only naming implementation.
## Configuration merge semantics
Merge configuration in this order:
1. toolkit defaults
2. repository `music.defaults`
3. dataset definition
4. command-line/per-task overrides
Merge rules should be explicit:
- scalar fields replace inherited values
- maps merge by key
- lists replace by default unless a field is documented as appendable
- empty strings mean "not set", not "clear this inherited field"
- booleans that need tri-state inheritance should use pointer/nullable semantics internally
- command-line overrides should be represented in result metadata so logs can show effective behavior
Reject unknown fields via strict YAML decode, matching the rest of the project config philosophy.
## Compatibility config shape
The current config:
```yaml
music:
prefixes:
envi/music/westgate: mus_wg_
```
should be treated as a compatibility shorthand equivalent to:
```yaml
music:
datasets:
envi_music:
source: envi/music
package:
mode: hak_asset
output_root: envi/music
naming:
scheme: nwn_bmu
prefixes:
westgate: mus_wg_
```
Exact normalized dataset ids may differ, but the migration must define how old `music.prefixes` maps into the new dataset model.
## Proposed expanded schema
Example target:
```yaml
music:
tools:
ffmpeg: "" # optional; fallback to SOW_FFMPEG then PATH
ffprobe: "" # optional; fallback to SOW_FFPROBE then PATH
defaults:
discover:
include:
- "**/*.mp3"
- "**/*.ogg"
passthrough:
- "**/*.bmu"
- "**/*.wav"
exclude: []
credits_file: CREDITS.md
naming:
scheme: nwn_bmu
max_stem_length: 16
prefix: ""
prefixes: {}
drop_words: []
reserved_stems_from_assets: true
metadata:
enabled: true
prefer_overlay: true
probe_tags:
title:
- title
artist:
- artist
- album_artist
- composer
rights:
- license
- license_url
- copyright
transcoding:
enabled: true
output_extension: .bmu
codec: libmp3lame
format: mp3
sample_rate: 44100
channels: 2
bitrate: 192k
strip_metadata: true
ffmpeg_args: []
credits:
enabled: true
output: .cache/credits
inventory: .cache/credits/credits.json
format: nwn_markdown_table
cache:
root: .cache/music
mode: content_hash
keep_staging: false
manifest:
enabled: true
output: .cache/music/manifest.json
format: json
validation:
require_credits_for_generated: false
reject_unknown_overlay_rows: true
reject_output_collisions: true
datasets:
westgate:
source: envi/music/westgate
output: envi/music/westgate
package:
mode: hak_asset
naming:
prefix: mus_wg_
```
Schema names can change during implementation, but the final shape should preserve these concepts.
# Pipeline architecture
## Suggested package layout
Prefer a dedicated internal subsystem instead of expanding `internal/pipeline/music.go`.
Suggested layout:
```text
internal/music/
config.go typed config, defaults, merge, validation
dataset.go resolved dataset model and effective settings
discover.go source discovery and file classification
metadata.go ffprobe integration and metadata normalization
credits.go markdown parsing/rendering and inventory generation
naming.go naming schemes, collision detection, compatibility prefix logic
transcode.go ffmpeg integration and output writing
cache.go content hashing and incremental state
manifest.go machine-readable dataset/stage manifests
validate.go dataset validation diagnostics
pipeline.go stage orchestration
result.go machine-readable stage/build result types
```
Then keep `internal/pipeline` responsible for HAK/module orchestration and call `internal/music` through a narrow adapter.
## Stage model
Each stage should have:
- typed input
- typed output
- deterministic ordering
- dry-run support where it can avoid writes
- changed/unchanged reporting
- structured diagnostics
- stable machine-readable result data
Suggested stages:
1. Resolve datasets and effective config.
2. Discover candidate files.
3. Load authored overlays.
4. Probe source metadata.
5. Plan output names and detect collisions.
6. Plan transformations.
7. Reuse cached outputs or transcode changed files.
8. Render generated credits.
9. Write dataset manifest/inventory.
10. Return generated package resources to the HAK pipeline.
11. Validate outputs and report diagnostics.
## Result model
Music command results should be structured enough for both CLI output and tests.
Include:
- dataset id
- source root
- output root
- files discovered
- files converted
- files reused from cache
- passthrough files
- skipped files
- generated resources
- credit artifacts
- manifest paths
- changed files
- warnings/errors
- source-to-output mappings
- effective naming/transcoding profile
Do not require the app layer to parse free-form progress strings.
# Incremental and cache behavior
## Content identity
Incremental rebuilds should be based on content and effective configuration, not only mtimes.
Cache keys should include:
- source content hash
- source relative path
- effective dataset id
- effective naming settings
- effective transcoding settings
- ffmpeg command/profile version
- toolkit music pipeline version
- manual overlay data that affects output metadata/name
Changing an output prefix, codec, bitrate, overlay output filename, or metadata rendering rules must invalidate the relevant planned output.
## Output writes
- Avoid rewriting unchanged credits, manifests, and transformed outputs.
- Write through temporary files and atomic rename where practical.
- Preserve deterministic JSON indentation and key ordering.
- Remove stale generated artifacts inside owned generated roots.
- Never remove authored source files.
- Never remove files outside configured owned cache/output roots.
## Plan-only behavior
Current HAK planning can trigger conversion work. The refactor should define stricter behavior:
- `music scan` and HAK `--plan-only` should not transcode by default.
- They may probe metadata only if needed and allowed.
- They should be able to produce planned mappings without writing transformed audio.
- If actual resource size is needed for HAK chunk planning, use cached transformed size when available and report when exact planning requires a build.
# CLI design expansion
Add first-class commands while preserving `build-haks` compatibility.
Suggested commands:
```text
sow-toolkit music list-datasets
sow-toolkit music scan [--dataset <id>] [--json]
sow-toolkit music build [--dataset <id>] [--dry-run] [--force]
sow-toolkit music credits [--dataset <id>] [--check]
sow-toolkit music validate [--dataset <id>] [--json]
sow-toolkit music manifest [--dataset <id>] [--check]
```
Integration commands:
```text
sow-toolkit build-haks
sow-toolkit build-haks --skip-music
sow-toolkit build-haks --music-dataset <id>
```
Command output should follow the existing log refactor direction:
- normal mode: phase summaries and counts
- verbose mode: mappings and per-file actions
- debug mode: resolved config, tool paths, ffmpeg args, cache keys
- quiet mode: final summary only
- `--json`: machine-readable result on stdout and human diagnostics on stderr
# Validation requirements
Validate repository config:
- duplicate dataset ids
- invalid dataset source/output paths
- absolute paths unless explicitly allowed
- path traversal outside project root for owned outputs/caches
- duplicate or ambiguous prefix mappings
- prefix too long for selected naming scheme
- unsupported output extension for selected package mode
- invalid codec/profile fields
- impossible inherited config combinations
- `package.mode: hak_asset` without an asset-relative output root
- unknown naming/transcoding/credits/manifest schemes
Validate discovered music:
- source files with unsupported extensions
- duplicate case-insensitive output stems
- output name collision with existing assets and generated outputs
- manual overlay rows referencing unknown originals or outputs
- manual overlay output filenames violating selected naming scheme
- metadata probe failures
- source paths differing only by case on case-insensitive targets
- generated resource names that exceed NWN constraints
- generated output extension/resource type compatibility
Validation severity should distinguish:
- errors that block deterministic build output
- warnings for incomplete credits or missing optional metadata
- info for skipped files and unchanged caches
# Fringe considerations
## Credits and licensing
- Preserve authored `CREDITS.md` files as source overlays.
- Generated credits should clearly identify generated versus authored sources in inventory.
- Missing artist/title/license can be warning-level or error-level per dataset.
- Overlay parsing should keep accepting backtick-wrapped filenames.
- Markdown parsing should continue to support escaped pipes and `<br>` line breaks.
- Generated credits should be deterministic across platforms.
- Consider whether aggregate credits inventory should include datasets disabled for build but enabled for credits.
## Filename and path edge cases
- Normalize separators to forward slashes in config, manifests, and logs.
- Treat paths case-insensitively for resource collision checks.
- Reject NUL bytes and unsafe relative paths.
- Define behavior for spaces, Unicode, apostrophes, brackets, years, and punctuation in stems.
- Keep existing slug/drop-word behavior for `nwn_bmu` compatibility, but make drop words configurable.
- Avoid Windows-reserved names and problematic trailing dots/spaces if outputs may be checked out or built on Windows.
- Detect collisions after truncation and suffixing.
- Validate suffixing still respects max stem length.
## Audio tooling
- Keep `SOW_FFMPEG` and `SOW_FFPROBE` as compatibility fallbacks.
- Add config-level tool paths for repositories that pin local toolchain binaries.
- Expose ffmpeg/ffprobe version in debug output and manifest metadata.
- Capture ffmpeg failures with enough context to diagnose source file, dataset, and command profile.
- Do not print enormous ffmpeg logs in normal mode unless the command fails.
- Decide whether `.ogg -> .bmu` should continue transcoding or become configurable passthrough for datasets that want native OGG output.
## HAK/resource integration
- Generated music resources must enter collision detection alongside authored resources.
- Source `.mp3`/`.ogg` files that are converted must be omitted from HAK packaging unless config says otherwise.
- Passthrough `.bmu` and `.wav` handling should remain distinct from generated outputs.
- Resource sorting should stay deterministic and reuse existing HAK resource ordering rules.
- Generated music should be represented in build manifests with enough source provenance to debug.
- HAK include globs should match generated output paths, not source paths, unless explicitly configured otherwise.
## Cross-repository behavior
- A repository with no `music` config should not pay a heavy ffmpeg/ffprobe cost.
- A repository with authored `.bmu` files but no generated music config should preserve current passthrough packaging.
- Multiple datasets should be able to target different output roots without collisions.
- Multiple repositories should be able to use different naming schemes without code changes.
- Dataset config should be portable between Linux and Windows path conventions.
- The toolkit should not assume Shadows Over Westgate names, HAK names, prefixes, or source layout.
# Migration strategy
## Phase 1: Extract without behavior changes
- Move existing private music functions from `internal/pipeline/music.go` into `internal/music`.
- Keep the current `prepareMusicAssets` adapter in `internal/pipeline` to avoid broad HAK changes.
- Preserve every existing test expectation.
- Add focused unit tests around moved naming, overlay, metadata, and credits functions.
## Phase 2: Introduce dataset config
- Extend `project.MusicConfig` with typed defaults and datasets.
- Keep `music.prefixes` as deprecated compatibility shorthand.
- Implement config normalization from legacy prefixes into a default dataset.
- Add strict validation for dataset ids, paths, naming, and prefix constraints.
- Update README examples.
## Phase 3: Add first-class music commands
- Wire `music scan`, `music build`, `music credits`, and `music validate` into `internal/app`.
- Reuse existing verbosity levels and summary formatting.
- Add JSON output support if the command result model is ready.
- Ensure `build-haks` calls the same service API as the music commands.
## Phase 4: Incremental generated output cache
- Replace temporary-only `.cache/music` staging with a content-addressed cache when configured.
- Keep cleanup behavior compatible for existing builds unless `cache.keep_staging` is enabled.
- Avoid conversion during dry-run/plan-only unless exact output artifacts are explicitly requested.
- Add stale generated artifact cleanup scoped only to owned roots.
## Phase 5: Repository migration
- Convert Shadows Over Westgate config from `music.prefixes` to explicit dataset config.
- Keep compatibility warning for one release window.
- Update consumer wrapper scripts only if command names change.
- Verify existing generated HAK manifests and credits inventories remain stable or document intentional diffs.
# Test plan
Add or preserve tests for:
- legacy `music.prefixes` compatibility
- explicit dataset config equivalent to current Westgate behavior
- dataset source outside `envi/music`
- multiple datasets with independent prefixes and outputs
- no music config with authored `.bmu` passthrough assets
- manual overlay output filename validation
- manual overlay references unknown source/output
- duplicate generated output collisions
- collision with existing authored asset stem
- max stem length and suffix collision behavior
- unchanged second build does not rewrite credits or manifests
- dry-run/plan-only does not transcode
- missing ffmpeg/ffprobe failure messages
- configured ffmpeg/ffprobe paths
- normal, verbose, quiet, debug, and JSON CLI output
- Windows-style path separators in config normalize deterministically
- generated credits and inventory are deterministic
# Acceptance criteria
- Music functionality is reusable through a dedicated toolkit subsystem.
- No repository-specific path or prefix is hardcoded in music implementation.
- Existing `build-haks` behavior remains compatible for Shadows Over Westgate.
- First-class music commands can scan, build, generate credits, and validate a configured dataset.
- Dataset configuration can express current Westgate behavior and at least one non-Westgate layout.
- Generated outputs are deterministic and unchanged files are not rewritten.
- Machine-readable artifacts remain JSON.
- Logs are concise by default and detailed under verbose/debug.
- Tests cover the compatibility path and the new dataset-driven path.