ShadowsOverWestgate/sow-tools
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 38 Activity

Phase 5: scaffold Crucible (sow-tools) — dispatcher + builder shims, container, PR-first CI, fail-closed (D11, D19).

Browse Source
This commit is contained in:
archvillainette
2026-06-11 20:36:13 +02:00
parent ff264f0d7c
commit 97f8427c4b
154 changed files with 948 additions and 76902 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/topdata/AUTO-INCLUDE_EXISTING_PART_MODELS_IN_2DA_GENERATION_CONTRACT.md
-155
View File
@@ -1,155 +0,0 @@
# AI Agent Contract: Auto-Include Existing Part Models in 2DA Generation
## Status Snapshot
Current state as of 2026-05-17:
- Superseded for current Shadows Over Westgate builds.
- The module no longer consumes a released parts manifest. Parts data is owned by
`assets/`, generated as HAK-side 2DA assets from `assets/topdata/data/parts`
plus a direct scan of `assets/content/part/**/*.mdl`, and packaged in
`sow_part`.
- This file remains as historical context for the legacy manifest-backed
consumer path.
## Objective
The native 2DA generation pipeline must automatically include existing part models already present in the **`sow-assets` repository** when building final parts tables.
This behavior is required for parity with the current asset set and must not depend on manually authored entries for models that already exist in repository assets.
---
## Source of Truth
The superseded manifest-backed native builder obtained existing model
information from a released artifact published by the **`sow-assets` repository**.
Resolution rules:
- if `topdata.assets` is configured to a local path, use that exact path
- otherwise derive the **`sow-assets` repository** from the module repo's Gitea origin
and fetch only the configured released manifest
Normal builds must not clone or scan the `sow-assets` repo tree. They may cache the
downloaded manifest locally for reuse, but the source of truth remains the released
manifest on Gitea.
Manifest-backed discovery is derived from:
```text
sow-assets/assets/part/**/*.mdl
```
The manifest generator may also accept the repo's legacy `assets/parts/` layout when
resolving the physical paths, but the native builder itself must consume the manifest,
not the repo tree.
---
## Required Scope
Process existing `.mdl` files for these parts table categories only:
- `belt`
- `bicep`
- `chest`
- `foot`
- `forearm`
- `leg`
- `neck`
- `pelvis`
- `robe`
- `shin`
- `shoulder`
Dataset mapping rules:
- `parts/legs` maps to asset category `leg`
- `parts/hand` remains unsupported and is not auto-generated
The scan must recurse through subdirectories under each applicable category path.
---
## Required Behavior
### 1. Manifest-backed discovery
For each supported part category, read the released manifest and discover all existing
numeric model IDs listed for that category.
### 2. Trailing numeric suffix extraction
The released manifest must be generated using trailing-numeric-suffix extraction from
the source `.mdl` filenames. Native topdata consumes the resulting numeric IDs only.
### 3. Row generation
If a trailing numeric suffix is present:
- use that numeric suffix as the output row ID
- emit a row for that part model in the final generated 2DA
### 4. Default emitted values
For rows generated from discovered existing models, set:
- `COSTMODIFIER = 0`
- `ACBONUS = 0.00`
Rows without discovered existing models may be intentionally emitted as dense
2DA null rows when configured. In that mode, missing rows must have
`COSTMODIFIER = ****` and `ACBONUS = ****` in the final generated 2DA.
### 5. Override precedence
If file-based overrides exist, they must take precedence over discovered defaults.
That means:
- manifest discovery provides the baseline row presence and default values
- override data may replace or augment those values
- override data wins in any conflict
---
## Non-Negotiable Requirements
- The implementation must read model presence from the **released parts manifest**,
not from assumption, hardcoded lists, or legacy output.
- Existing repository models with trailing numeric suffixes must be reflected in generated output even if they are not otherwise explicitly authored.
- Override application is mandatory and must win over auto-discovered defaults.
- This is replacement behavior for the native pipeline, not an optional enhancement.
---
## Acceptance Criteria
The superseded implementation was correct only if all of the following were true:
1. The builder fetches the configured released manifest artifact for part models.
2. Only the supported part categories are considered.
3. Manifest IDs derived from trailing numeric suffixes generate corresponding 2DA rows.
4. The numeric suffix is used as the row ID.
5. Auto-generated rows default to:
- `COSTMODIFIER = 0`
- `ACBONUS = 0.00`
6. Authored base rows remain exposed. Authored `ACBONUS = ****` rows remain
null unless model discovery activates the row.
7. Dense gaps without authored rows or discovered models are emitted as null
rows with `COSTMODIFIER = ****` and `ACBONUS = ****`.
8. File overrides are applied after discovery and take precedence.
9. The final output includes all eligible existing models present in the manifest.
10. Discovery does not require a sibling local `sow-assets` checkout or a Git clone.
---
## Implementation Directive
When generating native parts 2DAs, first scan the project-resolved assets repo for
supported part models, collect existing supported part models by category, extract
trailing numeric suffixes from filenames, emit rows using those suffixes as row IDs
with default `COSTMODIFIER = 0` and `ACBONUS = 0.00`, then apply overrides with
override values taking precedence.
internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md
-76
View File
@@ -1,76 +0,0 @@
# Class Feat Global Injection Contract
## Status Snapshot
Current state as of 2026-06-05:
- Preferred class feat injection policy is authored in
`topdata/data/classes/feats/global.json`.
- `global.json` rows flow through the ordinary class feat expansion pipeline,
including successor expansion, masterfeat expansion, class-skill filtering,
reference resolution, deduplication, and final 2DA emission.
- Legacy `topdata.class_feat_injections` YAML still loads only as a
compatibility path when no applicable class feat `global.json` exists.
- Toolkit hardcoded defaults are compatibility-only and must not be the normal
source of project policy.
## Objective
Native class feat generation must produce deterministic, legacy-equivalent
effective output while keeping project policy in topdata authoring files.
## Requirements
- For each class skill, inject the configured skill-focus masterfeat rows from
`classes/feats/global.json` using `FeatIndex.filter = "classskills"`.
- Inject `feat:literate` only when the class table does not already include
`feat:illiterate`.
- Inject the shared combat/menu feat rows declared in the manifest unless a
class already authors the equivalent `FeatIndex`.
- Explicit class feat rows take precedence over injected rows.
- Injections must be deterministic for identical inputs.
## Manifest Shape
```json
{
"position": "prepend",
"injections": [
{
"row": {
"FeatIndex": { "id": "feat:literate" },
"GrantedOnLevel": 1,
"List": 3,
"OnMenu": 0
},
"unless_present": [
{ "field": "FeatIndex", "id": "feat:illiterate" }
]
},
{
"row": {
"FeatIndex": {
"id": "masterfeats:skill_focus",
"filter": "classskills"
},
"GrantedOnLevel": -1,
"List": 1,
"OnMenu": 0
}
}
]
}
```
`position` defaults to `append`. The module uses `prepend` for class feat
injections to keep generated policy rows before per-class authored rows.
## Acceptance Criteria
- Class feat output contains the same effective shared feats as the prior YAML
policy.
- Literacy injection is skipped for illiterate classes.
- Skill-focus masterfeat rows expand only for class skills.
- YAML compatibility continues to work for consumers that have not migrated.
- A project with `classes/feats/global.json` does not also receive YAML or
hardcoded default class feat injections.
internal/topdata/FAMILY_EXPANSION_CONTRACT.md
-119
View File
@@ -1,119 +0,0 @@
# Generic Family Expansion Contract
## Status Snapshot
Current state as of 2026-05-13:
- Implemented and actively used by native feat generation.
- `family_expansion.go`, `native.go`, and `topdata_test.go` cover the current
underscore-family interpretation, generated family metadata, and reapplication
behavior.
- Remaining work is mostly consumer expansion: use the same primitive in more
datasets only when the authored data actually benefits from it.
Topdata now treats underscore expansion as a structural rule:
- `parent_child` means `child` is an expansion of `parent`
- underscores are for family structure, not simulated spaces
- existing canonical keys still win when lock or TLK state already established them
This is a global interpretation rule, not a wiki-only convention.
## Identity
- `weaponspecialization_club`
- parent: `weaponspecialization`
- child: `club`
- `gnome_rock`
- parent: `gnome`
- child: `rock`
- `toughness_10`
- parent: `toughness`
- child: `10`
Standalone keys without an underscore are treated as a parent identity with no child.
## Generated Family Files
Canonical generated families declare:
```json
{
"family": "weapon_focus",
"family_key": "weaponfocus",
"template": "masterfeats:weaponfocus",
"name_prefix": "Weapon Focus",
"label_prefix": "FEAT_WEAPON_FOCUS",
"constant_prefix": "FEAT_WEAPON_FOCUS",
"legacy_family_keys": ["epic_weapon_focus"],
"default_fields": {
"DESCRIPTION": {
"tlk": {
"key": "masterfeats:weaponfocus.description",
"text": "..."
}
},
"MINATTACKBONUS": "1",
"TOOLSCATEGORIES": "1"
},
"apply_after_modules": true,
"identity_source": "child_source_value",
"auto_prereq_fields": {
"PREREQFEAT1": "weaponfocus"
},
"child_source": {
"dataset": "baseitems",
"column": "WeaponFocusFeat"
}
}
```
Rules:
- `family` is an authored label and is not a hardcoded switch key
- `family_key` is the structural parent identity used in generated child keys
- `template` is optional for the primitive in general, but required by current
masterfeat-backed `feat` families
- `name_prefix`, `label_prefix`, and `constant_prefix` define how new child rows are
named when they are not already authored
- `template_fields` lists template-backed fields copied from the referenced template row
when authored; it is optional if the family owns its shared values directly
- `default_fields` applies authored shared/default values to every generated family row
and can be used to move family-wide shared values out of the template row
- `apply_after_modules: true` re-applies the generated family after normal module files,
so authored family-shared values remain authoritative even for legacy explicitly
authored child rows
- `child_ref_field` binds the generated child row back to the source row key, e.g.
`REQSKILL`
- `identity_source: "child_source_value"` tells the builder to reuse IDs/keys from the
source column when that column already carries feat identity
- `legacy_family_keys` declares previous generated-family prefixes that should donate
existing locked row IDs to the current family when a family is renamed
- `auto_prereq_fields` binds prerequisite fields to other authored families by
`family_key`, without code changes
- `allow_existing_only` limits expansion to already-authored child rows when a family is
intentionally partial
- `child_source.dataset` identifies the canonical dataset driving expansion
- `child_source.column` is used when expansion is gated by a non-null source field
- `child_source.predicate` is used when expansion depends on a named rule such as
accessibility
## Row Metadata
Generated rows can carry:
```json
{
"meta": {
"family": {
"parent": "weaponfocus",
"child": "club",
"source": "baseitems:club",
"template": "masterfeats:weaponfocus"
}
}
}
```
This metadata is builder-owned and does not affect emitted 2DA columns. It preserves
family structure for later phases without enabling wiki generation yet.
internal/topdata/FEAT_GENERATED_FAMILIES_CONTRACT.md
-174
View File
@@ -1,174 +0,0 @@
# `feat` Generated Families Contract
## Status Snapshot
Current state as of 2026-05-13:
- Implemented for the current native `feat` pipeline.
- Native tests cover generated-family expansion, family-owned shared defaults,
masterfeat-backed inheritance, and lock/TLK identity preservation.
- The one explicitly deferred item in this contract is still current:
`feat.2da` remains native-built but excluded from `compare-topdata`
reference-catalog coverage through `compare_reference: false`.
Canonical authored paths:
- `topdata/data/feat/base.json`
- `topdata/data/feat/lock.json`
- `topdata/data/feat/generated/skill_focus.json`
- `topdata/data/feat/generated/greater_skill_focus.json`
- `topdata/data/feat/generated/weapon_focus.json`
- `topdata/data/feat/generated/weapon_specialization.json`
- `topdata/data/feat/generated/greater_weapon_focus.json`
- `topdata/data/feat/generated/greater_weapon_specialization.json`
- `topdata/data/feat/generated/improved_critical.json`
- `topdata/data/feat/generated/overwhelming_critical.json`
- `topdata/data/feat/modules/activecombat/core.json`
- `topdata/data/feat/modules/activecombat/specialattacks.json`
- `topdata/data/feat/modules/class/core.json`
- `topdata/data/feat/modules/combat/core.json`
- `topdata/data/feat/modules/defensive/core.json`
- `topdata/data/feat/modules/magical/core.json`
- `topdata/data/feat/modules/other/core.json`
- `topdata/data/feat/modules/proficiency/core.json`
- `topdata/data/feat/modules/racial/core.json`
- `topdata/data/feat/modules/removedandhidden/core.json`
- `topdata/data/feat/modules/skillfeat/affinity.json`
- `topdata/data/feat/modules/skillfeat/focus.json`
- `topdata/data/feat/modules/skillfeat/greaterfocus.json`
- `topdata/data/feat/modules/skillfeat/other.json`
`skill_affinity` is no longer authored as a generated feat file. It is synthesized from
canonical `racialtypes` race grants.
Compact family-expansion shape:
```json
{
"family": "skill_focus",
"family_key": "skillfocus",
"template": "masterfeats:skillfocus",
"name_prefix": "Skill Focus",
"label_prefix": "FEAT_SKILL_FOCUS",
"constant_prefix": "FEAT_SKILL_FOCUS",
"default_fields": {
"DESCRIPTION": {
"tlk": {
"key": "masterfeats:skillfocus.description",
"text": "..."
}
},
"CRValue": "0.5",
"ReqSkillMinRanks": "1",
"TOOLSCATEGORIES": "6"
},
"apply_after_modules": true,
"child_ref_field": "REQSKILL",
"child_source": {
"dataset": "skills",
"predicate": "accessible"
},
"title_style": {
"child_case": "lower",
"child_parenthetical": "comma"
},
"overrides": {
"skills:craft_alchemy": {
"ICON": "ife_foc_alchm"
}
}
}
```
Weapon family shape:
```json
{
"family": "weapon_focus",
"family_key": "weaponfocus",
"template": "masterfeats:weaponfocus",
"name_prefix": "Weapon Focus",
"label_prefix": "FEAT_WEAPON_FOCUS",
"constant_prefix": "FEAT_WEAPON_FOCUS",
"legacy_family_keys": ["epic_weapon_focus"],
"default_fields": {
"DESCRIPTION": {
"tlk": {
"key": "masterfeats:weaponfocus.description",
"text": "..."
}
},
"MINATTACKBONUS": "1",
"TOOLSCATEGORIES": "1"
},
"apply_after_modules": true,
"identity_source": "child_source_value",
"child_source": {
"dataset": "baseitems",
"column": "WeaponFocusFeat"
},
"title_style": {
"child_case": "lower",
"child_parenthetical": "preserve"
},
"overrides": {
"baseitems:heavymace": {
"ICON": "ife_wepfoc_Lma"
}
}
}
```
Rules:
- any generated feat file with family-expansion fields is treated as an authored
family-expansion definition
- `family` is descriptive only; the builder does not hardcode family names
- family behavior is driven by authored fields such as `template_fields`,
`default_fields`, `apply_after_modules`, `identity_source`, `child_ref_field`,
`auto_prereq_fields`, and optional `title_style`
- `default_fields` is a family-wide shared layer; it applies to existing and newly
generated child rows before per-child overrides are merged
- `apply_after_modules: true` makes the generated family the final authoritative shared
layer for legacy explicitly authored child rows, so shared baselines can live in the
generated family file instead of `masterfeats` overrides
- `legacy_family_keys` declares old generated-family prefixes whose existing row IDs
should be inherited by the current family; this is how renamed families such as
`greater_skill_focus` retaining vanilla `epic_skill_focus` rows preserve hardcoded
engine behavior without hardcoding the rename in toolkit logic
- family-expansion files must declare `template`, `family_key`, and `child_source.dataset`
- `child_source.column` is used when expansion is gated by a non-null source field
- `child_source.predicate: "accessible"` currently means `HideFromLevelUp != 1`
- `title_style` only changes generated child display text while composing titles;
generated feat keys, TLK keys, row IDs, family metadata, and source references
remain tied to generated family identity
- generated family title text is authoritative for family-managed existing rows
as well as newly allocated rows, so in-game TLK names and generated wiki titles
are derived from the same normalized `FEAT` text
- `title_style.child_case` defaults to `preserve`; `lower` lowercases the resolved
child display text after parenthetical formatting
- `title_style.child_parenthetical` defaults to `preserve`; `comma` flattens one
child parenthetical suffix such as `Knowledge (local)` into
`Knowledge, local` before the child text is wrapped by the family title
- unsupported `title_style` values fail generated-family validation instead of
silently changing title behavior
- skill focus families intentionally do not set `allow_existing_only`, so any new
accessible skill automatically receives matching focus rows
- compact `overrides` are keyed by source dataset key:
- `skills:*` for skill families
- `baseitems:*` for weapon families
- generated rows carry `meta.family` so the underscore family rule is preserved even
before wiki generation exists
- emitted feat keys must prefer existing canonical lock/TLK identities over newly
derived child spellings
- manual and irregular feat content continues to live under `feat/modules/`
- generated families and module-authored feats are merged into the same final `feat.2da`
pipeline and share inheritance, override, TLK, and family-metadata behavior
Current rollout policy:
- `feat.2da` still builds natively
- `feat.2da` is still excluded from `compare-topdata` output-catalog self-check coverage
via `compare_reference: false`
- manual and irregular feat families remain authored directly in modules rather than
compact generated-family files
internal/topdata/INHERITANCE_CONTRACT.md
-78
View File
@@ -1,78 +0,0 @@
# Topdata Inheritance Contract
## Status Snapshot
Current state as of 2026-05-13:
- Implemented.
- `native.go` supports both row-sugar inheritance and generic object
inheritance, including cycle detection and missing-target failures.
- Validation and build regression coverage live in `topdata_test.go`.
Native topdata now supports two inheritance forms:
## 1. Row Sugar Compatibility
This is the existing narrow row helper:
```json
{
"PARENT": { "id": "custom:parent" },
"inherit": {
"from": "PARENT",
"fields": ["VALUE", "ICON"]
}
}
```
It remains supported for row-local field copying.
## 2. Generic Object Inheritance
This is the new reusable primitive:
```json
{
"inherit": {
"ref": "masterfeats:skillfocus"
},
"LABEL": "FEAT_SKILL_FOCUS_ATHLETICS"
}
```
Or for a nested object:
```json
{
"DETAILS": {
"inherit": {
"ref": "custom:parent",
"field": "DETAILS"
},
"meta": {
"cost": "9"
}
}
}
```
## Rules
- `inherit.ref` must use stable `dataset:key` identity.
- `inherit.field` is optional and selects an object-valued field on the target row.
- Generic inheritance can appear on any authored object, not only top-level rows.
- Merge precedence is `local overrides inherited`.
- Scalars replace inherited values.
- Arrays replace inherited values.
- Objects merge recursively unless the local object is an atomic topdata value object such as:
- TLK payloads
- row refs
- table refs
- Cycles fail the build.
- Missing targets fail the build.
## Current Intended Consumer
`feat` is the first dataset expected to use this primitive heavily, especially for
borrowing shared properties from `masterfeats` while keeping `feat`'s own dataset
contract and row model.
internal/topdata/MASTERFEATS_CONTRACT.md
-82
View File
@@ -1,82 +0,0 @@
# `masterfeats` Canonical Contract
## Status Snapshot
Current state as of 2026-05-13:
- Implemented as a first-class native dataset.
- The repository contains `topdata/data/masterfeats/`, migration/import support,
validator checks, and native output coverage for `masterfeats.2da`.
- Remaining work is only downstream adoption and regression maintenance as other
native datasets continue to consume `masterfeats` more heavily.
`masterfeats` is a stable native canonical family consumed directly by `feat`
generation, class-feat expansion, and inheritance/ref resolution.
Current canonical scope:
- dataset root: `topdata/data/masterfeats/`
- required files:
- `base.json`
- `lock.json`
- generated output:
- `masterfeats.2da`
## `base.json`
Canonical shape:
```json
{
"output": "masterfeats.2da",
"columns": ["LABEL", "STRREF", "DESCRIPTION", "..."],
"rows": [
{
"id": 0,
"key": "masterfeats:weaponfocus",
"LABEL": "WeaponFocus",
"STRREF": "6490",
"DESCRIPTION": "436"
}
]
}
```
Rules:
- `output` may be omitted; native dataset discovery deterministically derives
`masterfeats.2da` from `topdata/data/masterfeats/base.json`
- if `output` is present, it must be exactly `masterfeats.2da`
- `columns` must include `LABEL`, `STRREF`, and `DESCRIPTION`
- every canonical row must have a non-empty `key`
- row keys must start with `masterfeats:`
- inline TLK payloads are allowed in `STRREF` and `DESCRIPTION`
- numeric/text references are both allowed where the native TLK compiler already supports them
## `lock.json`
Canonical shape:
```json
{
"masterfeats:weaponfocus": 0
}
```
Rules:
- every key must start with `masterfeats:`
- every value must be numeric
- the lockfile remains the canonical stable id source for authored keys
## Module contributions
This family continues to support the shared canonical module shapes already used by native
datasets:
- `columns`
- `entries`
- `overrides`
Those shapes are validated generically in `topdata.go`; this contract only adds the
family-specific guarantees for `masterfeats`.
internal/topdata/appearance_migrate.go
-96
View File
@@ -1,96 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyAppearance(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "appearance")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "appearance")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
moduleNames := []string{
"cotblreaver.json",
"crawlingclaw.json",
"halfogre.json",
"zombieknight.json",
}
targetModulePaths := make([]string, 0, len(moduleNames))
for _, name := range moduleNames {
targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "appearance.2da"
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for i, path := range targetModulePaths {
writes = append(writes, struct {
path string
obj map[string]any
}{path: path, obj: moduleObjs[i]})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
internal/topdata/armor_migrate.go
-160
View File
@@ -1,160 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
"sort"
)
func importLegacyArmor(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "armor")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "armor")
targetPath := filepath.Join(targetDir, "armor.json")
if _, err := os.Stat(targetPath); err == nil {
obj, err := loadJSONObject(targetPath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
rows, err := mergeArmorRows(baseObj, filepath.Join(legacyDir, "modules"))
if err != nil {
return 0, err
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "base.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "lock.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "modules")); err != nil {
return 0, err
}
out := map[string]any{
"output": "armor.2da",
"columns": baseObj["columns"],
"rows": rows,
}
raw, err := json.MarshalIndent(out, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
return 0, err
}
return 1, nil
}
func mergeArmorRows(baseObj map[string]any, modulesDir string) ([]any, error) {
rawRows, ok := baseObj["rows"].([]any)
if !ok {
return nil, nil
}
rows := make([]map[string]any, 0, len(rawRows))
byID := map[int]map[string]any{}
for _, raw := range rawRows {
row, ok := deepCopyValue(raw).(map[string]any)
if !ok {
continue
}
delete(row, "key")
rawID, ok := row["id"]
if !ok {
continue
}
id, err := asInt(rawID)
if err != nil {
return nil, err
}
row["id"] = id
rows = append(rows, row)
byID[id] = row
}
modulePaths, err := collectModulePaths(modulesDir)
if err != nil {
return nil, err
}
for _, path := range modulePaths {
obj, err := loadJSONObject(path)
if err != nil {
return nil, err
}
if entries, ok := obj["entries"].(map[string]any); ok {
for _, raw := range entries {
row, ok := deepCopyValue(raw).(map[string]any)
if !ok {
continue
}
delete(row, "key")
rawID, ok := row["id"]
if !ok {
continue
}
id, err := asInt(rawID)
if err != nil {
return nil, err
}
row["id"] = id
rows = append(rows, row)
byID[id] = row
}
}
if overrides, ok := obj["overrides"].([]any); ok {
for _, raw := range overrides {
override, ok := raw.(map[string]any)
if !ok {
continue
}
rawID, ok := override["id"]
if !ok {
continue
}
id, err := asInt(rawID)
if err != nil {
return nil, err
}
row := byID[id]
if row == nil {
continue
}
for key, value := range override {
if key == "id" || key == "key" || key == "_tlk" || isMetadataField(key) {
continue
}
row[key] = deepCopyValue(value)
}
}
}
}
sort.Slice(rows, func(i, j int) bool {
return rows[i]["id"].(int) < rows[j]["id"].(int)
})
out := make([]any, 0, len(rows))
for _, row := range rows {
out = append(out, row)
}
return out, nil
}
internal/topdata/assets_repo.go
-46
View File
@@ -1,46 +0,0 @@
package topdata
import (
"fmt"
"os"
"os/exec"
"strings"
"gitea.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
func resolvePartsAssetsOverrideDir(p *project.Project) (string, error) {
spec := strings.TrimSpace(p.Config.TopData.Assets)
if spec == "" {
return "", nil
}
if looksLikeGitRepoSpec(spec) {
return "", fmt.Errorf("topdata.assets no longer supports git repo specs for parts discovery; use a local path override or the default released manifest")
}
path := p.TopDataAssetsDir()
if path == "" {
return "", nil
}
if _, err := os.Stat(path); err != nil {
return "", fmt.Errorf("configured topdata.assets path %s is not accessible: %w", path, err)
}
return path, nil
}
func looksLikeGitRepoSpec(value string) bool {
return strings.Contains(value, "://") ||
strings.HasPrefix(value, "git@") ||
strings.HasSuffix(value, ".git")
}
func gitOutput(dir string, args ...string) (string, error) {
cmd := exec.Command("git", args...)
if dir != "" {
cmd.Dir = dir
}
output, err := cmd.CombinedOutput()
if err != nil {
return "", fmt.Errorf("%v: %s", args, strings.TrimSpace(string(output)))
}
return strings.TrimSpace(string(output)), nil
}
internal/topdata/autogen.go
-1132
View File
File diff suppressed because it is too large Load Diff
internal/topdata/base_dialog_migrate.go
-41
View File
@@ -1,41 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyBaseDialog(referenceBuilderDir, sourceDir string) (int, error) {
legacyPath := filepath.Join(referenceBuilderDir, "tlk", "base.json")
if _, err := os.Stat(legacyPath); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetPath := filepath.Join(sourceDir, "base_dialog.json")
if fileExists(targetPath) {
current, err := loadJSONObject(targetPath)
if err == nil {
if entries, ok := current["entries"].(map[string]any); ok && len(entries) > 0 {
return 0, nil
}
}
}
obj, err := loadJSONObject(legacyPath)
if err != nil {
return 0, err
}
raw, err := json.MarshalIndent(obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
return 0, err
}
return 1, nil
}
internal/topdata/baseitems_migrate.go
-105
View File
@@ -1,105 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyBaseitems(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "baseitems")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "baseitems")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
moduleNames := []string{
"blunderbuss.json",
"coin.json",
"estoc.json",
"heavymace.json",
"maulandfalchion.json",
"ovr_baseitems.json",
"ovr_cloak255.json",
"ovr_helmet255.json",
"shortspear.json",
}
targetModulePaths := make([]string, 0, len(moduleNames))
for _, name := range moduleNames {
targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "baseitems.2da"
canonicalizeWikiMetadataDocument(baseObj)
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
canonicalizeWikiMetadataDocument(obj)
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for index, path := range targetModulePaths {
writes = append(writes, struct {
path string
obj map[string]any
}{
path: path,
obj: moduleObjs[index],
})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
internal/topdata/class_spellbooks.go
-373
View File
@@ -1,373 +0,0 @@
package topdata
import (
"fmt"
"path/filepath"
"slices"
"strconv"
"strings"
)
type classSpellbookSpec struct {
Path string
Key string
Class string
Levels map[int][]string
}
type classSpellbookPlan struct {
Spec classSpellbookSpec
Column string
SpellLevels map[string]int
}
func isClassSpellbookPath(path string) bool {
slashed := filepath.ToSlash(path)
return strings.Contains(slashed, "/data/classes/spellbooks/") && strings.HasSuffix(strings.ToLower(slashed), ".json")
}
func validateClassSpellbookShape(path string, obj map[string]any, report *ValidationReport) {
if _, ok := obj["class"]; !ok {
report.Diagnostics = append(report.Diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: "class spellbook requires class"})
}
if _, ok := obj["levels"]; !ok {
report.Diagnostics = append(report.Diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: "class spellbook requires levels"})
}
}
func validateClassSpellbooks(dataDir string, report *ValidationReport) {
collected, err := collectSpellbookValidationDatasets(dataDir)
if err != nil {
report.Diagnostics = append(report.Diagnostics, Diagnostic{
Severity: SeverityError,
Path: dataDir,
Message: fmt.Sprintf("validate class spellbooks: %v", err),
})
return
}
_, diagnostics := classSpellbookPlans(dataDir, collected)
report.Diagnostics = append(report.Diagnostics, diagnostics...)
if len(diagnostics) == 0 {
warnSpellModulesForManagedSpellbookColumns(dataDir, collected, report)
}
}
func collectSpellbookValidationDatasets(dataDir string) ([]nativeCollectedDataset, error) {
datasets, err := discoverNativeDatasets(dataDir)
if err != nil {
return nil, err
}
out := make([]nativeCollectedDataset, 0, 2)
for _, dataset := range datasets {
if dataset.Name != "classes/core" && dataset.Name != "spells" {
continue
}
collected, err := collectNativeDataset(dataset)
if err != nil {
return nil, err
}
out = append(out, collected)
}
return out, nil
}
func applyClassSpellbooks(sourceDir string, collected []nativeCollectedDataset) ([]nativeCollectedDataset, error) {
dataDir := filepath.Join(sourceDir, "data")
plans, diagnostics := classSpellbookPlans(dataDir, collected)
if len(diagnostics) > 0 {
return nil, fmt.Errorf("class spellbook validation failed: %s", diagnosticsTextForError(diagnostics))
}
if len(plans) == 0 {
return collected, nil
}
out := make([]nativeCollectedDataset, len(collected))
copy(out, collected)
spellsIndex := -1
for index := range out {
if out[index].Dataset.Name == "spells" {
spellsIndex = index
break
}
}
if spellsIndex == -1 {
return nil, fmt.Errorf("class spellbooks require canonical dataset %q", "spells")
}
spells := out[spellsIndex]
columns := append([]string(nil), spells.Columns...)
rows := make([]map[string]any, 0, len(spells.Rows))
for _, row := range spells.Rows {
rows = append(rows, cloneRowMap(row))
}
rowByKey := rowsByKey(rows)
for _, plan := range plans {
column := plan.Column
if _, ok := canonicalColumn(columns, column); !ok {
columns = append(columns, column)
ensureRowsExposeColumns(rows, columns)
}
for _, row := range rows {
row[column] = nullValue
}
for spellKey, level := range plan.SpellLevels {
row := rowByKey[spellKey]
if row == nil {
return nil, fmt.Errorf("%s: unknown spell %q", plan.Spec.Path, spellKey)
}
row[column] = level
}
}
spells.Columns = columns
spells.Rows = rows
out[spellsIndex] = spells
return out, nil
}
func classSpellbookPlans(dataDir string, collected []nativeCollectedDataset) ([]classSpellbookPlan, []Diagnostic) {
specs, diagnostics := loadClassSpellbookSpecs(filepath.Join(dataDir, "classes", "spellbooks"))
if len(specs) == 0 {
return nil, diagnostics
}
classes := collectedDatasetByName(collected, "classes/core")
spells := collectedDatasetByName(collected, "spells")
if classes == nil {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: dataDir, Message: `class spellbooks require canonical dataset "classes/core"`})
return nil, diagnostics
}
if spells == nil {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: dataDir, Message: `class spellbooks require canonical dataset "spells"`})
return nil, diagnostics
}
classRows := rowsByKey(classes.Rows)
spellRows := rowsByKey(spells.Rows)
seenColumns := map[string]string{}
plans := make([]classSpellbookPlan, 0, len(specs))
for _, spec := range specs {
classRow := classRows[spec.Class]
if classRow == nil {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: spec.Path, Message: fmt.Sprintf("unknown class %q", spec.Class)})
}
column := ""
if classRow != nil {
column, _ = classRow["SpellTableColumn"].(string)
column = strings.TrimSpace(column)
if column == "" || column == nullValue {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: spec.Path, Message: fmt.Sprintf("class %q has no SpellTableColumn", spec.Class)})
}
}
if column != "" && column != nullValue {
folded := strings.ToLower(column)
if previous, ok := seenColumns[folded]; ok {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: spec.Path, Message: fmt.Sprintf("SpellTableColumn %q is already managed by %s", column, previous)})
} else {
seenColumns[folded] = spec.Path
}
}
spellLevels := map[string]int{}
for _, level := range sortedIntKeys(spec.Levels) {
for _, spellKey := range spec.Levels[level] {
if spellRows[spellKey] == nil {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: spec.Path, Message: fmt.Sprintf("unknown spell %q", spellKey)})
continue
}
if previous, ok := spellLevels[spellKey]; ok {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: spec.Path, Message: fmt.Sprintf("spell %q appears at both level %d and level %d", spellKey, previous, level)})
continue
}
spellLevels[spellKey] = level
}
}
if column != "" && column != nullValue {
plans = append(plans, classSpellbookPlan{Spec: spec, Column: column, SpellLevels: spellLevels})
}
}
return plans, diagnostics
}
func warnSpellModulesForManagedSpellbookColumns(dataDir string, collected []nativeCollectedDataset, report *ValidationReport) {
plans, diagnostics := classSpellbookPlans(dataDir, collected)
if len(diagnostics) > 0 || len(plans) == 0 {
return
}
managed := map[string]classSpellbookPlan{}
for _, plan := range plans {
managed[strings.ToLower(plan.Column)] = plan
}
paths, err := collectModulePaths(filepath.Join(dataDir, "spells", "modules"))
if err != nil {
report.Diagnostics = append(report.Diagnostics, Diagnostic{Severity: SeverityError, Path: dataDir, Message: fmt.Sprintf("collect spell modules for spellbook warnings: %v", err)})
return
}
for _, path := range paths {
obj, err := loadJSONObject(path)
if err != nil {
continue
}
warnSpellbookManagedColumnsInValue(dataDir, path, obj["entries"], managed, report)
warnSpellbookManagedColumnsInValue(dataDir, path, obj["overrides"], managed, report)
warnSpellbookManagedColumnsInValue(dataDir, path, obj["rows"], managed, report)
}
}
func warnSpellbookManagedColumnsInValue(dataDir, path string, value any, managed map[string]classSpellbookPlan, report *ValidationReport) {
switch typed := value.(type) {
case map[string]any:
for _, key := range sortedKeys(typed) {
row, ok := typed[key].(map[string]any)
if !ok {
continue
}
warnSpellbookManagedColumnsInRow(dataDir, path, row, managed, report)
}
case []any:
for _, raw := range typed {
row, ok := raw.(map[string]any)
if !ok {
continue
}
warnSpellbookManagedColumnsInRow(dataDir, path, row, managed, report)
}
}
}
func warnSpellbookManagedColumnsInRow(dataDir, path string, row map[string]any, managed map[string]classSpellbookPlan, report *ValidationReport) {
for field, value := range row {
plan, ok := managed[strings.ToLower(field)]
if !ok || isNullLikeValue(value) {
continue
}
spellbookPath := displayTopdataPath(dataDir, plan.Spec.Path)
report.Diagnostics = append(report.Diagnostics, Diagnostic{
Severity: SeverityWarning,
Path: path,
Message: fmt.Sprintf(
"spell module authors class spell column %q managed by class spellbook %s; use %s instead",
plan.Column,
spellbookPath,
spellbookPath,
),
})
}
}
func displayTopdataPath(dataDir, path string) string {
topdataDir := filepath.Dir(dataDir)
rel, err := filepath.Rel(filepath.Dir(topdataDir), path)
if err != nil {
return filepath.ToSlash(path)
}
return filepath.ToSlash(rel)
}
func loadClassSpellbookSpecs(dir string) ([]classSpellbookSpec, []Diagnostic) {
paths, err := collectModulePaths(dir)
if err != nil {
return nil, []Diagnostic{{Severity: SeverityError, Path: dir, Message: err.Error()}}
}
specs := make([]classSpellbookSpec, 0, len(paths))
diagnostics := []Diagnostic{}
for _, path := range paths {
obj, err := loadJSONObject(path)
if err != nil {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: err.Error()})
continue
}
spec, specDiagnostics := parseClassSpellbookSpec(path, obj)
diagnostics = append(diagnostics, specDiagnostics...)
if spec.Class != "" && spec.Levels != nil {
specs = append(specs, spec)
}
}
return specs, diagnostics
}
func parseClassSpellbookSpec(path string, obj map[string]any) (classSpellbookSpec, []Diagnostic) {
diagnostics := []Diagnostic{}
spec := classSpellbookSpec{Path: path, Levels: map[int][]string{}}
if key, ok := obj["key"].(string); ok {
spec.Key = strings.TrimSpace(key)
}
classKey, ok := obj["class"].(string)
if !ok || strings.TrimSpace(classKey) == "" {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: "class spellbook requires class"})
} else {
spec.Class = strings.TrimSpace(classKey)
if !strings.HasPrefix(spec.Class, "classes:") {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: "class must use a classes: reference"})
}
}
rawLevels, ok := obj["levels"].(map[string]any)
if !ok {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: "class spellbook levels must be an object"})
return spec, diagnostics
}
for _, rawLevel := range sortedKeys(rawLevels) {
level, err := strconv.Atoi(rawLevel)
if err != nil || level < 0 {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: fmt.Sprintf("level key %q must be a nonnegative integer", rawLevel)})
continue
}
rawList, ok := rawLevels[rawLevel].([]any)
if !ok {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: fmt.Sprintf("level %d must be an array", level)})
continue
}
for index, rawSpell := range rawList {
spellKey, ok := rawSpell.(string)
if !ok || strings.TrimSpace(spellKey) == "" {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: fmt.Sprintf("level %d spell %d must be a spells: reference", level, index)})
continue
}
spellKey = strings.TrimSpace(spellKey)
if !strings.HasPrefix(spellKey, "spells:") {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: fmt.Sprintf("level %d spell %d must use a spells: reference", level, index)})
continue
}
spec.Levels[level] = append(spec.Levels[level], spellKey)
}
}
return spec, diagnostics
}
func collectedDatasetByName(collected []nativeCollectedDataset, name string) *nativeCollectedDataset {
for index := range collected {
if collected[index].Dataset.Name == name {
return &collected[index]
}
}
return nil
}
func rowsByKey(rows []map[string]any) map[string]map[string]any {
out := make(map[string]map[string]any, len(rows))
for _, row := range rows {
key, _ := row["key"].(string)
if key != "" {
out[key] = row
}
}
return out
}
func sortedIntKeys(values map[int][]string) []int {
keys := make([]int, 0, len(values))
for key := range values {
keys = append(keys, key)
}
slices.Sort(keys)
return keys
}
func diagnosticsTextForError(diags []Diagnostic) string {
messages := make([]string, 0, len(diags))
for _, diag := range diags {
messages = append(messages, diag.Message)
}
return strings.Join(messages, "; ")
}
internal/topdata/classes_migrate.go
-226
View File
@@ -1,226 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
"slices"
"strings"
)
var legacyClassesPlainFamilies = []string{"feats", "skills", "savthr", "bfeat", "pres"}
func importLegacyClasses(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
legacyRoot := filepath.Join(referenceBuilderDir, "data", "classes")
if _, err := os.Stat(legacyRoot); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetRoot := filepath.Join(dataDir, "classes")
if canonicalClassesPresent(targetRoot) {
return 0, nil
}
coreCollected, plainCollected, err := collectLegacyClassesDatasets(legacyRoot)
if err != nil {
return 0, err
}
tableKeyByOutput := map[string]string{}
for _, dataset := range plainCollected {
if dataset.TableKey == "" {
continue
}
registerLegacyClassesTableKey(tableKeyByOutput, dataset.TableKey, dataset.Dataset.OutputName)
}
coreRows := make([]map[string]any, 0, len(coreCollected.Rows))
for _, rawRow := range coreCollected.Rows {
row, ok := deepCopyValue(rawRow).(map[string]any)
if !ok {
continue
}
if legacyTLK != nil {
if _, _, err := inlineLegacyTLKValue(row, legacyTLK); err != nil {
return 0, err
}
}
rewriteLegacyClassesCoreRow(row, tableKeyByOutput)
coreRows = append(coreRows, row)
}
if err := removePathIfExists(targetRoot); err != nil {
return 0, err
}
if err := os.MkdirAll(filepath.Join(targetRoot, "core"), 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{
path: filepath.Join(targetRoot, "core", "base.json"),
obj: map[string]any{
"output": coreCollected.Dataset.OutputName,
"columns": stringSliceToAny(coreCollected.Columns),
"rows": rowsToAny(coreRows),
},
},
{
path: filepath.Join(targetRoot, "core", "lock.json"),
obj: anyMapInt(coreCollected.LockData),
},
}
for _, collected := range plainCollected {
obj := map[string]any{
"output": collected.Dataset.OutputName,
"columns": stringSliceToAny(collected.Columns),
"rows": rowsToAny(collected.Rows),
}
if strings.TrimSpace(collected.TableKey) != "" {
obj["key"] = collected.TableKey
}
writes = append(writes, struct {
path string
obj map[string]any
}{
path: filepath.Join(dataDir, collected.Dataset.Name+".json"),
obj: obj,
})
}
for _, write := range writes {
if err := os.MkdirAll(filepath.Dir(write.path), 0o755); err != nil {
return 0, err
}
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
func canonicalClassesPresent(targetRoot string) bool {
if !fileExists(filepath.Join(targetRoot, "core", "base.json")) || !fileExists(filepath.Join(targetRoot, "core", "lock.json")) {
return false
}
for _, family := range legacyClassesPlainFamilies {
ok, err := hasJSONFiles(filepath.Join(targetRoot, family))
if err != nil || !ok {
return false
}
}
return true
}
func collectLegacyClassesDatasets(legacyRoot string) (nativeCollectedDataset, []nativeCollectedDataset, error) {
coreCollected, err := collectBaseDataset(nativeDataset{
Name: "classes/core",
BasePath: filepath.Join(legacyRoot, "core", "base.json"),
LockPath: filepath.Join(legacyRoot, "core", "lock.json"),
ModulesDir: filepath.Join(legacyRoot, "core", "modules"),
OutputName: "classes.2da",
Spec: specForDataset("classes"),
})
if err != nil {
return nativeCollectedDataset{}, nil, err
}
coreCollected.Dataset.OutputName = "classes.2da"
plainCollected := make([]nativeCollectedDataset, 0)
for _, family := range legacyClassesPlainFamilies {
familyDir := filepath.Join(legacyRoot, family)
entries, err := os.ReadDir(familyDir)
if err != nil {
if os.IsNotExist(err) {
continue
}
return nativeCollectedDataset{}, nil, err
}
for _, entry := range entries {
if entry.IsDir() || filepath.Ext(entry.Name()) != ".json" || strings.HasPrefix(entry.Name(), ".") || entry.Name() == "lock.json" {
continue
}
filePath := filepath.Join(familyDir, entry.Name())
tableData, err := loadJSONObject(filePath)
if err != nil {
return nativeCollectedDataset{}, nil, err
}
outputName, _ := tableData["output"].(string)
if strings.TrimSpace(outputName) == "" {
outputName = strings.TrimSuffix(entry.Name(), filepath.Ext(entry.Name())) + ".2da"
}
collected, err := collectPlainDataset(nativeDataset{
Name: filepath.ToSlash(filepath.Join("classes", family, strings.TrimSuffix(entry.Name(), filepath.Ext(entry.Name())))),
BasePath: filePath,
LockPath: filepath.Join(familyDir, "lock.json"),
OutputName: outputName,
Spec: specForDataset(filepath.ToSlash(filepath.Join("classes", family, strings.TrimSuffix(entry.Name(), filepath.Ext(entry.Name()))))),
})
if err != nil {
return nativeCollectedDataset{}, nil, err
}
plainCollected = append(plainCollected, collected)
}
}
slices.SortFunc(plainCollected, func(a, b nativeCollectedDataset) int {
return strings.Compare(a.Dataset.Name, b.Dataset.Name)
})
return coreCollected, plainCollected, nil
}
func registerLegacyClassesTableKey(tableKeyByOutput map[string]string, tableKey, outputName string) {
trimmedOutput := strings.TrimSpace(outputName)
if trimmedOutput != "" {
tableKeyByOutput[trimmedOutput] = tableKey
}
trimmedStem := strings.TrimSpace(outputStem(outputName))
if trimmedStem != "" {
tableKeyByOutput[trimmedStem] = tableKey
}
}
func rewriteLegacyClassesCoreRow(row map[string]any, tableKeyByOutput map[string]string) {
for _, field := range []string{"FeatsTable", "SavingThrowTable", "SkillsTable", "BonusFeatsTable", "PreReqTable"} {
tableKey := legacyClassesTableKeyForValue(row[field], tableKeyByOutput)
if tableKey == "" {
continue
}
row[field] = map[string]any{"table": tableKey}
}
}
func legacyClassesTableKeyForValue(value any, tableKeyByOutput map[string]string) string {
switch typed := value.(type) {
case map[string]any:
tableKey, _ := typed["table"].(string)
return strings.TrimSpace(tableKey)
case string:
trimmed := strings.TrimSpace(typed)
if trimmed == "" || trimmed == nullValue {
return ""
}
if trimmed != strings.ToLower(trimmed) {
return ""
}
if tableKey, ok := tableKeyByOutput[trimmed]; ok {
return tableKey
}
return ""
default:
return ""
}
}
internal/topdata/cloakmodel_migrate.go
-6
View File
@@ -1,6 +0,0 @@
package topdata
func importLegacyCloakmodel(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "cloakmodel", "cloakmodel.2da", nil)
}
internal/topdata/convert.go
-888
View File
@@ -1,888 +0,0 @@
package topdata
import (
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"gitea.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
"gopkg.in/yaml.v3"
)
type convertOptions struct {
Namespace string
KeyFields []string
CollisionMode string
BaseDialog string
Type string
Name string
FilenamePrefixes map[string]string
}
func RunConvertCommand(args []string, stdout io.Writer) error {
if len(args) == 0 || args[0] == "--help" || args[0] == "-h" {
printConvertUsage(stdout)
return nil
}
switch args[0] {
case "2da-to-json":
if len(args) > 1 && (args[1] == "--help" || args[1] == "-h") {
_, _ = fmt.Fprintln(stdout, "usage: convert-topdata 2da-to-json [flags] <input.2da> <output.json>")
return nil
}
opts, input, output, err := parseConvertArgs(args[1:], false, "suffix")
if err != nil {
return err
}
data, err := parse2DAFile(input)
if err != nil {
return err
}
result, err := convert2DAToJSON(data, output, opts)
if err != nil {
return err
}
return writeJSONFile(output, result)
case "2da-to-module":
if len(args) > 1 && (args[1] == "--help" || args[1] == "-h") {
_, _ = fmt.Fprintln(stdout, "usage: convert-topdata 2da-to-module [--namespace <dataset>] [--name <module-name>] [--type entries|override] [flags] <input.2da> [output.json]")
return nil
}
opts, input, output, err := parseConvertArgs(args[1:], true, "error")
if err != nil {
return err
}
data, err := parse2DAFile(input)
if err != nil {
return err
}
if opts.Type == "override" {
result := map[string]any{"overrides": toOverridesRows(data.Rows)}
if err := writeJSONFile(output, result); err != nil {
return err
}
_, _ = fmt.Fprintf(stdout, "converted overrides: %d\n", len(result["overrides"].([]map[string]any)))
return nil
}
result, err := convert2DAToModule(data, output, opts)
if err != nil {
return err
}
if err := writeJSONFile(output, result); err != nil {
return err
}
_, _ = fmt.Fprintf(stdout, "converted entries: %d\n", len(result["entries"].(map[string]any)))
return nil
case "json-to-2da":
if len(args) > 1 && (args[1] == "--help" || args[1] == "-h") {
_, _ = fmt.Fprintln(stdout, "usage: convert-topdata json-to-2da <input.json> <output.2da>")
return nil
}
if len(args) != 3 {
return errors.New("usage: convert-topdata json-to-2da <input.json> <output.2da>")
}
data, err := readCanonicalJSON(args[1])
if err != nil {
return err
}
return write2DAFile(data, args[2])
default:
return fmt.Errorf("unknown convert-topdata subcommand %q", args[0])
}
}
func printConvertUsage(stdout io.Writer) {
_, _ = fmt.Fprintln(stdout, "usage: convert-topdata <2da-to-json|2da-to-module|json-to-2da> ...")
_, _ = fmt.Fprintln(stdout, "")
_, _ = fmt.Fprintln(stdout, "subcommands:")
_, _ = fmt.Fprintln(stdout, " 2da-to-json Convert a 2DA file into canonical JSON rows")
_, _ = fmt.Fprintln(stdout, " 2da-to-module Convert a 2DA file into module entries or override JSON")
_, _ = fmt.Fprintln(stdout, " json-to-2da Convert canonical JSON rows into a 2DA file")
_, _ = fmt.Fprintln(stdout, "")
_, _ = fmt.Fprintln(stdout, "2da-to-module notes:")
_, _ = fmt.Fprintln(stdout, " - --namespace is optional when topdata/templates/config.yaml declares it for the input table")
_, _ = fmt.Fprintln(stdout, " - --name controls inferred output naming: <prefix>_<namespace>_<name>.json")
_, _ = fmt.Fprintln(stdout, " - topdata/templates/config.yaml module_output.filename_prefixes can set entries/override prefixes")
_, _ = fmt.Fprintln(stdout, " - flags accept both --flag value and --flag=value forms")
_, _ = fmt.Fprintln(stdout, " - unkeyed rows now fail conversion instead of being skipped")
}
type parsed2DA struct {
Columns []string `json:"columns"`
Rows []map[string]any `json:"rows"`
}
func parseConvertArgs(args []string, allowFormat bool, defaultCollision string) (convertOptions, string, string, error) {
opts := convertOptions{
CollisionMode: defaultCollision,
Type: "entries",
}
positional := make([]string, 0, 2)
for index := 0; index < len(args); index++ {
arg := args[index]
switch arg {
case "--namespace":
index++
if index >= len(args) {
return opts, "", "", errors.New("--namespace requires a value")
}
opts.Namespace = args[index]
case "--key-field":
index++
if index >= len(args) {
return opts, "", "", errors.New("--key-field requires a value")
}
opts.KeyFields = append(opts.KeyFields, args[index])
case "--name":
index++
if index >= len(args) {
return opts, "", "", errors.New("--name requires a value")
}
opts.Name = args[index]
case "--collision":
index++
if index >= len(args) {
return opts, "", "", errors.New("--collision requires a value")
}
opts.CollisionMode = args[index]
case "--base-dialog":
index++
if index >= len(args) {
return opts, "", "", errors.New("--base-dialog requires a value")
}
opts.BaseDialog = args[index]
case "--format", "--type":
if !allowFormat {
return opts, "", "", fmt.Errorf("%s does not accept %s", positionalSafe(args), arg)
}
index++
if index >= len(args) {
return opts, "", "", fmt.Errorf("%s requires a value", arg)
}
opts.Type = args[index]
default:
if value, ok := parseInlineFlagValue(arg, "--namespace"); ok {
opts.Namespace = value
continue
}
if value, ok := parseInlineFlagValue(arg, "--key-field"); ok {
opts.KeyFields = append(opts.KeyFields, value)
continue
}
if value, ok := parseInlineFlagValue(arg, "--name"); ok {
opts.Name = value
continue
}
if value, ok := parseInlineFlagValue(arg, "--collision"); ok {
opts.CollisionMode = value
continue
}
if value, ok := parseInlineFlagValue(arg, "--base-dialog"); ok {
opts.BaseDialog = value
continue
}
if value, ok := parseInlineFlagValue(arg, "--format"); ok {
if !allowFormat {
return opts, "", "", fmt.Errorf("%s does not accept %s", positionalSafe(args), "--format")
}
opts.Type = value
continue
}
if value, ok := parseInlineFlagValue(arg, "--type"); ok {
if !allowFormat {
return opts, "", "", fmt.Errorf("%s does not accept %s", positionalSafe(args), "--type")
}
opts.Type = value
continue
}
if strings.HasPrefix(arg, "--") {
return opts, "", "", fmt.Errorf("unknown flag %q", arg)
}
positional = append(positional, arg)
}
}
if allowFormat {
if len(positional) < 1 || len(positional) > 2 {
return opts, "", "", errors.New("usage: convert-topdata 2da-to-module [--namespace <dataset>] [--name <module-name>] [--type entries|override] [flags] <input.2da> [output.json]")
}
} else if len(positional) != 2 {
return opts, "", "", errors.New("usage: convert-topdata 2da-to-json [flags] <input.2da> <output.json>")
}
if opts.CollisionMode != "suffix" && opts.CollisionMode != "error" {
return opts, "", "", fmt.Errorf("unsupported collision mode %q", opts.CollisionMode)
}
if allowFormat {
output := ""
if len(positional) == 2 {
output = positional[1]
}
ctx, err := resolveConvertContext(positional[0])
if err != nil {
return opts, "", "", err
}
if ctx.Project != nil {
if outputCtx, err := resolveConvertOutputContext(ctx.Project, output); err != nil {
return opts, "", "", err
} else if ctx.Template.Namespace == "" && len(ctx.Template.KeyFields) == 0 {
ctx.Template = outputCtx
}
}
if strings.TrimSpace(opts.Namespace) == "" {
opts.Namespace = ctx.Template.Namespace
}
if len(opts.KeyFields) == 0 && len(ctx.Template.KeyFields) > 0 {
opts.KeyFields = append(opts.KeyFields, ctx.Template.KeyFields...)
}
if len(opts.FilenamePrefixes) == 0 && len(ctx.Config.ModuleOutput.FilenamePrefixes) > 0 {
opts.FilenamePrefixes = cloneStringMap(ctx.Config.ModuleOutput.FilenamePrefixes)
}
if strings.TrimSpace(opts.Namespace) == "" {
return opts, "", "", errors.New("2da-to-module requires --namespace or a topdata/templates/config.yaml entry for the input table")
}
switch opts.Type {
case "entry":
opts.Type = "entries"
case "overrides":
opts.Type = "override"
}
if opts.Type != "entries" && opts.Type != "override" {
return opts, "", "", fmt.Errorf("unsupported module type %q", opts.Type)
}
if output == "" {
resolved, err := defaultModuleOutputPath(positional[0], opts, ctx.Project)
if err != nil {
return opts, "", "", err
}
output = resolved
}
return opts, positional[0], output, nil
}
ctx, err := resolveConvertContext(positional[0])
if err != nil {
return opts, "", "", err
}
if ctx.Project != nil {
if outputCtx, err := resolveConvertOutputContext(ctx.Project, positional[1]); err != nil {
return opts, "", "", err
} else if ctx.Template.Namespace == "" && len(ctx.Template.KeyFields) == 0 {
ctx.Template = outputCtx
}
}
if strings.TrimSpace(opts.Namespace) == "" {
opts.Namespace = ctx.Template.Namespace
}
if len(opts.KeyFields) == 0 && len(ctx.Template.KeyFields) > 0 {
opts.KeyFields = append(opts.KeyFields, ctx.Template.KeyFields...)
}
return opts, positional[0], positional[1], nil
}
func parseInlineFlagValue(arg, name string) (string, bool) {
prefix := name + "="
if !strings.HasPrefix(arg, prefix) {
return "", false
}
return strings.TrimSpace(arg[len(prefix):]), true
}
func positionalSafe(args []string) string {
if len(args) == 0 {
return "command"
}
return args[0]
}
func parse2DAFile(path string) (parsed2DA, error) {
raw, err := os.ReadFile(path)
if err != nil {
return parsed2DA{}, fmt.Errorf("read %s: %w", path, err)
}
lines := strings.Split(strings.ReplaceAll(string(raw), "\r\n", "\n"), "\n")
trimmed := make([]string, 0, len(lines))
for _, line := range lines {
line = strings.TrimSpace(line)
if line != "" {
trimmed = append(trimmed, line)
}
}
if len(trimmed) < 2 || !strings.HasPrefix(trimmed[0], "2DA") {
return parsed2DA{}, fmt.Errorf("%s: invalid 2DA file", path)
}
columns := split2DALine(trimmed[1])
if len(columns) == 0 {
return parsed2DA{}, fmt.Errorf("%s: missing 2DA columns", path)
}
rows := make([]map[string]any, 0, max(0, len(trimmed)-2))
seenIDs := map[int]struct{}{}
for _, line := range trimmed[2:] {
fields := split2DALine(line)
if len(fields) == 0 {
continue
}
rowID, err := strconv.Atoi(fields[0])
if err != nil {
return parsed2DA{}, fmt.Errorf("%s: invalid 2DA row id %q", path, fields[0])
}
if len(fields)-1 > len(columns) {
return parsed2DA{}, fmt.Errorf("%s: row %d has %d values but only %d columns are declared", path, rowID, len(fields)-1, len(columns))
}
if _, ok := seenIDs[rowID]; ok {
return parsed2DA{}, fmt.Errorf("%s: duplicate 2DA row id %d", path, rowID)
}
seenIDs[rowID] = struct{}{}
row := map[string]any{"id": rowID}
for index, column := range columns {
if index+1 < len(fields) {
row[column] = smartConvertScalar(fields[index+1])
continue
}
row[column] = nullValue
}
rows = append(rows, row)
}
return parsed2DA{Columns: columns, Rows: rows}, nil
}
func split2DALine(line string) []string {
fields := make([]string, 0)
var current strings.Builder
inQuotes := false
for _, ch := range line {
switch {
case ch == '"':
inQuotes = !inQuotes
case !inQuotes && (ch == '\t' || ch == ' '):
if current.Len() > 0 {
fields = append(fields, current.String())
current.Reset()
}
default:
current.WriteRune(ch)
}
}
if current.Len() > 0 {
fields = append(fields, current.String())
}
return fields
}
func smartConvertScalar(value string) any {
if value == "" {
return ""
}
if parsed, err := strconv.Atoi(value); err == nil {
return parsed
}
return value
}
func convert2DAToJSON(data parsed2DA, outputPath string, opts convertOptions) (map[string]any, error) {
rows, err := assignConvertedRowKeys(data.Rows, outputPath, opts)
if err != nil {
return nil, err
}
outRows := make([]map[string]any, 0, len(rows))
for _, row := range rows {
ordered := map[string]any{"id": row["id"]}
if key, ok := row["key"].(string); ok && key != "" {
ordered["key"] = key
}
for _, column := range data.Columns {
ordered[column] = row[column]
}
outRows = append(outRows, ordered)
}
return map[string]any{"columns": data.Columns, "rows": outRows}, nil
}
func convert2DAToModule(data parsed2DA, outputPath string, opts convertOptions) (map[string]any, error) {
rows, err := assignConvertedRowKeys(data.Rows, outputPath, opts)
if err != nil {
return nil, err
}
entries := map[string]any{}
missing := []string{}
for _, row := range rows {
key, _ := row["key"].(string)
if key == "" {
missing = append(missing, strconv.Itoa(row["id"].(int)))
continue
}
entry := map[string]any{}
for _, column := range data.Columns {
value := row[column]
if format2DAValue(value) == nullValue {
continue
}
entry[column] = value
}
entries[key] = entry
}
if len(missing) > 0 {
return nil, fmt.Errorf("unable to generate keys for row ids %s; declare --key-field or configure key_fields in topdata/templates/config.yaml", strings.Join(missing, ", "))
}
return map[string]any{"entries": entries}, nil
}
func toOverridesRows(rows []map[string]any) []map[string]any {
out := make([]map[string]any, 0, len(rows))
for _, row := range rows {
override := map[string]any{"id": row["id"]}
keys := make([]string, 0, len(row))
for key := range row {
if key == "id" || key == "key" {
continue
}
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
if format2DAValue(row[key]) == nullValue {
continue
}
override[key] = row[key]
}
out = append(out, override)
}
return out
}
func assignConvertedRowKeys(rows []map[string]any, outputPath string, opts convertOptions) ([]map[string]any, error) {
namespace := strings.TrimSpace(opts.Namespace)
out := make([]map[string]any, 0, len(rows))
used := map[string]struct{}{}
rowIDsByKey := map[string]int{}
for _, row := range rows {
cloned := cloneRowMap(row)
if namespace == "" {
out = append(out, cloned)
continue
}
candidates := convertedRowKeyCandidates(cloned, opts.KeyFields)
if len(candidates) == 0 {
out = append(out, cloned)
continue
}
key := ""
for _, candidate := range candidates {
candidateKey := namespace + ":" + candidate
if _, ok := used[candidateKey]; ok {
continue
}
key = candidateKey
break
}
if key == "" {
key = namespace + ":" + candidates[0]
if opts.CollisionMode == "error" {
return nil, fmt.Errorf("duplicate generated key %q for row %v (already used by row %d)", key, cloned["id"], rowIDsByKey[key])
}
key = fmt.Sprintf("%s_%v", key, cloned["id"])
}
used[key] = struct{}{}
rowID, _ := asInt(cloned["id"])
rowIDsByKey[key] = rowID
cloned["key"] = key
out = append(out, cloned)
}
return out, nil
}
var convertKeyWhitespace = regexp.MustCompile(`\s+`)
var convertKeyCleaner = regexp.MustCompile(`[^a-z0-9_]`)
func convertedRowKeyCandidates(row map[string]any, preferred []string) []string {
fields := preferred
if len(fields) == 0 {
fields = []string{"LABEL", "Label", "Name"}
} else {
for _, field := range fields {
value, ok := row[field]
if !ok {
return nil
}
text := strings.TrimSpace(format2DAValue(value))
if text == "" || text == nullValue {
return nil
}
}
}
values := make([]string, 0, len(fields))
seen := map[string]struct{}{}
candidates := make([]string, 0, len(fields))
for _, field := range fields {
value, ok := row[field]
if !ok {
continue
}
text := strings.TrimSpace(format2DAValue(value))
if text == "" || text == nullValue {
continue
}
normalized := normalizeConvertedKeyText(text)
if normalized == "" {
continue
}
values = append(values, normalized)
joined := strings.Join(values, "_")
if _, ok := seen[joined]; ok {
continue
}
seen[joined] = struct{}{}
candidates = append(candidates, joined)
}
return candidates
}
func normalizeConvertedKeyText(text string) string {
text = strings.TrimSpace(strings.ToLower(text))
text = convertKeyWhitespace.ReplaceAllString(text, "_")
text = convertKeyCleaner.ReplaceAllString(text, "")
text = strings.Trim(text, "_")
return text
}
type convertTemplateTable struct {
Namespace string `json:"namespace" yaml:"namespace"`
KeyFields []string `json:"key_fields" yaml:"key_fields"`
}
type convertTemplateConfig struct {
ModuleOutput convertModuleOutputConfig `json:"module_output" yaml:"module_output"`
Tables map[string]convertTemplateTable `json:"tables" yaml:"tables"`
}
type convertModuleOutputConfig struct {
FilenamePrefixes map[string]string `json:"filename_prefixes" yaml:"filename_prefixes"`
}
type convertContext struct {
Project *project.Project
Config convertTemplateConfig
Template convertTemplateTable
}
func resolveConvertContext(input string) (convertContext, error) {
cwd, err := os.Getwd()
if err != nil {
return convertContext{}, err
}
root, err := project.FindRoot(cwd)
if err != nil {
return convertContext{}, nil
}
p, err := project.Load(root)
if err != nil {
return convertContext{}, err
}
if !p.HasTopData() {
return convertContext{Project: p}, nil
}
config, err := loadConvertTemplateConfig(p)
if err != nil {
return convertContext{}, err
}
table, err := templateTableConfig(config, p, input)
if err != nil {
return convertContext{}, err
}
return convertContext{
Project: p,
Config: config,
Template: table,
}, nil
}
func loadTemplateTableConfig(p *project.Project, input string) (convertTemplateTable, error) {
config, err := loadConvertTemplateConfig(p)
if err != nil {
return convertTemplateTable{}, err
}
return templateTableConfig(config, p, input)
}
func templateTableConfig(config convertTemplateConfig, p *project.Project, input string) (convertTemplateTable, error) {
if len(config.Tables) == 0 {
return convertTemplateTable{}, nil
}
templatesDir := filepath.Join(p.TopDataSourceDir(), "templates")
inputAbs, err := filepath.Abs(input)
if err != nil {
return convertTemplateTable{}, err
}
templatesAbs, err := filepath.Abs(templatesDir)
if err != nil {
return convertTemplateTable{}, err
}
rel, err := filepath.Rel(templatesAbs, inputAbs)
if err != nil || strings.HasPrefix(rel, "..") || rel == "." {
return convertTemplateTable{}, nil
}
rel = filepath.ToSlash(rel)
base := filepath.Base(rel)
stem := strings.TrimSuffix(base, filepath.Ext(base))
for _, key := range []string{rel, base, stem} {
if table, ok := config.Tables[key]; ok {
return table, nil
}
}
return convertTemplateTable{}, nil
}
func loadConvertTemplateConfig(p *project.Project) (convertTemplateConfig, error) {
templatesDir := filepath.Join(p.TopDataSourceDir(), "templates")
for _, candidate := range []string{"config.yaml", "config.yml", "config.json"} {
configPath := filepath.Join(templatesDir, candidate)
raw, err := os.ReadFile(configPath)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
continue
}
return convertTemplateConfig{}, fmt.Errorf("read %s: %w", configPath, err)
}
var config convertTemplateConfig
switch filepath.Ext(configPath) {
case ".yaml", ".yml":
if err := yaml.Unmarshal(raw, &config); err != nil {
return convertTemplateConfig{}, fmt.Errorf("parse %s: %w", configPath, err)
}
default:
if err := json.Unmarshal(raw, &config); err != nil {
return convertTemplateConfig{}, fmt.Errorf("parse %s: %w", configPath, err)
}
}
if err := validateConvertTemplateConfig(config); err != nil {
return convertTemplateConfig{}, fmt.Errorf("validate %s: %w", configPath, err)
}
return config, nil
}
return convertTemplateConfig{}, nil
}
func validateConvertTemplateConfig(config convertTemplateConfig) error {
for moduleType, prefix := range config.ModuleOutput.FilenamePrefixes {
switch moduleType {
case "entries", "override":
default:
return fmt.Errorf("unsupported module_output.filename_prefixes key %q; expected entries or override", moduleType)
}
if normalizeModuleFilenamePrefix(prefix) == "" {
return fmt.Errorf("module_output.filename_prefixes.%s must contain at least one filename-safe character", moduleType)
}
}
return nil
}
func resolveConvertOutputContext(p *project.Project, output string) (convertTemplateTable, error) {
if p == nil || !p.HasTopData() || strings.TrimSpace(output) == "" {
return convertTemplateTable{}, nil
}
config, err := loadConvertTemplateConfig(p)
if err != nil {
return convertTemplateTable{}, err
}
if len(config.Tables) == 0 {
return convertTemplateTable{}, nil
}
dataDir := filepath.Join(p.TopDataSourceDir(), "data")
dataAbs, err := filepath.Abs(dataDir)
if err != nil {
return convertTemplateTable{}, err
}
outputAbs, err := filepath.Abs(output)
if err != nil {
return convertTemplateTable{}, err
}
rel, err := filepath.Rel(dataAbs, outputAbs)
if err != nil || rel == "." || strings.HasPrefix(rel, "..") {
return convertTemplateTable{}, nil
}
rel = filepath.ToSlash(rel)
bestMatch := ""
bestTable := convertTemplateTable{}
for _, table := range config.Tables {
namespacePath := namespaceToPath(table.Namespace)
if namespacePath == "" {
continue
}
if rel != namespacePath && !strings.HasPrefix(rel, namespacePath+"/") {
continue
}
if len(namespacePath) <= len(bestMatch) {
continue
}
bestMatch = namespacePath
bestTable = table
}
return bestTable, nil
}
func namespaceToPath(namespace string) string {
namespace = strings.TrimSpace(strings.ToLower(namespace))
if namespace == "" {
return ""
}
namespace = strings.ReplaceAll(namespace, "\\", "/")
namespace = strings.ReplaceAll(namespace, ":", "/")
namespace = convertKeyWhitespace.ReplaceAllString(namespace, "_")
namespace = strings.Trim(namespace, "/")
return namespace
}
func defaultModuleOutputPath(input string, opts convertOptions, p *project.Project) (string, error) {
if p == nil {
return "", errors.New("2da-to-module requires an explicit output path when not run inside a project root")
}
if !p.HasTopData() {
return "", errors.New("2da-to-module requires topdata to be configured when inferring an output path")
}
filename, err := defaultModuleFileName(input, opts, p)
if err != nil {
return "", err
}
modulesDir := filepath.Join(p.TopDataSourceDir(), "data", filepath.FromSlash(opts.Namespace), "modules")
return filepath.Join(modulesDir, filename), nil
}
func defaultModuleFileName(input string, opts convertOptions, p *project.Project) (string, error) {
if strings.TrimSpace(opts.Name) != "" {
name := normalizeConvertedKeyText(opts.Name)
if name == "" {
return "", fmt.Errorf("invalid --name %q", opts.Name)
}
namespace := normalizeModuleFilenameNamespace(opts.Namespace)
prefix := moduleFilenamePrefix(opts)
return fmt.Sprintf("%s_%s_%s.json", prefix, namespace, name), nil
}
if isTemplateInput(p, input) {
return "", errors.New("2da-to-module requires --name when converting from topdata/templates without an explicit output path")
}
return strings.TrimSuffix(filepath.Base(input), filepath.Ext(input)) + ".json", nil
}
func moduleFilenamePrefix(opts convertOptions) string {
moduleType := strings.TrimSpace(opts.Type)
if moduleType == "entry" {
moduleType = "entries"
}
if moduleType == "overrides" {
moduleType = "override"
}
if configured := strings.TrimSpace(opts.FilenamePrefixes[moduleType]); configured != "" {
if prefix := normalizeModuleFilenamePrefix(configured); prefix != "" {
return prefix
}
}
return "add"
}
func normalizeModuleFilenamePrefix(prefix string) string {
prefix = strings.TrimSpace(strings.ToLower(prefix))
prefix = strings.ReplaceAll(prefix, "/", "_")
prefix = strings.ReplaceAll(prefix, "\\", "_")
prefix = convertKeyWhitespace.ReplaceAllString(prefix, "_")
prefix = convertKeyCleaner.ReplaceAllString(prefix, "_")
prefix = strings.Trim(prefix, "_")
return prefix
}
func normalizeModuleFilenameNamespace(namespace string) string {
namespace = strings.TrimSpace(strings.ToLower(namespace))
namespace = strings.ReplaceAll(namespace, "/", "_")
namespace = strings.ReplaceAll(namespace, "\\", "_")
namespace = convertKeyWhitespace.ReplaceAllString(namespace, "_")
namespace = convertKeyCleaner.ReplaceAllString(namespace, "_")
namespace = strings.Trim(namespace, "_")
if namespace == "" {
return "module"
}
return namespace
}
func isTemplateInput(p *project.Project, input string) bool {
if p == nil || !p.HasTopData() {
return false
}
templatesAbs, err := filepath.Abs(filepath.Join(p.TopDataSourceDir(), "templates"))
if err != nil {
return false
}
inputAbs, err := filepath.Abs(input)
if err != nil {
return false
}
rel, err := filepath.Rel(templatesAbs, inputAbs)
if err != nil {
return false
}
return rel != "." && !strings.HasPrefix(rel, "..")
}
func readCanonicalJSON(path string) (parsed2DA, error) {
raw, err := os.ReadFile(path)
if err != nil {
return parsed2DA{}, err
}
var payload struct {
Columns []string `json:"columns"`
Rows []map[string]any `json:"rows"`
}
if err := json.Unmarshal(raw, &payload); err != nil {
return parsed2DA{}, err
}
return parsed2DA{Columns: payload.Columns, Rows: payload.Rows}, nil
}
func writeJSONFile(path string, payload any) error {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil && filepath.Dir(path) != "." {
return err
}
raw, err := json.MarshalIndent(payload, "", " ")
if err != nil {
return err
}
raw = append(raw, '\n')
return os.WriteFile(path, raw, 0o644)
}
func write2DAFile(data parsed2DA, path string) error {
rows := make([]map[string]any, 0, len(data.Rows))
for _, row := range data.Rows {
cloned := cloneRowMap(row)
if rowID, err := asInt(cloned["id"]); err == nil {
cloned["id"] = rowID
}
rows = append(rows, cloned)
}
sort.Slice(rows, func(i, j int) bool {
left, _ := asInt(rows[i]["id"])
right, _ := asInt(rows[j]["id"])
return left < right
})
table := map[string]any{
"columns": data.Columns,
"rows": rows,
}
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil && filepath.Dir(path) != "." {
return err
}
return write2DA(table, path, false, -1)
}
func cloneStringMap(in map[string]string) map[string]string {
out := make(map[string]string, len(in))
for key, value := range in {
out[key] = value
}
return out
}
internal/topdata/convert_test.go
-611
View File
@@ -1,611 +0,0 @@
package topdata
import (
"bytes"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
func TestRunConvertCommandInfersTemplateNamespaceAndOutput(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test
resref: test
paths:
source: src
assets: assets
build: build
topdata:
source: topdata
build: build/topdata
`)
mkdirAll(t, filepath.Join(root, "topdata", "templates"))
mkdirAll(t, filepath.Join(root, "topdata", "data", "appearance", "modules"))
writeFile(t, filepath.Join(root, "topdata", "templates", "config.yaml"), `
tables:
template_appearance:
namespace: appearance
key_fields:
- LABEL
`)
writeFile(t, filepath.Join(root, "topdata", "templates", "template_appearance.2da"), "2DA V2.0\n\nLABEL STRING_REF NAME\n960 \"Zombie, Ash, Hot\" **** Zombie_Ash_Hot\n961 \"Zombie, Ash, Done\" **** Zombie_Ash_Done\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
var stdout bytes.Buffer
if err := RunConvertCommand([]string{
"2da-to-module",
"--name", "ashzombies",
"topdata/templates/template_appearance.2da",
}, &stdout); err != nil {
t.Fatalf("RunConvertCommand failed: %v", err)
}
outputPath := filepath.Join(root, "topdata", "data", "appearance", "modules", "add_appearance_ashzombies.json")
raw, err := os.ReadFile(outputPath)
if err != nil {
t.Fatalf("read output: %v", err)
}
text := string(raw)
if !strings.Contains(text, `"appearance:zombie_ash_hot"`) || !strings.Contains(text, `"appearance:zombie_ash_done"`) {
t.Fatalf("unexpected output:\n%s", text)
}
if !strings.Contains(stdout.String(), "converted entries: 2") {
t.Fatalf("unexpected stdout: %s", stdout.String())
}
}
func TestRunConvertCommandInfersOverrideOutputPrefixFromTemplateConfig(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test
resref: test
paths:
source: src
assets: assets
build: build
topdata:
source: topdata
build: build/topdata
`)
mkdirAll(t, filepath.Join(root, "topdata", "templates"))
writeFile(t, filepath.Join(root, "topdata", "templates", "config.yaml"), `
module_output:
filename_prefixes:
entries: add
override: ovr
tables:
template_appearance:
namespace: appearance
key_fields:
- LABEL
`)
writeFile(t, filepath.Join(root, "topdata", "templates", "template_appearance.2da"), "2DA V2.0\n\nLABEL STRING_REF NAME\n960 \"Zombie, Ash, Hot\" **** Zombie_Ash_Hot\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
var stdout bytes.Buffer
if err := RunConvertCommand([]string{
"2da-to-module",
"--type", "override",
"--name", "ashzombies",
"topdata/templates/template_appearance.2da",
}, &stdout); err != nil {
t.Fatalf("RunConvertCommand failed: %v", err)
}
outputPath := filepath.Join(root, "topdata", "data", "appearance", "modules", "ovr_appearance_ashzombies.json")
raw, err := os.ReadFile(outputPath)
if err != nil {
t.Fatalf("read output: %v", err)
}
if !strings.Contains(string(raw), `"overrides"`) {
t.Fatalf("expected override payload, got:\n%s", string(raw))
}
if _, err := os.Stat(filepath.Join(root, "topdata", "data", "appearance", "modules", "add_appearance_ashzombies.json")); !os.IsNotExist(err) {
t.Fatalf("expected add-prefixed output to be absent, stat err: %v", err)
}
if !strings.Contains(stdout.String(), "converted overrides: 1") {
t.Fatalf("unexpected stdout: %s", stdout.String())
}
}
func TestRunConvertCommand2DAToJSONInfersTemplateKeys(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test
resref: test
paths:
source: src
assets: assets
build: build
topdata:
source: topdata
build: build/topdata
`)
mkdirAll(t, filepath.Join(root, "topdata", "templates"))
writeFile(t, filepath.Join(root, "topdata", "templates", "config.yaml"), `
tables:
template_appearance:
namespace: appearance
key_fields:
- LABEL
`)
inputPath := filepath.Join(root, "topdata", "templates", "template_appearance.2da")
writeFile(t, inputPath, "2DA V2.0\n\nLABEL STRING_REF NAME\n960 \"Zombie, Ash, Hot\" **** Zombie_Ash_Hot\n961 \"Zombie, Ash, Done\" **** Zombie_Ash_Done\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
outputPath := filepath.Join(root, "out", "appearance.json")
if err := RunConvertCommand([]string{
"2da-to-json",
"topdata/templates/template_appearance.2da",
outputPath,
}, &bytes.Buffer{}); err != nil {
t.Fatalf("RunConvertCommand failed: %v", err)
}
raw, err := os.ReadFile(outputPath)
if err != nil {
t.Fatalf("read output: %v", err)
}
var payload struct {
Rows []map[string]any `json:"rows"`
}
if err := json.Unmarshal(raw, &payload); err != nil {
t.Fatalf("unmarshal output: %v", err)
}
if got := payload.Rows[0]["key"]; got != "appearance:zombie_ash_hot" {
t.Fatalf("expected first key to be inferred from template config, got %#v", got)
}
if got := payload.Rows[1]["key"]; got != "appearance:zombie_ash_done" {
t.Fatalf("expected second key to be inferred from template config, got %#v", got)
}
}
func TestRunConvertCommand2DAToJSONInfersOutputDatasetKeysFromConfig(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test
resref: test
paths:
source: src
assets: assets
build: build
topdata:
source: topdata
build: build/topdata
`)
mkdirAll(t, filepath.Join(root, "topdata", "templates"))
mkdirAll(t, filepath.Join(root, "topdata", "data", "soundset"))
writeFile(t, filepath.Join(root, "topdata", "templates", "config.yaml"), `
tables:
template_soundset:
namespace: soundset
key_fields:
- RESREF
- LABEL
`)
inputPath := filepath.Join(root, "scratch_soundset.2da")
writeFile(t, inputPath, "2DA V2.0\n\nLABEL RESREF STRREF GENDER TYPE\n0 \"Bandit Male\" nw_bandit 100 1 4\n1 \"Bandit Female\" nw_bandit 101 2 4\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
outputPath := filepath.Join(root, "topdata", "data", "soundset", "base.json")
if err := RunConvertCommand([]string{
"2da-to-json",
"scratch_soundset.2da",
outputPath,
}, &bytes.Buffer{}); err != nil {
t.Fatalf("RunConvertCommand failed: %v", err)
}
raw, err := os.ReadFile(outputPath)
if err != nil {
t.Fatalf("read output: %v", err)
}
var payload struct {
Rows []map[string]any `json:"rows"`
}
if err := json.Unmarshal(raw, &payload); err != nil {
t.Fatalf("unmarshal output: %v", err)
}
if got := payload.Rows[0]["key"]; got != "soundset:nw_bandit" {
t.Fatalf("expected first soundset key, got %#v", got)
}
if got := payload.Rows[1]["key"]; got != "soundset:nw_bandit_bandit_female" {
t.Fatalf("expected second soundset key to use config disambiguation, got %#v", got)
}
}
func TestRunConvertCommand2DAToJSONInfersAmbientTemplateKeysFromResource(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test
resref: test
paths:
source: src
assets: assets
build: build
topdata:
source: topdata
build: build/topdata
`)
mkdirAll(t, filepath.Join(root, "topdata", "templates"))
writeFile(t, filepath.Join(root, "topdata", "templates", "config.yaml"), `
tables:
template_ambientmusic:
namespace: ambientmusic
key_fields:
- Resource
`)
inputPath := filepath.Join(root, "topdata", "templates", "template_ambientmusic.2da")
writeFile(t, inputPath, "2DA V2.0\n\nDescription DisplayName Resource Stinger1 Stinger2 Stinger3\n0 61901 **** **** **** **** ****\n1 61842 **** mus_ruralday1 **** **** ****\n2 61843 **** mus_ruralday2 **** **** ****\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
outputPath := filepath.Join(root, "out", "ambientmusic.json")
if err := RunConvertCommand([]string{
"2da-to-json",
"topdata/templates/template_ambientmusic.2da",
outputPath,
}, &bytes.Buffer{}); err != nil {
t.Fatalf("RunConvertCommand failed: %v", err)
}
raw, err := os.ReadFile(outputPath)
if err != nil {
t.Fatalf("read output: %v", err)
}
var payload struct {
Rows []map[string]any `json:"rows"`
}
if err := json.Unmarshal(raw, &payload); err != nil {
t.Fatalf("unmarshal output: %v", err)
}
if _, ok := payload.Rows[0]["key"]; ok {
t.Fatalf("expected null Resource row to remain unkeyed, got %#v", payload.Rows[0]["key"])
}
if got := payload.Rows[1]["key"]; got != "ambientmusic:mus_ruralday1" {
t.Fatalf("expected first populated resource row key, got %#v", got)
}
if got := payload.Rows[2]["key"]; got != "ambientmusic:mus_ruralday2" {
t.Fatalf("expected second populated resource row key, got %#v", got)
}
}
func TestRunConvertCommand2DAToJSONSuffixesDuplicateGeneratedKeys(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test
resref: test
paths:
source: src
assets: assets
build: build
topdata:
source: topdata
build: build/topdata
`)
inputPath := filepath.Join(root, "dup.2da")
writeFile(t, inputPath, "2DA V2.0\n\nResource\n1 cmp_reserved\n2 cmp_reserved\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
outputPath := filepath.Join(root, "out", "ambientmusic.json")
if err := RunConvertCommand([]string{
"2da-to-json",
"--namespace", "ambientmusic",
"--key-field", "Resource",
inputPath,
outputPath,
}, &bytes.Buffer{}); err != nil {
t.Fatalf("RunConvertCommand failed: %v", err)
}
raw, err := os.ReadFile(outputPath)
if err != nil {
t.Fatalf("read output: %v", err)
}
var payload struct {
Rows []map[string]any `json:"rows"`
}
if err := json.Unmarshal(raw, &payload); err != nil {
t.Fatalf("unmarshal output: %v", err)
}
if got := payload.Rows[0]["key"]; got != "ambientmusic:cmp_reserved" {
t.Fatalf("expected first duplicate key to remain unsuffixed, got %#v", got)
}
if got := payload.Rows[1]["key"]; got != "ambientmusic:cmp_reserved_2" {
t.Fatalf("expected second duplicate key to gain row-id suffix, got %#v", got)
}
}
func TestParseConvertArgsSupportsEqualsSyntax(t *testing.T) {
opts, input, output, err := parseConvertArgs([]string{
"--namespace=environment",
"--name=projectq",
"--type=entries",
"--collision=suffix",
"--key-field=Label",
"input.2da",
"output.json",
}, true, "error")
if err != nil {
t.Fatalf("parseConvertArgs failed: %v", err)
}
if opts.Namespace != "environment" {
t.Fatalf("expected namespace from equals syntax, got %q", opts.Namespace)
}
if opts.Name != "projectq" {
t.Fatalf("expected name from equals syntax, got %q", opts.Name)
}
if opts.Type != "entries" {
t.Fatalf("expected type from equals syntax, got %q", opts.Type)
}
if opts.CollisionMode != "suffix" {
t.Fatalf("expected collision from equals syntax, got %q", opts.CollisionMode)
}
if len(opts.KeyFields) != 1 || opts.KeyFields[0] != "Label" {
t.Fatalf("expected key field from equals syntax, got %#v", opts.KeyFields)
}
if input != "input.2da" || output != "output.json" {
t.Fatalf("unexpected positional parse result: input=%q output=%q", input, output)
}
}
func TestConvert2DAToJSONSkipsConfiguredKeysWhenAnyKeyFieldIsNull(t *testing.T) {
result, err := convert2DAToJSON(parsed2DA{
Columns: []string{"LABEL", "RESREF", "STRREF"},
Rows: []map[string]any{
{"id": 309, "LABEL": "Unused", "RESREF": "unused", "STRREF": 1},
{"id": 310, "LABEL": nullValue, "RESREF": "unused", "STRREF": 2},
},
}, "topdata/data/soundset/base.json", convertOptions{
Namespace: "soundset",
KeyFields: []string{"RESREF", "LABEL"},
CollisionMode: "error",
})
if err != nil {
t.Fatalf("convert2DAToJSON failed: %v", err)
}
rows := result["rows"].([]map[string]any)
if got := rows[0]["key"]; got != "soundset:unused" {
t.Fatalf("expected first row key to use the primary configured key field, got %#v", got)
}
if _, ok := rows[1]["key"]; ok {
t.Fatalf("expected second row to remain unkeyed when one configured key field is null, got %#v", rows[1]["key"])
}
}
func TestRunConvertCommandFailsOnUnkeyedRows(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.json"), `{
"module": {"name": "Test", "resref": "test"},
"paths": {"source": "src", "assets": "assets", "build": "build"},
"topdata": {"source": "topdata", "build": "build/topdata"}
}`+"\n")
mkdirAll(t, filepath.Join(root, "topdata", "templates"))
writeFile(t, filepath.Join(root, "topdata", "templates", "config.yaml"), `
tables:
template_appearance:
namespace: appearance
key_fields:
- STRING_REF
`)
writeFile(t, filepath.Join(root, "topdata", "templates", "template_appearance.2da"), "2DA V2.0\n\nLABEL STRING_REF NAME\n960 \"Zombie, Ash, Hot\" **** Zombie_Ash_Hot\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
err = RunConvertCommand([]string{
"2da-to-module",
"--name", "ashzombies",
"topdata/templates/template_appearance.2da",
}, &bytes.Buffer{})
if err == nil || !strings.Contains(err.Error(), "unable to generate keys for row ids 960") {
t.Fatalf("expected unkeyed row failure, got %v", err)
}
}
func TestRunConvertCommandFallsBackToLegacyTemplateJSONConfig(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test
resref: test
paths:
source: src
assets: assets
build: build
topdata:
source: topdata
build: build/topdata
`)
mkdirAll(t, filepath.Join(root, "topdata", "templates"))
writeFile(t, filepath.Join(root, "topdata", "templates", "config.json"), `{
"tables": {
"template_appearance": {
"namespace": "appearance",
"key_fields": ["LABEL"]
}
}
}`+"\n")
inputPath := filepath.Join(root, "topdata", "templates", "template_appearance.2da")
writeFile(t, inputPath, "2DA V2.0\n\nLABEL\n960 Zombie_Ash_Hot\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
outputPath := filepath.Join(root, "out", "appearance.json")
if err := RunConvertCommand([]string{
"2da-to-json",
"topdata/templates/template_appearance.2da",
outputPath,
}, &bytes.Buffer{}); err != nil {
t.Fatalf("RunConvertCommand failed: %v", err)
}
raw, err := os.ReadFile(outputPath)
if err != nil {
t.Fatalf("read output: %v", err)
}
if !strings.Contains(string(raw), `"appearance:zombie_ash_hot"`) {
t.Fatalf("expected legacy JSON template config fallback to be honored, got %s", string(raw))
}
}
func TestRunConvertCommandRejectsUnknownTemplateFilenamePrefixType(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test
resref: test
paths:
source: src
assets: assets
build: build
topdata:
source: topdata
build: build/topdata
`)
mkdirAll(t, filepath.Join(root, "topdata", "templates"))
writeFile(t, filepath.Join(root, "topdata", "templates", "config.yaml"), `
module_output:
filename_prefixes:
additions: add
tables:
template_appearance:
namespace: appearance
key_fields:
- LABEL
`)
writeFile(t, filepath.Join(root, "topdata", "templates", "template_appearance.2da"), "2DA V2.0\n\nLABEL\n960 Zombie_Ash_Hot\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
err = RunConvertCommand([]string{
"2da-to-module",
"--name", "ashzombies",
"topdata/templates/template_appearance.2da",
}, &bytes.Buffer{})
if err == nil || !strings.Contains(err.Error(), `unsupported module_output.filename_prefixes key "additions"`) {
t.Fatalf("expected invalid filename prefix type error, got %v", err)
}
}
func TestConvert2DAToModuleFailsOnDuplicateGeneratedKeysByDefault(t *testing.T) {
_, err := convert2DAToModule(parsed2DA{
Columns: []string{"LABEL"},
Rows: []map[string]any{
{"id": 1, "LABEL": "Zombie"},
{"id": 2, "LABEL": "Zombie"},
},
}, "topdata/data/appearance/modules/add_appearance_ashzombies.json", convertOptions{
Namespace: "appearance",
CollisionMode: "error",
})
if err == nil || !strings.Contains(err.Error(), `duplicate generated key "appearance:zombie"`) {
t.Fatalf("expected duplicate generated key error, got %v", err)
}
}
func TestParse2DAFileRejectsRowsWithTooManyFields(t *testing.T) {
root := t.TempDir()
inputPath := filepath.Join(root, "bad.2da")
writeFile(t, inputPath, "2DA V2.0\n\nLABEL NAME\n0 one two three\n")
_, err := parse2DAFile(inputPath)
if err == nil || !strings.Contains(err.Error(), "has 3 values but only 2 columns are declared") {
t.Fatalf("expected row width error, got %v", err)
}
}
internal/topdata/creaturespeed_migrate.go
-145
View File
@@ -1,145 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyCreaturespeed(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "creaturespeed")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "creaturespeed")
targetPath := filepath.Join(targetDir, "creaturespeed.json")
if _, err := os.Stat(targetPath); err == nil {
obj, err := loadJSONObject(targetPath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
rows, err := mergeCreaturespeedRows(baseObj, filepath.Join(legacyDir, "modules"))
if err != nil {
return 0, err
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "base.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "lock.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "modules")); err != nil {
return 0, err
}
out := map[string]any{
"output": "creaturespeed.2da",
"columns": baseObj["columns"],
"rows": rows,
}
raw, err := json.MarshalIndent(out, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
return 0, err
}
return 1, nil
}
func mergeCreaturespeedRows(baseObj map[string]any, modulesDir string) ([]any, error) {
rawRows, ok := baseObj["rows"].([]any)
if !ok {
return nil, nil
}
rows := make([]map[string]any, 0, len(rawRows))
byID := map[int]map[string]any{}
for _, raw := range rawRows {
row, ok := deepCopyValue(raw).(map[string]any)
if !ok {
continue
}
delete(row, "key")
rows = append(rows, row)
if rawID, ok := row["id"]; ok {
id, err := asInt(rawID)
if err != nil {
return nil, err
}
byID[id] = row
}
}
modulePaths, err := collectModulePaths(modulesDir)
if err != nil {
return nil, err
}
for _, path := range modulePaths {
obj, err := loadJSONObject(path)
if err != nil {
return nil, err
}
overrideList, ok := obj["overrides"].([]any)
if !ok {
continue
}
for _, raw := range overrideList {
override, ok := raw.(map[string]any)
if !ok {
continue
}
rawID, ok := override["id"]
if !ok {
continue
}
id, err := asInt(rawID)
if err != nil {
return nil, err
}
row := byID[id]
if row == nil {
continue
}
for key, value := range override {
if key == "id" || key == "key" || key == "_tlk" {
continue
}
row[key] = deepCopyValue(value)
}
}
}
out := make([]any, 0, len(rows))
for _, row := range rows {
out = append(out, row)
}
return out, nil
}
func removePathIfExists(path string) error {
if _, err := os.Stat(path); err != nil {
if os.IsNotExist(err) {
return nil
}
return err
}
return os.RemoveAll(path)
}
internal/topdata/damagetypes_migrate.go
-288
View File
@@ -1,288 +0,0 @@
package topdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
)
func importLegacyDamagetypes(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyRoot := filepath.Join(referenceBuilderDir, "data", "damagetypes")
if _, err := os.Stat(legacyRoot); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "damagetypes", "registry")
targetPath := filepath.Join(targetDir, "types.json")
if _, err := os.Stat(targetPath); err == nil {
obj, err := loadJSONObject(targetPath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
coreBase, err := loadJSONObject(filepath.Join(legacyRoot, "core", "base.json"))
if err != nil {
return 0, err
}
groupBase, err := loadJSONObject(filepath.Join(legacyRoot, "groups", "base.json"))
if err != nil {
return 0, err
}
hitvisualBase, err := loadJSONObject(filepath.Join(legacyRoot, "hitvisual", "base.json"))
if err != nil {
return 0, err
}
coreModule, err := loadJSONObject(filepath.Join(legacyRoot, "core", "modules", "add_eos.json"))
if err != nil {
return 0, err
}
coreLock, err := loadLockfile(filepath.Join(legacyRoot, "core", "lock.json"))
if err != nil {
return 0, err
}
groupModule, err := loadJSONObject(filepath.Join(legacyRoot, "groups", "modules", "add_eos.json"))
if err != nil {
return 0, err
}
groupLock, err := loadLockfile(filepath.Join(legacyRoot, "groups", "lock.json"))
if err != nil {
return 0, err
}
hitvisualModule, err := loadJSONObject(filepath.Join(legacyRoot, "hitvisual", "modules", "add_eos.json"))
if err != nil {
return 0, err
}
hitvisualLock, err := loadLockfile(filepath.Join(legacyRoot, "hitvisual", "lock.json"))
if err != nil {
return 0, err
}
rows, lockData, err := mergeLegacyDamagetypes(coreBase, groupBase, hitvisualBase, coreModule, coreLock, groupModule, groupLock, hitvisualModule, hitvisualLock)
if err != nil {
return 0, err
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(dataDir, "damagetypes", "core")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(dataDir, "damagetypes", "groups")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(dataDir, "damagetypes", "hitvisual")); err != nil {
return 0, err
}
typesOut := map[string]any{"rows": rows}
raw, err := json.MarshalIndent(typesOut, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
return 0, err
}
if err := saveLockfile(filepath.Join(targetDir, damagetypesRegistryLock), lockData); err != nil {
return 0, err
}
return 2, nil
}
func mergeLegacyDamagetypes(coreBase, groupBase, hitvisualBase, coreModule map[string]any, coreLock map[string]int, groupModule map[string]any, groupLock map[string]int, hitvisualModule map[string]any, hitvisualLock map[string]int) ([]map[string]any, map[string]int, error) {
coreRows, err := coerceRowMapSlice(coreBase["rows"])
if err != nil {
return nil, nil, err
}
groupRows, err := coerceRowMapSlice(groupBase["rows"])
if err != nil {
return nil, nil, err
}
hitRows, err := coerceRowMapSlice(hitvisualBase["rows"])
if err != nil {
return nil, nil, err
}
groupByID := map[int]map[string]any{}
for _, row := range groupRows {
id, err := asInt(row["id"])
if err != nil {
return nil, nil, err
}
groupByID[id] = row
}
hitByID := map[int]map[string]any{}
for _, row := range hitRows {
id, err := asInt(row["id"])
if err != nil {
return nil, nil, err
}
hitByID[id] = row
}
outRows := make([]map[string]any, 0, len(coreRows))
lockData := map[string]int{}
for _, core := range coreRows {
id, err := asInt(core["id"])
if err != nil {
return nil, nil, err
}
groupID, err := asInt(core["DamageTypeGroup"])
if err != nil {
return nil, nil, err
}
group := groupByID[groupID]
if group == nil {
return nil, nil, fmt.Errorf("core id %d references missing group %d", id, groupID)
}
hit := hitByID[id]
if hit == nil {
return nil, nil, fmt.Errorf("core id %d is missing hitvisual row", id)
}
key := "damagetype:" + normalizeDamagetypeKey(core["Label"])
lockData[key] = id
outRows = append(outRows, map[string]any{
"key": key,
"id": id,
"label": deepCopyValue(core["Label"]),
"charsheet_strref": deepCopyValue(core["CharsheetStrref"]),
"damage_type_group": strconvString(groupID),
"damage_ranged_projectile": deepCopyValue(core["DamageRangedProjectile"]),
"group_label": deepCopyValue(group["Label"]),
"feedback_strref": deepCopyValue(group["FeedbackStrref"]),
"color_r": deepCopyValue(group["ColorR"]),
"color_g": deepCopyValue(group["ColorG"]),
"color_b": deepCopyValue(group["ColorB"]),
"visual_effect_id": deepCopyValue(hit["VisualEffectID"]),
"ranged_effect_id": deepCopyValue(hit["RangedEffectID"]),
})
}
coreEntries, err := coerceEntriesMap(coreModule["entries"])
if err != nil {
return nil, nil, err
}
groupEntries, err := coerceEntriesMap(groupModule["entries"])
if err != nil {
return nil, nil, err
}
hitEntries, err := coerceEntriesMap(hitvisualModule["entries"])
if err != nil {
return nil, nil, err
}
keys := make([]string, 0, len(coreEntries))
for key := range coreEntries {
keys = append(keys, key)
}
sort.Slice(keys, func(i, j int) bool {
a := strings.TrimPrefix(keys[i], "damagetypes:")
b := strings.TrimPrefix(keys[j], "damagetypes:")
return a < b
})
for _, coreKey := range keys {
core := coreEntries[coreKey]
suffix := strings.TrimPrefix(coreKey, "damagetypes:")
group := groupEntries["damagetypegroups:"+suffix]
if group == nil {
return nil, nil, fmt.Errorf("%s: missing matching group entry", coreKey)
}
hit := hitEntries["damagehitvisual:"+suffix]
if hit == nil {
return nil, nil, fmt.Errorf("%s: missing matching hitvisual entry", coreKey)
}
id, ok := coreLock[coreKey]
if !ok {
return nil, nil, fmt.Errorf("%s: missing core lock id", coreKey)
}
groupKey := "damagetypegroups:" + suffix
groupID, ok := groupLock[groupKey]
if !ok {
return nil, nil, fmt.Errorf("%s: missing group lock id", groupKey)
}
hitKey := "damagehitvisual:" + suffix
hitID, ok := hitvisualLock[hitKey]
if !ok {
return nil, nil, fmt.Errorf("%s: missing hitvisual lock id", hitKey)
}
if hitID != id {
return nil, nil, fmt.Errorf("%s: core id %d does not match hitvisual id %d", suffix, id, hitID)
}
key := "damagetype:" + suffix
lockData[key] = id
outRows = append(outRows, map[string]any{
"key": key,
"id": id,
"label": deepCopyValue(core["Label"]),
"charsheet_strref": deepCopyValue(core["CharsheetStrref"]),
"damage_type_group": strconvString(groupID),
"damage_ranged_projectile": deepCopyValue(core["DamageRangedProjectile"]),
"group_label": deepCopyValue(group["Label"]),
"feedback_strref": deepCopyValue(group["FeedbackStrref"]),
"color_r": deepCopyValue(group["ColorR"]),
"color_g": deepCopyValue(group["ColorG"]),
"color_b": deepCopyValue(group["ColorB"]),
"visual_effect_id": deepCopyValue(hit["VisualEffectID"]),
"ranged_effect_id": deepCopyValue(hit["RangedEffectID"]),
})
}
sort.Slice(outRows, func(i, j int) bool {
return outRows[i]["id"].(int) < outRows[j]["id"].(int)
})
return outRows, lockData, nil
}
func coerceRowMapSlice(value any) ([]map[string]any, error) {
rawRows, ok := value.([]any)
if !ok {
return nil, fmt.Errorf("rows must be an array")
}
rows := make([]map[string]any, 0, len(rawRows))
for _, raw := range rawRows {
row, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("row must be an object")
}
rows = append(rows, row)
}
return rows, nil
}
func coerceEntriesMap(value any) (map[string]map[string]any, error) {
rawEntries, ok := value.(map[string]any)
if !ok {
return nil, fmt.Errorf("entries must be an object")
}
entries := make(map[string]map[string]any, len(rawEntries))
for key, raw := range rawEntries {
row, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("entry %s must be an object", key)
}
entries[key] = row
}
return entries, nil
}
func normalizeDamagetypeKey(value any) string {
text := strings.TrimSpace(strings.ToLower(format2DAValue(value)))
text = strings.TrimSuffix(text, "damage")
text = strings.ReplaceAll(text, " ", "")
text = strings.ReplaceAll(text, "_", "")
text = strings.ReplaceAll(text, "-", "")
return text
}
func strconvString(v int) string {
return fmt.Sprintf("%d", v)
}
internal/topdata/damagetypes_registry.go
-243
View File
@@ -1,243 +0,0 @@
package topdata
import (
"fmt"
"os"
"path/filepath"
"slices"
"strconv"
"strings"
)
const (
damagetypesRegistryDirName = "damagetypes/registry"
damagetypesRegistryLock = "lock.json"
)
type damagetypesRegistry struct {
RootPath string
LockPath string
LockData map[string]int
Types []map[string]any
}
func collectGeneratedRegistryDatasets(dataDir string) ([]nativeCollectedDataset, error) {
itempropsDatasets, err := collectItempropsRegistryDatasets(dataDir)
if err != nil {
return nil, err
}
damagetypeDatasets, err := collectDamagetypesRegistryDatasets(dataDir)
if err != nil {
return nil, err
}
racialtypesDatasets, err := collectRacialtypesRegistryDatasets(dataDir)
if err != nil {
return nil, err
}
out := make([]nativeCollectedDataset, 0, len(itempropsDatasets)+len(damagetypeDatasets)+len(racialtypesDatasets))
out = append(out, itempropsDatasets...)
out = append(out, damagetypeDatasets...)
out = append(out, racialtypesDatasets...)
return out, nil
}
func loadDamagetypesRegistry(dataDir string) (*damagetypesRegistry, error) {
root := filepath.Join(dataDir, filepath.FromSlash(damagetypesRegistryDirName))
info, err := os.Stat(root)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
if !info.IsDir() {
return nil, fmt.Errorf("damagetypes registry root must be a directory: %s", root)
}
lockData, err := loadLockfile(filepath.Join(root, damagetypesRegistryLock))
if err != nil {
return nil, err
}
types, err := loadRegistryRows(filepath.Join(root, "types.json"))
if err != nil {
return nil, err
}
return &damagetypesRegistry{
RootPath: root,
LockPath: filepath.Join(root, damagetypesRegistryLock),
LockData: lockData,
Types: types,
}, nil
}
func collectDamagetypesRegistryDatasets(dataDir string) ([]nativeCollectedDataset, error) {
registry, err := loadDamagetypesRegistry(dataDir)
if err != nil {
return nil, err
}
if registry == nil {
return nil, nil
}
typeIDs, lockModified, err := assignDamagetypeRegistryIDs(registry.Types, registry.LockData)
if err != nil {
return nil, err
}
if lockModified {
if err := saveLockfile(registry.LockPath, registry.LockData); err != nil {
return nil, err
}
}
typeRowsSorted := slices.Clone(registry.Types)
slices.SortFunc(typeRowsSorted, func(a, b map[string]any) int {
return typeIDs[stringField(a, "key")] - typeIDs[stringField(b, "key")]
})
coreRows := make([]map[string]any, 0, len(typeRowsSorted))
hitvisualRows := make([]map[string]any, 0, len(typeRowsSorted))
groupRowsByID := map[int]map[string]any{}
groupOrder := make([]int, 0)
for _, row := range typeRowsSorted {
key := stringField(row, "key")
rowID := typeIDs[key]
groupID, err := registryIntField(row, "damage_type_group")
if err != nil {
return nil, fmt.Errorf("%s: %w", key, err)
}
coreRows = append(coreRows, map[string]any{
"id": rowID,
"key": key,
"Label": registryStringField(row, "label"),
"CharsheetStrref": deepCopyValue(row["charsheet_strref"]),
"DamageTypeGroup": strconv.Itoa(groupID),
"DamageRangedProjectile": registryNullableField(row, "damage_ranged_projectile"),
})
hitvisualRows = append(hitvisualRows, map[string]any{
"id": rowID,
"key": strings.Replace(key, "damagetype:", "damagehitvisual:", 1),
"Label": registryStringField(row, "label"),
"VisualEffectID": registryNullableField(row, "visual_effect_id"),
"RangedEffectID": registryNullableField(row, "ranged_effect_id"),
})
if _, ok := groupRowsByID[groupID]; !ok {
groupRowsByID[groupID] = map[string]any{
"id": groupID,
"key": strings.Replace(key, "damagetype:", "damagetypegroup:", 1),
"Label": registryStringField(row, "group_label"),
"FeedbackStrref": deepCopyValue(row["feedback_strref"]),
"ColorR": registryNullableField(row, "color_r"),
"ColorG": registryNullableField(row, "color_g"),
"ColorB": registryNullableField(row, "color_b"),
}
groupOrder = append(groupOrder, groupID)
continue
}
existing := groupRowsByID[groupID]
for field, candidate := range map[string]any{
"Label": registryStringField(row, "group_label"),
"FeedbackStrref": deepCopyValue(row["feedback_strref"]),
"ColorR": registryNullableField(row, "color_r"),
"ColorG": registryNullableField(row, "color_g"),
"ColorB": registryNullableField(row, "color_b"),
} {
if format2DAValue(existing[field]) != format2DAValue(candidate) {
return nil, fmt.Errorf("%s: conflicting group projection for DamageTypeGroup %d field %s", key, groupID, field)
}
}
}
slices.Sort(groupOrder)
groupRows := make([]map[string]any, 0, len(groupOrder))
for _, id := range groupOrder {
groupRows = append(groupRows, groupRowsByID[id])
}
return []nativeCollectedDataset{
newGeneratedDataset("damagetypes/registry/core", "damagetypes.2da", []string{"Label", "CharsheetStrref", "DamageTypeGroup", "DamageRangedProjectile"}, coreRows),
newGeneratedDataset("damagetypes/registry/groups", "damagetypegroups.2da", []string{"Label", "FeedbackStrref", "ColorR", "ColorG", "ColorB"}, groupRows),
newGeneratedDataset("damagetypes/registry/hitvisual", "damagehitvisual.2da", []string{"Label", "VisualEffectID", "RangedEffectID"}, hitvisualRows),
}, nil
}
func assignDamagetypeRegistryIDs(rows []map[string]any, lockData map[string]int) (map[string]int, bool, error) {
ids := map[string]int{}
used := map[int]struct{}{}
for key, id := range lockData {
if strings.HasPrefix(key, "damagetype:") {
used[id] = struct{}{}
}
}
modified := false
for _, row := range rows {
key := stringField(row, "key")
if key == "" {
return nil, false, fmt.Errorf("registry row is missing key")
}
if !strings.HasPrefix(key, "damagetype:") {
return nil, false, fmt.Errorf("registry key %q must use prefix %q", key, "damagetype:")
}
if rawID, ok := row["id"]; ok {
id, err := asInt(rawID)
if err != nil {
return nil, false, fmt.Errorf("%s: invalid id: %w", key, err)
}
if existing, ok := lockData[key]; ok && existing != id {
return nil, false, fmt.Errorf("%s: explicit id %d does not match lock id %d", key, id, existing)
}
lockData[key] = id
ids[key] = id
used[id] = struct{}{}
modified = true
continue
}
if id, ok := lockData[key]; ok {
ids[key] = id
used[id] = struct{}{}
}
}
nextID := nextAvailableID(used)
for _, row := range rows {
key := stringField(row, "key")
if _, ok := ids[key]; ok {
continue
}
lockData[key] = nextID
ids[key] = nextID
used[nextID] = struct{}{}
nextID = nextAvailableID(used)
modified = true
}
return ids, modified, nil
}
func registryStringField(row map[string]any, field string) string {
text := stringField(row, field)
if text == "" {
return nullValue
}
return text
}
func registryNullableField(row map[string]any, field string) any {
value, ok := row[field]
if !ok {
return nullValue
}
switch typed := value.(type) {
case string:
if strings.TrimSpace(typed) == "" {
return nullValue
}
}
return deepCopyValue(value)
}
func registryIntField(row map[string]any, field string) (int, error) {
value, ok := row[field]
if !ok {
return 0, fmt.Errorf("missing %s", field)
}
return asInt(value)
}
internal/topdata/doortypes_migrate.go
-99
View File
@@ -1,99 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyDoortypes(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "doortypes")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "doortypes")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
moduleNames := []string{
"add_pld01.json",
"add_sic11.json",
"add_tapr.json",
"add_tdm01.json",
"add_tdx01.json",
"add_tei01.json",
"add_tfm01.json",
}
targetModulePaths := make([]string, 0, len(moduleNames))
for _, name := range moduleNames {
targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "doortypes.2da"
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for i, path := range targetModulePaths {
writes = append(writes, struct {
path string
obj map[string]any
}{path: path, obj: moduleObjs[i]})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
internal/topdata/editorconfig_format.go
-246
View File
@@ -1,246 +0,0 @@
package topdata
import (
"bufio"
"bytes"
"fmt"
"os"
"path/filepath"
"regexp"
"strconv"
"strings"
)
type jsonFileFormatting struct {
Indent string
LineEnding string
FinalNewline bool
}
type editorConfigDocument struct {
Dir string
Root bool
Sections []editorConfigSection
}
type editorConfigSection struct {
Pattern string
Properties map[string]string
}
func defaultJSONFileFormatting() jsonFileFormatting {
return jsonFileFormatting{
Indent: " ",
LineEnding: "\n",
FinalNewline: true,
}
}
func resolveJSONFileFormatting(path string) (jsonFileFormatting, error) {
formatting := defaultJSONFileFormatting()
documents, err := editorConfigDocumentsForPath(path)
if err != nil {
return jsonFileFormatting{}, err
}
for _, document := range documents {
relative, err := filepath.Rel(document.Dir, path)
if err != nil {
return jsonFileFormatting{}, fmt.Errorf("resolve editorconfig path for %s relative to %s: %w", path, document.Dir, err)
}
relative = filepath.ToSlash(relative)
if strings.HasPrefix(relative, "../") || relative == ".." {
continue
}
for _, section := range document.Sections {
matches, err := editorConfigPatternMatches(section.Pattern, relative)
if err != nil {
return jsonFileFormatting{}, fmt.Errorf("%s/.editorconfig section [%s]: %w", document.Dir, section.Pattern, err)
}
if !matches {
continue
}
next, err := applyEditorConfigProperties(formatting, section.Properties)
if err != nil {
return jsonFileFormatting{}, fmt.Errorf("%s/.editorconfig section [%s]: %w", document.Dir, section.Pattern, err)
}
formatting = next
}
}
return formatting, nil
}
func editorConfigDocumentsForPath(path string) ([]editorConfigDocument, error) {
dir := filepath.Dir(path)
documents := []editorConfigDocument{}
for {
configPath := filepath.Join(dir, ".editorconfig")
raw, err := os.ReadFile(configPath)
if err == nil {
document, err := parseEditorConfig(configPath, raw)
if err != nil {
return nil, err
}
document.Dir = dir
documents = append(documents, document)
if document.Root {
break
}
} else if !os.IsNotExist(err) {
return nil, fmt.Errorf("read %s: %w", configPath, err)
}
parent := filepath.Dir(dir)
if parent == dir {
break
}
dir = parent
}
for left, right := 0, len(documents)-1; left < right; left, right = left+1, right-1 {
documents[left], documents[right] = documents[right], documents[left]
}
return documents, nil
}
func parseEditorConfig(path string, raw []byte) (editorConfigDocument, error) {
document := editorConfigDocument{}
scanner := bufio.NewScanner(bytes.NewReader(raw))
var current *editorConfigSection
for lineNumber := 1; scanner.Scan(); lineNumber++ {
line := strings.TrimSpace(scanner.Text())
if line == "" || strings.HasPrefix(line, "#") || strings.HasPrefix(line, ";") {
continue
}
if strings.HasPrefix(line, "[") {
end := strings.LastIndex(line, "]")
if end < 0 {
return editorConfigDocument{}, fmt.Errorf("%s:%d: malformed section header", path, lineNumber)
}
section := editorConfigSection{
Pattern: strings.TrimSpace(line[1:end]),
Properties: map[string]string{},
}
document.Sections = append(document.Sections, section)
current = &document.Sections[len(document.Sections)-1]
continue
}
index := strings.IndexAny(line, "=:")
if index < 0 {
return editorConfigDocument{}, fmt.Errorf("%s:%d: expected key/value property", path, lineNumber)
}
key := strings.ToLower(strings.TrimSpace(line[:index]))
value := strings.TrimSpace(line[index+1:])
if current == nil {
if key == "root" {
document.Root = strings.EqualFold(value, "true")
}
continue
}
current.Properties[key] = value
}
if err := scanner.Err(); err != nil {
return editorConfigDocument{}, fmt.Errorf("scan %s: %w", path, err)
}
return document, nil
}
func applyEditorConfigProperties(formatting jsonFileFormatting, properties map[string]string) (jsonFileFormatting, error) {
style := strings.ToLower(strings.TrimSpace(properties["indent_style"]))
sizeText := strings.ToLower(strings.TrimSpace(properties["indent_size"]))
if style == "tab" {
formatting.Indent = "\t"
} else if style != "" && style != "space" && style != "unset" {
return jsonFileFormatting{}, fmt.Errorf("unsupported indent_style %q", properties["indent_style"])
}
if sizeText != "" && sizeText != "unset" && sizeText != "tab" {
size, err := strconv.Atoi(sizeText)
if err != nil || size < 1 {
return jsonFileFormatting{}, fmt.Errorf("indent_size must be a positive integer, got %q", properties["indent_size"])
}
if style != "tab" {
formatting.Indent = strings.Repeat(" ", size)
}
}
switch value := strings.ToLower(strings.TrimSpace(properties["end_of_line"])); value {
case "", "unset":
case "lf":
formatting.LineEnding = "\n"
case "crlf":
formatting.LineEnding = "\r\n"
case "cr":
formatting.LineEnding = "\r"
default:
return jsonFileFormatting{}, fmt.Errorf("unsupported end_of_line %q", properties["end_of_line"])
}
switch value := strings.ToLower(strings.TrimSpace(properties["insert_final_newline"])); value {
case "", "unset":
case "true":
formatting.FinalNewline = true
case "false":
formatting.FinalNewline = false
default:
return jsonFileFormatting{}, fmt.Errorf("insert_final_newline must be true or false, got %q", properties["insert_final_newline"])
}
return formatting, nil
}
func editorConfigPatternMatches(pattern, relativePath string) (bool, error) {
pattern = filepath.ToSlash(strings.TrimSpace(pattern))
pattern = strings.TrimPrefix(pattern, "/")
if pattern == "" {
return false, nil
}
if !strings.Contains(pattern, "/") {
relativePath = pathBase(relativePath)
}
expression, err := editorConfigGlobRegexp(pattern)
if err != nil {
return false, err
}
return regexp.MatchString(expression, relativePath)
}
func editorConfigGlobRegexp(pattern string) (string, error) {
var builder strings.Builder
builder.WriteByte('^')
for index := 0; index < len(pattern); {
char := pattern[index]
switch char {
case '*':
if index+1 < len(pattern) && pattern[index+1] == '*' {
if index+2 < len(pattern) && pattern[index+2] == '/' {
builder.WriteString("(?:.*/)?")
index += 3
} else {
builder.WriteString(".*")
index += 2
}
continue
}
builder.WriteString("[^/]*")
case '?':
builder.WriteString("[^/]")
case '.', '+', '(', ')', '|', '^', '$', '{', '}', '[', ']', '\\':
builder.WriteByte('\\')
builder.WriteByte(char)
default:
builder.WriteByte(char)
}
index++
}
builder.WriteByte('$')
expression := builder.String()
if _, err := regexp.Compile(expression); err != nil {
return "", err
}
return expression, nil
}
func pathBase(path string) string {
if index := strings.LastIndex(path, "/"); index >= 0 {
return path[index+1:]
}
return path
}
internal/topdata/expansion_native.go
-284
View File
@@ -1,284 +0,0 @@
package topdata
import (
"fmt"
"slices"
"strings"
)
func normalizeMetadataKey(key string) string {
replacer := strings.NewReplacer("-", "_", " ", "_")
return strings.ToLower(replacer.Replace(strings.TrimSpace(key)))
}
func parseRowMetadata(raw any) (map[string]any, error) {
if raw == nil {
return nil, nil
}
obj, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("meta must be an object")
}
if len(obj) == 0 {
return map[string]any{}, nil
}
meta := make(map[string]any, len(obj))
for key, value := range obj {
switch normalizeMetadataKey(key) {
case "family":
family, err := parseFamilyMetadata(value)
if err != nil {
return nil, fmt.Errorf("meta.family: %w", err)
}
meta["family"] = family
case "wiki":
wiki, err := parseWikiMetadata(value)
if err != nil {
return nil, fmt.Errorf("meta.wiki: %w", err)
}
if len(wiki) > 0 {
meta["wiki"] = wiki
}
default:
return nil, fmt.Errorf("unknown metadata key %q", key)
}
}
return meta, nil
}
func mergeExpansionData(collected []nativeCollectedDataset) ([]nativeCollectedDataset, error) {
datasetIndexMap := make(map[string]int)
for i, ds := range collected {
datasetIndexMap[ds.Dataset.Name] = i
datasetIndexMap[ds.Dataset.OutputName] = i
}
rowsWithExpansion := []struct {
datasetName string
row map[string]any
}{}
for _, ds := range collected {
for _, row := range ds.Rows {
if hasExpansionData(row) {
rowsWithExpansion = append(rowsWithExpansion, struct {
datasetName string
row map[string]any
}{ds.Dataset.Name, row})
}
}
}
if len(rowsWithExpansion) == 0 {
return collected, nil
}
modifiedTargets := map[int]struct{}{}
for _, src := range rowsWithExpansion {
expansion, _ := extractExpansionData(src.row)
for targetDatasetName, targetRows := range expansion.Data {
targetIndex, ok := datasetIndexMap[targetDatasetName]
if !ok {
outputName := targetDatasetName
if !strings.HasSuffix(outputName, ".2da") {
outputName = outputName + ".2da"
}
targetIndex, ok = datasetIndexMap[outputName]
}
if !ok {
return nil, fmt.Errorf("expansion targets unknown dataset %q", targetDatasetName)
}
targetDS := collected[targetIndex]
originalLockData, _ := loadLockfile(targetDS.Dataset.LockPath)
if originalLockData == nil {
originalLockData = map[string]int{}
}
lockModified := false
usedIDs := map[int]struct{}{}
usedKeys := map[string]struct{}{}
for _, rowID := range targetDS.LockData {
usedIDs[rowID] = struct{}{}
}
for _, row := range targetDS.Rows {
if id, ok := row["id"].(int); ok {
usedIDs[id] = struct{}{}
}
if key, ok := row["key"].(string); ok {
usedKeys[key] = struct{}{}
}
}
nextID := nextAvailableIDAtLeast(usedIDs, targetDS.Dataset.RowGenerationMinRow)
for _, targetRow := range targetRows {
var rowID int
var hasID bool
if rawID, ok := targetRow["id"]; ok {
switch typed := rawID.(type) {
case int:
rowID = typed
hasID = true
case float64:
rowID = int(typed)
hasID = true
case string:
parsed, err := asInt(typed)
if err == nil {
rowID = parsed
hasID = true
}
}
}
key, _ := targetRow["key"].(string)
if strings.TrimSpace(key) == "" {
return nil, fmt.Errorf("expansion into %s: injected rows must specify key", targetDatasetName)
}
if key != "" {
if existingID, exists := targetDS.LockData[key]; exists {
rowID = existingID
hasID = true
} else if existingID, exists := originalLockData[key]; exists && targetDS.Dataset.RowGeneration != "first_null_row" {
rowID = existingID
hasID = true
targetDS.LockData[key] = existingID
lockModified = true
targetDS.LockModified = true
}
if _, seen := usedKeys[key]; seen {
continue
}
usedKeys[key] = struct{}{}
}
if !hasID {
rowID = nextID
usedIDs[rowID] = struct{}{}
nextID = nextAvailableIDAtLeast(usedIDs, targetDS.Dataset.RowGenerationMinRow)
} else {
if _, exists := usedIDs[rowID]; exists && key == "" {
return nil, fmt.Errorf("expansion into %s: row id %d already exists", targetDatasetName, rowID)
}
usedIDs[rowID] = struct{}{}
}
if key != "" {
if _, exists := targetDS.LockData[key]; !exists {
targetDS.LockData[key] = rowID
lockModified = true
targetDS.LockAdded++
targetDS.LockModified = true
}
}
newRow := map[string]any{
"id": rowID,
}
for k, v := range targetRow {
if k != "id" {
newRow[k] = v
}
}
if key != "" {
newRow["key"] = key
}
targetDS.Rows = append(targetDS.Rows, newRow)
}
if lockModified {
modifiedTargets[targetIndex] = struct{}{}
}
slices.SortFunc(targetDS.Rows, func(a, b map[string]any) int {
return a["id"].(int) - b["id"].(int)
})
collected[targetIndex] = targetDS
}
if err := applyExpansionValueToRow(src.row, expansion.Value); err != nil {
return nil, err
}
}
for targetIndex := range modifiedTargets {
targetDS := collected[targetIndex]
referencedKeys, err := collectReferencedLockKeys(nativeDatasetSourceDir(targetDS.Dataset))
if err != nil {
return nil, fmt.Errorf("dataset %s: collect referenced keys: %w", targetDS.Dataset.Name, err)
}
if pruned, updated := pruneLockDataToActiveRows(targetDS.LockData, targetDS.Rows, referencedKeys, targetDS.RetiredKeys); pruned > 0 || updated > 0 {
targetDS.LockPruned += pruned
targetDS.LockModified = true
collected[targetIndex] = targetDS
}
}
return collected, nil
}
type expansionSpec struct {
Value string
Data map[string][]map[string]any
}
func hasExpansionData(row map[string]any) bool {
for _, value := range row {
if isExpansionValue(value) {
return true
}
}
return false
}
func isExpansionValue(value any) bool {
obj, ok := value.(map[string]any)
if !ok {
return false
}
_, hasValue := obj["value"]
_, hasData := obj["data"]
return hasValue && hasData
}
func extractExpansionData(row map[string]any) (expansionSpec, bool) {
for _, value := range row {
if isExpansionValue(value) {
obj := value.(map[string]any)
valueStr, _ := obj["value"].(string)
dataMap, _ := obj["data"].(map[string]any)
result := expansionSpec{
Value: valueStr,
Data: make(map[string][]map[string]any),
}
for datasetName, rawRows := range dataMap {
switch typed := rawRows.(type) {
case []any:
rows := make([]map[string]any, 0, len(typed))
for _, r := range typed {
if rowObj, ok := r.(map[string]any); ok {
rows = append(rows, rowObj)
}
}
result.Data[datasetName] = rows
case map[string]any:
result.Data[datasetName] = []map[string]any{typed}
}
}
return result, true
}
}
return expansionSpec{}, false
}
func applyExpansionValueToRow(row map[string]any, value string) error {
for field, oldValue := range row {
if isExpansionValue(oldValue) {
row[field] = value
}
}
return nil
}
internal/topdata/family_expansion.go
-345
View File
@@ -1,345 +0,0 @@
package topdata
import (
"fmt"
"strings"
)
type familyIdentity struct {
Parent string
Child string
}
type familyExpansionSource struct {
Dataset string
Column string
Predicate string
}
type familyExpansionTitleStyle struct {
ChildCase string
ChildParenthetical string
}
type familyExpansionSpec struct {
Family string
FamilyKey string
Template string
ChildSource familyExpansionSource
NamePrefix string
LabelPrefix string
ConstantPrefix string
TemplateFields []string
DefaultFields map[string]any
ApplyAfterModules bool
ChildRefField string
IdentitySource string
AllowExistingOnly bool
AutoPrereqFields map[string]string
LegacyFamilyKeys []string
TitleStyle familyExpansionTitleStyle
}
func splitFamilyExpansionIdentity(text string) familyIdentity {
text = strings.TrimSpace(text)
if text == "" {
return familyIdentity{}
}
if idx := strings.Index(text, "_"); idx > 0 && idx < len(text)-1 {
return familyIdentity{
Parent: text[:idx],
Child: text[idx+1:],
}
}
return familyIdentity{Parent: text}
}
func parseFamilyExpansionSource(raw any) (familyExpansionSource, error) {
if raw == nil {
return familyExpansionSource{}, fmt.Errorf("child_source is required")
}
obj, ok := raw.(map[string]any)
if !ok {
return familyExpansionSource{}, fmt.Errorf("child_source must be an object")
}
dataset, ok := obj["dataset"].(string)
if !ok || strings.TrimSpace(dataset) == "" {
return familyExpansionSource{}, fmt.Errorf("child_source.dataset must be a non-empty string")
}
source := familyExpansionSource{
Dataset: strings.TrimSpace(dataset),
}
if column, ok := obj["column"].(string); ok && strings.TrimSpace(column) != "" {
source.Column = strings.TrimSpace(column)
}
if predicate, ok := obj["predicate"].(string); ok && strings.TrimSpace(predicate) != "" {
source.Predicate = strings.TrimSpace(predicate)
}
return source, nil
}
func parseFamilyExpansionSpec(path string, obj map[string]any) (familyExpansionSpec, error) {
family, ok := obj["family"].(string)
if !ok || strings.TrimSpace(family) == "" {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: family must be a non-empty string", path)
}
familyKey, ok := obj["family_key"].(string)
if !ok || strings.TrimSpace(familyKey) == "" {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: family_key must be a non-empty string", path)
}
template, ok := obj["template"].(string)
if !ok || strings.TrimSpace(template) == "" {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: template must be a non-empty string", path)
}
source, err := parseFamilyExpansionSource(obj["child_source"])
if err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
namePrefix, _ := optionalTrimmedString(obj, "name_prefix")
labelPrefix, _ := optionalTrimmedString(obj, "label_prefix")
constantPrefix, _ := optionalTrimmedString(obj, "constant_prefix")
templateFields, err := parseOptionalStringArray(obj["template_fields"], "template_fields")
if err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
defaultFields, err := parseOptionalObject(obj["default_fields"], "default_fields")
if err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
applyAfterModules, err := parseOptionalBoolField(obj["apply_after_modules"], "apply_after_modules")
if err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
childRefField, _ := optionalTrimmedString(obj, "child_ref_field")
identitySource, _ := optionalTrimmedString(obj, "identity_source")
switch identitySource {
case "", "child_source_value":
default:
return familyExpansionSpec{}, fmt.Errorf("generated file %s: identity_source must be empty or child_source_value", path)
}
autoPrereqFields, err := parseOptionalStringMap(obj["auto_prereq_fields"], "auto_prereq_fields")
if err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
legacyFamilyKeys, err := parseOptionalStringArray(obj["legacy_family_keys"], "legacy_family_keys")
if err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
if err := validateLegacyFamilyKeys(familyKey, legacyFamilyKeys); err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
titleStyle, err := parseFamilyExpansionTitleStyle(obj["title_style"])
if err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
allowExistingOnly, err := parseOptionalBoolField(obj["allow_existing_only"], "allow_existing_only")
if err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
return familyExpansionSpec{
Family: strings.TrimSpace(family),
FamilyKey: strings.TrimSpace(familyKey),
Template: strings.TrimSpace(template),
ChildSource: source,
NamePrefix: namePrefix,
LabelPrefix: labelPrefix,
ConstantPrefix: constantPrefix,
TemplateFields: templateFields,
DefaultFields: defaultFields,
ApplyAfterModules: applyAfterModules,
ChildRefField: childRefField,
IdentitySource: identitySource,
AllowExistingOnly: allowExistingOnly,
AutoPrereqFields: autoPrereqFields,
LegacyFamilyKeys: legacyFamilyKeys,
TitleStyle: titleStyle,
}, nil
}
func parseFamilyExpansionTitleStyle(raw any) (familyExpansionTitleStyle, error) {
style := familyExpansionTitleStyle{
ChildCase: "preserve",
ChildParenthetical: "preserve",
}
if raw == nil {
return style, nil
}
obj, ok := raw.(map[string]any)
if !ok {
return familyExpansionTitleStyle{}, fmt.Errorf("title_style must be an object")
}
if childCase, ok := obj["child_case"].(string); ok && strings.TrimSpace(childCase) != "" {
style.ChildCase = strings.TrimSpace(childCase)
}
switch style.ChildCase {
case "preserve", "lower":
default:
return familyExpansionTitleStyle{}, fmt.Errorf("title_style.child_case must be preserve or lower")
}
if childParenthetical, ok := obj["child_parenthetical"].(string); ok && strings.TrimSpace(childParenthetical) != "" {
style.ChildParenthetical = strings.TrimSpace(childParenthetical)
}
switch style.ChildParenthetical {
case "preserve", "comma":
default:
return familyExpansionTitleStyle{}, fmt.Errorf("title_style.child_parenthetical must be preserve or comma")
}
return style, nil
}
func validateLegacyFamilyKeys(familyKey string, legacyFamilyKeys []string) error {
seen := map[string]string{}
normalizedFamilyKey := normalizeKeyIdentity(familyKey)
for _, legacyKey := range legacyFamilyKeys {
normalizedLegacyKey := normalizeKeyIdentity(legacyKey)
if normalizedLegacyKey == normalizedFamilyKey {
return fmt.Errorf("legacy_family_keys must not include family_key %q", familyKey)
}
if previous, ok := seen[normalizedLegacyKey]; ok {
return fmt.Errorf("legacy_family_keys contains duplicate-equivalent keys %q and %q", previous, legacyKey)
}
seen[normalizedLegacyKey] = legacyKey
}
return nil
}
func isFamilyExpansionObject(obj map[string]any) bool {
_, hasFamilyKey := obj["family_key"]
_, hasTemplate := obj["template"]
_, hasChildSource := obj["child_source"]
_, hasNamePrefix := obj["name_prefix"]
_, hasLabelPrefix := obj["label_prefix"]
_, hasConstantPrefix := obj["constant_prefix"]
_, hasTemplateFields := obj["template_fields"]
_, hasDefaultFields := obj["default_fields"]
_, hasApplyAfterModules := obj["apply_after_modules"]
_, hasChildRefField := obj["child_ref_field"]
_, hasIdentitySource := obj["identity_source"]
_, hasAllowExistingOnly := obj["allow_existing_only"]
_, hasAutoPrereqFields := obj["auto_prereq_fields"]
_, hasLegacyFamilyKeys := obj["legacy_family_keys"]
_, hasTitleStyle := obj["title_style"]
return hasFamilyKey || hasTemplate || hasChildSource || hasNamePrefix || hasLabelPrefix ||
hasConstantPrefix || hasTemplateFields || hasDefaultFields || hasApplyAfterModules || hasChildRefField ||
hasIdentitySource || hasAllowExistingOnly || hasAutoPrereqFields || hasLegacyFamilyKeys || hasTitleStyle
}
func optionalTrimmedString(obj map[string]any, field string) (string, bool) {
text, ok := obj[field].(string)
if !ok || strings.TrimSpace(text) == "" {
return "", false
}
return strings.TrimSpace(text), true
}
func parseOptionalStringArray(raw any, field string) ([]string, error) {
if raw == nil {
return nil, nil
}
items, ok := raw.([]any)
if !ok {
return nil, fmt.Errorf("%s must be an array of strings", field)
}
out := make([]string, 0, len(items))
for _, item := range items {
text, ok := item.(string)
if !ok || strings.TrimSpace(text) == "" {
return nil, fmt.Errorf("%s must contain only non-empty strings", field)
}
out = append(out, strings.TrimSpace(text))
}
return out, nil
}
func parseOptionalObject(raw any, field string) (map[string]any, error) {
if raw == nil {
return nil, nil
}
obj, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s must be an object", field)
}
return obj, nil
}
func parseOptionalStringMap(raw any, field string) (map[string]string, error) {
if raw == nil {
return nil, nil
}
obj, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s must be an object of strings", field)
}
out := make(map[string]string, len(obj))
for key, value := range obj {
text, ok := value.(string)
if !ok || strings.TrimSpace(text) == "" {
return nil, fmt.Errorf("%s must contain only non-empty string values", field)
}
out[key] = strings.TrimSpace(text)
}
return out, nil
}
func parseOptionalBoolField(raw any, field string) (bool, error) {
if raw == nil {
return false, nil
}
value, ok := raw.(bool)
if !ok {
return false, fmt.Errorf("%s must be a boolean", field)
}
return value, nil
}
func familyMetadata(parent, child, source, template string) map[string]any {
meta := map[string]any{
"parent": strings.TrimSpace(parent),
}
if strings.TrimSpace(child) != "" {
meta["child"] = strings.TrimSpace(child)
}
if strings.TrimSpace(source) != "" {
meta["source"] = strings.TrimSpace(source)
}
if strings.TrimSpace(template) != "" {
meta["template"] = strings.TrimSpace(template)
}
return map[string]any{"family": meta}
}
func parseFamilyMetadata(raw any) (map[string]any, error) {
obj, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("must be an object")
}
parent, ok := obj["parent"].(string)
if !ok || strings.TrimSpace(parent) == "" {
return nil, fmt.Errorf("parent must be a non-empty string")
}
meta := map[string]any{
"parent": strings.TrimSpace(parent),
}
if child, ok := obj["child"].(string); ok && strings.TrimSpace(child) != "" {
meta["child"] = strings.TrimSpace(child)
}
if source, ok := obj["source"].(string); ok && strings.TrimSpace(source) != "" {
meta["source"] = strings.TrimSpace(source)
}
if template, ok := obj["template"].(string); ok && strings.TrimSpace(template) != "" {
meta["template"] = strings.TrimSpace(template)
}
return meta, nil
}
func childTokenFromExpandedKey(key, parent string) string {
key = strings.TrimSpace(key)
key = strings.TrimPrefix(key, "feat:")
if strings.HasPrefix(key, parent+"_") {
return strings.TrimPrefix(key, parent+"_")
}
if strings.HasPrefix(key, parent) {
return strings.TrimPrefix(key, parent)
}
return ""
}
internal/topdata/feat_migrate.go
-64
View File
@@ -1,64 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyFeat(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "feat")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
legacyBasePath := filepath.Join(legacyDir, "base.json")
if _, err := os.Stat(legacyBasePath); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "feat")
targetBasePath := filepath.Join(targetDir, "base.json")
var targetObj map[string]any
if _, err := os.Stat(targetBasePath); err == nil {
targetObj, err = loadJSONObject(targetBasePath)
if err != nil {
return 0, err
}
} else if !os.IsNotExist(err) {
return 0, err
}
if targetObj != nil {
if rows, ok := targetObj["rows"].([]any); ok && len(rows) > 0 {
return 0, nil
}
}
baseObj, err := loadJSONObject(legacyBasePath)
if err != nil {
return 0, err
}
baseObj["output"] = "feat.2da"
baseObj["compare_reference"] = false
canonicalizeWikiMetadataDocument(baseObj)
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
raw, err := json.MarshalIndent(baseObj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetBasePath, raw, 0o644); err != nil {
return 0, err
}
return 1, nil
}
internal/topdata/generated_assets.go
-205
View File
@@ -1,205 +0,0 @@
package topdata
import (
"fmt"
"os"
"path/filepath"
"strings"
"gitea.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
type Generated2DAAsset struct {
Rel string
SourcePath string
}
func BuildGenerated2DAAssets(p *project.Project, progress func(string)) ([]Generated2DAAsset, error) {
if progress == nil {
progress = func(string) {}
}
configs := p.EffectiveConfig().Generated.TopData2DA
if len(configs) == 0 {
return nil, nil
}
results := make([]Generated2DAAsset, 0)
for _, cfg := range configs {
progress(fmt.Sprintf("Building generated 2DA assets for %s...", cfg.ID))
generated, err := buildGenerated2DAAssetGroup(p, cfg, progress)
if err != nil {
return nil, err
}
results = append(results, generated...)
}
return results, nil
}
func buildGenerated2DAAssetGroup(p *project.Project, cfg project.GeneratedTopData2DAConfig, progress func(string)) ([]Generated2DAAsset, error) {
sourceDir := filepath.Join(p.Root, filepath.FromSlash(strings.TrimSpace(cfg.Source)))
dataDir := filepath.Join(sourceDir, "data")
outputDir := filepath.Join(p.Root, filepath.FromSlash(strings.TrimSpace(cfg.Output)))
datasets, err := discoverNativeDatasets(dataDir)
if err != nil {
return nil, fmt.Errorf("discover generated topdata 2DA datasets %s: %w", cfg.ID, err)
}
selected := make([]nativeDataset, 0, len(datasets))
for _, dataset := range datasets {
if generatedDatasetIncluded(dataset.Name, cfg.IncludeDatasets) {
selected = append(selected, dataset)
}
}
if len(selected) == 0 {
return nil, fmt.Errorf("generated topdata 2DA config %s matched no datasets under %s", cfg.ID, dataDir)
}
collected := make([]nativeCollectedDataset, 0, len(selected))
for _, dataset := range selected {
collectedDataset, err := collectNativeDataset(dataset)
if err != nil {
return nil, err
}
collected = append(collected, collectedDataset)
}
consumer := cfg.Autogen
consumer.LocalOverrideRoot = p.AssetsDir()
collected, err = applyAutogenConsumersForGeneratedAssets(collected, consumer, progress)
if err != nil {
return nil, err
}
collected, err = normalizePartsRowsACBonus(collected, consumer.PartsRows)
if err != nil {
return nil, err
}
collected, err = applyPartOverridesWithConfig(sourceDir, collected, consumer.PartsRows)
if err != nil {
return nil, err
}
if err := rejectGenerated2DATLKValues(cfg.ID, collected); err != nil {
return nil, err
}
if _, _, err := saveNativeLockfiles(collected); err != nil {
return nil, fmt.Errorf("save generated topdata 2DA lockfiles %s: %w", cfg.ID, err)
}
tableRegistry, err := newResolvedTableRegistry(collected)
if err != nil {
return nil, err
}
keyToID, rowByKey := generated2DAGlobalRows(collected)
if err := os.RemoveAll(outputDir); err != nil {
return nil, fmt.Errorf("clean generated topdata 2DA output %s: %w", outputDir, err)
}
if err := os.MkdirAll(outputDir, 0o755); err != nil {
return nil, fmt.Errorf("create generated topdata 2DA output %s: %w", outputDir, err)
}
results := make([]Generated2DAAsset, 0, len(collected))
for _, dataset := range collected {
compiled, err := resolveNativeDataset(dataset, keyToID, rowByKey, tableRegistry, nil, project.TopDataClassFeatInjectionConfig{}, nil)
if err != nil {
return nil, err
}
outputPath := filepath.Join(outputDir, dataset.Dataset.OutputName)
denseRows := dataset.Dataset.Kind == nativeDatasetBase ||
(consumer.Mode == "parts_rows" && isPartsDataset(dataset.Dataset.Name))
if err := write2DA(compiled, outputPath, denseRows, -1); err != nil {
return nil, err
}
results = append(results, Generated2DAAsset{
Rel: filepath.ToSlash(filepath.Join(cfg.PackageRoot, dataset.Dataset.OutputName)),
SourcePath: outputPath,
})
}
return results, nil
}
func applyAutogenConsumersForGeneratedAssets(collected []nativeCollectedDataset, consumer project.AutogenConsumerConfig, progress func(string)) ([]nativeCollectedDataset, error) {
if strings.TrimSpace(consumer.Mode) == "" {
return collected, nil
}
if progress == nil {
progress = func(string) {}
}
if !autogenConsumerTargetsCollectedDataset(collected, consumer) {
return collected, nil
}
if progress != nil {
progress(fmt.Sprintf("Scanning local generated 2DA autogen input for %s from %s...", consumer.ID, consumer.LocalOverrideRoot))
}
entries, err := scanLocalAutogenEntries(consumer.LocalOverrideRoot, consumer)
if err != nil {
return nil, err
}
switch consumer.Mode {
case "parts_rows":
return augmentWithAutogeneratedPartsWithConfig(collected, autogenPartsInventory(entries), consumer.PartsRows), nil
case "cachedmodels_rows":
return augmentWithAutogeneratedCachedModels(collected, entries)
default:
return nil, fmt.Errorf("unsupported generated topdata 2DA autogen mode %q", consumer.Mode)
}
}
func generatedDatasetIncluded(name string, patterns []string) bool {
name = filepath.ToSlash(strings.TrimSpace(name))
for _, pattern := range patterns {
pattern = filepath.ToSlash(strings.TrimSpace(pattern))
if pattern == "" {
continue
}
if pattern == name {
return true
}
if strings.HasSuffix(pattern, "/**") {
prefix := strings.TrimSuffix(pattern, "/**")
if name == prefix || strings.HasPrefix(name, prefix+"/") {
return true
}
}
}
return false
}
func generated2DAGlobalRows(collected []nativeCollectedDataset) (map[string]int, map[string]map[string]any) {
keyToID := map[string]int{}
rowByKey := map[string]map[string]any{}
for _, dataset := range collected {
for key, rowID := range dataset.LockData {
keyToID[key] = rowID
}
for _, row := range dataset.Rows {
key, _ := row["key"].(string)
if key == "" {
continue
}
if rowID, ok := row["id"].(int); ok {
keyToID[key] = rowID
rowByKey[key] = row
}
}
}
return keyToID, rowByKey
}
func rejectGenerated2DATLKValues(configID string, collected []nativeCollectedDataset) error {
for _, dataset := range collected {
for _, row := range dataset.Rows {
for field, value := range row {
if field == "id" || field == "key" {
continue
}
allowBare := columnMatchesSpec(dataset.Dataset.Spec, field)
if _, ok, err := parseTLKPayload(value, allowBare); err != nil {
return fmt.Errorf("generated topdata 2DA config %s dataset %s field %s has invalid TLK payload: %w", configID, dataset.Dataset.Name, field, err)
} else if ok {
return fmt.Errorf("generated topdata 2DA config %s dataset %s field %s uses TLK-backed text; 2DA-only generated asset builds do not generate TLK output", configID, dataset.Dataset.Name, field)
}
}
}
}
return nil
}
internal/topdata/genericdoors_migrate.go
-6
View File
@@ -1,6 +0,0 @@
package topdata
func importLegacyGenericdoors(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "genericdoors", "genericdoors.2da", nil)
}
internal/topdata/itemprops_registry.go
-1214
View File
File diff suppressed because it is too large Load Diff
internal/topdata/legacy_dataset_migrate.go
-176
View File
@@ -1,176 +0,0 @@
package topdata
import (
"encoding/json"
"io/fs"
"os"
"path/filepath"
"sort"
"strings"
)
type legacyDatasetTransform func(relativePath string, obj map[string]any)
func importLegacyDatasetMirror(referenceBuilderDir, dataDir, datasetName, outputName string, transform legacyDatasetTransform) (int, error) {
legacyDir := filepath.Join(referenceBuilderDir, "data", datasetName)
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, datasetName)
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
if outputName != "" {
baseObj["output"] = outputName
}
if transform != nil {
transform("base.json", baseObj)
}
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
if transform != nil {
transform("lock.json", lockObj)
}
moduleRelPaths, err := collectJSONRelativePaths(filepath.Join(legacyDir, "modules"))
if err != nil {
return 0, err
}
moduleObjs := make(map[string]map[string]any, len(moduleRelPaths))
for _, relPath := range moduleRelPaths {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", relPath))
if err != nil {
return 0, err
}
normalizedRel := filepath.ToSlash(relPath)
if transform != nil {
transform(filepath.ToSlash(filepath.Join("modules", normalizedRel)), obj)
}
moduleObjs[normalizedRel] = obj
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
updated := 0
for _, write := range []struct {
path string
obj map[string]any
}{
{path: filepath.Join(targetDir, "base.json"), obj: baseObj},
{path: filepath.Join(targetDir, "lock.json"), obj: lockObj},
} {
changed, err := writeJSONObjectIfChanged(write.path, write.obj)
if err != nil {
return 0, err
}
if changed {
updated++
}
}
targetModulesDir := filepath.Join(targetDir, "modules")
if len(moduleObjs) > 0 {
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
}
for relPath, obj := range moduleObjs {
targetPath := filepath.Join(targetModulesDir, filepath.FromSlash(relPath))
if err := os.MkdirAll(filepath.Dir(targetPath), 0o755); err != nil {
return 0, err
}
changed, err := writeJSONObjectIfChanged(targetPath, obj)
if err != nil {
return 0, err
}
if changed {
updated++
}
}
staleModules, err := collectJSONRelativePaths(targetModulesDir)
if err != nil {
return 0, err
}
for _, relPath := range staleModules {
normalizedRel := filepath.ToSlash(relPath)
if _, ok := moduleObjs[normalizedRel]; ok {
continue
}
if err := os.Remove(filepath.Join(targetModulesDir, filepath.FromSlash(normalizedRel))); err != nil && !os.IsNotExist(err) {
return 0, err
}
updated++
}
return updated, nil
}
func collectJSONRelativePaths(root string) ([]string, error) {
if _, err := os.Stat(root); err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
var relPaths []string
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
return nil
}
if !strings.EqualFold(filepath.Ext(d.Name()), ".json") {
return nil
}
relPath, err := filepath.Rel(root, path)
if err != nil {
return err
}
relPaths = append(relPaths, filepath.ToSlash(relPath))
return nil
})
if err != nil {
return nil, err
}
sort.Strings(relPaths)
return relPaths, nil
}
func writeJSONObjectIfChanged(path string, obj map[string]any) (bool, error) {
raw, err := json.MarshalIndent(obj, "", " ")
if err != nil {
return false, err
}
raw = append(raw, '\n')
current, err := os.ReadFile(path)
if err == nil && string(current) == string(raw) {
return false, nil
}
if err != nil && !os.IsNotExist(err) {
return false, err
}
if err := os.WriteFile(path, raw, 0o644); err != nil {
return false, err
}
return true, nil
}
func fileExists(path string) bool {
_, err := os.Stat(path)
return err == nil
}
internal/topdata/loadscreens_migrate.go
-6
View File
@@ -1,6 +0,0 @@
package topdata
func importLegacyLoadscreens(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "loadscreens", "loadscreens.2da", nil)
}
internal/topdata/masterfeats_migrate.go
-237
View File
@@ -1,237 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
"strconv"
)
func importLegacyMasterfeats(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
legacyDir := filepath.Join(referenceBuilderDir, "data", "feat", "masterfeats")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "masterfeats")
targetPath := filepath.Join(targetDir, "base.json")
legacyPlainPath := filepath.Join(targetDir, "masterfeats.json")
if _, err := os.Stat(targetPath); err == nil {
obj, err := loadJSONObject(targetPath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
collected, err := importLegacyMasterfeatsDataset(legacyDir)
if err != nil {
return 0, err
}
referenceIDs, err := collectReferenceLockIDs(filepath.Join(referenceBuilderDir, "data"))
if err != nil {
return 0, err
}
rows := make([]any, 0, len(collected.Rows))
for _, row := range collected.Rows {
copyRow, ok := deepCopyValue(row).(map[string]any)
if !ok {
continue
}
if legacyTLK != nil {
if _, _, err := inlineLegacyTLKValue(copyRow, legacyTLK); err != nil {
return 0, err
}
}
copyRow = rewriteImportedIDRefs(copyRow, referenceIDs)
rows = append(rows, copyRow)
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{
path: targetPath,
obj: map[string]any{
"output": collected.Dataset.OutputName,
"columns": stringSliceToAny(collected.Columns),
"rows": rows,
},
},
{
path: filepath.Join(targetDir, "lock.json"),
obj: anyMapInt(collected.LockData),
},
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
updatedFiles := len(writes)
if err := os.Remove(legacyPlainPath); err == nil {
updatedFiles++
} else if err != nil && !os.IsNotExist(err) {
return 0, err
}
return updatedFiles, nil
}
func importLegacyMasterfeatsDataset(legacyDir string) (nativeCollectedDataset, error) {
dataset := nativeDataset{
Name: "masterfeats",
BasePath: filepath.Join(legacyDir, "masterfeats.json"),
LockPath: filepath.Join(legacyDir, "lock.json"),
Spec: specForDataset("masterfeats"),
}
tableData, err := loadJSONObject(dataset.BasePath)
if err != nil {
return nativeCollectedDataset{}, err
}
columns, err := parseColumns(tableData, dataset.Name)
if err != nil {
return nativeCollectedDataset{}, err
}
rawRows, ok := tableData["rows"].([]any)
if !ok {
return nativeCollectedDataset{}, nil
}
lockData, err := loadLockfile(dataset.LockPath)
if err != nil {
return nativeCollectedDataset{}, err
}
usedIDs := map[int]struct{}{}
for _, rowID := range lockData {
usedIDs[rowID] = struct{}{}
}
rows := make([]map[string]any, 0, len(rawRows))
explicitID := make([]bool, 0, len(rawRows))
for index, raw := range rawRows {
rowObj, ok := raw.(map[string]any)
if !ok {
continue
}
rowID := index
if value, ok := rowObj["id"]; ok {
parsed, err := asInt(value)
if err != nil {
return nativeCollectedDataset{}, err
}
rowID = parsed
}
row := map[string]any{"id": rowID}
for _, column := range columns {
row[column] = nullValue
}
for key, value := range rowObj {
switch key {
case "id":
case "key":
if keyText, ok := value.(string); ok && keyText != "" {
row["key"] = keyText
}
default:
columnName, ok := canonicalColumn(columns, key)
if !ok {
continue
}
row[columnName] = value
}
}
rows = append(rows, row)
_, hasExplicitID := rowObj["id"]
explicitID = append(explicitID, hasExplicitID)
if hasExplicitID {
usedIDs[rowID] = struct{}{}
}
}
nextID := nextAvailableID(usedIDs)
lockModified := false
for index, row := range rows {
key, hasKey := row["key"].(string)
if hasKey && key != "" {
if lockedID, ok := lockData[key]; ok {
row["id"] = lockedID
continue
}
if explicitID[index] {
lockData[key] = row["id"].(int)
} else {
row["id"] = nextID
lockData[key] = nextID
usedIDs[nextID] = struct{}{}
nextID = nextAvailableID(usedIDs)
}
lockModified = true
continue
}
if !explicitID[index] {
row["id"] = index
}
}
if lockModified {
_ = lockModified
}
return nativeCollectedDataset{
Dataset: nativeDataset{
Name: dataset.Name,
OutputName: "masterfeats.2da",
Columns: columns,
Spec: dataset.Spec,
},
Columns: columns,
Rows: rows,
LockData: lockData,
}, nil
}
func rewriteImportedIDRefs(value any, ids map[string]int) map[string]any {
row, ok := rewriteImportedIDRefsValue(value, ids).(map[string]any)
if !ok {
return map[string]any{}
}
return row
}
func rewriteImportedIDRefsValue(value any, ids map[string]int) any {
switch typed := value.(type) {
case map[string]any:
if len(typed) == 1 {
if rawID, ok := typed["id"]; ok {
if key, ok := rawID.(string); ok {
if resolved, ok := ids[key]; ok {
return strconv.Itoa(resolved)
}
}
}
}
out := make(map[string]any, len(typed))
for key, child := range typed {
out[key] = rewriteImportedIDRefsValue(child, ids)
}
return out
case []any:
out := make([]any, 0, len(typed))
for _, child := range typed {
out = append(out, rewriteImportedIDRefsValue(child, ids))
}
return out
default:
return value
}
}
internal/topdata/metadata_wiki.go
-190
View File
@@ -1,190 +0,0 @@
package topdata
import (
"fmt"
"strings"
)
func isNullLike(value any) bool {
if value == nil {
return true
}
text, ok := value.(string)
if !ok {
return false
}
return strings.TrimSpace(text) == nullValue
}
func normalizeWikiMetadataKey(key string) string {
switch normalizeMetadataKey(key) {
case "reqfeats", "req_feats":
return "reqfeats"
case "generate", "wikigenerate":
return "generate"
case "canonical", "wikicanonical":
return "canonical"
case "status", "wikistatus":
return "status"
case "progressionnotes", "progression_notes", "classprogressionnotes", "class_progression_notes":
return "progression_notes"
default:
return ""
}
}
func legacyWikiMetadataField(key string) string {
switch normalizeMetadataKey(key) {
case "wikireqfeats":
return "reqfeats"
case "wikigenerate":
return "generate"
case "wikicanonical":
return "canonical"
case "wikistatus":
return "status"
default:
return ""
}
}
func parseWikiMetadata(raw any) (map[string]any, error) {
obj, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("must be an object")
}
wiki := map[string]any{}
for key, value := range obj {
normalized := normalizeWikiMetadataKey(key)
if normalized == "" {
return nil, fmt.Errorf("unknown field %q", key)
}
if isNullLike(value) {
continue
}
wiki[normalized] = cloneAuthoringValue(value)
}
return wiki, nil
}
func mergeMetadataMaps(base, overlay map[string]any) map[string]any {
if len(base) == 0 && len(overlay) == 0 {
return nil
}
out := map[string]any{}
for key, value := range base {
out[key] = cloneAuthoringValue(value)
}
for key, value := range overlay {
baseValue, hasBase := out[key]
baseMap, baseIsMap := baseValue.(map[string]any)
valueMap, valueIsMap := value.(map[string]any)
if hasBase && baseIsMap && valueIsMap {
out[key] = mergeMetadataMaps(baseMap, valueMap)
continue
}
out[key] = cloneAuthoringValue(value)
}
if len(out) == 0 {
return nil
}
return out
}
func parseExistingMetadata(row map[string]any) (map[string]any, error) {
rawMeta, ok := lookupField(row, "meta")
if !ok {
return nil, nil
}
return parseRowMetadata(rawMeta)
}
func setLegacyWikiMetadata(row map[string]any, field string, value any) error {
wikiKey := legacyWikiMetadataField(field)
if wikiKey == "" {
return nil
}
meta, err := parseExistingMetadata(row)
if err != nil {
return err
}
if meta == nil {
meta = map[string]any{}
}
wiki, _ := meta["wiki"].(map[string]any)
if wiki == nil {
wiki = map[string]any{}
}
if isNullLike(value) {
delete(wiki, wikiKey)
} else {
wiki[wikiKey] = cloneAuthoringValue(value)
}
if len(wiki) == 0 {
delete(meta, "wiki")
} else {
meta["wiki"] = wiki
}
if len(meta) == 0 {
delete(row, "meta")
} else {
row["meta"] = meta
}
return nil
}
func canonicalizeWikiMetadataDocument(obj map[string]any) {
if rawColumns, ok := obj["columns"].([]any); ok {
filtered := make([]any, 0, len(rawColumns))
for _, raw := range rawColumns {
column, ok := raw.(string)
if ok && isAuthoringOnlyField(column) {
continue
}
filtered = append(filtered, raw)
}
obj["columns"] = filtered
}
if rows, ok := obj["rows"].([]any); ok {
for _, raw := range rows {
row, ok := raw.(map[string]any)
if !ok {
continue
}
canonicalizeWikiMetadataRow(row)
}
}
if entries, ok := obj["entries"].(map[string]any); ok {
for _, raw := range entries {
row, ok := raw.(map[string]any)
if !ok {
continue
}
canonicalizeWikiMetadataRow(row)
}
}
if overrides, ok := obj["overrides"].([]any); ok {
for _, raw := range overrides {
row, ok := raw.(map[string]any)
if !ok {
continue
}
canonicalizeWikiMetadataRow(row)
}
}
}
func canonicalizeWikiMetadataRow(row map[string]any) {
for _, field := range []string{"WIKIREQFEATS", "WIKIGENERATE", "WIKICANONICAL", "WIKISTATUS"} {
value, ok := row[field]
if !ok {
continue
}
_ = setLegacyWikiMetadata(row, field, value)
delete(row, field)
}
}
internal/topdata/migrate.go
-415
View File
@@ -1,415 +0,0 @@
package topdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"gitea.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
type NormalizeResult struct {
StatePath string
ScannedFiles int
UpdatedFiles int
InlineValues int
RemainingFiles int
RemainingLegacyRefs int
}
func NormalizeProject(p *project.Project) (NormalizeResult, error) {
if !p.HasTopData() {
return NormalizeResult{}, fmt.Errorf("topdata is not configured for this project")
}
sourceDir := p.TopDataSourceDir()
dataDir := filepath.Join(sourceDir, "data")
migrationRoot := resolveMigrationInputRoot(sourceDir, p.TopDataReferenceBuilderDir())
legacyTLK, err := loadLegacyTLK(filepath.Join(sourceDir, "tlk"))
if err != nil {
return NormalizeResult{}, err
}
if migrationRoot != "" {
migrationTLK, err := loadLegacyTLK(filepath.Join(migrationRoot, "tlk"))
if err != nil {
return NormalizeResult{}, err
}
legacyTLK = mergeLegacyTLK(migrationTLK, legacyTLK)
}
baseDialogImported, err := importLegacyBaseDialog(migrationRoot, sourceDir)
if err != nil {
return NormalizeResult{}, err
}
itempropsImported, err := importLegacyItempropsRegistry(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
masterfeatsImported, err := importLegacyMasterfeats(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
featImported, err := importLegacyFeat(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
creaturespeedImported, err := importLegacyCreaturespeed(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
armorImported, err := importLegacyArmor(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
baseitemsImported, err := importLegacyBaseitems(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
damagetypesImported, err := importLegacyDamagetypes(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
skyboxesImported, err := importLegacySkyboxes(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
vfxPersistentImported, err := importLegacyVFXPersistent(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
loadscreensImported, err := importLegacyLoadscreens(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
progfxImported, err := importLegacyProgfx(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
genericdoorsImported, err := importLegacyGenericdoors(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
cloakmodelImported, err := importLegacyCloakmodel(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
repadjustImported, err := importLegacyRepadjust(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
wingmodelImported, err := importLegacyWingmodel(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
tailmodelImported, err := importLegacyTailmodel(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
doortypesImported, err := importLegacyDoortypes(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
visualeffectsImported, err := importLegacyVisualeffects(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
appearanceImported, err := importLegacyAppearance(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
placeablesImported, err := importLegacyPlaceables(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
rulesetImported, err := importLegacyRuleset(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
skillsImported, err := importLegacySkills(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
spellsImported, err := importLegacySpells(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
racialtypesImported, err := importLegacyRacialtypes(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
classesImported, err := importLegacyClasses(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
portraitsImported, err := importLegacyPortraits(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
paths, err := collectDataJSONPaths(dataDir)
if err != nil {
return NormalizeResult{}, err
}
result := NormalizeResult{
StatePath: filepath.Join(sourceDir, tlkStateFile),
ScannedFiles: len(paths),
UpdatedFiles: baseDialogImported + itempropsImported + masterfeatsImported + featImported + creaturespeedImported + armorImported + baseitemsImported + damagetypesImported + skyboxesImported + vfxPersistentImported + loadscreensImported + progfxImported + genericdoorsImported + cloakmodelImported + repadjustImported + wingmodelImported + tailmodelImported + doortypesImported + visualeffectsImported + appearanceImported + placeablesImported + rulesetImported + skillsImported + spellsImported + racialtypesImported + classesImported + portraitsImported,
}
state, err := loadTLKState(result.StatePath)
if err != nil {
return result, err
}
if state.Entries == nil {
state.Entries = map[string]tlkStateMapping{}
}
if legacyTLK != nil && state.Language == "" {
state.Language = legacyTLK.Language
}
if legacyTLK != nil {
for key, id := range legacyTLK.Lock {
if _, ok := state.Entries[key]; !ok {
state.Entries[key] = tlkStateMapping{ID: id}
}
}
}
if legacyTLK != nil {
for _, path := range paths {
updated, count, err := inlineLegacyTLKFile(path, legacyTLK)
if err != nil {
return result, err
}
if updated {
result.UpdatedFiles++
result.InlineValues += count
}
}
}
if err := saveTLKState(result.StatePath, state); err != nil {
return result, err
}
if err := removeLegacyTLKAuthoredInput(filepath.Join(sourceDir, "tlk")); err != nil {
return result, err
}
remainingFiles, remainingRefs, err := countLegacyTLKRefs(paths)
if err != nil {
return result, err
}
result.RemainingFiles = remainingFiles
result.RemainingLegacyRefs = remainingRefs
return result, nil
}
func mergeLegacyTLK(base, override *legacyTLKData) *legacyTLKData {
if base == nil && override == nil {
return nil
}
merged := &legacyTLKData{
Language: "en",
Entries: map[string]tlkEntryData{},
Lock: map[string]int{},
}
if base != nil {
if base.Language != "" {
merged.Language = base.Language
}
for key, entry := range base.Entries {
merged.Entries[key] = entry
}
for key, id := range base.Lock {
merged.Lock[key] = id
}
}
if override != nil {
if override.Language != "" {
merged.Language = override.Language
}
for key, entry := range override.Entries {
merged.Entries[key] = entry
}
for key, id := range override.Lock {
merged.Lock[key] = id
}
}
return merged
}
func removeLegacyTLKAuthoredInput(tlkDir string) error {
modulesDir := filepath.Join(tlkDir, "modules")
if err := os.RemoveAll(modulesDir); err != nil {
return fmt.Errorf("remove %s: %w", modulesDir, err)
}
lockPath := filepath.Join(tlkDir, "lock.json")
if err := os.Remove(lockPath); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("remove %s: %w", lockPath, err)
}
gitkeepPath := filepath.Join(tlkDir, ".gitkeep")
if err := os.Remove(gitkeepPath); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("remove %s: %w", gitkeepPath, err)
}
if err := os.Remove(tlkDir); err != nil && !os.IsNotExist(err) {
if !strings.Contains(err.Error(), "directory not empty") {
return fmt.Errorf("remove %s: %w", tlkDir, err)
}
}
return nil
}
func resolveMigrationInputRoot(sourceDir, referenceBuilderDir string) string {
if strings.TrimSpace(referenceBuilderDir) != "" {
return referenceBuilderDir
}
snapshotRoot := filepath.Join(sourceDir, "migration_snapshot")
if info, err := os.Stat(snapshotRoot); err == nil && info.IsDir() {
return snapshotRoot
}
return ""
}
func inlineLegacyTLKFile(path string, legacy *legacyTLKData) (bool, int, error) {
obj, err := loadJSONObject(path)
if err != nil {
return false, 0, err
}
updated, count, err := inlineLegacyTLKValue(obj, legacy)
if err != nil {
return false, 0, err
}
if !updated {
return false, 0, nil
}
raw, err := json.MarshalIndent(obj, "", " ")
if err != nil {
return false, 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(path, raw, 0o644); err != nil {
return false, 0, err
}
return true, count, nil
}
func inlineLegacyTLKValue(value any, legacy *legacyTLKData) (bool, int, error) {
switch typed := value.(type) {
case map[string]any:
if rawTLK, ok := typed["tlk"]; ok {
key, ok := rawTLK.(string)
if ok {
entry, ok := legacy.Entries[key]
if !ok {
return false, 0, nil
}
payload := map[string]any{
"key": key,
"text": entry.Text,
}
if entry.SoundResRef != "" {
payload["sound_resref"] = entry.SoundResRef
}
if entry.SoundLength != 0 {
payload["sound_length"] = entry.SoundLength
}
typed["tlk"] = payload
return true, 1, nil
}
}
updated := false
total := 0
for key, child := range typed {
childUpdated, childCount, err := inlineLegacyTLKValue(child, legacy)
if err != nil {
return false, 0, err
}
if childUpdated {
typed[key] = child
updated = true
total += childCount
}
}
return updated, total, nil
case []any:
updated := false
total := 0
for index, child := range typed {
childUpdated, childCount, err := inlineLegacyTLKValue(child, legacy)
if err != nil {
return false, 0, err
}
if childUpdated {
typed[index] = child
updated = true
total += childCount
}
}
return updated, total, nil
default:
return false, 0, nil
}
}
func collectDataJSONPaths(dataDir string) ([]string, error) {
paths := make([]string, 0)
err := filepath.WalkDir(dataDir, func(path string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
return nil
}
if strings.HasSuffix(strings.ToLower(d.Name()), ".json") && d.Name() != "lock.json" {
paths = append(paths, path)
}
return nil
})
if err != nil {
return nil, err
}
return paths, nil
}
func countLegacyTLKRefs(paths []string) (int, int, error) {
files := 0
refs := 0
for _, path := range paths {
obj, err := loadJSONObject(path)
if err != nil {
return 0, 0, err
}
count := countLegacyTLKRefsInValue(obj)
if count == 0 {
continue
}
files++
refs += count
}
return files, refs, nil
}
func countLegacyTLKRefsInValue(value any) int {
switch typed := value.(type) {
case map[string]any:
total := 0
if rawTLK, ok := typed["tlk"]; ok {
if _, ok := rawTLK.(string); ok {
total++
}
}
for _, child := range typed {
total += countLegacyTLKRefsInValue(child)
}
return total
case []any:
total := 0
for _, child := range typed {
total += countLegacyTLKRefsInValue(child)
}
return total
default:
return 0
}
}
internal/topdata/native.go
-6936
View File
File diff suppressed because it is too large Load Diff
internal/topdata/parts_discovery.go
-692
View File
@@ -1,692 +0,0 @@
package topdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"regexp"
"slices"
"strconv"
"strings"
"gitea.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
// supportedPartCategories defines the part types that should be auto-discovered
// from the sow-assets repository.
var supportedPartCategories = []string{
"belt",
"bicep",
"chest",
"foot",
"forearm",
"leg",
"neck",
"pelvis",
"robe",
"shin",
"shoulder",
}
var partDatasetToAssetCategory = map[string]string{
"belt": "belt",
"bicep": "bicep",
"chest": "chest",
"foot": "foot",
"forearm": "forearm",
"legs": "leg",
"neck": "neck",
"pelvis": "pelvis",
"robe": "robe",
"shin": "shin",
"shoulder": "shoulder",
}
// trailingNumberRegex matches trailing digits at the end of a string
var trailingNumberRegex = regexp.MustCompile(`(\d+)$`)
// isPartsDataset checks if a dataset name corresponds to a parts dataset
func isPartsDataset(datasetName string) bool {
// Check if the dataset is under parts/ directory
if strings.HasPrefix(datasetName, "parts/") {
return true
}
// Check if it's exactly a parts file (e.g., "parts/belt")
parts := strings.Split(datasetName, "/")
if len(parts) == 2 && parts[0] == "parts" {
return true
}
return false
}
func isAutogenEligiblePartsDataset(dataset nativeCollectedDataset) bool {
if !isPartsDataset(dataset.Dataset.Name) {
return false
}
for _, row := range dataset.Rows {
key, ok := row["key"].(string)
if ok && strings.TrimSpace(key) != "" {
return false
}
}
return true
}
// extractPartCategory extracts the part category from a dataset name
// e.g., "parts/belt" -> "belt", "parts/bicep" -> "bicep"
func extractPartCategory(datasetName string) string {
parts := strings.Split(datasetName, "/")
if len(parts) >= 2 {
return parts[len(parts)-1]
}
return ""
}
func partAssetCategoryForDataset(datasetName string) string {
category := extractPartCategory(datasetName)
if mapped, ok := partDatasetToAssetCategory[category]; ok {
return mapped
}
return category
}
// isSupportedPartCategory checks if the given category is in the supported list
func isSupportedPartCategory(category string) bool {
for _, c := range supportedPartCategories {
if c == category {
return true
}
}
return false
}
// extractTrailingNumericSuffix extracts the trailing numeric suffix from a filename stem
// e.g., "pfa0_belt018" -> 18, "pfa0_belt171" -> 171
// Returns 0 and false if no trailing numeric suffix is found
func extractTrailingNumericSuffix(filenameStem string) (int, bool) {
matches := trailingNumberRegex.FindStringSubmatch(filenameStem)
if len(matches) < 2 {
return 0, false
}
num, err := strconv.Atoi(matches[1])
if err != nil {
return 0, false
}
return num, true
}
// DiscoveredPartModel represents a discovered model file with its row ID
type DiscoveredPartModel struct {
RowID int
Filename string
}
// DiscoverPartModels scans the assets/part directory for .mdl files
// and returns a map of rowID -> default row data for the given category.
// The assetsDir should be the root assets directory (containing the part/ subdirectory).
func DiscoverPartModels(assetsDir string, category string) (map[int]*DiscoveredPartModel, error) {
result := make(map[int]*DiscoveredPartModel)
if !isSupportedPartCategory(category) {
return result, nil
}
partRoot, err := resolvePartsRoot(assetsDir)
if err != nil {
return nil, err
}
if partRoot == "" {
return result, nil
}
// Construct the path to the part category directory
partDir := filepath.Join(partRoot, category)
// Check if the directory exists
info, err := os.Stat(partDir)
if err != nil {
if os.IsNotExist(err) {
return result, nil
}
return nil, fmt.Errorf("failed to access part directory %s: %w", partDir, err)
}
if !info.IsDir() {
return nil, fmt.Errorf("part directory %s is not a directory", partDir)
}
// Walk the directory and find .mdl files
err = filepath.Walk(partDir, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.IsDir() {
return nil
}
// Only process .mdl files
if !strings.HasSuffix(strings.ToLower(info.Name()), ".mdl") {
return nil
}
// Extract the filename stem (without extension)
filenameStem := strings.TrimSuffix(info.Name(), filepath.Ext(info.Name()))
// Extract trailing numeric suffix
rowID, hasSuffix := extractTrailingNumericSuffix(filenameStem)
if !hasSuffix {
return nil
}
// Store the discovered model
if existing, ok := result[rowID]; ok {
// If we already have this row ID, keep the first one found (alphabetically)
if path < existing.Filename {
result[rowID] = &DiscoveredPartModel{
RowID: rowID,
Filename: path,
}
}
} else {
result[rowID] = &DiscoveredPartModel{
RowID: rowID,
Filename: path,
}
}
return nil
})
if err != nil {
return nil, fmt.Errorf("failed to scan part directory %s: %w", partDir, err)
}
return result, nil
}
// CreateDefaultPartRow creates a default row for a discovered part model.
// Base defaults are COSTMODIFIER=0 and ACBONUS=0.00. Any robe HIDE* columns
// present in the dataset columns default to 0 as well.
func CreateDefaultPartRow(rowID int) map[string]any {
return CreateDefaultPartRowForColumns(rowID, nil)
}
func CreateDefaultPartRowForColumns(rowID int, columns []string) map[string]any {
return createDefaultPartRowForDataset(rowID, "", columns, project.PartsRowsConfig{})
}
func createDefaultPartRowForDataset(rowID int, datasetName string, columns []string, cfg project.PartsRowsConfig) map[string]any {
row := map[string]any{
"id": rowID,
"COSTMODIFIER": "0",
"ACBONUS": "0.00",
}
for field, value := range cfg.RowDefaults {
if strings.TrimSpace(field) == "" {
continue
}
row[field] = value
}
if value, ok := configuredPartsRowsACBonusValue(rowID, datasetName, cfg); ok {
row["ACBONUS"] = value
}
for _, column := range columns {
if strings.HasPrefix(column, "HIDE") {
row[column] = "0"
}
}
return row
}
func isUnsetPartValue(value any) bool {
switch typed := value.(type) {
case nil:
return true
case string:
return typed == "" || typed == "****"
}
return false
}
func applyDiscoveredPartDefaults(row map[string]any, columns []string) {
applyDiscoveredPartDefaultsWithConfig(row, "", columns, project.PartsRowsConfig{})
}
func applyDiscoveredPartDefaultsWithConfig(row map[string]any, datasetName string, columns []string, cfg project.PartsRowsConfig) {
if isUnsetPartValue(row["COSTMODIFIER"]) {
row["COSTMODIFIER"] = "0"
}
for field, value := range cfg.RowDefaults {
if strings.TrimSpace(field) == "" {
continue
}
if isUnsetPartValue(row[field]) {
row[field] = value
}
}
if isUnsetPartValue(row["ACBONUS"]) {
row["ACBONUS"] = "0.00"
}
if value, ok := configuredPartsRowsACBonusValue(rowIDFromPartRow(row), datasetName, cfg); ok {
row["ACBONUS"] = value
}
for _, column := range columns {
if !strings.HasPrefix(column, "HIDE") {
continue
}
if isUnsetPartValue(row[column]) {
row[column] = "0"
}
}
}
func rowIDFromPartRow(row map[string]any) int {
rowID, _ := row["id"].(int)
return rowID
}
func resolvePartsRoot(root string) (string, error) {
if strings.TrimSpace(root) == "" {
return "", nil
}
candidates := []string{
filepath.Join(root, "assets", "parts"),
filepath.Join(root, "assets", "part"),
filepath.Join(root, "parts"),
filepath.Join(root, "part"),
}
for _, candidate := range candidates {
info, err := os.Stat(candidate)
if err == nil && info.IsDir() {
return candidate, nil
}
if err != nil && !os.IsNotExist(err) {
return "", fmt.Errorf("failed to access parts root %s: %w", candidate, err)
}
}
return "", nil
}
// scanAutogeneratedParts scans the assets/part directory for all supported part categories
// and returns a map of category -> set of row IDs that should be auto-generated.
// The assetsDir parameter should be the path to the assets directory that contains the part/ subdirectory.
func scanAutogeneratedParts(assetsDir string) (map[string]map[int]struct{}, error) {
if assetsDir == "" {
return nil, nil
}
result := make(map[string]map[int]struct{})
for _, category := range supportedPartCategories {
discovered, err := DiscoverPartModels(assetsDir, category)
if err != nil {
return nil, err
}
// Always include the category, even if no IDs were discovered
ids := make(map[int]struct{}, len(discovered))
for rowID := range discovered {
ids[rowID] = struct{}{}
}
result[category] = ids
}
return result, nil
}
func resolveAutogeneratedPartsInventory(p *project.Project, progress func(string)) (map[string]map[int]struct{}, error) {
overrideDir, err := resolvePartsAssetsOverrideDir(p)
if err != nil {
return nil, err
}
if overrideDir != "" {
if progress != nil {
progress(fmt.Sprintf("Scanning part models from local topdata.assets override %s...", overrideDir))
}
return scanAutogeneratedParts(overrideDir)
}
return resolveReleasedPartsManifest(p, progress)
}
func hasCollectedPartsDatasets(collected []nativeCollectedDataset) bool {
for _, dataset := range collected {
if isPartsDataset(dataset.Dataset.Name) {
return true
}
}
return false
}
func loadPartOverrides(path string) ([]map[string]any, error) {
if _, err := os.Stat(path); err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
obj, err := loadJSONObject(path)
if err != nil {
return nil, err
}
rawOverrides, ok := obj["overrides"]
if !ok {
return nil, nil
}
overrideList, ok := rawOverrides.([]any)
if !ok {
return nil, fmt.Errorf("%s: overrides must be an array", path)
}
overrides := make([]map[string]any, 0, len(overrideList))
for index, rawOverride := range overrideList {
override, ok := rawOverride.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: override %d must be an object", path, index)
}
overrides = append(overrides, override)
}
return overrides, nil
}
func applyPartOverrides(sourceDir string, collected []nativeCollectedDataset) ([]nativeCollectedDataset, error) {
return applyPartOverridesWithConfig(sourceDir, collected, project.PartsRowsConfig{})
}
func applyPartOverridesWithConfig(sourceDir string, collected []nativeCollectedDataset, cfg project.PartsRowsConfig) ([]nativeCollectedDataset, error) {
result := make([]nativeCollectedDataset, len(collected))
copy(result, collected)
for i, dataset := range result {
if !strings.HasPrefix(dataset.Dataset.Name, "parts/") {
continue
}
category := extractPartCategory(dataset.Dataset.Name)
if category == "" {
continue
}
overrides, err := loadPartOverridesForCategory(sourceDir, category)
if err != nil {
return nil, err
}
if len(overrides) == 0 {
continue
}
rows := make([]map[string]any, len(dataset.Rows))
rowByID := make(map[int]map[string]any, len(dataset.Rows))
for index, row := range dataset.Rows {
cloned := cloneRowMap(row)
rows[index] = cloned
if rowID, ok := cloned["id"].(int); ok {
rowByID[rowID] = cloned
}
}
for index, override := range overrides {
rawID, ok := override["id"]
if !ok {
return nil, fmt.Errorf("parts/%s override %d is missing id", category, index)
}
rowID, err := asInt(rawID)
if err != nil {
return nil, fmt.Errorf("parts/%s override %d id is not numeric", category, index)
}
row, ok := rowByID[rowID]
if !ok {
row = createDefaultPartRowForDataset(rowID, dataset.Dataset.Name, dataset.Columns, cfg)
rows = append(rows, row)
rowByID[rowID] = row
}
for field, value := range override {
if field == "id" {
continue
}
row[field] = normalizePartOverrideValue(value)
}
}
slices.SortFunc(rows, func(a, b map[string]any) int {
idA := a["id"].(int)
idB := b["id"].(int)
return idA - idB
})
result[i].Rows = rows
}
return result, nil
}
func normalizePartsRowsACBonus(collected []nativeCollectedDataset, cfg project.PartsRowsConfig) ([]nativeCollectedDataset, error) {
if !partsRowsACBonusConfigured(cfg) && len(cfg.RowDefaults) == 0 {
return collected, nil
}
result := make([]nativeCollectedDataset, len(collected))
copy(result, collected)
for i, dataset := range result {
if !strings.HasPrefix(dataset.Dataset.Name, "parts/") {
continue
}
rows := make([]map[string]any, len(dataset.Rows))
for index, row := range dataset.Rows {
cloned := cloneRowMap(row)
for field, value := range cfg.RowDefaults {
if strings.TrimSpace(field) == "" {
continue
}
if isUnsetPartValue(cloned[field]) {
cloned[field] = value
}
}
rowID, ok := cloned["id"].(int)
if isUnsetPartValue(cloned["ACBONUS"]) {
cloned["COSTMODIFIER"] = nullValue
} else if ok {
if value, configured := configuredPartsRowsACBonusValue(rowID, dataset.Dataset.Name, cfg); configured {
cloned["ACBONUS"] = value
}
}
rows[index] = cloned
}
result[i].Rows = rows
}
return result, nil
}
func partsRowsACBonusConfigured(cfg project.PartsRowsConfig) bool {
if strings.TrimSpace(cfg.ACBonus.Default.Strategy) != "" {
return true
}
if len(cfg.ACBonus.Datasets) > 0 {
return true
}
for _, dataset := range cfg.Datasets {
if strings.TrimSpace(dataset.ACBonus.Strategy) != "" {
return true
}
}
return false
}
func configuredPartsRowsACBonusValue(rowID int, datasetName string, cfg project.PartsRowsConfig) (string, bool) {
policy, ok := partsRowsACBonusPolicyForDataset(datasetName, cfg)
if !ok {
return "", false
}
switch strings.TrimSpace(policy.Strategy) {
case "ascending_row_id_sort_key":
format := strings.TrimSpace(policy.Format)
if format == "" {
format = "%.2f"
}
return fmt.Sprintf(format, float64(rowID)/float64(policy.Divisor)), true
case "descending_row_id_sort_key":
format := strings.TrimSpace(policy.Format)
if format == "" {
format = "%.2f"
}
return fmt.Sprintf(format, float64(policy.MaxRowID-rowID)/float64(policy.Divisor)), true
case "fixed":
return policy.Value, true
default:
return "", false
}
}
func partsRowsACBonusPolicyForDataset(datasetName string, cfg project.PartsRowsConfig) (project.PartsRowsACBonusPolicy, bool) {
category := extractPartCategory(datasetName)
if policy, ok := cfg.ACBonus.Datasets[category]; ok && strings.TrimSpace(policy.Strategy) != "" {
return policy, true
}
if dataset, ok := cfg.Datasets[category]; ok && strings.TrimSpace(dataset.ACBonus.Strategy) != "" {
return dataset.ACBonus, true
}
if strings.TrimSpace(cfg.ACBonus.Default.Strategy) != "" {
return cfg.ACBonus.Default, true
}
return project.PartsRowsACBonusPolicy{}, false
}
func loadPartOverridesForCategory(sourceDir, category string) ([]map[string]any, error) {
overrides := []map[string]any{}
legacyPath := filepath.Join(sourceDir, "data", "parts", "overrides", category+".json")
legacyOverrides, err := loadPartOverrides(legacyPath)
if err != nil {
return nil, err
}
overrides = append(overrides, legacyOverrides...)
moduleDir := filepath.Join(sourceDir, "data", "parts", "modules")
modulePaths, err := partModuleOverridePaths(moduleDir, category)
if err != nil {
return nil, err
}
for _, path := range modulePaths {
moduleOverrides, err := loadPartOverrides(path)
if err != nil {
return nil, err
}
overrides = append(overrides, moduleOverrides...)
}
return overrides, nil
}
func partModuleOverridePaths(moduleDir, category string) ([]string, error) {
info, err := os.Stat(moduleDir)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
if !info.IsDir() {
return nil, fmt.Errorf("parts module override path %s is not a directory", moduleDir)
}
var paths []string
err = filepath.WalkDir(moduleDir, func(path string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
return nil
}
if strings.ToLower(filepath.Ext(path)) != ".json" {
return nil
}
if partModuleOverrideMatchesCategory(path, category) {
paths = append(paths, path)
}
return nil
})
if err != nil {
return nil, err
}
slices.Sort(paths)
return paths, nil
}
func partModuleOverrideMatchesCategory(path, category string) bool {
stem := strings.TrimSuffix(strings.ToLower(filepath.Base(path)), strings.ToLower(filepath.Ext(path)))
category = strings.ToLower(strings.TrimSpace(category))
return stem == category ||
strings.HasPrefix(stem, category+"_") ||
strings.HasSuffix(stem, "_"+category) ||
strings.Contains(stem, "_"+category+"_")
}
func normalizePartOverrideValue(value any) any {
switch typed := value.(type) {
case float64:
if typed == float64(int(typed)) {
return strconv.Itoa(int(typed))
}
case int:
return strconv.Itoa(typed)
case json.Number:
return typed.String()
}
return value
}
// augmentWithAutogeneratedParts applies repository-backed discovery defaults.
// Missing discovered IDs are added with default values. Existing discovered rows
// retain authored values unless they still use placeholder values, in which case
// discovery activates them with engine-visible defaults.
func augmentWithAutogeneratedParts(collected []nativeCollectedDataset, autogenerated map[string]map[int]struct{}) []nativeCollectedDataset {
return augmentWithAutogeneratedPartsWithConfig(collected, autogenerated, project.PartsRowsConfig{})
}
func augmentWithAutogeneratedPartsWithConfig(collected []nativeCollectedDataset, autogenerated map[string]map[int]struct{}, cfg project.PartsRowsConfig) []nativeCollectedDataset {
if len(autogenerated) == 0 {
return collected
}
result := make([]nativeCollectedDataset, len(collected))
copy(result, collected)
for i, dataset := range collected {
if !isAutogenEligiblePartsDataset(dataset) {
continue
}
category := partAssetCategoryForDataset(dataset.Dataset.Name)
if !isSupportedPartCategory(category) {
continue
}
ids, ok := autogenerated[category]
if !ok || len(ids) == 0 {
continue
}
// Build a map of existing row IDs.
existingRows := make(map[int]map[string]any, len(dataset.Rows))
for _, row := range dataset.Rows {
if id, ok := row["id"].(int); ok {
existingRows[id] = row
}
}
// Add rows for discovered IDs that don't already exist.
newRows := make([]map[string]any, 0, len(dataset.Rows))
newRows = append(newRows, dataset.Rows...)
for rowID := range ids {
if row, exists := existingRows[rowID]; exists {
applyDiscoveredPartDefaultsWithConfig(row, dataset.Dataset.Name, dataset.Columns, cfg)
continue
}
if _, exists := existingRows[rowID]; !exists {
newRow := createDefaultPartRowForDataset(rowID, dataset.Dataset.Name, dataset.Columns, cfg)
newRows = append(newRows, newRow)
}
}
// Sort rows by ID
slices.SortFunc(newRows, func(a, b map[string]any) int {
idA := a["id"].(int)
idB := b["id"].(int)
return idA - idB
})
result[i].Rows = newRows
}
return result
}
internal/topdata/parts_manifest.go
-287
View File
@@ -1,287 +0,0 @@
package topdata
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"os"
"path/filepath"
"strings"
"time"
"gitea.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
const (
defaultSowAssetsRepoOwner = "ShadowsOverWestgate"
defaultSowAssetsRepoName = "sow-assets"
defaultGiteaBaseURL = "https://gitea.westgate.pw"
partsManifestReleaseTag = "parts-manifest-current"
partsManifestAssetName = "sow-parts-manifest.json"
partsManifestCacheFileName = "sow-parts-manifest.json"
partsManifestCacheMaxAge = time.Hour
partsManifestRequestTimeout = 30 * time.Second
)
type partsManifest struct {
Repo string `json:"repo"`
Ref string `json:"ref"`
GeneratedAt string `json:"generated_at"`
Categories map[string][]int `json:"categories"`
}
type giteaRepoSpec struct {
BaseURL string
Owner string
Repo string
}
var partsManifestHTTPClient = &http.Client{Timeout: partsManifestRequestTimeout}
func resolveReleasedPartsManifest(p *project.Project, progress func(string)) (map[string]map[int]struct{}, error) {
if progress == nil {
progress = func(string) {}
}
cachePath := p.AutogenCachePath(partsManifestCacheFileName)
if strings.TrimSpace(os.Getenv(p.PartsManifestRefreshEnv())) == "" {
manifest, fresh, err := readFreshPartsManifestCache(cachePath, partsManifestCacheMaxAge)
if err != nil {
progress(fmt.Sprintf("Ignoring cached parts manifest: %v", err))
} else if fresh {
progress(fmt.Sprintf("Using cached released parts manifest from %s...", cachePath))
return inventoryFromPartsManifest(manifest), nil
}
}
spec, err := deriveSowAssetsRepoSpec(p)
if err != nil {
return nil, err
}
manifestURL, err := resolvePartsManifestAssetURL(spec)
if err != nil {
return nil, err
}
progress(fmt.Sprintf("Fetching released parts manifest from %s...", manifestURL))
manifest, err := fetchPartsManifest(manifestURL)
if err != nil {
return nil, err
}
if err := writePartsManifestCache(cachePath, manifest); err != nil {
return nil, err
}
return inventoryFromPartsManifest(manifest), nil
}
func deriveSowAssetsRepoSpec(p *project.Project) (giteaRepoSpec, error) {
serverOverride := strings.TrimSpace(os.Getenv(p.AssetsServerURLEnv()))
repoOverride := strings.TrimSpace(os.Getenv(p.AssetsRepoEnv()))
remote, err := gitOutput(p.Root, "remote", "get-url", "origin")
spec := giteaRepoSpec{BaseURL: defaultGiteaBaseURL, Owner: defaultSowAssetsRepoOwner, Repo: defaultSowAssetsRepoName}
if err == nil && strings.TrimSpace(remote) != "" {
if derived, ok := parseRepoSpec(strings.TrimSpace(remote)); ok {
derived.Repo = defaultSowAssetsRepoName
spec = derived
}
}
if serverOverride != "" {
spec.BaseURL = strings.TrimRight(serverOverride, "/")
}
if repoOverride != "" {
owner, repo, ok := parseOwnerRepo(repoOverride)
if !ok {
return giteaRepoSpec{}, fmt.Errorf("invalid SOW_ASSETS_REPO %q; expected owner/repo", repoOverride)
}
spec.Owner = owner
spec.Repo = repo
}
return spec, nil
}
func parseOwnerRepo(raw string) (string, string, bool) {
parts := strings.Split(strings.Trim(strings.TrimSuffix(raw, ".git"), "/"), "/")
if len(parts) < 2 {
return "", "", false
}
owner := strings.TrimSpace(parts[len(parts)-2])
repo := strings.TrimSpace(parts[len(parts)-1])
if owner == "" || repo == "" {
return "", "", false
}
return owner, repo, true
}
func parseRepoSpec(raw string) (giteaRepoSpec, bool) {
if strings.HasPrefix(raw, "ssh://") || strings.HasPrefix(raw, "http://") || strings.HasPrefix(raw, "https://") {
u, err := url.Parse(raw)
if err != nil || u.Host == "" {
return giteaRepoSpec{}, false
}
parts := strings.Split(strings.Trim(strings.TrimSuffix(u.Path, ".git"), "/"), "/")
if len(parts) < 2 {
return giteaRepoSpec{}, false
}
base := (&url.URL{Scheme: "https", Host: normalizeGiteaHTTPHost(u.Hostname())}).String()
if u.Scheme == "http" || u.Scheme == "https" {
base = (&url.URL{Scheme: u.Scheme, Host: u.Host}).String()
}
return giteaRepoSpec{BaseURL: strings.TrimRight(base, "/"), Owner: parts[len(parts)-2], Repo: parts[len(parts)-1]}, true
}
if strings.HasPrefix(raw, "git@") {
withoutUser := strings.TrimPrefix(raw, "git@")
hostAndPath := strings.SplitN(withoutUser, ":", 2)
if len(hostAndPath) != 2 {
return giteaRepoSpec{}, false
}
parts := strings.Split(strings.Trim(strings.TrimSuffix(hostAndPath[1], ".git"), "/"), "/")
if len(parts) < 2 {
return giteaRepoSpec{}, false
}
base := (&url.URL{Scheme: "https", Host: normalizeGiteaHTTPHost(hostAndPath[0])}).String()
return giteaRepoSpec{BaseURL: strings.TrimRight(base, "/"), Owner: parts[len(parts)-2], Repo: parts[len(parts)-1]}, true
}
return giteaRepoSpec{}, false
}
func normalizeGiteaHTTPHost(host string) string {
host = strings.TrimSpace(host)
if strings.HasPrefix(host, "git-ssh.") {
return "gitea." + strings.TrimPrefix(host, "git-ssh.")
}
return host
}
func resolvePartsManifestAssetURL(spec giteaRepoSpec) (string, error) {
apiURL := fmt.Sprintf("%s/api/v1/repos/%s/%s/releases/tags/%s", strings.TrimRight(spec.BaseURL, "/"), url.PathEscape(spec.Owner), url.PathEscape(spec.Repo), url.PathEscape(partsManifestReleaseTag))
var release struct {
Assets []struct {
Name string `json:"name"`
BrowserDownloadURL string `json:"browser_download_url"`
DownloadURL string `json:"download_url"`
} `json:"assets"`
}
if err := fetchJSON(apiURL, &release); err != nil {
return "", fmt.Errorf("fetch parts manifest release metadata: %w", err)
}
for _, asset := range release.Assets {
if asset.Name != partsManifestAssetName {
continue
}
if asset.DownloadURL != "" {
return asset.DownloadURL, nil
}
if asset.BrowserDownloadURL != "" {
return asset.BrowserDownloadURL, nil
}
}
return "", fmt.Errorf("parts manifest asset %s not found in release %s/%s:%s", partsManifestAssetName, spec.Owner, spec.Repo, partsManifestReleaseTag)
}
func fetchPartsManifest(manifestURL string) (*partsManifest, error) {
var manifest partsManifest
if err := fetchJSON(manifestURL, &manifest); err != nil {
return nil, fmt.Errorf("download parts manifest: %w", err)
}
if len(manifest.Categories) == 0 {
return nil, fmt.Errorf("download parts manifest: categories is empty")
}
return &manifest, nil
}
func readFreshPartsManifestCache(path string, maxAge time.Duration) (*partsManifest, bool, error) {
info, err := os.Stat(path)
if err != nil {
if os.IsNotExist(err) {
return nil, false, nil
}
return nil, false, err
}
if maxAge > 0 && time.Since(info.ModTime()) > maxAge {
return nil, false, nil
}
raw, err := os.ReadFile(path)
if err != nil {
return nil, false, err
}
var manifest partsManifest
if err := json.Unmarshal(raw, &manifest); err != nil {
return nil, false, err
}
if len(manifest.Categories) == 0 {
return nil, false, fmt.Errorf("categories is empty")
}
return &manifest, true, nil
}
func fetchJSON(target string, out any) error {
req, err := http.NewRequest(http.MethodGet, target, nil)
if err != nil {
return err
}
if token := strings.TrimSpace(os.Getenv("SOW_ASSETS_TOKEN")); token != "" {
req.Header.Set("Authorization", "token "+token)
} else if token := strings.TrimSpace(os.Getenv("GITEA_TOKEN")); token != "" {
req.Header.Set("Authorization", "token "+token)
}
resp, err := partsManifestHTTPClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
if resp.StatusCode == http.StatusUnauthorized || resp.StatusCode == http.StatusForbidden {
return fmt.Errorf("HTTP %d from %s; set SOW_ASSETS_TOKEN or GITEA_TOKEN if the repo is private", resp.StatusCode, target)
}
return fmt.Errorf("HTTP %d from %s: %s", resp.StatusCode, target, strings.TrimSpace(string(body)))
}
decoder := json.NewDecoder(resp.Body)
if err := decoder.Decode(out); err != nil {
return err
}
return nil
}
func writePartsManifestCache(path string, manifest *partsManifest) error {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return fmt.Errorf("create parts manifest cache dir: %w", err)
}
payload, err := json.MarshalIndent(manifest, "", " ")
if err != nil {
return fmt.Errorf("marshal parts manifest cache: %w", err)
}
payload = append(payload, '\n')
current, err := os.ReadFile(path)
if err == nil && bytes.Equal(current, payload) {
now := time.Now()
if err := os.Chtimes(path, now, now); err != nil {
return fmt.Errorf("refresh parts manifest cache timestamp: %w", err)
}
return nil
}
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("read parts manifest cache: %w", err)
}
tmpPath := path + ".tmp"
if err := os.WriteFile(tmpPath, payload, 0o644); err != nil {
return fmt.Errorf("write parts manifest cache: %w", err)
}
if err := os.Rename(tmpPath, path); err != nil {
return fmt.Errorf("replace parts manifest cache: %w", err)
}
return nil
}
func inventoryFromPartsManifest(manifest *partsManifest) map[string]map[int]struct{} {
result := make(map[string]map[int]struct{}, len(supportedPartCategories))
for _, category := range supportedPartCategories {
ids := make(map[int]struct{})
for _, rowID := range manifest.Categories[category] {
ids[rowID] = struct{}{}
}
result[category] = ids
}
return result
}
internal/topdata/parts_manifest_test.go
-26
View File
@@ -1,26 +0,0 @@
package topdata
import "testing"
func TestParseRepoSpecNormalizesWestgateSSHHostToPublicGiteaHost(t *testing.T) {
spec, ok := parseRepoSpec("ssh://git@git-ssh.westgate.pw:2222/ShadowsOverWestgate/sow-module.git")
if !ok {
t.Fatal("expected SSH remote to parse")
}
if spec.BaseURL != "https://gitea.westgate.pw" {
t.Fatalf("expected public Gitea base URL, got %q", spec.BaseURL)
}
if spec.Owner != "ShadowsOverWestgate" || spec.Repo != "sow-module" {
t.Fatalf("unexpected repo spec: %#v", spec)
}
}
func TestParseRepoSpecKeepsHTTPRemoteHost(t *testing.T) {
spec, ok := parseRepoSpec("https://example.invalid/owner/repo.git")
if !ok {
t.Fatal("expected HTTPS remote to parse")
}
if spec.BaseURL != "https://example.invalid" {
t.Fatalf("expected HTTPS base URL to be preserved, got %q", spec.BaseURL)
}
}
internal/topdata/placeables_migrate.go
-6
View File
@@ -1,6 +0,0 @@
package topdata
func importLegacyPlaceables(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "placeables", "placeables.2da", nil)
}
internal/topdata/portraits_migrate.go
-8
View File
@@ -1,8 +0,0 @@
package topdata
func importLegacyPortraits(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
if legacyTLK == nil {
return 0, nil
}
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "portraits", "portraits.2da", nil)
}
internal/topdata/progfx_migrate.go
-6
View File
@@ -1,6 +0,0 @@
package topdata
func importLegacyProgfx(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "progfx", "progfx.2da", nil)
}
internal/topdata/racialtypes_migrate.go
-282
View File
@@ -1,282 +0,0 @@
package topdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"slices"
"strings"
)
type legacyRacialtypesFeatTable struct {
Output string
Rows []map[string]any
}
func importLegacyRacialtypes(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
if strings.TrimSpace(referenceBuilderDir) == "" {
return 0, nil
}
legacyCoreDir := filepath.Join(referenceBuilderDir, "data", "racialtypes", "core")
legacyFeatsDir := filepath.Join(referenceBuilderDir, "data", "racialtypes", "feats")
if _, err := os.Stat(legacyCoreDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
if _, err := os.Stat(legacyFeatsDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetRoot := filepath.Join(dataDir, "racialtypes", "registry")
targetBasePath := filepath.Join(targetRoot, "base.json")
targetLockPath := filepath.Join(targetRoot, "lock.json")
targetRacesDir := filepath.Join(targetRoot, "races")
if fileExists(targetBasePath) && fileExists(targetLockPath) {
entries, err := os.ReadDir(targetRacesDir)
if err == nil {
raceFiles := 0
for _, entry := range entries {
if !entry.IsDir() && strings.HasSuffix(strings.ToLower(entry.Name()), ".json") {
raceFiles++
}
}
if raceFiles == 16 {
return 0, nil
}
}
}
baseColumns, baseRows, lockData, raceRows, err := collectLegacyRacialtypesRegistry(legacyCoreDir, legacyFeatsDir, legacyTLK)
if err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(dataDir, "racialtypes", "core")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(dataDir, "racialtypes", "feats")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetRoot, "base_rows.json")); err != nil {
return 0, err
}
if err := removePathIfExists(targetRacesDir); err != nil {
return 0, err
}
if err := os.MkdirAll(targetRacesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj any
}{
{
path: targetBasePath,
obj: map[string]any{
"columns": stringSliceToAny(baseColumns),
"rows": rowsToAny(baseRows),
},
},
{
path: targetLockPath,
obj: anyMapInt(lockData),
},
}
raceKeys := make([]string, 0, len(raceRows))
for key := range raceRows {
raceKeys = append(raceKeys, key)
}
slices.Sort(raceKeys)
for _, key := range raceKeys {
fileName := strings.TrimPrefix(key, "racialtypes:") + ".json"
writes = append(writes, struct {
path string
obj any
}{
path: filepath.Join(targetRacesDir, fileName),
obj: raceRows[key],
})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
func collectLegacyRacialtypesRegistry(coreDir, featsDir string, legacyTLK *legacyTLKData) ([]string, []map[string]any, map[string]int, map[string]map[string]any, error) {
coreCollected, err := collectBaseDataset(nativeDataset{
Name: "racialtypes",
BasePath: filepath.Join(coreDir, "base.json"),
LockPath: filepath.Join(coreDir, "lock.json"),
ModulesDir: filepath.Join(coreDir, "modules"),
Spec: specForDataset("racialtypes"),
})
if err != nil {
return nil, nil, nil, nil, err
}
featTables, err := loadLegacyRacialtypesFeatTables(featsDir)
if err != nil {
return nil, nil, nil, nil, err
}
baseRows := make([]map[string]any, 0)
raceRows := map[string]map[string]any{}
lockData := map[string]int{}
for key, id := range coreCollected.LockData {
if strings.HasPrefix(key, "racialtypes:") {
lockData[key] = id
}
}
for _, collectedRow := range coreCollected.Rows {
row, ok := deepCopyValue(collectedRow).(map[string]any)
if !ok {
continue
}
if legacyTLK != nil {
if _, _, err := inlineLegacyTLKValue(row, legacyTLK); err != nil {
return nil, nil, nil, nil, err
}
}
key, _ := row["key"].(string)
if !strings.HasPrefix(key, "racialtypes:") {
assignRacialtypesBaseRowKey(row)
if key, _ := row["key"].(string); key != "" {
rowID, err := asInt(row["id"])
if err != nil {
return nil, nil, nil, nil, err
}
lockData[key] = rowID
}
baseRows = append(baseRows, row)
continue
}
delete(row, "id")
delete(row, "key")
raceFile := map[string]any{
"key": key,
"core": row,
}
if tableKey := extractRacialtypesFeatTableKey(row["FeatsTable"]); tableKey != "" {
table, ok := featTables[tableKey]
if !ok {
return nil, nil, nil, nil, fmt.Errorf("%s: unknown feat table %s", key, tableKey)
}
delete(row, "FeatsTable")
raceFile["feat_output"] = table.Output
raceFile["feats"] = rowsToAny(table.Rows)
}
raceRows[key] = raceFile
}
slices.SortFunc(baseRows, func(a, b map[string]any) int {
left, _ := asInt(a["id"])
right, _ := asInt(b["id"])
return left - right
})
return coreCollected.Columns, baseRows, lockData, raceRows, nil
}
func loadLegacyRacialtypesFeatTables(featsDir string) (map[string]legacyRacialtypesFeatTable, error) {
entries, err := os.ReadDir(featsDir)
if err != nil {
return nil, err
}
tables := map[string]legacyRacialtypesFeatTable{}
for _, entry := range entries {
if entry.IsDir() || filepath.Ext(entry.Name()) != ".json" {
continue
}
obj, err := loadJSONObject(filepath.Join(featsDir, entry.Name()))
if err != nil {
return nil, err
}
tableKey, _ := obj["key"].(string)
outputName, _ := obj["output"].(string)
rawRows, ok := obj["rows"].([]any)
if tableKey == "" || outputName == "" || !ok {
return nil, fmt.Errorf("%s: racialtypes feat table must define key, output, and rows", entry.Name())
}
rows := make([]map[string]any, 0, len(rawRows))
for index, raw := range rawRows {
row, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: row %d must be an object", entry.Name(), index)
}
rows = append(rows, row)
}
tables[tableKey] = legacyRacialtypesFeatTable{
Output: outputName,
Rows: rows,
}
}
return tables, nil
}
func extractRacialtypesFeatTableKey(value any) string {
obj, ok := value.(map[string]any)
if !ok {
return ""
}
tableKey, _ := obj["table"].(string)
return tableKey
}
func assignRacialtypesBaseRowKey(row map[string]any) {
name, _ := row["Name"].(string)
if strings.TrimSpace(name) == "" || name == nullValue {
delete(row, "key")
return
}
label, _ := row["Label"].(string)
keySuffix := normalizeRacialtypesKeySuffix(label)
if keySuffix == "" {
delete(row, "key")
return
}
row["key"] = "racialtypes:" + keySuffix
}
func normalizeRacialtypesKeySuffix(label string) string {
normalized := strings.ToLower(strings.TrimSpace(label))
if normalized == "" {
return ""
}
var b strings.Builder
lastUnderscore := false
for _, ch := range normalized {
isAlphaNum := (ch >= 'a' && ch <= 'z') || (ch >= '0' && ch <= '9')
if isAlphaNum {
b.WriteRune(ch)
lastUnderscore = false
continue
}
if lastUnderscore {
continue
}
b.WriteByte('_')
lastUnderscore = true
}
return strings.Trim(b.String(), "_")
}
internal/topdata/racialtypes_registry.go
-270
View File
@@ -1,270 +0,0 @@
package topdata
import (
"fmt"
"io/fs"
"os"
"path/filepath"
"slices"
"strings"
)
const (
racialtypesRegistryDirName = "racialtypes/registry"
racialtypesRegistryLock = "lock.json"
)
type racialtypesRegistry struct {
RootPath string
LockPath string
LockData map[string]int
BaseColumns []string
BaseRows []map[string]any
Races []map[string]any
}
func loadRacialtypesRegistry(dataDir string) (*racialtypesRegistry, error) {
root := filepath.Join(dataDir, filepath.FromSlash(racialtypesRegistryDirName))
info, err := os.Stat(root)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
if !info.IsDir() {
return nil, fmt.Errorf("racialtypes registry root must be a directory: %s", root)
}
lockPath := filepath.Join(root, racialtypesRegistryLock)
lockData, err := loadLockfile(lockPath)
if err != nil {
return nil, err
}
basePath := filepath.Join(root, "base.json")
baseObj, err := loadJSONObject(basePath)
if err != nil {
return nil, err
}
baseColumns, err := parseColumns(baseObj, "racialtypes/registry/core")
if err != nil {
return nil, err
}
rawBaseRows, ok := baseObj["rows"].([]any)
if !ok {
return nil, fmt.Errorf("%s: rows must be an array", basePath)
}
baseRows := make([]map[string]any, 0, len(rawBaseRows))
for index, raw := range rawBaseRows {
row, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: row %d must be an object", basePath, index)
}
baseRows = append(baseRows, row)
}
raceRows, err := loadRacialtypesRegistryRaceRows(filepath.Join(root, "races"))
if err != nil {
return nil, err
}
return &racialtypesRegistry{
RootPath: root,
LockPath: lockPath,
LockData: lockData,
BaseColumns: baseColumns,
BaseRows: baseRows,
Races: raceRows,
}, nil
}
func loadRacialtypesRegistryRaceRows(dir string) ([]map[string]any, error) {
info, err := os.Stat(dir)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
if !info.IsDir() {
return nil, fmt.Errorf("racialtypes registry races path must be a directory: %s", dir)
}
paths := make([]string, 0)
err = filepath.WalkDir(dir, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
return nil
}
if strings.HasSuffix(strings.ToLower(d.Name()), ".json") {
paths = append(paths, path)
}
return nil
})
if err != nil {
return nil, err
}
slices.Sort(paths)
rows := make([]map[string]any, 0, len(paths))
for _, path := range paths {
obj, err := loadJSONObject(path)
if err != nil {
return nil, err
}
rows = append(rows, obj)
}
return rows, nil
}
func collectRacialtypesRegistryDatasets(dataDir string) ([]nativeCollectedDataset, error) {
registry, err := loadRacialtypesRegistry(dataDir)
if err != nil {
return nil, err
}
if registry == nil {
return nil, nil
}
lockModified, err := syncRacialtypesBaseLockData(registry.BaseRows, registry.LockData)
if err != nil {
return nil, err
}
raceIDs, raceLockModified, err := assignRegistryIDs(registry.Races, registry.LockData, "racialtypes:")
if err != nil {
return nil, err
}
lockModified = lockModified || raceLockModified
if lockModified {
if err := saveLockfile(registry.LockPath, registry.LockData); err != nil {
return nil, err
}
}
coreDataset := nativeDataset{
Kind: nativeDatasetBase,
Name: "racialtypes/registry/core",
OutputName: "racialtypes.2da",
Spec: specForDataset("racialtypes"),
}
featDataset := nativeDataset{
Kind: nativeDatasetPlain,
Name: "racialtypes/registry/feats",
Spec: specForDataset("racialtypes"),
}
coreRows := make([]map[string]any, 0, len(registry.BaseRows)+len(registry.Races))
rowIDSeen := map[int]string{}
for index, raw := range registry.BaseRows {
row, err := canonicalizeBaseRow(coreDataset, registry.BaseColumns, raw, index)
if err != nil {
return nil, fmt.Errorf("racialtypes registry base row %d: %w", index, err)
}
rowID := row["id"].(int)
if existing, ok := rowIDSeen[rowID]; ok {
return nil, fmt.Errorf("racialtypes registry base row id %d conflicts with %s", rowID, existing)
}
rowIDSeen[rowID] = "base"
coreRows = append(coreRows, row)
}
datasets := make([]nativeCollectedDataset, 0, 16)
for _, race := range registry.Races {
key := stringField(race, "key")
if key == "" {
return nil, fmt.Errorf("racialtypes registry row is missing key")
}
rowID, ok := raceIDs[key]
if !ok {
return nil, fmt.Errorf("racialtypes registry row %s is missing lock id", key)
}
if existing, ok := rowIDSeen[rowID]; ok {
return nil, fmt.Errorf("racialtypes registry row id %d for %s conflicts with %s", rowID, key, existing)
}
rowIDSeen[rowID] = key
coreRaw, ok := race["core"].(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: core must be an object", key)
}
row, err := canonicalizeEntry(coreDataset, registry.BaseColumns, rowID, key, coreRaw)
if err != nil {
return nil, fmt.Errorf("%s: %w", key, err)
}
featOutput, _ := race["feat_output"].(string)
if featOutput != "" {
row["FeatsTable"] = outputStem(featOutput)
}
coreRows = append(coreRows, row)
if featOutput == "" {
continue
}
rawFeats, ok := race["feats"].([]any)
if !ok {
return nil, fmt.Errorf("%s: feats must be an array when feat_output is set", key)
}
featRows := make([]map[string]any, 0, len(rawFeats))
for index, rawFeat := range rawFeats {
featRow, ok := rawFeat.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: feat row %d must be an object", key, index)
}
row, err := canonicalizeBaseRow(featDataset, []string{"FeatLabel", "FeatIndex"}, featRow, index)
if err != nil {
return nil, fmt.Errorf("%s: feat row %d: %w", key, index, err)
}
featRows = append(featRows, row)
}
datasets = append(datasets, newGeneratedDataset(
"racialtypes/registry/"+strings.TrimPrefix(key, "racialtypes:"),
featOutput,
[]string{"FeatLabel", "FeatIndex"},
featRows,
))
}
slices.SortFunc(coreRows, func(a, b map[string]any) int {
return a["id"].(int) - b["id"].(int)
})
slices.SortFunc(datasets, func(a, b nativeCollectedDataset) int {
return strings.Compare(a.Dataset.OutputName, b.Dataset.OutputName)
})
coreCollected := newGeneratedDataset("racialtypes/registry/core", "racialtypes.2da", registry.BaseColumns, coreRows)
out := make([]nativeCollectedDataset, 0, len(datasets)+1)
out = append(out, coreCollected)
out = append(out, datasets...)
return out, nil
}
func syncRacialtypesBaseLockData(rows []map[string]any, lockData map[string]int) (bool, error) {
modified := false
for _, raw := range rows {
key := stringField(raw, "key")
if key == "" {
continue
}
rowID, ok := raw["id"]
if !ok {
return false, fmt.Errorf("racialtypes registry base row %s is missing id", key)
}
id, err := asInt(rowID)
if err != nil {
return false, fmt.Errorf("racialtypes registry base row %s has invalid id: %w", key, err)
}
if existing, ok := lockData[key]; ok {
if existing != id {
return false, fmt.Errorf("racialtypes registry base row %s id %d does not match lock id %d", key, id, existing)
}
continue
}
lockData[key] = id
modified = true
}
return modified, nil
}
internal/topdata/repadjust_migrate.go
-63
View File
@@ -1,63 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyRepadjust(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "repadjust")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "repadjust")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
if fileExists(targetBasePath) && fileExists(targetLockPath) {
obj, err := loadJSONObject(targetBasePath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "repadjust.2da"
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
internal/topdata/ruleset_migrate.go
-171
View File
@@ -1,171 +0,0 @@
package topdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
)
func importLegacyRuleset(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "ruleset")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "ruleset")
targetPath := filepath.Join(targetDir, "ruleset.json")
if _, err := os.Stat(targetPath); err == nil {
obj, err := loadJSONObject(targetPath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
rows, err := mergeRulesetRows(baseObj, filepath.Join(legacyDir, "modules"))
if err != nil {
return 0, err
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "base.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "lock.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "modules")); err != nil {
return 0, err
}
out := map[string]any{
"output": "ruleset.2da",
"columns": baseObj["columns"],
"rows": rows,
}
raw, err := json.MarshalIndent(out, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
return 0, err
}
return 1, nil
}
func mergeRulesetRows(baseObj map[string]any, modulesDir string) ([]any, error) {
rawRows, ok := baseObj["rows"].([]any)
if !ok {
return nil, nil
}
rows := make([]map[string]any, 0, len(rawRows))
byLabel := map[string]map[string]any{}
for _, raw := range rawRows {
row, ok := deepCopyValue(raw).(map[string]any)
if !ok {
continue
}
delete(row, "key")
if rawID, ok := row["id"]; ok {
id, err := asInt(rawID)
if err != nil {
return nil, err
}
row["id"] = id
}
if label, ok := row["Label"].(string); ok && label != "" && label != "****" {
if _, exists := byLabel[label]; exists {
return nil, fmt.Errorf("duplicate ruleset label %q", label)
}
byLabel[label] = row
}
rows = append(rows, row)
}
modulePaths, err := collectModulePaths(modulesDir)
if err != nil {
return nil, err
}
for _, path := range modulePaths {
obj, err := loadJSONObject(path)
if err != nil {
return nil, err
}
overrideList, ok := obj["overrides"].([]any)
if !ok {
continue
}
for _, raw := range overrideList {
override, ok := raw.(map[string]any)
if !ok {
continue
}
matchObj, ok := override["match"].(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: ruleset override is missing match object", path)
}
rawLabel, ok := matchObj["Label"]
if !ok {
return nil, fmt.Errorf("%s: ruleset override match is missing Label", path)
}
labels, err := normalizeRulesetMatchLabels(rawLabel)
if err != nil {
return nil, fmt.Errorf("%s: %w", path, err)
}
for _, label := range labels {
row, ok := byLabel[label]
if !ok {
return nil, fmt.Errorf("%s: ruleset override target label %q was not found", path, label)
}
for key, value := range override {
if key == "id" || key == "key" || key == "_tlk" || isMetadataField(key) || key == "match" {
continue
}
row[key] = deepCopyValue(value)
}
}
}
}
sort.Slice(rows, func(i, j int) bool {
return rows[i]["id"].(int) < rows[j]["id"].(int)
})
out := make([]any, 0, len(rows))
for _, row := range rows {
out = append(out, row)
}
return out, nil
}
func normalizeRulesetMatchLabels(raw any) ([]string, error) {
switch typed := raw.(type) {
case string:
return []string{typed}, nil
case []any:
out := make([]string, 0, len(typed))
for _, item := range typed {
label, ok := item.(string)
if !ok {
return nil, fmt.Errorf("ruleset match.Label entries must be strings")
}
out = append(out, label)
}
return out, nil
default:
return nil, fmt.Errorf("ruleset match.Label must be a string or string array")
}
}
internal/topdata/skills_migrate.go
-140
View File
@@ -1,140 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacySkills(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "skills")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "skills")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
moduleNames := []string{
"add_athletics.json",
"add_disguise.json",
"add_linguistics.json",
"add_perception.json",
"add_scriptcraft.json",
"add_sensemotive.json",
"add_stealth.json",
"add_survival.json",
"add_userope.json",
filepath.Join("craft", "add_craftalchemy.json"),
filepath.Join("craft", "add_craftcooking.json"),
filepath.Join("craft", "add_craftjewelry.json"),
filepath.Join("craft", "add_craftleatherworking.json"),
filepath.Join("craft", "add_craftstonework.json"),
filepath.Join("craft", "add_crafttextiles.json"),
filepath.Join("craft", "ovr_craftarmorsmithing.json"),
filepath.Join("craft", "ovr_crafttinkering.json"),
filepath.Join("craft", "ovr_craftweaponsmithing.json"),
filepath.Join("craft", "ovr_craftwoodworking.json"),
filepath.Join("knowledge", "add_knowledgearcana.json"),
filepath.Join("knowledge", "add_knowledgearchitecture.json"),
filepath.Join("knowledge", "add_knowledgedungeoneering.json"),
filepath.Join("knowledge", "add_knowledgegeography.json"),
filepath.Join("knowledge", "add_knowledgehistory.json"),
filepath.Join("knowledge", "add_knowledgelocal.json"),
filepath.Join("knowledge", "add_knowledgenature.json"),
filepath.Join("knowledge", "add_knowledgenobility.json"),
filepath.Join("knowledge", "add_knowledgeplanar.json"),
filepath.Join("knowledge", "add_knowledgereligion.json"),
"ovr_acrobatics.json",
"ovr_animalhandling.json",
"ovr_appraise.json",
"ovr_concentration.json",
"ovr_disabledevice.json",
"ovr_heal.json",
"ovr_hiddenskills.json",
"ovr_influence.json",
"ovr_openlock.json",
"ovr_parry.json",
"ovr_searchtowis.json",
"ovr_sleightofhand.json",
"ovr_taunttointimidate.json",
"rmv_removedskills.json",
}
targetModulePaths := make([]string, 0, len(moduleNames))
for _, name := range moduleNames {
targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "skills.2da"
canonicalizeWikiMetadataDocument(baseObj)
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
canonicalizeWikiMetadataDocument(obj)
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for i, path := range targetModulePaths {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return 0, err
}
writes = append(writes, struct {
path string
obj map[string]any
}{path: path, obj: moduleObjs[i]})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
internal/topdata/skyboxes_migrate.go
-6
View File
@@ -1,6 +0,0 @@
package topdata
func importLegacySkyboxes(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "skyboxes", "skyboxes.2da", nil)
}
internal/topdata/spells_migrate.go
-103
View File
@@ -1,103 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacySpells(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "spells")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "spells")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
moduleNames := []string{
"specialattacks.json",
"ovr_fixduplicates.json",
filepath.Join("bardicmusic", "fascinate.json"),
filepath.Join("bardicmusic", "inspirecompetence.json"),
filepath.Join("bardicmusic", "inspirecourage.json"),
filepath.Join("bardicmusic", "inspiregreatness.json"),
filepath.Join("bardicmusic", "inspireheroics.json"),
filepath.Join("bardicmusic", "songoffreedom.json"),
}
targetModulePaths := make([]string, 0, len(moduleNames))
for _, name := range moduleNames {
targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "spells.2da"
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for i, path := range targetModulePaths {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return 0, err
}
writes = append(writes, struct {
path string
obj map[string]any
}{path: path, obj: moduleObjs[i]})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
internal/topdata/tailmodel_migrate.go
-86
View File
@@ -1,86 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyTailmodel(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "tailmodel")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "tailmodel")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
targetModulePaths := []string{
filepath.Join(targetModulesDir, "add.json"),
filepath.Join(targetModulesDir, "ovr_tailprefixes.json"),
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "tailmodel.2da"
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleNames := []string{"add.json", "ovr_tailprefixes.json"}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
{path: targetModulePaths[0], obj: moduleObjs[0]},
{path: targetModulePaths[1], obj: moduleObjs[1]},
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
internal/topdata/tlk_native.go
-729
View File
@@ -1,729 +0,0 @@
package topdata
import (
"bytes"
"encoding/binary"
"encoding/json"
"fmt"
"math"
"os"
"path/filepath"
"slices"
"strconv"
"strings"
"golang.org/x/text/encoding/charmap"
)
const (
defaultTLKName = "sow_tlk.tlk"
tlkStateFile = ".tlk_state.json"
customTLKBase = 0x01000000
)
type textRole string
const (
textRoleName textRole = "name"
textRoleDescription textRole = "description"
)
type datasetTextSpec struct {
NameFields []string
DescriptionFields []string
}
var datasetTextSpecs = map[string]datasetTextSpec{
"baseitems": {
NameFields: []string{"Name"},
DescriptionFields: []string{"Description"},
},
"classes": {
NameFields: []string{"Short", "Name", "Plural", "Lower", "CLASS"},
DescriptionFields: []string{"Description"},
},
"feat": {
NameFields: []string{"FEAT"},
DescriptionFields: []string{"DESCRIPTION"},
},
"masterfeats": {
NameFields: []string{"STRREF"},
DescriptionFields: []string{"DESCRIPTION"},
},
"itemprops": {
NameFields: []string{"Name", "StringRef"},
DescriptionFields: []string{"Description", "GameStrRef"},
},
"racialtypes": {
NameFields: []string{"Name", "RACIALTYPE"},
DescriptionFields: []string{"Description"},
},
"skills": {
NameFields: []string{"Name", "SKILL"},
DescriptionFields: []string{"Description"},
},
"spells": {
NameFields: []string{"Name", "SPELL"},
DescriptionFields: []string{"SpellDesc", "DESCRIPTION"},
},
}
type tlkEntryData struct {
Text string
SoundResRef string
SoundLength float32
}
type tlkStateDocument struct {
Version int `json:"version"`
Language string `json:"language"`
Entries map[string]tlkStateMapping `json:"entries"`
}
type tlkStateMapping struct {
ID int `json:"id"`
Retired bool `json:"retired,omitempty"`
}
type tlkCompiledValue struct {
Value any
TLKKey string
}
type tlkCompiler struct {
language string
statePath string
state tlkStateDocument
legacy map[string]tlkEntryData
active map[string]tlkEntryData
activeKeys map[string]struct{}
reservedByID map[int]string
nextID int
}
func newTLKCompiler(sourceDir string, legacy *legacyTLKData) (*tlkCompiler, error) {
statePath := filepath.Join(sourceDir, tlkStateFile)
state, err := loadTLKState(statePath)
if err != nil {
return nil, err
}
baseDialog, err := loadBaseDialogData(filepath.Join(sourceDir, "base_dialog.json"))
if err != nil {
return nil, err
}
legacy = mergeLegacyTLK(baseDialog, legacy)
language := state.Language
if language == "" {
language = "en"
}
if legacy != nil && legacy.Language != "" {
if state.Language == "" {
language = legacy.Language
}
for key, id := range legacy.Lock {
if _, ok := state.Entries[key]; !ok {
state.Entries[key] = tlkStateMapping{ID: id}
}
}
}
reserved := map[int]string{}
nextID := 0
for key, entry := range state.Entries {
if owner, ok := reserved[entry.ID]; ok && owner != key {
return nil, fmt.Errorf("tlk state id collision between %q and %q", owner, key)
}
reserved[entry.ID] = key
if entry.ID >= nextID {
nextID = entry.ID + 1
}
}
compiler := &tlkCompiler{
language: language,
statePath: statePath,
state: state,
legacy: map[string]tlkEntryData{},
active: map[string]tlkEntryData{},
activeKeys: map[string]struct{}{},
reservedByID: reserved,
nextID: nextID,
}
if legacy != nil {
for key, entry := range legacy.Entries {
compiler.legacy[key] = entry
}
}
return compiler, nil
}
func loadBaseDialogData(path string) (*legacyTLKData, error) {
if _, err := os.Stat(path); err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, fmt.Errorf("stat %s: %w", path, err)
}
obj, err := loadJSONObject(path)
if err != nil {
return nil, err
}
result := &legacyTLKData{
Language: "en",
Entries: map[string]tlkEntryData{},
Lock: map[string]int{},
}
if language, ok := obj["language"].(string); ok && strings.TrimSpace(language) != "" {
result.Language = language
}
rawEntries, ok := obj["entries"]
if !ok {
return result, nil
}
entries, err := normalizeBaseDialogEntries(rawEntries)
if err != nil {
return nil, err
}
normalized, err := normalizeLegacyTLKEntries(entries)
if err != nil {
return nil, err
}
for key, entry := range normalized {
result.Entries[key] = entry
}
return result, nil
}
func normalizeBaseDialogEntries(raw any) (map[string]any, error) {
switch typed := raw.(type) {
case map[string]any:
return typed, nil
case []any:
entries := make(map[string]any, len(typed))
for index, item := range typed {
obj, ok := item.(map[string]any)
if !ok {
return nil, fmt.Errorf("base_dialog.json entry %d must be an object", index)
}
rawID, ok := obj["id"]
if !ok {
return nil, fmt.Errorf("base_dialog.json entry %d is missing id", index)
}
key, err := legacyDialogIDKey(rawID)
if err != nil {
return nil, fmt.Errorf("base_dialog.json entry %d: %w", index, err)
}
entry := map[string]any{}
if text, ok := obj["text"]; ok {
entry["text"] = text
}
if sound, ok := obj["sound_resref"]; ok {
entry["sound_resref"] = sound
}
if length, ok := obj["sound_length"]; ok {
entry["sound_length"] = length
}
entries[key] = entry
}
return entries, nil
default:
return nil, fmt.Errorf("base_dialog.json entries must be an object or array")
}
}
func legacyDialogIDKey(raw any) (string, error) {
switch typed := raw.(type) {
case string:
if strings.TrimSpace(typed) == "" {
return "", fmt.Errorf("id must not be empty")
}
return typed, nil
case float64:
return strconv.Itoa(int(typed)), nil
case int:
return strconv.Itoa(typed), nil
default:
return "", fmt.Errorf("id must be a string or number")
}
}
func loadTLKState(path string) (tlkStateDocument, error) {
doc := tlkStateDocument{
Version: 1,
Entries: map[string]tlkStateMapping{},
}
raw, err := os.ReadFile(path)
if err != nil {
if os.IsNotExist(err) {
return doc, nil
}
return doc, fmt.Errorf("read %s: %w", path, err)
}
if err := json.Unmarshal(raw, &doc); err != nil {
return doc, fmt.Errorf("parse %s: %w", path, err)
}
if doc.Version == 0 {
doc.Version = 1
}
if doc.Entries == nil {
doc.Entries = map[string]tlkStateMapping{}
}
return doc, nil
}
func saveTLKState(path string, doc tlkStateDocument) error {
doc.Version = 1
if doc.Language == "" {
doc.Language = "en"
}
if doc.Entries == nil {
doc.Entries = map[string]tlkStateMapping{}
}
raw, err := json.MarshalIndent(doc, "", " ")
if err != nil {
return fmt.Errorf("marshal tlk state: %w", err)
}
raw = append(raw, '\n')
current, err := os.ReadFile(path)
if err == nil && bytes.Equal(current, raw) {
return nil
}
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("read %s: %w", path, err)
}
if err := os.WriteFile(path, raw, 0o644); err != nil {
return fmt.Errorf("write %s: %w", path, err)
}
return nil
}
func (c *tlkCompiler) resolveLegacyKey(key string) (tlkCompiledValue, error) {
if _, ok := c.legacy[key]; !ok {
if active, ok := c.active[key]; ok {
return tlkCompiledValue{
Value: c.customStrrefForKey(key),
TLKKey: key,
}, c.markActive(key, active)
}
if len(c.legacy) == 0 {
return tlkCompiledValue{}, fmt.Errorf("unknown TLK reference: %s (inline TLK text is required for native builds)", key)
}
return tlkCompiledValue{}, fmt.Errorf("unknown TLK reference: %s", key)
}
if err := c.markActive(key, c.legacy[key]); err != nil {
return tlkCompiledValue{}, err
}
return tlkCompiledValue{
Value: c.customStrrefForKey(key),
TLKKey: key,
}, nil
}
func (c *tlkCompiler) registerInline(key string, entry tlkEntryData) (tlkCompiledValue, error) {
if err := c.markActive(key, entry); err != nil {
return tlkCompiledValue{}, err
}
return tlkCompiledValue{
Value: c.customStrrefForKey(key),
TLKKey: key,
}, nil
}
func (c *tlkCompiler) customStrrefForKey(key string) int {
return customTLKBase + c.state.Entries[key].ID
}
func (c *tlkCompiler) markActive(key string, entry tlkEntryData) error {
mapping, ok := c.state.Entries[key]
if !ok {
mapping = tlkStateMapping{ID: c.allocateID(), Retired: false}
c.state.Entries[key] = mapping
}
existing, ok := c.active[key]
if ok && existing != entry {
return fmt.Errorf("conflicting TLK content for key %q", key)
}
c.active[key] = entry
c.activeKeys[key] = struct{}{}
mapping.Retired = false
c.state.Entries[key] = mapping
return nil
}
func (c *tlkCompiler) allocateID() int {
id := c.nextID
c.nextID++
return id
}
func (c *tlkCompiler) finish(outputDir, tlkName string) (int, error) {
if err := os.MkdirAll(outputDir, 0o755); err != nil {
return 0, fmt.Errorf("create tlk output dir: %w", err)
}
for key, mapping := range c.state.Entries {
if _, ok := c.activeKeys[key]; ok {
mapping.Retired = false
} else {
mapping.Retired = true
}
c.state.Entries[key] = mapping
}
pruneRetiredGeneratedFeatTLKEntries(filepath.Dir(c.statePath), c.state.Entries)
if err := saveTLKState(c.statePath, c.state); err != nil {
return 0, err
}
maxID := -1
for key := range c.activeKeys {
id := c.state.Entries[key].ID
if id > maxID {
maxID = id
}
}
entries := make([]tlkEntryData, maxID+1)
for key := range c.activeKeys {
id := c.state.Entries[key].ID
entries[id] = c.active[key]
}
if strings.TrimSpace(tlkName) == "" {
tlkName = defaultTLKName
}
if err := writeTLKBinary(filepath.Join(outputDir, tlkName), entries, c.language); err != nil {
return 0, err
}
if maxID < 0 {
return 1, nil
}
return 1, nil
}
func pruneRetiredGeneratedFeatTLKEntries(sourceDir string, entries map[string]tlkStateMapping) {
prefixes := collectGeneratedFeatTLKPrefixes(sourceDir)
for key, mapping := range entries {
if !mapping.Retired || !isRetiredGeneratedFeatTLKKey(key, prefixes) {
continue
}
delete(entries, key)
}
}
func isRetiredGeneratedFeatTLKKey(key string, prefixes []string) bool {
if !strings.HasPrefix(key, "feat:") {
return false
}
for _, prefix := range prefixes {
if strings.HasPrefix(key, prefix) {
return true
}
}
return false
}
func collectGeneratedFeatTLKPrefixes(sourceDir string) []string {
featGeneratedDir := filepath.Join(sourceDir, "data", "feat", "generated")
paths, err := collectModulePaths(featGeneratedDir)
if err != nil {
return nil
}
prefixes := make([]string, 0, len(paths))
seen := map[string]struct{}{}
for _, path := range paths {
obj, err := loadJSONObject(path)
if err != nil || !isFamilyExpansionObject(obj) {
continue
}
spec, err := parseFamilyExpansionSpec(path, obj)
if err != nil || spec.FamilyKey == "" {
continue
}
prefix := "feat:" + spec.FamilyKey + "_"
if _, ok := seen[prefix]; ok {
continue
}
seen[prefix] = struct{}{}
prefixes = append(prefixes, prefix)
}
slices.Sort(prefixes)
return prefixes
}
func writeTLKBinary(path string, entries []tlkEntryData, language string) error {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return fmt.Errorf("create tlk output parent: %w", err)
}
file, err := os.Create(path)
if err != nil {
return fmt.Errorf("create tlk output: %w", err)
}
defer file.Close()
langID, err := languageID(language)
if err != nil {
return err
}
if _, err := file.Write([]byte("TLK V3.0")); err != nil {
return err
}
if err := binary.Write(file, binary.LittleEndian, uint32(langID)); err != nil {
return err
}
if err := binary.Write(file, binary.LittleEndian, uint32(len(entries))); err != nil {
return err
}
if err := binary.Write(file, binary.LittleEndian, uint32(20+len(entries)*40)); err != nil {
return err
}
stringData := make([][]byte, len(entries))
offset := 0
for index, entry := range entries {
flags := uint32(0)
if entry.Text != "" {
flags |= 0x1
}
if entry.SoundResRef != "" {
flags |= 0x2
}
if entry.SoundLength != 0 {
flags |= 0x4
}
encodedText, err := charmap.Windows1252.NewEncoder().Bytes([]byte(entry.Text))
if err != nil {
return fmt.Errorf("encode tlk text at row %d: %w", index, err)
}
stringData[index] = encodedText
soundBytes := make([]byte, 16)
if len(entry.SoundResRef) > 16 {
return fmt.Errorf("sound resref at row %d exceeds 16 characters", index)
}
copy(soundBytes, []byte(entry.SoundResRef))
if err := binary.Write(file, binary.LittleEndian, flags); err != nil {
return err
}
if _, err := file.Write(soundBytes); err != nil {
return err
}
if err := binary.Write(file, binary.LittleEndian, uint32(0)); err != nil {
return err
}
if err := binary.Write(file, binary.LittleEndian, uint32(0)); err != nil {
return err
}
if err := binary.Write(file, binary.LittleEndian, uint32(offset)); err != nil {
return err
}
if err := binary.Write(file, binary.LittleEndian, uint32(len(encodedText))); err != nil {
return err
}
if err := binary.Write(file, binary.LittleEndian, entry.SoundLength); err != nil {
return err
}
offset += len(encodedText)
}
for _, encoded := range stringData {
if _, err := file.Write(encoded); err != nil {
return err
}
}
return nil
}
func languageID(code string) (int, error) {
switch strings.ToLower(strings.TrimSpace(code)) {
case "", "en":
return 0, nil
case "fr":
return 1, nil
case "de":
return 2, nil
case "it":
return 3, nil
case "es":
return 4, nil
case "pl":
return 5, nil
default:
return 0, fmt.Errorf("unknown TLK language code %q", code)
}
}
func specForDataset(name string) datasetTextSpec {
best := datasetTextSpecs[name]
bestLength := 0
for prefix, spec := range datasetTextSpecs {
if name != prefix && !strings.HasPrefix(name, prefix+"/") {
continue
}
if len(prefix) <= bestLength {
continue
}
best = spec
bestLength = len(prefix)
}
return best
}
func roleForField(spec datasetTextSpec, field string) textRole {
for _, candidate := range spec.NameFields {
if strings.EqualFold(candidate, field) {
return textRoleName
}
}
for _, candidate := range spec.DescriptionFields {
if strings.EqualFold(candidate, field) {
return textRoleDescription
}
}
return textRole(strings.ToLower(field))
}
func columnForRole(columns []string, spec datasetTextSpec, role textRole) string {
var candidates []string
switch role {
case textRoleName:
candidates = spec.NameFields
case textRoleDescription:
candidates = spec.DescriptionFields
default:
return ""
}
for _, candidate := range candidates {
if column, ok := canonicalColumn(columns, candidate); ok {
return column
}
}
return ""
}
type parsedTLKPayload struct {
Key string
Text string
Ref string
RefField string
SoundResRef string
SoundLength float32
}
func parseTLKPayload(value any, allowBare bool) (parsedTLKPayload, bool, error) {
switch typed := value.(type) {
case map[string]any:
if rawTLK, ok := typed["tlk"]; ok {
if key, ok := rawTLK.(string); ok {
return parsedTLKPayload{Key: key}, true, nil
}
return parseTLKPayload(rawTLK, true)
}
if allowBare || hasNonRefTLKPayloadKeys(typed) {
if _, ok := typed["text"]; ok {
return buildParsedTLKPayload(typed)
}
if _, ok := typed["ref"]; ok {
return buildParsedTLKPayload(typed)
}
if _, ok := typed["key"]; ok {
return buildParsedTLKPayload(typed)
}
}
return parsedTLKPayload{}, false, nil
case string:
return parsedTLKPayload{}, false, nil
default:
return parsedTLKPayload{}, false, nil
}
}
func hasNonRefTLKPayloadKeys(obj map[string]any) bool {
for _, key := range []string{"text", "key", "sound_resref", "sound_length"} {
if _, ok := obj[key]; ok {
return true
}
}
return false
}
func buildParsedTLKPayload(obj map[string]any) (parsedTLKPayload, bool, error) {
payload := parsedTLKPayload{}
if rawKey, ok := obj["key"]; ok {
key, ok := rawKey.(string)
if !ok {
return payload, false, fmt.Errorf("tlk key must be a string")
}
payload.Key = key
}
if rawText, ok := obj["text"]; ok {
text, ok := rawText.(string)
if !ok {
return payload, false, fmt.Errorf("tlk text must be a string")
}
payload.Text = text
}
if rawRef, ok := obj["ref"]; ok {
ref, ok := rawRef.(string)
if !ok {
return payload, false, fmt.Errorf("tlk ref must be a string")
}
payload.Ref = ref
}
if rawField, ok := obj["field"]; ok {
field, ok := rawField.(string)
if !ok {
return payload, false, fmt.Errorf("tlk field must be a string")
}
payload.RefField = field
}
if rawSound, ok := obj["sound_resref"]; ok {
sound, ok := rawSound.(string)
if !ok {
return payload, false, fmt.Errorf("tlk sound_resref must be a string")
}
payload.SoundResRef = sound
}
if rawLength, ok := obj["sound_length"]; ok {
switch typed := rawLength.(type) {
case float64:
payload.SoundLength = float32(typed)
case int:
payload.SoundLength = float32(typed)
default:
return payload, false, fmt.Errorf("tlk sound_length must be numeric")
}
}
if payload.Ref != "" && payload.RefField == "" {
return payload, false, fmt.Errorf("tlk ref requires field")
}
if payload.Ref != "" && payload.Text != "" {
return payload, false, fmt.Errorf("tlk payload cannot contain both text and ref")
}
if payload.Ref == "" && payload.Text == "" && payload.Key == "" {
return payload, false, fmt.Errorf("tlk payload must contain text, ref, or key")
}
return payload, true, nil
}
func deriveAutoTLKKey(rowIdentity string, field string) string {
fieldText := strings.ToLower(strings.TrimSpace(field))
if fieldText == "" {
fieldText = "text"
}
return rowIdentity + "." + fieldText
}
func sortedKeys[M ~map[string]V, V any](input M) []string {
keys := make([]string, 0, len(input))
for key := range input {
keys = append(keys, key)
}
slices.Sort(keys)
return keys
}
func almostEqualFloat32(a, b float32) bool {
return math.Abs(float64(a-b)) < 0.0001
}
internal/topdata/top_package.go
-713
View File
@@ -1,713 +0,0 @@
package topdata
import (
"errors"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"slices"
"strings"
"time"
"gitea.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
"gitea.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
const (
PackageHAKResRef = "sow_top"
PackageHAKFileName = PackageHAKResRef + ".hak"
PackageTLKFileName = defaultTLKName
)
func compiled2DAOutputDir(p *project.Project) string {
return p.TopDataCompiled2DADir()
}
func topDataBuildUsesRepoCache(p *project.Project) bool {
return p.TopDataUsesRepoCache()
}
func compiledTLKOutputPath(p *project.Project) string {
return p.TopDataCompiledTLKPath()
}
func compiledTLKOutputDir(p *project.Project) string {
return p.TopDataCompiledTLKDir()
}
func wikiOutputRootDir(p *project.Project) string {
return p.TopDataWikiRootDir()
}
func wikiOutputPagesDir(p *project.Project) string {
return p.TopDataWikiPagesDir()
}
func wikiOutputStatePath(p *project.Project) string {
return p.TopDataWikiStatePath()
}
func legacyCompiled2DAOutputDir(p *project.Project) string {
return filepath.Join(p.TopDataSourceDir(), "assets", "2da")
}
func legacyWikiOutputRootDir(p *project.Project) string {
return filepath.Join(p.TopDataSourceDir(), legacyWikiRootDirName)
}
func BuildPackage(p *project.Project, progress func(string)) (PackageResult, error) {
if progress == nil {
progress = func(string) {}
}
nativeResult, err := packagedBuildResult(p)
if err != nil {
return PackageResult{}, err
}
return packageBuiltTopData(p, nativeResult, progress)
}
func BuildAndPackage(p *project.Project, progress func(string)) (PackageResult, error) {
return BuildAndPackageWithOptions(p, BuildAndPackageOptions{}, progress)
}
type BuildAndPackageOptions struct {
Force bool
BuildWiki bool
}
func BuildAndPackageWithOptions(p *project.Project, opts BuildAndPackageOptions, progress func(string)) (PackageResult, error) {
if progress == nil {
progress = func(string) {}
}
if !opts.Force {
incrementalResult, ok, err := currentTopPackageResult(p)
if err != nil {
return PackageResult{}, err
}
if ok {
progress("Topdata outputs are current; skipping native build and top package refresh.")
return incrementalResult, nil
}
nativeResult, ok, err := currentCompiledBuildResult(p)
if err != nil {
return PackageResult{}, err
}
if ok {
progress(fmt.Sprintf("Compiled topdata outputs are current; refreshing %s and %s only.", p.TopDataPackageHAKName(), p.TopDataPackageTLKName()))
return packageBuiltTopData(p, nativeResult, progress)
}
}
return buildAndPackageFull(p, opts, progress)
}
func buildAndPackageFull(p *project.Project, opts BuildAndPackageOptions, progress func(string)) (PackageResult, error) {
nativeResult, err := BuildNativeWithOptions(p, NativeBuildOptions{BuildWiki: opts.BuildWiki, Force: opts.Force}, progress)
if err != nil {
return PackageResult{}, err
}
return packageBuiltTopData(p, nativeResult, progress)
}
func BuildPackageFromBuild(p *project.Project, nativeResult BuildResult, progress func(string)) (PackageResult, error) {
if progress == nil {
progress = func(string) {}
}
return packageBuiltTopData(p, nativeResult, progress)
}
func packageBuiltTopData(p *project.Project, nativeResult BuildResult, progress func(string)) (PackageResult, error) {
if progress == nil {
progress = func(string) {}
}
outputHAK := p.TopDataPackageHAKPath()
outputTLK := p.TopDataPackageTLKPath()
progress(fmt.Sprintf("Packaging compiled topdata resources into %s...", filepath.Base(outputHAK)))
resources, assetFiles, err := collectTopPackageResources(p, nativeResult.Output2DADir)
if err != nil {
return PackageResult{}, err
}
if err := writeHAK(outputHAK, resources); err != nil {
return PackageResult{}, err
}
progress(fmt.Sprintf("Preparing compiled TLK release output %s...", filepath.Base(outputTLK)))
sourceTLK := compiledTLKOutputPath(p)
if nativeResult.OutputTLKDir != "" {
sourceTLK, err = resolveCompiledTLKPath(nativeResult.OutputTLKDir)
if err != nil {
return PackageResult{}, err
}
}
if err := copyFile(outputTLK, sourceTLK); err != nil {
return PackageResult{}, err
}
return PackageResult{
Mode: nativeResult.Mode,
Output2DADir: nativeResult.Output2DADir,
OutputTLKDir: nativeResult.OutputTLKDir,
OutputHAKPath: outputHAK,
OutputTLKPath: outputTLK,
Files2DA: nativeResult.Files2DA,
FilesTLK: nativeResult.FilesTLK,
HAKResources: len(resources),
AssetFiles: assetFiles,
WikiOutputDir: nativeResult.WikiOutputDir,
WikiPages: nativeResult.WikiPages,
WikiStatus: nativeResult.WikiStatus,
}, nil
}
func collectTopPackageResources(p *project.Project, compiled2DADir string) ([]erf.Resource, int, error) {
resourceByKey := map[string]erf.Resource{}
resourceSourceByKey := map[string]string{}
entries, err := os.ReadDir(compiled2DADir)
if err != nil {
return nil, 0, fmt.Errorf("read compiled 2da output dir: %w", err)
}
for _, entry := range entries {
if entry.IsDir() || !strings.EqualFold(filepath.Ext(entry.Name()), ".2da") {
continue
}
path := filepath.Join(compiled2DADir, entry.Name())
resource, err := topPackageResourceFromPath(path)
if err != nil {
return nil, 0, err
}
key := topPackageResourceKey(resource)
resourceByKey[key] = resource
resourceSourceByKey[key] = path
}
assetFiles := 0
assetsDir := filepath.Join(p.TopDataSourceDir(), "assets")
skipDirs := map[string]struct{}{
filepath.Clean(compiled2DADir): {},
filepath.Clean(legacyCompiled2DAOutputDir(p)): {},
}
info, err := os.Stat(assetsDir)
if err == nil && info.IsDir() {
err = filepath.WalkDir(assetsDir, func(path string, d os.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if d.IsDir() {
if _, ok := skipDirs[filepath.Clean(path)]; ok {
return filepath.SkipDir
}
}
if d.IsDir() {
return nil
}
if strings.HasPrefix(filepath.Base(path), ".") {
return nil
}
resource, err := topPackageResourceFromPath(path)
if err != nil {
return err
}
key := topPackageResourceKey(resource)
if existing, ok := resourceSourceByKey[key]; ok {
return fmt.Errorf("topdata asset %s collides with %s for top package resource %s", path, existing, key)
}
resourceByKey[key] = resource
resourceSourceByKey[key] = path
assetFiles++
return nil
})
if err != nil {
return nil, 0, err
}
} else if err != nil && !os.IsNotExist(err) {
return nil, 0, fmt.Errorf("stat topdata assets dir: %w", err)
}
keys := make([]string, 0, len(resourceByKey))
for key := range resourceByKey {
keys = append(keys, key)
}
slices.Sort(keys)
resources := make([]erf.Resource, 0, len(keys))
for _, key := range keys {
resources = append(resources, resourceByKey[key])
}
return resources, assetFiles, nil
}
func shouldSkipTopDataSourceDir(path string, skipDirs map[string]struct{}) bool {
_, ok := skipDirs[filepath.Clean(path)]
return ok
}
func topDataSourceSkipDirs(p *project.Project) map[string]struct{} {
return map[string]struct{}{
filepath.Clean(filepath.Join(p.TopDataSourceDir(), "templates")): {},
filepath.Clean(legacyWikiOutputRootDir(p)): {},
filepath.Clean(legacyCompiled2DAOutputDir(p)): {},
filepath.Clean(wikiOutputRootDir(p)): {},
}
}
func newestTopDataSource(p *project.Project) (time.Time, string, error) {
newest := time.Time{}
newestPath := ""
skipDirs := topDataSourceSkipDirs(p)
sourceDir := p.TopDataSourceDir()
err := filepath.WalkDir(sourceDir, func(path string, d fs.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if d.IsDir() && shouldSkipTopDataSourceDir(path, skipDirs) {
return filepath.SkipDir
}
if path == sourceDir {
return nil
}
info, err := d.Info()
if err != nil {
return err
}
if newest.IsZero() || info.ModTime().After(newest) {
newest = info.ModTime()
newestPath = path
}
return nil
})
if err != nil {
return time.Time{}, "", fmt.Errorf("scan topdata sources: %w", err)
}
return newest, newestPath, nil
}
func newestTopDataInput(p *project.Project, now time.Time) (time.Time, string, error) {
newest, newestPath, err := newestTopDataSource(p)
if err != nil {
return time.Time{}, "", err
}
autogenTime, autogenPath, err := newestAutogenInput(p, now)
if err != nil {
return time.Time{}, "", err
}
if autogenTime.After(newest) {
return autogenTime, autogenPath, nil
}
return newest, newestPath, nil
}
func newestAutogenInput(p *project.Project, now time.Time) (time.Time, string, error) {
newest := time.Time{}
newestPath := ""
for _, consumer := range p.Config.Autogen.Consumers {
overrideRoot, err := resolveAutogenLocalOverrideRoot(p, consumer)
if err != nil {
return time.Time{}, "", err
}
if overrideRoot != "" {
candidateRoot := filepath.Join(overrideRoot, consumer.Root)
candidateTime, candidatePath, err := newestMatchingAutogenOverrideInput(candidateRoot, consumer.Include)
if err != nil {
return time.Time{}, "", err
}
if candidateTime.After(newest) {
newest = candidateTime
newestPath = candidatePath
}
continue
}
cachePath := p.AutogenCachePath(consumer.Manifest.CacheName)
candidateTime, candidatePath, err := newestReleasedAutogenManifestInput(p, consumer, now, cachePath)
if err != nil {
return time.Time{}, "", err
}
if candidateTime.After(newest) {
newest = candidateTime
newestPath = candidatePath
}
}
return newest, newestPath, nil
}
func newestMatchingAutogenOverrideInput(scanRoot string, include []string) (time.Time, string, error) {
info, err := os.Stat(scanRoot)
if err != nil {
if os.IsNotExist(err) {
return time.Time{}, "", nil
}
return time.Time{}, "", fmt.Errorf("stat autogen override root %s: %w", scanRoot, err)
}
if !info.IsDir() {
return time.Time{}, "", fmt.Errorf("autogen override root %s is not a directory", scanRoot)
}
newest := time.Time{}
newestPath := ""
err = filepath.WalkDir(scanRoot, func(path string, d fs.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if d.IsDir() {
return nil
}
rel, err := filepath.Rel(scanRoot, path)
if err != nil {
return err
}
if !matchesAutogenInclude(filepath.ToSlash(rel), include) {
return nil
}
info, err := d.Info()
if err != nil {
return err
}
if info.ModTime().After(newest) {
newest = info.ModTime()
newestPath = path
}
return nil
})
if err != nil {
return time.Time{}, "", fmt.Errorf("scan autogen override %s: %w", scanRoot, err)
}
return newest, newestPath, nil
}
func topPackageResourceFromPath(path string) (erf.Resource, error) {
extension := strings.TrimPrefix(strings.ToLower(filepath.Ext(path)), ".")
resourceType, ok := erf.HAKResourceTypeForExtension(extension)
if !ok {
return erf.Resource{}, fmt.Errorf("unsupported top package HAK resource extension %q", filepath.Ext(path))
}
info, err := os.Stat(path)
if err != nil {
return erf.Resource{}, fmt.Errorf("stat %s: %w", path, err)
}
return erf.Resource{
Name: strings.ToLower(strings.TrimSuffix(filepath.Base(path), filepath.Ext(path))),
Type: resourceType,
SourcePath: path,
Size: info.Size(),
}, nil
}
func topPackageResourceKey(resource erf.Resource) string {
extension, ok := erf.ExtensionForResourceType(resource.Type)
if !ok {
return strings.ToLower(resource.Name)
}
return strings.ToLower(resource.Name) + "." + strings.TrimPrefix(strings.ToLower(extension), ".")
}
func writeHAK(path string, resources []erf.Resource) error {
output, err := os.Create(path)
if err != nil {
return fmt.Errorf("create top package hak: %w", err)
}
defer output.Close()
if err := erf.Write(output, erf.New("HAK ", resources)); err != nil {
return fmt.Errorf("write top package hak: %w", err)
}
return nil
}
func resolveCompiledTLKPath(outputDir string) (string, error) {
entries, err := os.ReadDir(outputDir)
if err != nil {
return "", fmt.Errorf("read compiled tlk output dir: %w", err)
}
var tlkFiles []string
for _, entry := range entries {
if entry.IsDir() || !strings.EqualFold(filepath.Ext(entry.Name()), ".tlk") {
continue
}
tlkFiles = append(tlkFiles, filepath.Join(outputDir, entry.Name()))
}
if len(tlkFiles) != 1 {
return "", fmt.Errorf("expected exactly 1 compiled tlk output, found %d", len(tlkFiles))
}
return tlkFiles[0], nil
}
func copyFile(dst, src string) error {
if samePath(dst, src) {
return nil
}
input, err := os.Open(src)
if err != nil {
return fmt.Errorf("open %s: %w", src, err)
}
defer input.Close()
output, err := os.Create(dst)
if err != nil {
return fmt.Errorf("create %s: %w", dst, err)
}
defer output.Close()
if _, err := io.Copy(output, input); err != nil {
return fmt.Errorf("copy %s to %s: %w", src, dst, err)
}
return nil
}
func samePath(left, right string) bool {
leftAbs, leftErr := filepath.Abs(left)
rightAbs, rightErr := filepath.Abs(right)
if leftErr != nil || rightErr != nil {
return left == right
}
return leftAbs == rightAbs
}
func packagedBuildResult(p *project.Project) (BuildResult, error) {
if !p.HasTopData() {
return BuildResult{}, errors.New("topdata is not configured for this project")
}
output2DA := compiled2DAOutputDir(p)
outputTLK := compiledTLKOutputDir(p)
if _, err := os.Stat(output2DA); err != nil {
return BuildResult{}, fmt.Errorf("topdata build output missing: run build-topdata first")
}
if _, err := os.Stat(compiledTLKOutputPath(p)); err != nil {
return BuildResult{}, fmt.Errorf("topdata tlk output missing: run build-topdata first")
}
files2DA, oldest2DA, err := countCompiledFiles(output2DA, ".2da")
if err != nil {
return BuildResult{}, err
}
filesTLK := 1
tlkInfo, err := os.Stat(compiledTLKOutputPath(p))
if err != nil {
return BuildResult{}, fmt.Errorf("stat compiled tlk output %s: %w", compiledTLKOutputPath(p), err)
}
oldestTLK := tlkInfo.ModTime()
if files2DA == 0 {
return BuildResult{}, fmt.Errorf("topdata build output missing: run build-topdata first")
}
if err := validateCompiled2DAOutputCatalog(p, output2DA); err != nil {
return BuildResult{}, err
}
newestSource, newestPath, err := newestTopDataInput(p, time.Now())
if err != nil {
return BuildResult{}, err
}
oldestBuild := oldest2DA
if oldestBuild.IsZero() || (!oldestTLK.IsZero() && oldestTLK.Before(oldestBuild)) {
oldestBuild = oldestTLK
}
if !newestSource.IsZero() && !oldestBuild.IsZero() && newestSource.After(oldestBuild) {
return BuildResult{}, fmt.Errorf("topdata build output is older than source file %s; run build-topdata first", newestPath)
}
return BuildResult{
Mode: "native",
Output2DADir: output2DA,
OutputTLKDir: outputTLK,
Files2DA: files2DA,
FilesTLK: filesTLK,
}, nil
}
func validateCompiled2DAOutputCatalog(p *project.Project, output2DA string) error {
expected, err := expectedCompiled2DAOutputs(p)
if err != nil {
return err
}
actual, err := compiled2DAOutputNames(output2DA)
if err != nil {
return err
}
missing := make([]string, 0)
for outputName := range expected {
if _, ok := actual[outputName]; !ok {
missing = append(missing, outputName)
}
}
stale := make([]string, 0)
for outputName := range actual {
if _, ok := expected[outputName]; !ok {
stale = append(stale, outputName)
}
}
slices.Sort(missing)
slices.Sort(stale)
if len(missing) > 0 {
return fmt.Errorf("topdata build output is missing compiled 2da %s; run build-topdata first", strings.Join(missing, ", "))
}
if len(stale) > 0 {
return fmt.Errorf("topdata build output contains stale compiled 2da %s; run build-topdata first", strings.Join(stale, ", "))
}
return nil
}
func expectedCompiled2DAOutputs(p *project.Project) (map[string]struct{}, error) {
dataDir := filepath.Join(p.TopDataSourceDir(), "data")
datasets, err := discoverNativeDatasets(dataDir)
if err != nil {
return nil, err
}
registryDatasets, err := collectGeneratedRegistryDatasets(dataDir)
if err != nil {
return nil, err
}
outputs := make(map[string]struct{}, len(datasets)+len(registryDatasets))
for _, dataset := range datasets {
outputs[dataset.OutputName] = struct{}{}
}
for _, dataset := range registryDatasets {
outputs[dataset.Dataset.OutputName] = struct{}{}
}
return outputs, nil
}
func compiled2DAOutputNames(output2DA string) (map[string]struct{}, error) {
entries, err := os.ReadDir(output2DA)
if err != nil {
return nil, fmt.Errorf("read compiled 2da output dir %s: %w", output2DA, err)
}
outputs := make(map[string]struct{})
for _, entry := range entries {
if entry.IsDir() || !strings.EqualFold(filepath.Ext(entry.Name()), ".2da") {
continue
}
outputs[entry.Name()] = struct{}{}
}
return outputs, nil
}
func currentCompiledBuildResult(p *project.Project) (BuildResult, bool, error) {
nativeResult, err := packagedBuildResult(p)
if err == nil {
return nativeResult, true, nil
}
if strings.Contains(err.Error(), "run build-topdata first") {
return BuildResult{}, false, nil
}
return BuildResult{}, false, err
}
func currentTopPackageResult(p *project.Project) (PackageResult, bool, error) {
nativeResult, ok, err := currentCompiledBuildResult(p)
if err != nil || !ok {
return PackageResult{}, ok, err
}
outputHAK := p.TopDataPackageHAKPath()
outputTLK := p.TopDataPackageTLKPath()
hakInfo, err := os.Stat(outputHAK)
if err != nil {
if os.IsNotExist(err) {
return PackageResult{}, false, nil
}
return PackageResult{}, false, fmt.Errorf("stat top package hak %s: %w", outputHAK, err)
}
tlkInfo, err := os.Stat(outputTLK)
if err != nil {
if os.IsNotExist(err) {
return PackageResult{}, false, nil
}
return PackageResult{}, false, fmt.Errorf("stat top package tlk %s: %w", outputTLK, err)
}
newestSource, _, err := newestTopDataInput(p, time.Now())
if err != nil {
return PackageResult{}, false, err
}
newestCompiled, err := newestFileModTime(nativeResult.Output2DADir, ".2da")
if err != nil {
return PackageResult{}, false, err
}
compiledTLKInfo, err := os.Stat(compiledTLKOutputPath(p))
if err != nil {
return PackageResult{}, false, fmt.Errorf("stat compiled tlk output %s: %w", compiledTLKOutputPath(p), err)
}
if compiledTLKInfo.ModTime().After(newestCompiled) {
newestCompiled = compiledTLKInfo.ModTime()
}
newestInput := newestSource
if newestCompiled.After(newestInput) {
newestInput = newestCompiled
}
oldestPackage := hakInfo.ModTime()
if tlkInfo.ModTime().Before(oldestPackage) {
oldestPackage = tlkInfo.ModTime()
}
if !newestInput.IsZero() && newestInput.After(oldestPackage) {
return PackageResult{}, false, nil
}
resources, assetFiles, err := collectTopPackageResources(p, nativeResult.Output2DADir)
if err != nil {
return PackageResult{}, false, err
}
return PackageResult{
Mode: "incremental-noop",
Output2DADir: nativeResult.Output2DADir,
OutputTLKDir: nativeResult.OutputTLKDir,
OutputHAKPath: outputHAK,
OutputTLKPath: outputTLK,
Files2DA: nativeResult.Files2DA,
FilesTLK: nativeResult.FilesTLK,
HAKResources: len(resources),
AssetFiles: assetFiles,
WikiOutputDir: nativeResult.WikiOutputDir,
WikiPages: nativeResult.WikiPages,
WikiStatus: nativeResult.WikiStatus,
}, true, nil
}
func newestFileModTime(dir, extension string) (time.Time, error) {
entries, err := os.ReadDir(dir)
if err != nil {
return time.Time{}, fmt.Errorf("read output dir %s: %w", dir, err)
}
newest := time.Time{}
for _, entry := range entries {
if entry.IsDir() || !strings.EqualFold(filepath.Ext(entry.Name()), extension) {
continue
}
info, err := entry.Info()
if err != nil {
return time.Time{}, fmt.Errorf("stat output %s: %w", filepath.Join(dir, entry.Name()), err)
}
if newest.IsZero() || info.ModTime().After(newest) {
newest = info.ModTime()
}
}
return newest, nil
}
func countCompiledFiles(dir, extension string) (int, time.Time, error) {
entries, err := os.ReadDir(dir)
if err != nil {
return 0, time.Time{}, fmt.Errorf("read compiled output dir %s: %w", dir, err)
}
count := 0
oldest := time.Time{}
for _, entry := range entries {
if entry.IsDir() || !strings.EqualFold(filepath.Ext(entry.Name()), extension) {
continue
}
info, err := entry.Info()
if err != nil {
return 0, time.Time{}, fmt.Errorf("stat compiled output %s: %w", filepath.Join(dir, entry.Name()), err)
}
count++
if oldest.IsZero() || info.ModTime().Before(oldest) {
oldest = info.ModTime()
}
}
return count, oldest, nil
}
internal/topdata/topdata.go
-3009
View File
File diff suppressed because it is too large Load Diff
internal/topdata/topdata_test.go
-15115
View File
File diff suppressed because it is too large Load Diff
internal/topdata/vfx_persistent_migrate.go
-198
View File
@@ -1,198 +0,0 @@
package topdata
import (
"os"
"path/filepath"
"strings"
)
func importLegacyVFXPersistent(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
if strings.TrimSpace(referenceBuilderDir) == "" {
return 0, nil
}
legacyDir := filepath.Join(referenceBuilderDir, "data", "vfx_persistent")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "vfx_persistent")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
targetModulePath := filepath.Join(targetModulesDir, "freeformaoes.json")
if fileExists(targetBasePath) && fileExists(targetLockPath) && fileExists(targetModulePath) {
baseObj, baseErr := loadJSONObject(targetBasePath)
lockObj, lockErr := loadJSONObject(targetLockPath)
if baseErr == nil && lockErr == nil && vfxPersistentImportComplete(baseObj, lockObj) {
legacyModulePaths, err := collectJSONRelativePaths(filepath.Join(legacyDir, "modules"))
if err == nil {
targetModulePaths, err := collectJSONRelativePaths(targetModulesDir)
if err == nil && equalStringSlices(legacyModulePaths, targetModulePaths) {
return 0, nil
}
}
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
lockObj, err := synthesizeVFXPersistentKeys(baseObj)
if err != nil {
return 0, err
}
legacyLockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
for key, value := range legacyLockObj {
lockObj[key] = value
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
updated := 0
for _, write := range []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
} {
changed, err := writeJSONObjectIfChanged(write.path, write.obj)
if err != nil {
return 0, err
}
if changed {
updated++
}
}
moduleRelPaths, err := collectJSONRelativePaths(filepath.Join(legacyDir, "modules"))
if err != nil {
return 0, err
}
for _, relPath := range moduleRelPaths {
moduleObj, err := loadJSONObject(filepath.Join(legacyDir, "modules", filepath.FromSlash(relPath)))
if err != nil {
return 0, err
}
targetPath := filepath.Join(targetModulesDir, filepath.FromSlash(relPath))
if err := os.MkdirAll(filepath.Dir(targetPath), 0o755); err != nil {
return 0, err
}
changed, err := writeJSONObjectIfChanged(targetPath, moduleObj)
if err != nil {
return 0, err
}
if changed {
updated++
}
}
targetModulePaths, err := collectJSONRelativePaths(targetModulesDir)
if err != nil {
return 0, err
}
legacyModules := make(map[string]struct{}, len(moduleRelPaths))
for _, relPath := range moduleRelPaths {
legacyModules[relPath] = struct{}{}
}
for _, relPath := range targetModulePaths {
if _, ok := legacyModules[relPath]; ok {
continue
}
if err := os.Remove(filepath.Join(targetModulesDir, filepath.FromSlash(relPath))); err != nil && !os.IsNotExist(err) {
return 0, err
}
updated++
}
return updated, nil
}
func vfxPersistentImportComplete(baseObj, lockObj map[string]any) bool {
if !vfxPersistentRowsHaveKeys(baseObj) {
return false
}
expected := map[string]int{
"vfx_persistent:blankradius5": 47,
"vfx_persistent:blankradius10": 48,
"vfx_persistent:blankradius15": 49,
"vfx_persistent:blankradius20": 50,
"vfx_persistent:blankradius25": 51,
"vfx_persistent:blankradius30": 52,
}
for key, want := range expected {
raw, ok := lockObj[key]
if !ok {
return false
}
got, err := asInt(raw)
if err != nil || got != want {
return false
}
}
return true
}
func synthesizeVFXPersistentKeys(baseObj map[string]any) (map[string]any, error) {
rawRows, ok := baseObj["rows"].([]any)
if !ok {
return map[string]any{}, nil
}
lockObj := map[string]any{}
for _, raw := range rawRows {
row, ok := raw.(map[string]any)
if !ok {
continue
}
label, _ := row["LABEL"].(string)
if label == "" {
continue
}
key := "vfx_persistent:" + strings.ToLower(label)
row["key"] = key
if rawID, ok := row["id"]; ok {
lockObj[key] = rawID
}
}
return lockObj, nil
}
func vfxPersistentRowsHaveKeys(baseObj map[string]any) bool {
rawRows, ok := baseObj["rows"].([]any)
if !ok {
return false
}
for _, raw := range rawRows {
row, ok := raw.(map[string]any)
if !ok {
return false
}
key, _ := row["key"].(string)
if key == "" {
return false
}
}
return true
}
func equalStringSlices(left, right []string) bool {
if len(left) != len(right) {
return false
}
for i := range left {
if left[i] != right[i] {
return false
}
}
return true
}
internal/topdata/visualeffects_migrate.go
-6
View File
@@ -1,6 +0,0 @@
package topdata
func importLegacyVisualeffects(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "visualeffects", "visualeffects.2da", nil)
}
internal/topdata/wiki_data_providers.go
-518
View File
@@ -1,518 +0,0 @@
package topdata
import (
"fmt"
"os"
"strconv"
"strings"
"gopkg.in/yaml.v3"
)
type wikiDataProvidersDocument struct {
Providers map[string]wikiDataProviderDefinition `json:"providers" yaml:"providers"`
}
type wikiDataProviderDefinition struct {
Source string `json:"source" yaml:"source"`
Levels string `json:"levels" yaml:"levels"`
ClassFeats wikiDataProviderClassFeatConfig `json:"class_feats" yaml:"class_feats"`
Attacks wikiDataProviderAttacksConfig `json:"attacks" yaml:"attacks"`
Facts wikiDataProviderFactsConfig `json:"facts" yaml:"facts"`
Provider string `json:"provider" yaml:"provider"`
Columns int `json:"columns" yaml:"columns"`
ItemsPerColumn int `json:"items_per_column" yaml:"items_per_column"`
ForceColumns bool `json:"force_columns" yaml:"force_columns"`
}
type wikiDataProviderClassFeatConfig struct {
Fields map[string]wikiDataProviderProjectionField `json:"fields" yaml:"fields"`
}
type wikiDataProviderProjectionField struct {
When string `json:"when" yaml:"when"`
}
type wikiDataProviderFactsConfig struct {
Rows []wikiDataProviderFactRow `json:"rows" yaml:"rows"`
}
type wikiDataProviderFactRow struct {
Key string `json:"key" yaml:"key"`
Label string `json:"label" yaml:"label"`
Value string `json:"value" yaml:"value"`
When string `json:"when" yaml:"when"`
}
type wikiDataProviderAttacksConfig struct {
Field string `json:"field" yaml:"field"`
CountField string `json:"count_field" yaml:"count_field"`
Mode string `json:"mode" yaml:"mode"`
Step int `json:"step" yaml:"step"`
Attacks int `json:"attacks" yaml:"attacks"`
Thresholds []wikiDataProviderAttackThreshold `json:"thresholds" yaml:"thresholds"`
Overrides []wikiDataProviderAttackOverrideRule `json:"overrides" yaml:"overrides"`
}
type wikiDataProviderAttackThreshold struct {
MinBAB int `json:"min_bab" yaml:"min_bab"`
Attacks int `json:"attacks" yaml:"attacks"`
}
type wikiDataProviderAttackOverrideRule struct {
Classes []string `json:"classes" yaml:"classes"`
Levels any `json:"levels" yaml:"levels"`
Attacks int `json:"attacks" yaml:"attacks"`
Add int `json:"add" yaml:"add"`
}
func loadWikiDataProviders(path string) (map[string]wikiDataProviderDefinition, error) {
raw, err := os.ReadFile(path)
if err != nil {
if os.IsNotExist(err) {
return map[string]wikiDataProviderDefinition{}, nil
}
return nil, fmt.Errorf("read wiki data providers %s: %w", path, err)
}
var doc wikiDataProvidersDocument
if err := yaml.Unmarshal(raw, &doc); err != nil {
return nil, fmt.Errorf("parse wiki data providers %s: %w", path, err)
}
for name, def := range doc.Providers {
if strings.TrimSpace(name) == "" {
return nil, fmt.Errorf("wiki data providers %s: provider name must not be empty", path)
}
if strings.TrimSpace(def.Source) == "" {
return nil, fmt.Errorf("wiki data providers %s: provider %s missing source", path, name)
}
source := strings.TrimSpace(def.Source)
switch source {
case "class_progression":
for field, projection := range def.ClassFeats.Fields {
if strings.TrimSpace(field) == "" {
return nil, fmt.Errorf("wiki data providers %s: provider %s has class feat projection without field name", path, name)
}
if strings.TrimSpace(projection.When) == "" {
return nil, fmt.Errorf("wiki data providers %s: provider %s class feat projection %s missing when", path, name, field)
}
}
if err := validateWikiDataProviderAttacks(path, name, def.Attacks); err != nil {
return nil, err
}
case "columns":
if len(def.ClassFeats.Fields) > 0 {
return nil, fmt.Errorf("wiki data providers %s: provider %s class feat projections require source class_progression", path, name)
}
if strings.TrimSpace(def.Attacks.Mode) != "" {
return nil, fmt.Errorf("wiki data providers %s: provider %s attacks config requires source class_progression", path, name)
}
if strings.TrimSpace(def.Provider) == "" {
return nil, fmt.Errorf("wiki data providers %s: provider %s columns provider missing provider", path, name)
}
if strings.TrimSpace(def.Provider) == name {
return nil, fmt.Errorf("wiki data providers %s: provider %s columns provider must not reference itself", path, name)
}
if def.Columns <= 0 {
return nil, fmt.Errorf("wiki data providers %s: provider %s columns must be a positive integer", path, name)
}
if def.ItemsPerColumn <= 0 {
return nil, fmt.Errorf("wiki data providers %s: provider %s items_per_column must be a positive integer", path, name)
}
case "facts":
if len(def.Facts.Rows) == 0 {
return nil, fmt.Errorf("wiki data providers %s: provider %s facts source requires rows", path, name)
}
for index, row := range def.Facts.Rows {
if strings.TrimSpace(row.Label) == "" {
return nil, fmt.Errorf("wiki data providers %s: provider %s facts row %d missing label", path, name, index+1)
}
if strings.TrimSpace(row.Value) == "" {
return nil, fmt.Errorf("wiki data providers %s: provider %s facts row %d missing value", path, name, index+1)
}
}
case "class_feats", "class_skills", "class_summary", "class_spellbook", "race_feats", "race_name_forms":
if len(def.ClassFeats.Fields) > 0 {
return nil, fmt.Errorf("wiki data providers %s: provider %s class feat projections require source class_progression", path, name)
}
if strings.TrimSpace(def.Attacks.Mode) != "" {
return nil, fmt.Errorf("wiki data providers %s: provider %s attacks config requires source class_progression", path, name)
}
default:
return nil, fmt.Errorf("wiki data providers %s: provider %s has unknown source %q", path, name, def.Source)
}
}
for name, def := range doc.Providers {
if strings.TrimSpace(def.Source) != "columns" {
continue
}
if _, ok := doc.Providers[strings.TrimSpace(def.Provider)]; !ok {
return nil, fmt.Errorf("wiki data providers %s: provider %s columns provider references missing provider %q", path, name, def.Provider)
}
}
if doc.Providers == nil {
return map[string]wikiDataProviderDefinition{}, nil
}
return doc.Providers, nil
}
func (ctx *wikiContext) wikiDataProviderRows(name string, page wikiTemplatePage) ([]map[string]any, error) {
def, ok := ctx.wikiDataProviders[name]
if !ok {
return nil, fmt.Errorf("missing wiki data provider %q in %s", name, ctx.wikiDataPath)
}
switch strings.TrimSpace(def.Source) {
case "columns":
sourceRows, err := ctx.wikiDataProviderRows(strings.TrimSpace(def.Provider), page)
if err != nil {
return nil, err
}
return wikiColumnProviderRows(sourceRows, def.Columns, def.ItemsPerColumn, def.ForceColumns), nil
case "class_progression":
if page.Category != "classes" {
return nil, nil
}
return ctx.classProgressionRows(def.Levels, def.ClassFeats.Fields, def.Attacks, page.Key, page.Row)
case "facts":
return ctx.wikiFactRows(name, def.Facts, page)
case "class_feats":
if page.Category != "classes" {
return nil, nil
}
return ctx.classFeatRows(page.Row), nil
case "class_skills":
if page.Category != "classes" {
return nil, nil
}
return ctx.classSkillRows(page.Row), nil
case "class_summary":
if page.Category != "classes" {
return nil, nil
}
return ctx.classSummaryRows(page.Row), nil
case "class_spellbook":
if page.Category != "classes" {
return nil, nil
}
return ctx.classSpellbookRows(page.Key, page.Row), nil
case "race_feats":
if page.Category != "racialtypes" {
return nil, nil
}
return ctx.raceFeatRows(page.Row), nil
case "race_name_forms":
if page.Category != "racialtypes" {
return nil, nil
}
return ctx.raceNameFormRows(page.Row), nil
default:
return nil, fmt.Errorf("unknown wiki data provider source %q", def.Source)
}
}
func (ctx *wikiContext) wikiFactRows(providerName string, cfg wikiDataProviderFactsConfig, page wikiTemplatePage) ([]map[string]any, error) {
out := []map[string]any{}
renderCtx := wikiTableRenderContext{Page: page, TableName: providerName, Path: ctx.wikiDataPath}
for index, spec := range cfg.Rows {
if strings.TrimSpace(spec.When) != "" {
value, err := ctx.evalWikiDataProviderTemplate(spec.When, page.Row, renderCtx)
if err != nil {
return nil, fmt.Errorf("facts provider %s row %d condition %q: %w", providerName, index+1, spec.When, err)
}
if !truthyWikiTableValue(value) {
continue
}
}
value, err := ctx.evalWikiDataProviderTemplate(spec.Value, page.Row, renderCtx)
if err != nil {
return nil, fmt.Errorf("facts provider %s row %d value %q: %w", providerName, index+1, spec.Value, err)
}
text := stringifyWikiTableValue(value)
if strings.TrimSpace(text) == "" || strings.TrimSpace(text) == nullValue {
continue
}
key := strings.TrimSpace(spec.Key)
if key == "" {
key = wikiFactKeyFromLabel(spec.Label)
}
out = append(out, map[string]any{
"Key": key,
"Label": spec.Label,
"Value": text,
})
}
return out, nil
}
func (ctx *wikiContext) evalWikiDataProviderTemplate(template string, row map[string]any, renderCtx wikiTableRenderContext) (any, error) {
template = strings.TrimSpace(template)
if strings.HasPrefix(template, "{{") && strings.HasSuffix(template, "}}") {
expr := strings.TrimSpace(strings.TrimSuffix(strings.TrimPrefix(template, "{{"), "}}"))
parser := wikiExprParser{tokens: tokenizeWikiExpr(expr), row: row, ctx: ctx, renderCtx: renderCtx, allowMissingFields: true}
value, err := parser.parseComparison()
if err != nil {
return nil, err
}
if parser.peek().kind != wikiExprEOF {
return nil, fmt.Errorf("unexpected token %q", parser.peek().text)
}
return value, nil
}
if strings.Contains(template, "{{") {
return nil, fmt.Errorf("mixed literal/expression templates are not supported")
}
return template, nil
}
func wikiFactKeyFromLabel(label string) string {
key := strings.ToLower(strings.TrimSpace(label))
key = strings.Map(func(r rune) rune {
switch {
case r >= 'a' && r <= 'z':
return r
case r >= '0' && r <= '9':
return r
case r == ' ' || r == '-' || r == '_':
return '_'
default:
return -1
}
}, key)
for strings.Contains(key, "__") {
key = strings.ReplaceAll(key, "__", "_")
}
return strings.Trim(key, "_")
}
func validateWikiDataProviderAttacks(path, provider string, cfg wikiDataProviderAttacksConfig) error {
if strings.TrimSpace(cfg.Mode) == "" {
return nil
}
switch strings.TrimSpace(cfg.Mode) {
case "fixed":
if cfg.Attacks < 1 {
return fmt.Errorf("wiki data providers %s: provider %s attacks fixed mode requires positive attacks", path, provider)
}
case "bab_thresholds":
if len(cfg.Thresholds) == 0 {
return fmt.Errorf("wiki data providers %s: provider %s attacks bab_thresholds mode requires thresholds", path, provider)
}
for i, threshold := range cfg.Thresholds {
if threshold.MinBAB < 0 {
return fmt.Errorf("wiki data providers %s: provider %s attacks threshold %d min_bab must be non-negative", path, provider, i+1)
}
if threshold.Attacks < 1 {
return fmt.Errorf("wiki data providers %s: provider %s attacks threshold %d attacks must be positive", path, provider, i+1)
}
}
default:
return fmt.Errorf("wiki data providers %s: provider %s attacks has unknown mode %q", path, provider, cfg.Mode)
}
for i, override := range cfg.Overrides {
if len(override.Classes) == 0 {
return fmt.Errorf("wiki data providers %s: provider %s attacks override %d missing classes", path, provider, i+1)
}
for _, class := range override.Classes {
if strings.TrimSpace(class) == "" {
return fmt.Errorf("wiki data providers %s: provider %s attacks override %d has empty class", path, provider, i+1)
}
}
if _, err := parseWikiAttackLevels(override.Levels); err != nil {
return fmt.Errorf("wiki data providers %s: provider %s attacks override %d bad levels: %w", path, provider, i+1, err)
}
if override.Attacks != 0 && override.Add != 0 {
return fmt.Errorf("wiki data providers %s: provider %s attacks override %d must not set both attacks and add", path, provider, i+1)
}
if override.Attacks == 0 && override.Add == 0 {
return fmt.Errorf("wiki data providers %s: provider %s attacks override %d must set attacks or add", path, provider, i+1)
}
if override.Attacks < 0 {
return fmt.Errorf("wiki data providers %s: provider %s attacks override %d attacks must be positive", path, provider, i+1)
}
}
return nil
}
func parseWikiAttackLevels(raw any) (map[int]struct{}, error) {
out := map[int]struct{}{}
addLevel := func(value any) error {
switch typed := value.(type) {
case int:
if typed < 1 {
return fmt.Errorf("level must be positive")
}
out[typed] = struct{}{}
return nil
case int64:
if typed < 1 {
return fmt.Errorf("level must be positive")
}
out[int(typed)] = struct{}{}
return nil
case float64:
level := int(typed)
if typed != float64(level) || level < 1 {
return fmt.Errorf("level must be a positive integer")
}
out[level] = struct{}{}
return nil
case string:
text := strings.TrimSpace(typed)
if text == "" {
return fmt.Errorf("level must not be empty")
}
if strings.Contains(text, "-") {
start, end, err := parseLevelRange(text)
if err != nil {
return err
}
for level := start; level <= end; level++ {
out[level] = struct{}{}
}
return nil
}
level, err := strconv.Atoi(text)
if err != nil || level < 1 {
return fmt.Errorf("level must be a positive integer")
}
out[level] = struct{}{}
return nil
default:
return fmt.Errorf("unsupported level value")
}
}
switch typed := raw.(type) {
case nil:
return nil, fmt.Errorf("levels is required")
case []any:
if len(typed) == 0 {
return nil, fmt.Errorf("levels must not be empty")
}
for _, value := range typed {
if err := addLevel(value); err != nil {
return nil, err
}
}
default:
if err := addLevel(raw); err != nil {
return nil, err
}
}
return out, nil
}
func wikiColumnProviderRows(sourceRows []map[string]any, configuredColumns, itemsPerColumn int, forceColumns bool) []map[string]any {
if configuredColumns <= 0 || itemsPerColumn <= 0 {
return nil
}
needed := 0
if len(sourceRows) > 0 {
needed = (len(sourceRows) + itemsPerColumn - 1) / itemsPerColumn
}
total := needed
if forceColumns && configuredColumns > total {
total = configuredColumns
}
if total == 0 {
return nil
}
rows := make([]map[string]any, 0, total)
for i := 0; i < total; i++ {
start := i * itemsPerColumn
end := start + itemsPerColumn
if start > len(sourceRows) {
start = len(sourceRows)
}
if end > len(sourceRows) {
end = len(sourceRows)
}
items := make([]map[string]any, 0, end-start)
for _, source := range sourceRows[start:end] {
items = append(items, cloneRowMap(source))
}
rows = append(rows, map[string]any{
"Index": i + 1,
"Count": len(items),
"Items": items,
})
}
return rows
}
func (ctx *wikiContext) classSkillRows(classRow map[string]any) []map[string]any {
table := ctx.tableForValue(fieldValue(classRow, "SkillsTable"), ctx.classSkillTables)
if table == nil {
return nil
}
rows := []map[string]any{}
for _, source := range table.Rows {
if stringValue(source, "ClassSkill") != "1" {
continue
}
key := resolveReferenceKey(fieldValue(source, "SkillIndex"), ctx.skillIDToKey)
if key == "" {
key = resolveReferenceKey(fieldValue(source, "SkillIndex"), nil)
}
if key == "" || !ctx.canReferenceWikiKey(key) {
continue
}
row := cloneRowMap(source)
row["SkillKey"] = key
row["SkillName"] = ctx.resolveSkillName(key)
if skillRow := ctx.skillRows[key]; skillRow != nil {
row["SkillAbility"] = stringValue(skillRow, "KeyAbility")
}
rows = append(rows, row)
}
return rows
}
func (ctx *wikiContext) raceFeatRows(raceRow map[string]any) []map[string]any {
values := []any{}
switch typed := fieldValue(raceRow, "FeatsTable").(type) {
case []any:
values = append(values, typed...)
case []string:
for _, value := range typed {
values = append(values, value)
}
default:
if table := ctx.tableForValue(typed, ctx.raceFeatTables); table != nil {
for _, row := range table.Rows {
values = append(values, fieldValue(row, "FeatIndex"))
}
}
}
rows := []map[string]any{}
for _, value := range values {
key := resolveReferenceKey(value, ctx.featIDToKey)
if key == "" {
key = resolveReferenceKey(value, nil)
}
if key == "" || !ctx.canReferenceWikiKey(key) {
continue
}
rows = append(rows, map[string]any{
"FeatKey": key,
"FeatName": ctx.resolveFeatName(key),
})
}
return rows
}
func (ctx *wikiContext) raceNameFormRows(raceRow map[string]any) []map[string]any {
candidates := []struct {
label string
field string
}{
{"Plural", "NamePlural"},
{"Converted Name", "ConverName"},
{"Lower Name", "ConverNameLower"},
}
rows := []map[string]any{}
for _, candidate := range candidates {
if value := ctx.resolveTextValue(fieldValue(raceRow, candidate.field), nil); value != "" {
rows = append(rows, map[string]any{"Label": candidate.label, "Value": value})
}
}
return rows
}
internal/topdata/wiki_deploy.go
-1861
View File
File diff suppressed because it is too large Load Diff
internal/topdata/wiki_deploy_test.go
-2387
View File
File diff suppressed because it is too large Load Diff
internal/topdata/wiki_formatters.go
-286
View File
@@ -1,286 +0,0 @@
package topdata
import (
"fmt"
"html"
"strings"
)
func (ctx *wikiContext) renderWikiFormatter(spec string, page wikiTemplatePage) (string, error) {
name := strings.Fields(spec)
if len(name) == 0 {
return "", fmt.Errorf("render wiki template for %s: empty formatter", page.PageID)
}
switch name[0] {
case "Description":
return renderWikiTemplatePlainHTML(ctx.resolveRowDescription(page.Category, page.Row)), nil
case "FactsTable":
return ctx.renderFactsHTML(page.Category, page.Key, page.Row), nil
case "Prerequisites":
return renderWikiFactTableHTML([]wikiHTMLFact{{Label: "Prerequisites", Value: ctx.renderFeatPrerequisites(page.Row)}}), nil
case "SkillList":
if page.Category != "classes" {
return "", nil
}
return ctx.renderClassSkillListHTML(page.Row), nil
case "ClassFeatureTable":
if page.Category != "classes" {
return "", nil
}
return ctx.renderClassFeatureTableHTML(page.Row), nil
case "FeatProgression":
if page.Category != "classes" {
return "", nil
}
return ctx.renderClassFeatProgressionHTML(page.Row), nil
case "SpellTable", "SavesProgression", "BABProgression":
return "", nil
case "RaceFeatList":
return renderWikiFactTableHTML([]wikiHTMLFact{{Label: "Racial Feats", Value: ctx.renderRaceFeatList(page.Row)}}), nil
case "StatusCallout":
return buildWikiStatusBlock(page.Status, page.Category), nil
case "RaceNameForms":
return ctx.renderRaceNameFormsHTML(page.Row), nil
case "UserBottomFallback":
return "", nil
default:
return "", fmt.Errorf("render wiki template for %s: unknown formatter %q", page.PageID, name[0])
}
}
type wikiHTMLFact struct {
Label string
Value string
}
func (ctx *wikiContext) renderFactsHTML(category, key string, row map[string]any) string {
facts := []wikiHTMLFact{}
add := func(label, value string) {
if strings.TrimSpace(value) != "" {
facts = append(facts, wikiHTMLFact{Label: label, Value: value})
}
}
switch category {
case "feat":
for _, fact := range ctx.renderFeatFactsHTML(row) {
add(fact.Label, fact.Value)
}
case "skills":
add("Key Ability", stringValue(row, "KeyAbility"))
add("Untrained", yesNoValue(stringValue(row, "Untrained")))
add("Armor Check Penalty", yesNoValue(stringValue(row, "ArmorCheckPenalty")))
case "spells":
add("Constant", stringValue(row, "Constant"))
case "baseitems":
add("Item Class", stringValue(row, "ItemClass"))
add("Weapon Type", stringValue(row, "WeaponType"))
add("Required Feats", ctx.renderReferenceList(ctx.wikiReqFeats(row)))
add("Base Item Stats", ctx.resolveTextValue(fieldValue(row, "BaseItemStatRef"), nil))
case "classes":
add("Hit Die", "d"+stringValue(row, "HitDie"))
add("Skill Points", stringValue(row, "SkillPointBase"))
add("Primary Ability", stringValue(row, "PrimaryAbil"))
case "racialtypes":
add("Ability Adjustments", formatAbilityAdjustments(row))
add("Favored Class", ctx.renderReference(fieldValue(row, "Favored"), ctx.classIDToKey, ctx.resolveClassName))
add("Favored Enemy", ctx.renderReference(fieldValue(row, "FavoredEnemyFeat"), ctx.featIDToKey, ctx.resolveFeatName))
add("Racial Feats", ctx.renderRaceFeatList(row))
}
_ = key
return renderWikiFactTableHTML(facts)
}
func (ctx *wikiContext) renderFeatFactsHTML(row map[string]any) []wikiHTMLFact {
return []wikiHTMLFact{
{Label: "Prerequisites", Value: ctx.renderFeatPrerequisites(row)},
{Label: "Minimum Attack Bonus", Value: stringValue(row, "MINATTACKBONUS")},
{Label: "Minimum Spell Level", Value: stringValue(row, "MINSPELLLVL")},
{Label: "Minimum Fortitude Save", Value: stringValue(row, "MinFortSave")},
}
}
func renderWikiFactTableHTML(facts []wikiHTMLFact) string {
rows := []string{}
for _, fact := range facts {
if strings.TrimSpace(fact.Value) == "" {
continue
}
rows = append(rows, `<tr><th scope="row">`+html.EscapeString(fact.Label)+`</th><td>`+renderMultilineInlineHTML(fact.Value)+`</td></tr>`)
}
if len(rows) == 0 {
return ""
}
return `<table class="wiki-facts"><tbody>` + strings.Join(rows, "\n") + `</tbody></table>`
}
func renderMultilineInlineHTML(text string) string {
lines := []string{}
for _, line := range strings.Split(strings.ReplaceAll(strings.TrimSpace(text), "\r\n", "\n"), "\n") {
line = strings.TrimSpace(line)
if line != "" {
lines = append(lines, renderInlineHTML(line))
}
}
return strings.Join(lines, "<br>")
}
func renderWikiListHTML(class string, items []string) string {
rendered := []string{}
for _, item := range items {
if strings.TrimSpace(item) != "" {
rendered = append(rendered, "<li>"+renderInlineHTML(item)+"</li>")
}
}
if len(rendered) == 0 {
return ""
}
classAttr := ""
if class != "" {
classAttr = ` class="` + html.EscapeString(class) + `"`
}
return "<ul" + classAttr + ">" + strings.Join(rendered, "\n") + "</ul>"
}
func (ctx *wikiContext) renderClassSkillList(row map[string]any) string {
table := ctx.tableForValue(fieldValue(row, "SkillsTable"), ctx.classSkillTables)
if table == nil {
return ""
}
skills := []string{}
for _, skillRow := range table.Rows {
if stringValue(skillRow, "ClassSkill") != "1" {
continue
}
name := ctx.renderReference(fieldValue(skillRow, "SkillIndex"), ctx.skillIDToKey, ctx.resolveSkillName)
if name != "" {
skills = append(skills, name)
}
}
if len(skills) == 0 {
return ""
}
return "==== Class Skills ====\n\n * " + strings.Join(skills, ", ")
}
func (ctx *wikiContext) renderClassSkillListHTML(row map[string]any) string {
table := ctx.tableForValue(fieldValue(row, "SkillsTable"), ctx.classSkillTables)
if table == nil {
return ""
}
skills := []string{}
for _, skillRow := range table.Rows {
if stringValue(skillRow, "ClassSkill") != "1" {
continue
}
name := ctx.renderReference(fieldValue(skillRow, "SkillIndex"), ctx.skillIDToKey, ctx.resolveSkillName)
if name != "" {
skills = append(skills, name)
}
}
if len(skills) == 0 {
return ""
}
return `<h4>Class Skills</h4>` + renderWikiListHTML("wiki-list wiki-class-skills", skills)
}
func (ctx *wikiContext) renderClassFeatProgression(row map[string]any) string {
table := ctx.tableForValue(fieldValue(row, "FeatsTable"), ctx.classFeatTables)
if table == nil {
return ""
}
lines := []string{}
byLevel := map[string][]string{}
selectable := []string{}
for _, featRow := range table.Rows {
name := ctx.renderReference(fieldValue(featRow, "FeatIndex"), ctx.featIDToKey, ctx.resolveFeatName)
if name == "" {
name = ctx.renderReference(fieldValue(featRow, "FeatIndex"), nil, func(string) string { return "" })
}
if name == "" {
continue
}
level := stringValue(featRow, "GrantedOnLevel")
if level == "" || strings.HasPrefix(level, "-") {
selectable = append(selectable, name)
continue
}
byLevel[level] = append(byLevel[level], name)
}
if len(byLevel) > 0 {
lines = append(lines, "==== Granted Feats ====", "")
for _, level := range sortedKeys(byLevel) {
lines = append(lines, " * Level "+level+": "+strings.Join(byLevel[level], ", "))
}
lines = append(lines, "")
}
if len(selectable) > 0 {
lines = append(lines, "==== Selectable Feats ====", "", " * "+strings.Join(selectable, ", "), "")
}
return strings.TrimSpace(strings.Join(lines, "\n"))
}
func (ctx *wikiContext) renderClassFeatProgressionHTML(row map[string]any) string {
table := ctx.tableForValue(fieldValue(row, "FeatsTable"), ctx.classFeatTables)
if table == nil {
return ""
}
byLevel := map[string][]string{}
selectable := []string{}
for _, featRow := range table.Rows {
name := ctx.renderReference(fieldValue(featRow, "FeatIndex"), ctx.featIDToKey, ctx.resolveFeatName)
if name == "" {
name = ctx.renderReference(fieldValue(featRow, "FeatIndex"), nil, func(string) string { return "" })
}
if name == "" {
continue
}
level := stringValue(featRow, "GrantedOnLevel")
if level == "" || strings.HasPrefix(level, "-") {
selectable = append(selectable, name)
continue
}
byLevel[level] = append(byLevel[level], name)
}
parts := []string{}
if len(byLevel) > 0 {
items := []string{}
for _, level := range sortedKeys(byLevel) {
items = append(items, "Level "+level+": "+strings.Join(byLevel[level], ", "))
}
parts = append(parts, `<h4>Granted Feats</h4>`+renderWikiListHTML("wiki-list wiki-class-granted-feats", items))
}
if len(selectable) > 0 {
parts = append(parts, `<h4>Selectable Feats</h4>`+renderWikiListHTML("wiki-list wiki-class-selectable-feats", []string{strings.Join(selectable, ", ")}))
}
return strings.Join(parts, "\n")
}
func (ctx *wikiContext) renderClassFeatureTable(row map[string]any) string {
table := ctx.tableForValue(fieldValue(row, "BonusFeatsTable"), ctx.classBonusTables)
if table == nil || len(table.Rows) == 0 {
return ""
}
return "==== Bonus Feats ====\n\n * Bonus feat table: " + table.OutputStem
}
func (ctx *wikiContext) renderClassFeatureTableHTML(row map[string]any) string {
table := ctx.tableForValue(fieldValue(row, "BonusFeatsTable"), ctx.classBonusTables)
if table == nil || len(table.Rows) == 0 {
return ""
}
return `<h4>Bonus Feats</h4>` + renderWikiListHTML("wiki-list wiki-class-bonus-feats", []string{"Bonus feat table: " + table.OutputStem})
}
func (ctx *wikiContext) renderRaceNameFormsHTML(row map[string]any) string {
facts := []wikiHTMLFact{}
if plural := ctx.resolveTextValue(fieldValue(row, "NamePlural"), nil); plural != "" {
facts = append(facts, wikiHTMLFact{Label: "Plural", Value: plural})
}
if converted := ctx.resolveTextValue(fieldValue(row, "ConverName"), nil); converted != "" {
facts = append(facts, wikiHTMLFact{Label: "Converted Name", Value: converted})
}
if lower := ctx.resolveTextValue(fieldValue(row, "ConverNameLower"), nil); lower != "" {
facts = append(facts, wikiHTMLFact{Label: "Lower Name", Value: lower})
}
return renderWikiFactTableHTML(facts)
}
internal/topdata/wiki_native.go
-2159
View File
File diff suppressed because it is too large Load Diff
internal/topdata/wiki_native_test.go
-3008
View File
File diff suppressed because it is too large Load Diff
internal/topdata/wiki_page_paths.go
-77
View File
@@ -1,77 +0,0 @@
package topdata
import (
"fmt"
"os"
"strings"
"gopkg.in/yaml.v3"
)
type wikiPagePathDocument struct {
PagePaths wikiPagePaths `json:"page_paths" yaml:"page_paths"`
}
type wikiPagePaths struct {
SlugOverrides []wikiPageSlugOverride `json:"slug_overrides" yaml:"slug_overrides"`
}
type wikiPageSlugOverride struct {
PageID string `json:"page_id" yaml:"page_id"`
Slug string `json:"slug" yaml:"slug"`
}
func loadWikiPagePathDeclarations(path string) (map[string]string, error) {
raw, err := os.ReadFile(path)
if os.IsNotExist(err) {
return map[string]string{}, nil
}
if err != nil {
return nil, err
}
return parseWikiPagePathDeclarations(raw, path)
}
func parseWikiPagePathDeclarations(raw []byte, path string) (map[string]string, error) {
var doc wikiPagePathDocument
if err := yaml.Unmarshal(raw, &doc); err != nil {
return nil, fmt.Errorf("parse wiki page paths %s: %w", path, err)
}
if len(doc.PagePaths.SlugOverrides) > 0 {
return nil, fmt.Errorf("parse wiki page paths %s: page_paths.slug_overrides is retired; generated public wiki paths are title-derived", path)
}
overrides := map[string]string{}
for _, override := range doc.PagePaths.SlugOverrides {
pageID := strings.TrimSpace(override.PageID)
slug := strings.TrimSpace(override.Slug)
if pageID == "" {
return nil, fmt.Errorf("parse wiki page paths %s: slug override has empty page_id", path)
}
if !isCanonicalWikiSlug(slug) {
return nil, fmt.Errorf("parse wiki page paths %s: page_id %q has invalid slug %q", path, pageID, override.Slug)
}
if _, ok := overrides[pageID]; ok {
return nil, fmt.Errorf("parse wiki page paths %s: duplicate page_id %q", path, pageID)
}
overrides[pageID] = slug
}
return overrides, nil
}
func isCanonicalWikiSlug(value string) bool {
if value == "" || value != strings.ToLower(value) {
return false
}
parts := strings.Split(value, "-")
for _, part := range parts {
if part == "" {
return false
}
for _, r := range part {
if !((r >= 'a' && r <= 'z') || (r >= '0' && r <= '9')) {
return false
}
}
}
return true
}
internal/topdata/wiki_preserve.go
-54
View File
@@ -1,54 +0,0 @@
package topdata
import "strings"
func normalizePreservedSectionBodies(content string) string {
var out strings.Builder
offset := 0
for {
startRel := strings.Index(content[offset:], manualStartMarkerPrefix)
if startRel < 0 {
out.WriteString(content[offset:])
break
}
start := offset + startRel
startEndRel := strings.Index(content[start:], "-->")
if startEndRel < 0 {
out.WriteString(content[offset:])
break
}
startMarker := content[start : start+startEndRel+len("-->")]
id := markerID(startMarker)
if id == "" {
out.WriteString(content[offset : start+len(startMarker)])
offset = start + len(startMarker)
continue
}
endPrefix := manualEndMarkerPrefix + " id=\"" + id + "\""
endRel := strings.Index(content[start+len(startMarker):], endPrefix)
if endRel < 0 {
out.WriteString(content[offset:])
break
}
end := start + len(startMarker) + endRel
endEndRel := strings.Index(content[end:], "-->")
if endEndRel < 0 {
out.WriteString(content[offset:])
break
}
endMarker := content[end : end+endEndRel+len("-->")]
out.WriteString(content[offset:start])
out.WriteString(startMarker)
out.WriteString("\n<!-- preserved-section-body -->\n")
out.WriteString(endMarker)
offset = end + len(endMarker)
}
return out.String()
}
func defaultWikiManualSections() []wikiManualSection {
return []wikiManualSection{
{ID: "user_top", Alias: "UserTopSection", Heading: "", InitialHTML: "<p></p>"},
{ID: "user_bottom", Alias: "UserBottomSection", Heading: "", InitialHTML: "<p></p>"},
}
}
internal/topdata/wiki_slug.go
-210
View File
@@ -1,210 +0,0 @@
package topdata
import (
"html"
"strings"
"unicode"
"golang.org/x/text/unicode/norm"
)
var reservedCanonicalWikiFirstSegments = map[string]struct{}{
"admin": {},
"api": {},
"category": {},
"compose": {},
"edit": {},
"namespace": {},
"search": {},
}
func canonicalWikiSegment(value string) string {
value = html.UnescapeString(strings.TrimSpace(value))
var b strings.Builder
lastSeparator := false
for _, r := range norm.NFKD.String(value) {
transliterated := transliterateWikiSlugRune(unicode.ToLower(r))
switch {
case (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9'):
b.WriteRune(r)
lastSeparator = false
case r == '_':
if b.Len() > 0 && !lastSeparator {
b.WriteRune(r)
lastSeparator = true
}
case unicode.Is(unicode.Mn, r):
continue
case isWikiSlugJoinerPunctuation(r):
continue
case transliterated != "":
b.WriteString(matchWikiTransliterationCase(r, transliterated))
lastSeparator = false
default:
if b.Len() > 0 && !lastSeparator {
b.WriteByte('_')
lastSeparator = true
}
}
}
return strings.Trim(b.String(), "_")
}
func canonicalWikiSegmentFoldedKey(value string) string {
return strings.Join(strings.Fields(strings.ToLower(strings.ReplaceAll(canonicalWikiSegment(value), "_", " "))), " ")
}
func canonicalWikiSlashPath(value string) string {
segments := []string{}
for _, part := range strings.Split(strings.Trim(strings.TrimSpace(value), "/"), "/") {
canonical := canonicalWikiSegment(part)
if canonical != "" {
segments = append(segments, canonical)
}
}
return strings.Join(segments, "/")
}
func canonicalWikiSlashPathFoldedKey(value string) string {
segments := []string{}
for _, part := range strings.Split(strings.Trim(strings.TrimSpace(value), "/"), "/") {
folded := canonicalWikiSegmentFoldedKey(part)
if folded != "" {
segments = append(segments, folded)
}
}
return strings.Join(segments, "/")
}
func canonicalWikiSlashPathSegments(value string) []string {
segments := []string{}
for _, part := range strings.Split(strings.Trim(strings.TrimSpace(value), "/"), "/") {
canonical := canonicalWikiSegment(part)
if canonical != "" {
segments = append(segments, canonical)
}
}
return segments
}
func isReservedCanonicalWikiFirstSegment(path string) bool {
segments := canonicalWikiSlashPathSegments(path)
if len(segments) == 0 {
return false
}
first := strings.ToLower(segments[0])
if isCanonicalWikiNumericSegment(first) {
return true
}
_, reserved := reservedCanonicalWikiFirstSegments[first]
return reserved
}
func isCanonicalWikiNumericSegment(segment string) bool {
if segment == "" {
return false
}
for _, r := range segment {
if r < '0' || r > '9' {
return false
}
}
return true
}
func isNestedCanonicalWikiPath(path string) bool {
return len(canonicalWikiSlashPathSegments(path)) > 2
}
func matchWikiTransliterationCase(r rune, value string) string {
if unicode.IsUpper(r) {
if len(value) == 1 {
return strings.ToUpper(value)
}
return strings.ToUpper(value[:1]) + value[1:]
}
return value
}
func canonicalWikiPath(segments ...string) string {
out := []string{}
for _, segment := range segments {
canonical := canonicalWikiSegment(segment)
if canonical != "" {
out = append(out, canonical)
}
}
return strings.Join(out, "/")
}
func slugifyWikiText(value string, fallback string) string {
value = html.UnescapeString(strings.TrimSpace(value))
var b strings.Builder
lastDash := false
for _, r := range norm.NFKD.String(strings.ToLower(value)) {
transliterated := transliterateWikiSlugRune(r)
switch {
case r >= 'a' && r <= 'z':
b.WriteRune(r)
lastDash = false
case r >= '0' && r <= '9':
b.WriteRune(r)
lastDash = false
case unicode.Is(unicode.Mn, r):
continue
case isWikiSlugJoinerPunctuation(r):
continue
case transliterated != "":
b.WriteString(transliterated)
lastDash = false
default:
if b.Len() > 0 && !lastDash {
b.WriteByte('-')
lastDash = true
}
}
}
slug := strings.Trim(b.String(), "-")
if slug == "" {
return fallback
}
return slug
}
func isWikiSlugJoinerPunctuation(r rune) bool {
switch r {
case '\'', '"', '`', '´', '‘', '’', '‚', '‛', '“', '”', '„', '‟', '‹', '›', '«', '»', '′', '″', '＇', '＂':
return true
default:
return false
}
}
func transliterateWikiSlugRune(r rune) string {
switch r {
case 'æ':
return "ae"
case 'œ':
return "oe"
case 'ø':
return "o"
case 'ß':
return "ss"
case 'þ':
return "th"
case 'ð', 'đ':
return "d"
case 'ł':
return "l"
case 'ħ':
return "h"
case 'ı':
return "i"
case 'ŋ':
return "n"
case 'ŧ':
return "t"
default:
return ""
}
}
internal/topdata/wiki_tables.go
-1749
View File
File diff suppressed because it is too large Load Diff
internal/topdata/wiki_template.go
-433
View File
@@ -1,433 +0,0 @@
package topdata
import (
"fmt"
"html"
"os"
"path/filepath"
"strings"
)
type wikiTemplatePage struct {
PageID string
Category string
Key string
Title string
Status string
Row map[string]any
ManualSections []wikiManualSection
}
func (ctx *wikiContext) renderWikiPageTemplate(category string, page wikiTemplatePage) (string, error) {
name := wikiTemplateName(category)
source, path := ctx.loadWikiTemplateSource(name)
if source == "" && name != "default.html" {
source, path = ctx.loadWikiTemplateSource("default.html")
}
if source == "" {
source = builtinWikiTemplate(category)
path = "builtin:" + name
}
return ctx.renderWikiTemplateString(path, source, page)
}
func wikiTemplateName(category string) string {
switch category {
case "classes", "feat", "skills", "spells", "racialtypes", "baseitems":
return category + ".html"
default:
return "default.html"
}
}
func (ctx *wikiContext) loadWikiTemplateSource(name string) (string, string) {
if strings.TrimSpace(ctx.templateDir) == "" {
return "", ""
}
path := filepath.Join(ctx.templateDir, filepath.FromSlash(name))
raw, err := os.ReadFile(path)
if err != nil {
return "", path
}
return string(raw), path
}
func builtinWikiTemplate(category string) string {
switch category {
case "classes":
return `<h1>{{title}}</h1>
{{UserTopSection}}
{{format:StatusCallout}}
{{format:Description}}
{{format:FactsTable}}
{{format:SkillList}}
{{format:FeatProgression}}
{{format:UserBottomFallback}}`
case "racialtypes":
return `<h1>{{title}}</h1>
{{UserTopSection}}
{{format:StatusCallout}}
{{format:Description}}
{{format:FactsTable}}
{{format:RaceNameForms}}
{{format:UserBottomFallback}}`
default:
return `<h1>{{title}}</h1>
{{UserTopSection}}
{{format:StatusCallout}}
{{format:Description}}
{{format:FactsTable}}
{{format:UserBottomFallback}}`
}
}
func (ctx *wikiContext) renderWikiTemplateString(path, source string, page wikiTemplatePage) (string, error) {
state := &wikiTemplateRenderState{
path: path,
page: page,
seenSections: map[string]struct{}{},
active: true,
}
rendered, offset, closed, err := ctx.renderWikiTemplateRange(source, 0, "", state)
if err != nil {
return "", err
}
if closed {
return "", fmt.Errorf("render wiki template %s for %s: unexpected closing block", path, page.PageID)
}
if offset != len(source) {
return "", fmt.Errorf("render wiki template %s for %s: template parse stopped early", path, page.PageID)
}
return strings.TrimSpace(rendered), nil
}
type wikiTemplateRenderState struct {
path string
page wikiTemplatePage
row map[string]any
dot any
seenSections map[string]struct{}
active bool
}
func (ctx *wikiContext) renderWikiTemplateRange(source string, offset int, stopBlock string, state *wikiTemplateRenderState) (string, int, bool, error) {
var out strings.Builder
for {
start := strings.Index(source[offset:], "{{")
if start < 0 {
if stopBlock != "" {
return "", offset, false, fmt.Errorf("render wiki template %s for %s: missing {{/%s}}", state.path, state.page.PageID, stopBlock)
}
if state.active {
out.WriteString(source[offset:])
}
return out.String(), len(source), false, nil
}
start += offset
end := strings.Index(source[start+2:], "}}")
if end < 0 {
return "", offset, false, fmt.Errorf("render wiki template %s for %s: unclosed placeholder", state.path, state.page.PageID)
}
end += start + 2
if state.active {
out.WriteString(source[offset:start])
}
token := strings.TrimSpace(source[start+2 : end])
if strings.HasPrefix(token, "/") {
name := strings.TrimSpace(strings.TrimPrefix(token, "/"))
if stopBlock == "" || name != stopBlock {
return "", offset, false, fmt.Errorf("render wiki template %s for %s: unexpected {{/%s}}", state.path, state.page.PageID, name)
}
return out.String(), end + 2, true, nil
}
if strings.HasPrefix(token, "#each ") {
providerName := strings.TrimSpace(strings.TrimPrefix(token, "#each "))
body, nextOffset, err := extractWikiTemplateBlock(source, end+2, "each", state.path, state.page.PageID)
if err != nil {
return "", offset, false, err
}
if state.active {
items, err := ctx.resolveWikiTemplateEachItems(providerName, state)
if err != nil {
return "", offset, false, fmt.Errorf("render wiki template %s for %s: %w", state.path, state.page.PageID, err)
}
for _, item := range items {
child := *state
child.row = item.row
child.dot = item.dot
rendered, childOffset, closed, err := ctx.renderWikiTemplateRange(body, 0, "", &child)
if err != nil {
return "", offset, false, err
}
if closed || childOffset != len(body) {
return "", offset, false, fmt.Errorf("render wiki template %s for %s: malformed each body", state.path, state.page.PageID)
}
out.WriteString(rendered)
}
}
offset = nextOffset
continue
}
if strings.HasPrefix(token, "#if ") {
expr := strings.TrimSpace(strings.TrimPrefix(token, "#if "))
body, nextOffset, err := extractWikiTemplateBlock(source, end+2, "if", state.path, state.page.PageID)
if err != nil {
return "", offset, false, err
}
if state.active {
value, err := ctx.evalWikiTemplateExpression(expr, state)
if err != nil {
return "", offset, false, err
}
if truthyWikiTableValue(value) {
child := *state
rendered, childOffset, closed, err := ctx.renderWikiTemplateRange(body, 0, "", &child)
if err != nil {
return "", offset, false, err
}
if closed || childOffset != len(body) {
return "", offset, false, fmt.Errorf("render wiki template %s for %s: malformed if body", state.path, state.page.PageID)
}
out.WriteString(rendered)
}
}
offset = nextOffset
continue
}
if strings.HasPrefix(token, "#with ") {
expr := strings.TrimSpace(strings.TrimPrefix(token, "#with "))
body, nextOffset, err := extractWikiTemplateBlock(source, end+2, "with", state.path, state.page.PageID)
if err != nil {
return "", offset, false, err
}
if state.active {
value, err := ctx.evalWikiTemplateExpression(expr, state)
if err != nil {
return "", offset, false, err
}
if truthyWikiTableValue(value) {
child := *state
child.dot = value
rendered, childOffset, closed, err := ctx.renderWikiTemplateRange(body, 0, "", &child)
if err != nil {
return "", offset, false, err
}
if closed || childOffset != len(body) {
return "", offset, false, fmt.Errorf("render wiki template %s for %s: malformed with body", state.path, state.page.PageID)
}
out.WriteString(rendered)
}
}
offset = nextOffset
continue
}
if !state.active {
offset = end + 2
continue
}
rendered, sectionID, err := ctx.renderWikiTemplateTokenWithState(token, state)
if err != nil {
return "", offset, false, err
}
if sectionID != "" {
if _, exists := state.seenSections[sectionID]; exists {
return "", offset, false, fmt.Errorf("render wiki template %s for %s: duplicate preserved section %q", state.path, state.page.PageID, sectionID)
}
state.seenSections[sectionID] = struct{}{}
}
out.WriteString(rendered)
offset = end + 2
}
}
type wikiTemplateEachItem struct {
row map[string]any
dot any
}
func (ctx *wikiContext) resolveWikiTemplateEachItems(name string, state *wikiTemplateRenderState) ([]wikiTemplateEachItem, error) {
if value, ok := ctx.resolveWikiTemplateScopedValue(name, state); ok {
return wikiTemplateEachItemsFromValue(value), nil
}
rows, err := ctx.wikiDataProviderRows(name, state.page)
if err != nil {
return nil, err
}
items := make([]wikiTemplateEachItem, 0, len(rows))
for _, row := range rows {
items = append(items, wikiTemplateEachItem{row: row, dot: row})
}
return items, nil
}
func wikiTemplateEachItemsFromValue(value any) []wikiTemplateEachItem {
switch typed := value.(type) {
case []map[string]any:
items := make([]wikiTemplateEachItem, 0, len(typed))
for _, row := range typed {
items = append(items, wikiTemplateEachItem{row: row, dot: row})
}
return items
case []any:
items := make([]wikiTemplateEachItem, 0, len(typed))
for _, value := range typed {
if row, ok := value.(map[string]any); ok {
items = append(items, wikiTemplateEachItem{row: row, dot: row})
continue
}
items = append(items, wikiTemplateEachItem{dot: value})
}
return items
case []string:
items := make([]wikiTemplateEachItem, 0, len(typed))
for _, value := range typed {
items = append(items, wikiTemplateEachItem{dot: value})
}
return items
default:
return nil
}
}
func (ctx *wikiContext) renderWikiTemplateTokenWithState(token string, state *wikiTemplateRenderState) (string, string, error) {
if strings.HasPrefix(token, "format:") || strings.HasPrefix(token, "table:") || strings.HasPrefix(token, "field:") || strings.HasPrefix(token, "section:") {
rendered, sectionID, err := ctx.renderWikiTemplateToken(state.path, token, state.page)
return rendered, sectionID, err
}
if strings.Contains(token, "|") || strings.Contains(token, "(") || token == "." || strings.HasPrefix(token, "page.") {
value, err := ctx.evalWikiTemplateExpression(token, state)
if err != nil {
return "", "", err
}
return stringifyWikiTemplateOutput(value), "", nil
}
if state.row != nil {
if _, ok := ctx.resolveWikiTemplateScopedValue(token, state); ok {
value, err := ctx.evalWikiTemplateExpression(token, state)
if err != nil {
return "", "", err
}
return stringifyWikiTemplateOutput(value), "", nil
}
}
rendered, sectionID, err := ctx.renderWikiTemplateToken(state.path, token, state.page)
return rendered, sectionID, err
}
func (ctx *wikiContext) renderWikiTemplateToken(path, token string, page wikiTemplatePage) (string, string, error) {
switch token {
case "title":
return html.EscapeString(page.Title), "", nil
case "page_id":
return html.EscapeString(page.PageID), "", nil
case "status":
return html.EscapeString(page.Status), "", nil
}
if section, ok := wikiManualSectionForAlias(page.ManualSections, token); ok {
return renderWikiManualSection(section), section.ID, nil
}
if strings.HasPrefix(token, "section:") {
sectionName := strings.TrimSpace(strings.TrimPrefix(token, "section:"))
if section, ok := wikiManualSectionForAlias(page.ManualSections, sectionName); ok {
return renderWikiManualSection(section), section.ID, nil
}
return "", "", nil
}
if strings.HasPrefix(token, "format:") {
rendered, err := ctx.renderWikiFormatter(strings.TrimSpace(strings.TrimPrefix(token, "format:")), page)
return rendered, "", err
}
if strings.HasPrefix(token, "table:") {
rendered, err := ctx.renderConfiguredWikiTable(path, strings.TrimSpace(strings.TrimPrefix(token, "table:")), page)
return rendered, "", err
}
if strings.HasPrefix(token, "field:") {
rendered, err := ctx.renderExplicitWikiField(path, strings.TrimSpace(strings.TrimPrefix(token, "field:")), page)
return rendered, "", err
}
rendered, err := ctx.renderRequiredWikiField(path, token, page)
return rendered, "", err
}
func wikiManualSectionForAlias(sections []wikiManualSection, alias string) (wikiManualSection, bool) {
for _, section := range sections {
if section.Alias == alias {
return section, true
}
}
switch alias {
case "UserTopSection":
for _, section := range sections {
if section.ID == "user_top" {
return section, true
}
}
case "UserBottomSection":
for _, section := range sections {
if section.ID == "user_bottom" {
return section, true
}
}
}
return wikiManualSection{}, false
}
func renderWikiManualSection(section wikiManualSection) string {
return strings.Join([]string{
"<!-- sow-topdata-wiki:manual:start id=\"" + html.EscapeString(section.ID) + "\" -->",
strings.TrimSpace(section.InitialHTML),
"<!-- sow-topdata-wiki:manual:end id=\"" + html.EscapeString(section.ID) + "\" -->",
}, "\n")
}
func (ctx *wikiContext) renderExplicitWikiField(path, spec string, page wikiTemplatePage) (string, error) {
fieldSpec, defaultValue, hasDefault := strings.Cut(spec, "|default=")
fieldSpec = strings.TrimSpace(fieldSpec)
value, ok := ctx.resolveWikiTemplateField(fieldSpec, page)
if !ok {
if hasDefault {
return html.EscapeString(defaultValue), nil
}
return "", fmt.Errorf("render wiki template %s for %s: missing field %q", path, page.PageID, fieldSpec)
}
return html.EscapeString(value), nil
}
func (ctx *wikiContext) renderRequiredWikiField(path, fieldSpec string, page wikiTemplatePage) (string, error) {
value, ok := ctx.resolveWikiTemplateField(fieldSpec, page)
if !ok {
return "", fmt.Errorf("render wiki template %s for %s: missing field %q", path, page.PageID, fieldSpec)
}
return html.EscapeString(value), nil
}
func (ctx *wikiContext) resolveWikiTemplateField(fieldSpec string, page wikiTemplatePage) (string, bool) {
fieldSpec = strings.TrimSpace(fieldSpec)
if strings.HasSuffix(fieldSpec, ".text") {
base := strings.TrimSuffix(fieldSpec, ".text")
value, ok := lookupField(page.Row, base)
if !ok {
return "", false
}
text := ctx.resolveTextValue(value, nil)
return text, text != ""
}
value, ok := lookupField(page.Row, fieldSpec)
if !ok || value == nil {
return "", false
}
switch typed := value.(type) {
case string:
if typed == "" || typed == nullValue {
return "", false
}
return typed, true
case int:
return fmt.Sprintf("%d", typed), true
case float64:
return fmt.Sprintf("%d", int(typed)), true
default:
text := ctx.resolveTextValue(typed, nil)
return text, text != ""
}
}
internal/topdata/wiki_template_expr.go
-629
View File
@@ -1,629 +0,0 @@
package topdata
import (
"fmt"
"html"
"strconv"
"strings"
"unicode"
)
type wikiTemplateHTML string
func extractWikiTemplateBlock(source string, offset int, blockName, path, pageID string) (string, int, error) {
depth := 1
bodyStart := offset
for offset < len(source) {
start := strings.Index(source[offset:], "{{")
if start < 0 {
break
}
start += offset
end := strings.Index(source[start+2:], "}}")
if end < 0 {
return "", offset, fmt.Errorf("render wiki template %s for %s: unclosed placeholder", path, pageID)
}
end += start + 2
token := strings.TrimSpace(source[start+2 : end])
if token == "#"+blockName || strings.HasPrefix(token, "#"+blockName+" ") {
depth++
} else if token == "/"+blockName {
depth--
if depth == 0 {
return source[bodyStart:start], end + 2, nil
}
}
offset = end + 2
}
return "", offset, fmt.Errorf("render wiki template %s for %s: missing {{/%s}}", path, pageID, blockName)
}
func (ctx *wikiContext) evalWikiTemplateExpression(expr string, state *wikiTemplateRenderState) (any, error) {
parts := splitWikiTemplatePipes(expr)
if len(parts) == 0 {
return "", nil
}
value, err := ctx.evalWikiTemplateBase(strings.TrimSpace(parts[0]), state)
if err != nil {
return nil, err
}
for _, rawFilter := range parts[1:] {
value, err = ctx.applyWikiTemplateFilter(value, strings.TrimSpace(rawFilter), state)
if err != nil {
return nil, err
}
}
return value, nil
}
func splitWikiTemplatePipes(expr string) []string {
parts := []string{}
start := 0
inQuote := false
escaped := false
for i := 0; i < len(expr); i++ {
switch {
case escaped:
escaped = false
case expr[i] == '\\' && inQuote:
escaped = true
case expr[i] == '"':
inQuote = !inQuote
case expr[i] == '|' && !inQuote:
parts = append(parts, expr[start:i])
start = i + 1
}
}
parts = append(parts, expr[start:])
return parts
}
func (ctx *wikiContext) evalWikiTemplateBase(expr string, state *wikiTemplateRenderState) (any, error) {
if expr == "." {
return state.dot, nil
}
if value, ok := ctx.resolveWikiTemplateScopedValue(expr, state); ok {
return value, nil
}
renderCtx := wikiTableRenderContext{Page: state.page, Path: state.path}
row := state.page.Row
if state.row != nil {
row = state.row
}
parser := wikiExprParser{tokens: tokenizeWikiExpr(expr), row: row, ctx: ctx, renderCtx: renderCtx}
value, err := parser.parseComparison()
if err != nil {
return nil, fmt.Errorf("render wiki template %s for %s expression %q: %w", state.path, state.page.PageID, expr, err)
}
if parser.peek().kind != wikiExprEOF {
return nil, fmt.Errorf("render wiki template %s for %s expression %q: unexpected token %q", state.path, state.page.PageID, expr, parser.peek().text)
}
return value, nil
}
func (ctx *wikiContext) resolveWikiTemplateScopedValue(name string, state *wikiTemplateRenderState) (any, bool) {
name = strings.TrimSpace(name)
if strings.HasPrefix(name, "page.") {
field := strings.TrimPrefix(name, "page.")
switch field {
case "title", "Title":
return state.page.Title, true
case "page_id", "PageID":
return state.page.PageID, true
case "status", "Status":
return state.page.Status, true
case "category", "Category":
return state.page.Category, true
case "key", "Key":
return state.page.Key, true
default:
if value, ok := lookupField(state.page.Row, field); ok {
return value, true
}
return nil, false
}
}
if state.row != nil {
if value, ok := lookupField(state.row, name); ok {
return value, true
}
}
if value, ok := lookupField(state.page.Row, name); ok {
return value, true
}
switch name {
case "title":
return state.page.Title, true
case "page_id":
return state.page.PageID, true
case "status":
return state.page.Status, true
}
return nil, false
}
func (ctx *wikiContext) applyWikiTemplateFilter(value any, filter string, state *wikiTemplateRenderState) (any, error) {
name, argText, hasArg := strings.Cut(filter, ":")
name = strings.TrimSpace(name)
args, err := parseWikiTemplateFilterArgs(argText)
if hasArg && err != nil {
return nil, fmt.Errorf("render wiki template %s for %s filter %q: %w", state.path, state.page.PageID, filter, err)
}
switch name {
case "text":
return ctx.resolveTextValue(value, nil), nil
case "trim":
return strings.TrimSpace(stringifyWikiTemplatePlain(value)), nil
case "after":
return afterFirstMarker(stringifyWikiTemplatePlain(value), firstArg(args)), nil
case "before":
return beforeFirstMarker(stringifyWikiTemplatePlain(value), firstArg(args)), nil
case "between":
text := stringifyWikiTemplatePlain(value)
return beforeFirstMarker(afterFirstMarker(text, firstArg(args)), secondArg(args)), nil
case "paragraph":
return selectWikiTemplateParagraphs(stringifyWikiTemplatePlain(value), firstArg(args)), nil
case "paragraphs":
return selectWikiTemplateParagraphs(stringifyWikiTemplatePlain(value), firstArg(args)), nil
case "linebreaks":
return wikiTemplateHTML(strings.ReplaceAll(html.EscapeString(stringifyWikiTemplatePlain(value)), "\n", "<br>")), nil
case "html":
return wikiTemplateHTML(renderWikiTemplatePlainHTML(stringifyWikiTemplatePlain(value))), nil
case "wiki":
return wikiTemplateHTML(renderWikiLinks(stringifyWikiTemplatePlain(value))), nil
case "lower":
return strings.ToLower(stringifyWikiTemplatePlain(value)), nil
case "title":
return strings.Title(stringifyWikiTemplatePlain(value)), nil
case "sentence":
text := stringifyWikiTemplatePlain(value)
for i, r := range text {
if unicode.IsLetter(r) {
return text[:i] + string(unicode.ToUpper(r)) + text[i+len(string(r)):], nil
}
}
return text, nil
case "join":
return joinWikiTemplateValue(value, firstArg(args)), nil
case "count":
return countWikiTemplateValue(value), nil
case "alignments":
return ctx.formatWikiTemplateAlignments(value)
case "list":
return wikiTemplateHTML(renderWikiTemplateList(value, firstArg(args))), nil
case "ordinal":
return ordinalWikiTableValue(numericWikiTableValue(value)), nil
case "default":
if strings.TrimSpace(stringifyWikiTemplatePlain(value)) == "" {
return firstArg(args), nil
}
return value, nil
case "ref":
return ctx.renderTemplateReference(value, firstArg(args)), nil
case "link":
return ctx.renderTemplateLink(value, firstArg(args), state), nil
default:
return nil, fmt.Errorf("render wiki template %s for %s: unknown filter %q", state.path, state.page.PageID, name)
}
}
func parseWikiTemplateFilterArgs(raw string) ([]string, error) {
raw = strings.TrimSpace(raw)
if raw == "" {
return nil, nil
}
args := []string{}
for len(raw) > 0 {
raw = strings.TrimSpace(raw)
if !strings.HasPrefix(raw, `"`) {
if idx := strings.Index(raw, ","); idx >= 0 {
args = append(args, strings.TrimSpace(raw[:idx]))
raw = raw[idx+1:]
continue
}
args = append(args, strings.TrimSpace(raw))
break
}
var b strings.Builder
escaped := false
i := 1
for ; i < len(raw); i++ {
if escaped {
b.WriteByte(raw[i])
escaped = false
continue
}
if raw[i] == '\\' {
escaped = true
continue
}
if raw[i] == '"' {
break
}
b.WriteByte(raw[i])
}
if i >= len(raw) {
return nil, fmt.Errorf("unterminated quoted argument")
}
args = append(args, b.String())
raw = strings.TrimSpace(raw[i+1:])
if strings.HasPrefix(raw, ",") {
raw = raw[1:]
} else if raw != "" {
return nil, fmt.Errorf("unexpected argument text %q", raw)
}
}
return args, nil
}
func firstArg(args []string) string {
if len(args) == 0 {
return ""
}
return args[0]
}
func secondArg(args []string) string {
if len(args) < 2 {
return ""
}
return args[1]
}
func afterFirstMarker(text, marker string) string {
if marker == "" {
return text
}
idx := strings.Index(text, marker)
if idx < 0 {
return ""
}
return text[idx+len(marker):]
}
func beforeFirstMarker(text, marker string) string {
if marker == "" {
return text
}
idx := strings.Index(text, marker)
if idx < 0 {
return text
}
return text[:idx]
}
func selectWikiTemplateParagraphs(text, selector string) string {
paragraphs := splitWikiTemplateParagraphs(text)
if len(paragraphs) == 0 {
return ""
}
switch strings.TrimSpace(selector) {
case "first":
return paragraphs[0]
case "last":
return paragraphs[len(paragraphs)-1]
case "all", "":
return strings.Join(paragraphs, "\n\n")
default:
selected := []string{}
for _, part := range strings.Split(selector, ",") {
switch strings.TrimSpace(part) {
case "first":
selected = append(selected, paragraphs[0])
case "last":
selected = append(selected, paragraphs[len(paragraphs)-1])
default:
if idx, err := strconv.Atoi(strings.TrimSpace(part)); err == nil && idx > 0 && idx <= len(paragraphs) {
selected = append(selected, paragraphs[idx-1])
}
}
}
return strings.Join(selected, "\n\n")
}
}
func splitWikiTemplateParagraphs(text string) []string {
normalized := strings.ReplaceAll(text, "\r\n", "\n")
chunks := strings.Split(normalized, "\n\n")
out := []string{}
for _, chunk := range chunks {
if trimmed := strings.TrimSpace(chunk); trimmed != "" {
out = append(out, trimmed)
}
}
return out
}
func renderWikiTemplatePlainHTML(text string) string {
paragraphs := splitWikiTemplateParagraphs(text)
if len(paragraphs) == 0 {
return ""
}
out := strings.Builder{}
for _, paragraph := range paragraphs {
escaped := html.EscapeString(paragraph)
escaped = strings.ReplaceAll(escaped, "\n", "<br>")
out.WriteString("<p>")
out.WriteString(escaped)
out.WriteString("</p>")
}
return out.String()
}
func joinWikiTemplateValue(value any, sep string) string {
switch typed := value.(type) {
case []string:
return strings.Join(typed, sep)
case []any:
parts := make([]string, 0, len(typed))
for _, item := range typed {
if text := stringifyWikiTemplatePlain(item); text != "" {
parts = append(parts, text)
}
}
return strings.Join(parts, sep)
default:
return stringifyWikiTemplatePlain(value)
}
}
func (ctx *wikiContext) formatWikiTemplateAlignments(value any) (any, error) {
values, err := wikiTemplateAlignmentValues(value)
if err != nil {
return "", err
}
if len(values) == 0 || (len(values) == 1 && values[0] == 0) {
return "Any", nil
}
labels := make([]string, 0, len(values))
for _, value := range values {
if value == 0 {
continue
}
label, ok := wikiTemplateAlignmentLabel(value)
if !ok {
return "", fmt.Errorf("unknown alignment value 0x%02X", value)
}
labels = append(labels, ctx.renderWikiTemplateAlignmentLabel(label))
}
if len(labels) == 0 {
return "Any", nil
}
rendered := strings.Join(labels, ", ")
if ctx != nil && strings.TrimSpace(ctx.alignmentLinkFormat) == "html_anchor" {
return wikiTemplateHTML(rendered), nil
}
return rendered, nil
}
func (ctx *wikiContext) renderWikiTemplateAlignmentLabel(label string) string {
pattern := ""
format := "wiki_markup"
if ctx != nil {
pattern = strings.TrimSpace(ctx.alignmentLinkPattern)
if strings.TrimSpace(ctx.alignmentLinkFormat) != "" {
format = strings.TrimSpace(ctx.alignmentLinkFormat)
}
}
if pattern == "" {
return label
}
target := strings.ReplaceAll(pattern, "{alignment}", wikiTemplateAlignmentTargetSegment(label))
target = strings.ReplaceAll(target, "{label}", label)
if strings.TrimSpace(target) == "" {
return label
}
if format == "html_anchor" {
return `<a class="wiki-internal-link" href="` + html.EscapeString(target) + `">` + html.EscapeString(label) + `</a>`
}
return "[[" + target + "|" + label + "]]"
}
func wikiTemplateAlignmentTargetSegment(label string) string {
return strings.ReplaceAll(strings.TrimSpace(label), " ", "_")
}
func wikiTemplateAlignmentValues(value any) ([]int, error) {
if isNullLike(value) {
return []int{0}, nil
}
switch typed := value.(type) {
case map[string]any:
_, hasAlignments := typed["alignments"]
_, hasList := typed["list"]
if hasAlignments && hasList {
return nil, fmt.Errorf("alignment list object cannot contain both alignments and list")
}
for key := range typed {
if key != "alignments" && key != "list" {
return nil, fmt.Errorf("alignment list object contains unsupported key %q", key)
}
}
if hasAlignments {
return wikiTemplateAlignmentValues(typed["alignments"])
}
return wikiTemplateAlignmentValues(typed["list"])
case string:
trimmed := strings.TrimSpace(typed)
if trimmed == "" || trimmed == "0" || strings.EqualFold(trimmed, "0x00") {
return []int{0}, nil
}
if strings.HasPrefix(strings.ToLower(trimmed), "0x") {
hexText := trimmed[2:]
if len(hexText)%2 != 0 {
return nil, fmt.Errorf("alignment hex list %q must contain two-character chunks", typed)
}
values := make([]int, 0, len(hexText)/2)
for index := 0; index < len(hexText); index += 2 {
parsed, err := strconv.ParseInt(hexText[index:index+2], 16, 0)
if err != nil {
return nil, fmt.Errorf("alignment hex list %q contains invalid chunk %q", typed, hexText[index:index+2])
}
values = append(values, int(parsed))
}
return values, nil
}
if strings.Contains(trimmed, ",") {
parts := strings.Split(trimmed, ",")
values := make([]int, 0, len(parts))
for _, part := range parts {
parsed, err := parseConfiguredAlignmentValue(part)
if err != nil {
return nil, err
}
values = append(values, parsed)
}
return values, nil
}
parsed, err := parseConfiguredAlignmentValue(trimmed)
if err != nil {
return nil, err
}
return []int{parsed}, nil
case []any:
values := make([]int, 0, len(typed))
for _, item := range typed {
parsed, err := parseConfiguredAlignmentValue(item)
if err != nil {
return nil, err
}
values = append(values, parsed)
}
return values, nil
default:
parsed, err := parseConfiguredAlignmentValue(value)
if err != nil {
return nil, err
}
return []int{parsed}, nil
}
}
func wikiTemplateAlignmentLabel(value int) (string, bool) {
switch value {
case 0x0A:
return "Lawful Good", true
case 0x09:
return "Neutral Good", true
case 0x0C:
return "Chaotic Good", true
case 0x03:
return "Lawful Neutral", true
case 0x01:
return "True Neutral", true
case 0x05:
return "Chaotic Neutral", true
case 0x12:
return "Lawful Evil", true
case 0x11:
return "Neutral Evil", true
case 0x14:
return "Chaotic Evil", true
default:
return "", false
}
}
func renderWikiTemplateList(value any, class string) string {
items := wikiTemplateListItems(value)
if len(items) == 0 {
return ""
}
var out strings.Builder
out.WriteString("<ul")
out.WriteString(attrClass(class))
out.WriteString(">")
for _, item := range items {
out.WriteString("<li>")
out.WriteString(renderWikiLinks(item))
out.WriteString("</li>")
}
out.WriteString("</ul>")
return out.String()
}
func wikiTemplateListItems(value any) []string {
switch typed := value.(type) {
case []string:
out := make([]string, 0, len(typed))
for _, item := range typed {
if text := strings.TrimSpace(item); text != "" {
out = append(out, text)
}
}
return out
case []any:
out := make([]string, 0, len(typed))
for _, item := range typed {
if text := strings.TrimSpace(stringifyWikiTemplatePlain(item)); text != "" {
out = append(out, text)
}
}
return out
default:
if text := strings.TrimSpace(stringifyWikiTemplatePlain(value)); text != "" {
return []string{text}
}
return nil
}
}
func (ctx *wikiContext) renderTemplateReference(value any, dataset string) string {
return ctx.renderReference(value, ctx.wikiIDMapForDataset(dataset), ctx.wikiNameResolverForDataset(dataset))
}
func (ctx *wikiContext) wikiNameResolverForDataset(dataset string) func(string) string {
switch dataset {
case "classes":
return ctx.resolveClassName
case "feat":
return ctx.resolveFeatName
case "skills":
return ctx.resolveSkillName
case "spells":
return ctx.resolveSpellName
case "racialtypes":
return ctx.resolveRaceName
case "baseitems":
return ctx.resolveBaseItemName
default:
return func(string) string { return "" }
}
}
func (ctx *wikiContext) renderTemplateLink(value any, labelField string, state *wikiTemplateRenderState) string {
key := stringifyWikiTemplatePlain(value)
label := labelField
if state.row != nil {
if raw, ok := lookupField(state.row, labelField); ok {
label = stringifyWikiTemplatePlain(raw)
}
}
if label == "" {
label = key
}
if key == "" {
return label
}
if !ctx.canReferenceWikiKey(key) {
return ""
}
if target := ctx.publicWikiTargetForKey(key, label); target != "" {
key = target
}
return "[[" + key + "|" + label + "]]"
}
func stringifyWikiTemplateOutput(value any) string {
if htmlValue, ok := value.(wikiTemplateHTML); ok {
return string(htmlValue)
}
return html.EscapeString(stringifyWikiTemplatePlain(value))
}
func stringifyWikiTemplatePlain(value any) string {
if htmlValue, ok := value.(wikiTemplateHTML); ok {
return string(htmlValue)
}
return stringifyWikiTableValue(value)
}
internal/topdata/wiki_templates/classes.txt
-17
View File
@@ -1,17 +0,0 @@
====== {{name}} ======
{{status}}
{{description}}
{{facts}}
===== Progression =====
{{progression}}
<WRAP center round important>
**Auto-generated content.** This section is rebuilt from game data on each release. Edits above this line will be lost on the next build.
</WRAP>
===== Notes =====
internal/topdata/wiki_templates/feat.txt
-13
View File
@@ -1,13 +0,0 @@
====== {{name}} ======
{{status}}
{{description}}
{{facts}}
<WRAP center round important>
**Auto-generated content.** This section is rebuilt from game data on each release. Edits above this line will be lost on the next build.
</WRAP>
===== Notes =====
internal/topdata/wiki_templates/items.txt
-13
View File
@@ -1,13 +0,0 @@
====== {{name}} ======
{{status}}
{{description}}
{{facts}}
<WRAP center round important>
**Auto-generated content.** This section is rebuilt from game data on each release. Edits above this line will be lost on the next build.
</WRAP>
===== Notes =====
internal/topdata/wiki_templates/races.txt
-15
View File
@@ -1,15 +0,0 @@
====== {{name}} ======
{{status}}
{{nameforms}}
{{description}}
{{facts}}
<WRAP center round important>
**Auto-generated content.** This section is rebuilt from game data on each release. Edits above this line will be lost on the next build.
</WRAP>
===== Notes =====
internal/topdata/wiki_templates/skills.txt
-13
View File
@@ -1,13 +0,0 @@
====== {{name}} ======
{{status}}
{{description}}
{{facts}}
<WRAP center round important>
**Auto-generated content.** This section is rebuilt from game data on each release. Edits above this line will be lost on the next build.
</WRAP>
===== Notes =====
internal/topdata/wiki_templates/spells.txt
-13
View File
@@ -1,13 +0,0 @@
====== {{name}} ======
{{status}}
{{description}}
{{facts}}
<WRAP center round important>
**Auto-generated content.** This section is rebuilt from game data on each release. Edits above this line will be lost on the next build.
</WRAP>
===== Notes =====
internal/topdata/wiki_visibility.go
-407
View File
@@ -1,407 +0,0 @@
package topdata
import (
"fmt"
"os"
"sort"
"strings"
"gopkg.in/yaml.v3"
)
type wikiVisibilityDefinitions struct {
ReferenceSets map[string]wikiVisibilityReferenceSet `json:"reference_sets" yaml:"reference_sets"`
Datasets map[string]wikiVisibilityDataset `json:"datasets" yaml:"datasets"`
}
type wikiVisibilityDataset struct {
Eligibility wikiVisibilityPredicate `json:"eligibility" yaml:"eligibility"`
Include []wikiVisibilityRule `json:"include" yaml:"include"`
Exclude []wikiVisibilityRule `json:"exclude" yaml:"exclude"`
}
type wikiVisibilityPredicate struct {
When string `json:"when" yaml:"when"`
}
type wikiVisibilityRule struct {
When string `json:"when" yaml:"when"`
ReferenceSet string `json:"reference_set" yaml:"reference_set"`
}
type wikiVisibilityReferenceSet struct {
TargetDataset string `json:"target_dataset" yaml:"target_dataset"`
Sources []wikiVisibilityReferenceSource `json:"sources" yaml:"sources"`
}
type wikiVisibilityReferenceSource struct {
Dataset string `json:"dataset" yaml:"dataset"`
TableField string `json:"table_field" yaml:"table_field"`
TableKind string `json:"table_kind" yaml:"table_kind"`
RowRefField string `json:"row_ref_field" yaml:"row_ref_field"`
ExpandMasterFeat bool `json:"expand_masterfeat" yaml:"expand_masterfeat"`
}
func loadWikiVisibilityDefinitions(path string) (*wikiVisibilityDefinitions, error) {
raw, err := os.ReadFile(path)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, fmt.Errorf("read wiki visibility definitions %s: %w", path, err)
}
policy, err := parseWikiVisibilityDefinitions(raw, path)
if err != nil {
return nil, err
}
return &policy, nil
}
func parseWikiVisibilityDefinitions(raw []byte, path string) (wikiVisibilityDefinitions, error) {
var policy wikiVisibilityDefinitions
if err := yaml.Unmarshal(raw, &policy); err != nil {
return wikiVisibilityDefinitions{}, fmt.Errorf("parse wiki visibility definitions %s: %w", path, err)
}
if policy.ReferenceSets == nil {
policy.ReferenceSets = map[string]wikiVisibilityReferenceSet{}
}
if policy.Datasets == nil {
policy.Datasets = map[string]wikiVisibilityDataset{}
}
for dataset, definition := range policy.Datasets {
if !isWikiVisibilityDataset(dataset) {
return wikiVisibilityDefinitions{}, fmt.Errorf("wiki visibility definitions %s: unknown dataset %q", path, dataset)
}
if strings.TrimSpace(definition.Eligibility.When) != "" {
if err := validateWikiVisibilityExpression(definition.Eligibility.When); err != nil {
return wikiVisibilityDefinitions{}, fmt.Errorf("wiki visibility definitions %s: dataset %s eligibility: %w", path, dataset, err)
}
}
for _, rules := range [][]wikiVisibilityRule{definition.Include, definition.Exclude} {
for _, rule := range rules {
if strings.TrimSpace(rule.When) == "" && strings.TrimSpace(rule.ReferenceSet) == "" {
return wikiVisibilityDefinitions{}, fmt.Errorf("wiki visibility definitions %s: dataset %s rule requires when or reference_set", path, dataset)
}
if strings.TrimSpace(rule.When) != "" {
if err := validateWikiVisibilityExpression(rule.When); err != nil {
return wikiVisibilityDefinitions{}, fmt.Errorf("wiki visibility definitions %s: dataset %s rule: %w", path, dataset, err)
}
}
if set := strings.TrimSpace(rule.ReferenceSet); set != "" {
referenceSet, ok := policy.ReferenceSets[set]
if !ok {
return wikiVisibilityDefinitions{}, fmt.Errorf("wiki visibility definitions %s: dataset %s references unknown reference_set %q", path, dataset, set)
}
if referenceSet.TargetDataset != dataset {
return wikiVisibilityDefinitions{}, fmt.Errorf("wiki visibility definitions %s: dataset %s cannot use reference_set %q targeting %s", path, dataset, set, referenceSet.TargetDataset)
}
}
}
}
}
for name, referenceSet := range policy.ReferenceSets {
if !isWikiVisibilityDataset(referenceSet.TargetDataset) {
return wikiVisibilityDefinitions{}, fmt.Errorf("wiki visibility definitions %s: reference_set %s has unknown target_dataset %q", path, name, referenceSet.TargetDataset)
}
if len(referenceSet.Sources) == 0 {
return wikiVisibilityDefinitions{}, fmt.Errorf("wiki visibility definitions %s: reference_set %s requires sources", path, name)
}
for _, source := range referenceSet.Sources {
if !isWikiVisibilityDataset(source.Dataset) {
return wikiVisibilityDefinitions{}, fmt.Errorf("wiki visibility definitions %s: reference_set %s has unknown source dataset %q", path, name, source.Dataset)
}
if strings.TrimSpace(source.TableField) == "" || strings.TrimSpace(source.RowRefField) == "" {
return wikiVisibilityDefinitions{}, fmt.Errorf("wiki visibility definitions %s: reference_set %s source %s requires table_field and row_ref_field", path, name, source.Dataset)
}
switch source.TableKind {
case "class_feats", "class_bonus_feats", "race_feats":
default:
return wikiVisibilityDefinitions{}, fmt.Errorf("wiki visibility definitions %s: reference_set %s source %s has unsupported table_kind %q", path, name, source.Dataset, source.TableKind)
}
}
}
return policy, nil
}
func isWikiVisibilityDataset(name string) bool {
switch name {
case "baseitems", "classes", "feat", "racialtypes", "skills", "spells":
return true
default:
return false
}
}
func validateWikiVisibilityExpression(expression string) error {
parser := wikiExprParser{
tokens: tokenizeWikiExpr(wikiExpressionBody(expression)),
row: map[string]any{},
allowMissingFields: true,
}
if _, err := parser.parseComparison(); err != nil {
return err
}
if parser.peek().kind != wikiExprEOF {
return fmt.Errorf("unexpected token %q", parser.peek().text)
}
return nil
}
func wikiExpressionBody(expression string) string {
expression = strings.TrimSpace(expression)
if strings.HasPrefix(expression, "{{") && strings.HasSuffix(expression, "}}") {
return strings.TrimSpace(strings.TrimSuffix(strings.TrimPrefix(expression, "{{"), "}}"))
}
return expression
}
func (ctx *wikiContext) loadWikiVisibility(path string) error {
policy, err := loadWikiVisibilityDefinitions(path)
if err != nil {
return err
}
ctx.visibilityPolicy = policy
if policy == nil {
ctx.visibleWikiKeys = nil
return nil
}
ctx.visibleWikiKeys, err = ctx.buildWikiVisibilityIndex(policy)
return err
}
func (ctx *wikiContext) buildWikiVisibilityIndex(policy *wikiVisibilityDefinitions) (map[string]bool, error) {
visible := map[string]bool{}
for _, dataset := range []string{"classes", "racialtypes", "skills", "spells", "baseitems", "feat"} {
definition, ok := policy.Datasets[dataset]
if !ok {
continue
}
rows := ctx.wikiRowsForDataset(dataset)
for _, key := range sortedKeys(rows) {
row := rows[key]
include, err := ctx.evaluateWikiVisibilityWithoutSets(dataset, key, row, definition)
if err != nil {
return nil, err
}
visible[key] = include
}
}
referenceSets := ctx.collectWikiVisibilityReferenceSets(policy, visible)
previousVisible := ctx.visibleWikiKeys
ctx.visibleWikiKeys = visible
defer func() {
ctx.visibleWikiKeys = previousVisible
}()
for pass := 0; pass <= len(visible); pass++ {
changed := false
for _, dataset := range []string{"classes", "racialtypes", "skills", "spells", "baseitems", "feat"} {
definition, ok := policy.Datasets[dataset]
if !ok {
continue
}
rows := ctx.wikiRowsForDataset(dataset)
for _, key := range sortedKeys(rows) {
include, err := ctx.evaluateWikiVisibility(dataset, key, rows[key], definition, referenceSets)
if err != nil {
return nil, err
}
if visible[key] != include {
changed = true
}
visible[key] = include
}
}
if !changed {
return visible, nil
}
}
return nil, fmt.Errorf("wiki visibility index did not converge")
}
func (ctx *wikiContext) evaluateWikiVisibilityWithoutSets(dataset, key string, row map[string]any, definition wikiVisibilityDataset) (bool, error) {
return ctx.evaluateWikiVisibility(dataset, key, row, definition, map[string]map[string]struct{}{})
}
func (ctx *wikiContext) evaluateWikiVisibility(dataset, key string, row map[string]any, definition wikiVisibilityDataset, referenceSets map[string]map[string]struct{}) (bool, error) {
if row == nil || !ctx.wikiRowCanRender(dataset, row) {
return false, nil
}
if expression := strings.TrimSpace(definition.Eligibility.When); expression != "" {
eligible, err := ctx.evalWikiVisibilityPredicate(expression, row)
if err != nil {
return false, fmt.Errorf("evaluate wiki visibility dataset %s eligibility for %s: %w", dataset, key, err)
}
if !eligible {
return false, nil
}
}
included := len(definition.Include) == 0
for _, rule := range definition.Include {
match, err := ctx.matchWikiVisibilityRule(rule, key, row, referenceSets)
if err != nil {
return false, fmt.Errorf("evaluate wiki visibility dataset %s include for %s: %w", dataset, key, err)
}
included = included || match
}
excluded := false
for _, rule := range definition.Exclude {
match, err := ctx.matchWikiVisibilityRule(rule, key, row, referenceSets)
if err != nil {
return false, fmt.Errorf("evaluate wiki visibility dataset %s exclude for %s: %w", dataset, key, err)
}
excluded = excluded || match
}
switch ctx.wikiGenerateOverride(row) {
case "0":
return false, nil
case "1":
return true, nil
default:
return included && !excluded, nil
}
}
func (ctx *wikiContext) matchWikiVisibilityRule(rule wikiVisibilityRule, key string, row map[string]any, referenceSets map[string]map[string]struct{}) (bool, error) {
match := false
if expression := strings.TrimSpace(rule.When); expression != "" {
value, err := ctx.evalWikiVisibilityPredicate(expression, row)
if err != nil {
return false, err
}
match = match || value
}
if set := strings.TrimSpace(rule.ReferenceSet); set != "" {
_, ok := referenceSets[set][key]
match = match || ok
}
return match, nil
}
func (ctx *wikiContext) evalWikiVisibilityPredicate(expression string, row map[string]any) (bool, error) {
parser := wikiExprParser{
tokens: tokenizeWikiExpr(wikiExpressionBody(expression)),
row: row,
ctx: ctx,
allowMissingFields: true,
}
value, err := parser.parseComparison()
if err != nil {
return false, err
}
if parser.peek().kind != wikiExprEOF {
return false, fmt.Errorf("unexpected token %q", parser.peek().text)
}
return truthyWikiTableValue(value), nil
}
func (ctx *wikiContext) collectWikiVisibilityReferenceSets(policy *wikiVisibilityDefinitions, visible map[string]bool) map[string]map[string]struct{} {
out := map[string]map[string]struct{}{}
names := make([]string, 0, len(policy.ReferenceSets))
for name := range policy.ReferenceSets {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
keys := map[string]struct{}{}
for _, source := range policy.ReferenceSets[name].Sources {
for key, row := range ctx.wikiRowsForDataset(source.Dataset) {
if !visible[key] {
continue
}
table := ctx.tableForValue(fieldValue(row, source.TableField), ctx.wikiVisibilityTables(source.TableKind))
if table == nil {
continue
}
for _, tableRow := range table.Rows {
refKey := resolveReferenceKey(fieldValue(tableRow, source.RowRefField), ctx.wikiIDMapForDataset(policy.ReferenceSets[name].TargetDataset))
if refKey == "" {
refKey = resolveReferenceKey(fieldValue(tableRow, source.RowRefField), nil)
}
if refKey == "" {
continue
}
if source.ExpandMasterFeat && strings.HasPrefix(refKey, "masterfeats:") {
for _, featKey := range ctx.masterFeatGroupKeys(refKey) {
keys[featKey] = struct{}{}
}
continue
}
keys[refKey] = struct{}{}
}
}
}
out[name] = keys
}
return out
}
func (ctx *wikiContext) wikiRowsForDataset(dataset string) map[string]map[string]any {
switch dataset {
case "baseitems":
return ctx.baseitemRows
case "classes":
return ctx.classRows
case "feat":
return ctx.featRows
case "racialtypes":
return ctx.raceRows
case "skills":
return ctx.skillRows
case "spells":
return ctx.spellRows
default:
return nil
}
}
func (ctx *wikiContext) wikiIDMapForDataset(dataset string) map[int]string {
switch dataset {
case "classes":
return ctx.classIDToKey
case "feat":
return ctx.featIDToKey
case "racialtypes":
return ctx.raceIDToKey
case "skills":
return ctx.skillIDToKey
default:
return nil
}
}
func (ctx *wikiContext) wikiVisibilityTables(kind string) map[string]wikiTable {
switch kind {
case "class_feats":
return ctx.classFeatTables
case "class_bonus_feats":
return ctx.classBonusTables
case "race_feats":
return ctx.raceFeatTables
default:
return nil
}
}
func (ctx *wikiContext) masterFeatGroupKeys(group string) []string {
keys := []string{}
for key, row := range ctx.featRows {
if masterfeatGroupKey(fieldValue(row, "MASTERFEAT"), ctx.masterfeatRows) == group {
keys = append(keys, key)
}
}
sort.Strings(keys)
return keys
}
func (ctx *wikiContext) canReferenceWikiKey(key string) bool {
if key == "" || ctx.visibilityPolicy == nil {
return true
}
visible, configured := ctx.visibleWikiKeys[key]
if configured {
return visible
}
dataset := strings.SplitN(key, ":", 2)[0]
_, configured = ctx.visibilityPolicy.Datasets[dataset]
return !configured
}
internal/topdata/wingmodel_migrate.go
-86
View File
@@ -1,86 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyWingmodel(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "wingmodel")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "wingmodel")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
targetModulePaths := []string{
filepath.Join(targetModulesDir, "add.json"),
filepath.Join(targetModulesDir, "ovr_wingprefixes.json"),
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "wingmodel.2da"
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleNames := []string{"add.json", "ovr_wingprefixes.json"}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
{path: targetModulePaths[0], obj: moduleObjs[0]},
{path: targetModulePaths[1], obj: moduleObjs[1]},
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}