Compare commits
58
Commits
v0.2.3
...
9b7be2c76e
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9b7be2c76e | ||
|
|
56d4054118 | ||
|
|
682f920114 | ||
|
|
2f860ca9e4 | ||
|
|
1c2acc5530 | ||
|
|
fa32dd411f | ||
|
|
7437653f14 | ||
|
|
f9051a2c01 | ||
|
|
a131b25e5b | ||
|
|
4d03085996 | ||
|
|
2f4685e470 | ||
|
|
00f467b932 | ||
|
|
22eecd1a41 | ||
|
|
ed6945d308 | ||
|
|
3f500dabc8 | ||
|
|
8e7cead5c0 | ||
|
|
4d38967078 | ||
|
|
7b481ca0c0 | ||
|
|
b058846e16 | ||
|
|
e509b7a90a | ||
|
|
0d12674838 | ||
|
|
bc8fa3a6fe | ||
|
|
24e57457b0 | ||
|
|
43fe5e0373 | ||
|
|
4ee28fee4f | ||
|
|
5ebef57160 | ||
|
|
87fd3d8c04 | ||
|
|
e9860c12c6 | ||
|
|
825fff8b67 | ||
|
|
754375fb08 | ||
|
|
8a90713122 | ||
|
|
895f63a81c | ||
|
|
018b0f7686 | ||
|
|
cec3466779 | ||
|
|
9357b30994 | ||
|
|
448ebc74d4 | ||
|
|
bf787a6458 | ||
|
|
dd92379b68 | ||
|
|
06e5893734 | ||
|
|
332a24fd3e | ||
|
|
da6bf17f69 | ||
|
|
396236f242 | ||
|
|
f257672427 | ||
|
|
6afac1a4d9 | ||
|
|
79595e55be | ||
|
|
eefe00ebc5 | ||
|
|
eb5d15061c | ||
|
|
5c46824ebd | ||
|
|
2ea7959693 | ||
|
|
3315f8b7eb | ||
|
|
faee2cde95 | ||
|
|
d2f7c420ba | ||
|
|
97f4c00393 | ||
|
|
4bab09412d | ||
|
|
13c8ced5e8 | ||
|
|
f1fd03ee83 | ||
|
|
cdbbba3181 | ||
|
|
223b9831c5 |
@@ -0,0 +1,89 @@
|
|||||||
|
---
|
||||||
|
name: code-review
|
||||||
|
description: Review the changes since a fixed point (commit, branch, tag, or merge-base) along two axes — Standards (does the code follow this repo's documented coding standards?) and Spec (does the code match what the originating issue/PRD asked for?). Runs both reviews in parallel sub-agents and reports them side by side. Use when the user wants to review a branch, a PR, work-in-progress changes, or asks to "review since X".
|
||||||
|
---
|
||||||
|
|
||||||
|
Two-axis review of the diff between `HEAD` and a fixed point the user supplies:
|
||||||
|
|
||||||
|
- **Standards** — does the code conform to this repo's documented coding standards?
|
||||||
|
- **Spec** — does the code faithfully implement the originating issue / PRD / spec?
|
||||||
|
|
||||||
|
Both axes run as **parallel sub-agents** so they don't pollute each other's context, then this skill aggregates their findings.
|
||||||
|
|
||||||
|
The issue tracker should have been provided to you — run `/setup-matt-pocock-skills` if `docs/agents/issue-tracker.md` is missing.
|
||||||
|
|
||||||
|
## Process
|
||||||
|
|
||||||
|
### 1. Pin the fixed point
|
||||||
|
|
||||||
|
Whatever the user said is the fixed point — a commit SHA, branch name, tag, `main`, `HEAD~5`, etc. If they didn't specify one, ask for it.
|
||||||
|
|
||||||
|
Capture the diff command once: `git diff <fixed-point>...HEAD` (three-dot, so the comparison is against the merge-base). Also note the list of commits via `git log <fixed-point>..HEAD --oneline`.
|
||||||
|
|
||||||
|
Before going further, confirm the fixed point resolves (`git rev-parse <fixed-point>`) and the diff is non-empty. A bad ref or empty diff should fail here — not inside two parallel sub-agents.
|
||||||
|
|
||||||
|
### 2. Identify the spec source
|
||||||
|
|
||||||
|
Look for the originating spec, in this order:
|
||||||
|
|
||||||
|
1. Issue references in the commit messages (`#123`, `Closes #45`, GitLab `!67`, etc.) — fetch via the workflow in `docs/agents/issue-tracker.md`.
|
||||||
|
2. A path the user passed as an argument.
|
||||||
|
3. A PRD/spec file under `docs/`, `specs/`, or `.scratch/` matching the branch name or feature.
|
||||||
|
4. If nothing is found, ask the user where the spec is. If they say there isn't one, the **Spec** sub-agent will skip and report "no spec available".
|
||||||
|
|
||||||
|
### 3. Identify the standards sources
|
||||||
|
|
||||||
|
Anything in the repo that documents how code should be written, such as `CODING_STANDARDS.md` or `CONTRIBUTING.md`.
|
||||||
|
|
||||||
|
On top of whatever the repo documents, the Standards axis always carries the **smell baseline** below — a fixed set of Fowler code smells (_Refactoring_, ch.3) that applies even when a repo documents nothing. Two rules bind it:
|
||||||
|
|
||||||
|
- **The repo overrides.** A documented repo standard always wins; where it endorses something the baseline would flag, suppress the smell.
|
||||||
|
- **Always a judgement call.** Each smell is a labelled heuristic ("possible Feature Envy"), never a hard violation — and, like any standard here, skip anything tooling already enforces.
|
||||||
|
|
||||||
|
Each smell reads *what it is* → *how to fix*; match it against the diff:
|
||||||
|
|
||||||
|
- **Mysterious Name** — a function, variable, or type whose name doesn't reveal what it does or holds. → rename it; if no honest name comes, the design's murky.
|
||||||
|
- **Duplicated Code** — the same logic shape appears in more than one hunk or file in the change. → extract the shared shape, call it from both.
|
||||||
|
- **Feature Envy** — a method that reaches into another object's data more than its own. → move the method onto the data it envies.
|
||||||
|
- **Data Clumps** — the same few fields or params keep travelling together (a type wanting to be born). → bundle them into one type, pass that.
|
||||||
|
- **Primitive Obsession** — a primitive or string standing in for a domain concept that deserves its own type. → give the concept its own small type.
|
||||||
|
- **Repeated Switches** — the same `switch`/`if`-cascade on the same type recurs across the change. → replace with polymorphism, or one map both sites share.
|
||||||
|
- **Shotgun Surgery** — one logical change forces scattered edits across many files in the diff. → gather what changes together into one module.
|
||||||
|
- **Divergent Change** — one file or module is edited for several unrelated reasons. → split so each module changes for one reason.
|
||||||
|
- **Speculative Generality** — abstraction, parameters, or hooks added for needs the spec doesn't have. → delete it; inline back until a real need shows.
|
||||||
|
- **Message Chains** — long `a.b().c().d()` navigation the caller shouldn't depend on. → hide the walk behind one method on the first object.
|
||||||
|
- **Middle Man** — a class or function that mostly just delegates onward. → cut it, call the real target direct.
|
||||||
|
- **Refused Bequest** — a subclass or implementer that ignores or overrides most of what it inherits. → drop the inheritance, use composition.
|
||||||
|
|
||||||
|
### 4. Spawn both sub-agents in parallel
|
||||||
|
|
||||||
|
Send a single message with two `Agent` tool calls. Use the `general-purpose` subagent for both.
|
||||||
|
|
||||||
|
**Standards sub-agent prompt** — include:
|
||||||
|
|
||||||
|
- The full diff command and commit list.
|
||||||
|
- The list of standards-source files you found in step 3, **plus the smell baseline from step 3** pasted in full — the sub-agent has no other access to it.
|
||||||
|
- The brief: "Report — per file/hunk where relevant — (a) every place the diff violates a documented standard: cite the standard (file + the rule); and (b) any baseline smell you spot: name it and quote the hunk. Distinguish hard violations from judgement calls — documented-standard breaches can be hard, but baseline smells are always judgement calls, and a documented repo standard overrides the baseline. Skip anything tooling enforces. Under 400 words."
|
||||||
|
|
||||||
|
**Spec sub-agent prompt** — include:
|
||||||
|
|
||||||
|
- The diff command and commit list.
|
||||||
|
- The path or fetched contents of the spec.
|
||||||
|
- The brief: "Report: (a) requirements the spec asked for that are missing or partial; (b) behaviour in the diff that wasn't asked for (scope creep); (c) requirements that look implemented but where the implementation looks wrong. Quote the spec line for each finding. Under 400 words."
|
||||||
|
|
||||||
|
If the spec is missing, skip the Spec sub-agent and note this in the final report.
|
||||||
|
|
||||||
|
### 5. Aggregate
|
||||||
|
|
||||||
|
Present the two reports under `## Standards` and `## Spec` headings, verbatim or lightly cleaned. Do **not** merge or rerank findings — the two axes are deliberately separate (see _Why two axes_).
|
||||||
|
|
||||||
|
End with a one-line summary: total findings per axis, and the worst issue _within each axis_ (if any). Don't pick a single winner across axes — that's the reranking the separation exists to prevent.
|
||||||
|
|
||||||
|
## Why two axes
|
||||||
|
|
||||||
|
A change can pass one axis and fail the other:
|
||||||
|
|
||||||
|
- Code that follows every standard but implements the wrong thing → **Standards pass, Spec fail.**
|
||||||
|
- Code that does exactly what the issue asked but breaks the project's conventions → **Spec pass, Standards fail.**
|
||||||
|
|
||||||
|
Reporting them separately stops one axis from masking the other.
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
# Deepening
|
||||||
|
|
||||||
|
How to deepen a cluster of shallow modules safely, given its dependencies. Assumes the vocabulary in [SKILL.md](SKILL.md) — **module**, **interface**, **seam**, **adapter**.
|
||||||
|
|
||||||
|
## Dependency categories
|
||||||
|
|
||||||
|
When assessing a candidate for deepening, classify its dependencies. The category determines how the deepened module is tested across its seam.
|
||||||
|
|
||||||
|
### 1. In-process
|
||||||
|
|
||||||
|
Pure computation, in-memory state, no I/O. Always deepenable — merge the modules and test through the new interface directly. No adapter needed.
|
||||||
|
|
||||||
|
### 2. Local-substitutable
|
||||||
|
|
||||||
|
Dependencies that have local test stand-ins (PGLite for Postgres, in-memory filesystem). Deepenable if the stand-in exists. The deepened module is tested with the stand-in running in the test suite. The seam is internal; no port at the module's external interface.
|
||||||
|
|
||||||
|
### 3. Remote but owned (Ports & Adapters)
|
||||||
|
|
||||||
|
Your own services across a network boundary (microservices, internal APIs). Define a **port** (interface) at the seam. The deep module owns the logic; the transport is injected as an **adapter**. Tests use an in-memory adapter. Production uses an HTTP/gRPC/queue adapter.
|
||||||
|
|
||||||
|
Recommendation shape: *"Define a port at the seam, implement an HTTP adapter for production and an in-memory adapter for testing, so the logic sits in one deep module even though it's deployed across a network."*
|
||||||
|
|
||||||
|
### 4. True external (Mock)
|
||||||
|
|
||||||
|
Third-party services (Stripe, Twilio, etc.) you don't control. The deepened module takes the external dependency as an injected port; tests provide a mock adapter.
|
||||||
|
|
||||||
|
## Seam discipline
|
||||||
|
|
||||||
|
- **One adapter means a hypothetical seam. Two adapters means a real one.** Don't introduce a port unless at least two adapters are justified (typically production + test). A single-adapter seam is just indirection.
|
||||||
|
- **Internal seams vs external seams.** A deep module can have internal seams (private to its implementation, used by its own tests) as well as the external seam at its interface. Don't expose internal seams through the interface just because tests use them.
|
||||||
|
|
||||||
|
## Testing strategy: replace, don't layer
|
||||||
|
|
||||||
|
- Old unit tests on shallow modules become waste once tests at the deepened module's interface exist — delete them.
|
||||||
|
- Write new tests at the deepened module's interface. The **interface is the test surface**.
|
||||||
|
- Tests assert on observable outcomes through the interface, not internal state.
|
||||||
|
- Tests should survive internal refactors — they describe behaviour, not implementation. If a test has to change when the implementation changes, it's testing past the interface.
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# Design It Twice
|
||||||
|
|
||||||
|
When the user wants to explore alternative interfaces for a chosen deepening candidate, use this parallel sub-agent pattern. Based on "Design It Twice" (Ousterhout) — your first idea is unlikely to be the best.
|
||||||
|
|
||||||
|
Uses the vocabulary in [SKILL.md](SKILL.md) — **module**, **interface**, **seam**, **adapter**, **leverage**.
|
||||||
|
|
||||||
|
## Process
|
||||||
|
|
||||||
|
### 1. Frame the problem space
|
||||||
|
|
||||||
|
Before spawning sub-agents, write a user-facing explanation of the problem space for the chosen candidate:
|
||||||
|
|
||||||
|
- The constraints any new interface would need to satisfy
|
||||||
|
- The dependencies it would rely on, and which category they fall into (see [DEEPENING.md](DEEPENING.md))
|
||||||
|
- A rough illustrative code sketch to ground the constraints — not a proposal, just a way to make the constraints concrete
|
||||||
|
|
||||||
|
Show this to the user, then immediately proceed to Step 2. The user reads and thinks while the sub-agents work in parallel.
|
||||||
|
|
||||||
|
### 2. Spawn sub-agents
|
||||||
|
|
||||||
|
Spawn 3+ sub-agents in parallel using the Agent tool. Each must produce a **radically different** interface for the deepened module.
|
||||||
|
|
||||||
|
Prompt each sub-agent with a separate technical brief (file paths, coupling details, dependency category from [DEEPENING.md](DEEPENING.md), what sits behind the seam). The brief is independent of the user-facing problem-space explanation in Step 1. Give each agent a different design constraint:
|
||||||
|
|
||||||
|
- Agent 1: "Minimize the interface — aim for 1–3 entry points max. Maximise leverage per entry point."
|
||||||
|
- Agent 2: "Maximise flexibility — support many use cases and extension."
|
||||||
|
- Agent 3: "Optimise for the most common caller — make the default case trivial."
|
||||||
|
- Agent 4 (if applicable): "Design around ports & adapters for cross-seam dependencies."
|
||||||
|
|
||||||
|
Include both [SKILL.md](SKILL.md) vocabulary and CONTEXT.md vocabulary in the brief so each sub-agent names things consistently with the architecture language and the project's domain language.
|
||||||
|
|
||||||
|
Each sub-agent outputs:
|
||||||
|
|
||||||
|
1. Interface (types, methods, params — plus invariants, ordering, error modes)
|
||||||
|
2. Usage example showing how callers use it
|
||||||
|
3. What the implementation hides behind the seam
|
||||||
|
4. Dependency strategy and adapters (see [DEEPENING.md](DEEPENING.md))
|
||||||
|
5. Trade-offs — where leverage is high, where it's thin
|
||||||
|
|
||||||
|
### 3. Present and compare
|
||||||
|
|
||||||
|
Present designs sequentially so the user can absorb each one, then compare them in prose. Contrast by **depth** (leverage at the interface), **locality** (where change concentrates), and **seam placement**.
|
||||||
|
|
||||||
|
After comparing, give your own recommendation: which design you think is strongest and why. If elements from different designs would combine well, propose a hybrid. Be opinionated — the user wants a strong read, not a menu.
|
||||||
@@ -0,0 +1,114 @@
|
|||||||
|
---
|
||||||
|
name: codebase-design
|
||||||
|
description: Shared vocabulary for designing deep modules. Use when the user wants to design or improve a module's interface, find deepening opportunities, decide where a seam goes, make code more testable or AI-navigable, or when another skill needs the deep-module vocabulary.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Codebase Design
|
||||||
|
|
||||||
|
Design **deep modules**: a lot of behaviour behind a small interface, placed at a clean seam, testable through that interface. Use this language and these principles wherever code is being designed or restructured. The aim is leverage for callers, locality for maintainers, and testability for everyone.
|
||||||
|
|
||||||
|
## Glossary
|
||||||
|
|
||||||
|
Use these terms exactly — don't substitute "component," "service," "API," or "boundary." Consistent language is the whole point.
|
||||||
|
|
||||||
|
**Module** — anything with an interface and an implementation. Deliberately scale-agnostic: a function, class, package, or tier-spanning slice. _Avoid_: unit, component, service.
|
||||||
|
|
||||||
|
**Interface** — everything a caller must know to use the module correctly: the type signature, but also invariants, ordering constraints, error modes, required configuration, and performance characteristics. _Avoid_: API, signature (too narrow — they refer only to the type-level surface).
|
||||||
|
|
||||||
|
**Implementation** — what's inside a module, its body of code. Distinct from **Adapter**: a thing can be a small adapter with a large implementation (a Postgres repo) or a large adapter with a small implementation (an in-memory fake). Reach for "adapter" when the seam is the topic; "implementation" otherwise.
|
||||||
|
|
||||||
|
**Depth** — leverage at the interface: the amount of behaviour a caller (or test) can exercise per unit of interface they have to learn. A module is **deep** when a large amount of behaviour sits behind a small interface, **shallow** when the interface is nearly as complex as the implementation.
|
||||||
|
|
||||||
|
**Seam** _(Michael Feathers)_ — a place where you can alter behaviour without editing in that place; the *location* at which a module's interface lives. Where to put the seam is its own design decision, distinct from what goes behind it. _Avoid_: boundary (overloaded with DDD's bounded context).
|
||||||
|
|
||||||
|
**Adapter** — a concrete thing that satisfies an interface at a seam. Describes *role* (what slot it fills), not substance (what's inside).
|
||||||
|
|
||||||
|
**Leverage** — what callers get from depth: more capability per unit of interface they learn. One implementation pays back across N call sites and M tests.
|
||||||
|
|
||||||
|
**Locality** — what maintainers get from depth: change, bugs, knowledge, and verification concentrate in one place rather than spreading across callers. Fix once, fixed everywhere.
|
||||||
|
|
||||||
|
## Deep vs shallow
|
||||||
|
|
||||||
|
**Deep module** = small interface + lots of implementation:
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────┐
|
||||||
|
│ Small Interface │ ← Few methods, simple params
|
||||||
|
├─────────────────────┤
|
||||||
|
│ │
|
||||||
|
│ Deep Implementation│ ← Complex logic hidden
|
||||||
|
│ │
|
||||||
|
└─────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
**Shallow module** = large interface + little implementation (avoid):
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────┐
|
||||||
|
│ Large Interface │ ← Many methods, complex params
|
||||||
|
├─────────────────────────────────┤
|
||||||
|
│ Thin Implementation │ ← Just passes through
|
||||||
|
└─────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
When designing an interface, ask:
|
||||||
|
|
||||||
|
- Can I reduce the number of methods?
|
||||||
|
- Can I simplify the parameters?
|
||||||
|
- Can I hide more complexity inside?
|
||||||
|
|
||||||
|
## Principles
|
||||||
|
|
||||||
|
- **Depth is a property of the interface, not the implementation.** A deep module can be internally composed of small, mockable, swappable parts — they just aren't part of the interface. A module can have **internal seams** (private to its implementation, used by its own tests) as well as the **external seam** at its interface.
|
||||||
|
- **The deletion test.** Imagine deleting the module. If complexity vanishes, it was a pass-through. If complexity reappears across N callers, it was earning its keep.
|
||||||
|
- **The interface is the test surface.** Callers and tests cross the same seam. If you want to test *past* the interface, the module is probably the wrong shape.
|
||||||
|
- **One adapter means a hypothetical seam. Two adapters means a real one.** Don't introduce a seam unless something actually varies across it.
|
||||||
|
|
||||||
|
## Designing for testability
|
||||||
|
|
||||||
|
Good interfaces make testing natural:
|
||||||
|
|
||||||
|
1. **Accept dependencies, don't create them.**
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Testable
|
||||||
|
function processOrder(order, paymentGateway) {}
|
||||||
|
|
||||||
|
// Hard to test
|
||||||
|
function processOrder(order) {
|
||||||
|
const gateway = new StripeGateway();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Return results, don't produce side effects.**
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Testable
|
||||||
|
function calculateDiscount(cart): Discount {}
|
||||||
|
|
||||||
|
// Hard to test
|
||||||
|
function applyDiscount(cart): void {
|
||||||
|
cart.total -= discount;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Small surface area.** Fewer methods = fewer tests needed. Fewer params = simpler test setup.
|
||||||
|
|
||||||
|
## Relationships
|
||||||
|
|
||||||
|
- A **Module** has exactly one **Interface** (the surface it presents to callers and tests).
|
||||||
|
- **Depth** is a property of a **Module**, measured against its **Interface**.
|
||||||
|
- A **Seam** is where a **Module**'s **Interface** lives.
|
||||||
|
- An **Adapter** sits at a **Seam** and satisfies the **Interface**.
|
||||||
|
- **Depth** produces **Leverage** for callers and **Locality** for maintainers.
|
||||||
|
|
||||||
|
## Rejected framings
|
||||||
|
|
||||||
|
- **Depth as ratio of implementation-lines to interface-lines** (Ousterhout): rewards padding the implementation. We use depth-as-leverage instead.
|
||||||
|
- **"Interface" as the TypeScript `interface` keyword or a class's public methods**: too narrow — interface here includes every fact a caller must know.
|
||||||
|
- **"Boundary"**: overloaded with DDD's bounded context. Say **seam** or **interface**.
|
||||||
|
|
||||||
|
## Going deeper
|
||||||
|
|
||||||
|
- **Deepening a cluster given its dependencies** — see [DEEPENING.md](DEEPENING.md): dependency categories, seam discipline, and replace-don't-layer testing.
|
||||||
|
- **Exploring alternative interfaces** — see [DESIGN-IT-TWICE.md](DESIGN-IT-TWICE.md): spin up parallel sub-agents to design the interface several radically different ways, then compare on depth, locality, and seam placement.
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
---
|
||||||
|
name: data-driven-design
|
||||||
|
description: Use when adding a mode, switch case, config entry, special-cased name, or per-column/per-dataset handler to a tool that processes authored data — or when a tool "knows" a project's layout, dataset names, key prefixes, or format quirks and could not be open sourced as-is.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Data-Driven Design
|
||||||
|
|
||||||
|
The tool implements **general mechanisms**; the data declares **all specific knowledge**. Every time code names a thing that lives in data — a dataset, a column, a format quirk — the tool stops being a tool and becomes a private extension of one project's layout.
|
||||||
|
|
||||||
|
**The test:** could this repo be open sourced and build a *different* project's data without code changes? Every hardcoded name, mode, and special case is a "no".
|
||||||
|
|
||||||
|
## The two smells
|
||||||
|
|
||||||
|
**1. The mode zoo.** A new data shape arrives and you add a named mode plus a switch case:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- {column: AvailableHeadsMale, mode: external_hex_list, hex_width: 3}
|
||||||
|
- {column: PreferredAlignments, mode: alignment_set_mask, hex_width: 3}
|
||||||
|
- {column: ProficiencyFeats, mode: id_hex_list, hex_width: 4}
|
||||||
|
```
|
||||||
|
|
||||||
|
Three "modes" are one concept — *a set of ids with an engine-side encoding* — wearing three names. Each mode is code the data could have been. The fix is a smaller vocabulary of orthogonal primitives declared where the column is defined:
|
||||||
|
|
||||||
|
```json
|
||||||
|
"EquipSlots": {"shape": "id_set", "encode": "bitmask", "max": 31}
|
||||||
|
```
|
||||||
|
|
||||||
|
One parser for `id_set`, one encoder per encoding. Adding the next column is a data edit, zero code. The mode zoo *shrinks*: N special cases collapse into shape × encoding.
|
||||||
|
|
||||||
|
**2. Names in code.** `datasetRows("skills")`, `case "skills":`, `TrimPrefix(key, "skills:")` — the tool assumes the project's layout. When data moves (`skills` → `skills/core`), the tool breaks even though the data is self-consistent. Names must flow *from* the data: the spec that references a dataset names it; a key's namespace is whatever precedes its own colon. If code needs a name, some piece of data already knows it — read it from there.
|
||||||
|
|
||||||
|
## Deciding
|
||||||
|
|
||||||
|
Ask of every constant, case, and config knob: **whose knowledge is this?**
|
||||||
|
|
||||||
|
- *The mechanism's* (how to parse JSON, allocate rows, write a table) → code.
|
||||||
|
- *The data's* (which columns exist, how the engine encodes them, what things are called) → data, even if code is faster to write today.
|
||||||
|
|
||||||
|
Generalize only when it deletes: a primitive that replaces N modes is lazy; a framework "for future shapes" is not. One new shape may take its case *if* the case is spelled as a reusable primitive the next shape can declare.
|
||||||
|
|
||||||
|
## Rationalizations
|
||||||
|
|
||||||
|
| Excuse | Reality |
|
||||||
|
|---|---|
|
||||||
|
| "Just follow the existing pattern, it's proven" | The pattern *is* the defect. Each repetition raises the cost of the real fix. |
|
||||||
|
| "N cases isn't a crisis yet" | The crisis is per-case: every one couples the tool to one project forever. |
|
||||||
|
| "Release is tonight, generalize later" | Later never comes; tonight's mode is tomorrow's baseline. Declaring shape in data is usually the *same* hour of work. |
|
||||||
|
| "This name never changes" | Today's rename broke exactly such a name. Data moves; mechanisms shouldn't care. |
|
||||||
|
|
||||||
|
## Red flags — stop and re-shape
|
||||||
|
|
||||||
|
- A config entry that names both a dataset and a column to select behavior
|
||||||
|
- A new value in a `mode`/`kind`/`type` string enum with its own switch arm
|
||||||
|
- A literal in code that also appears in the data tree (`"skills"`, `"skills:"`)
|
||||||
|
- Per-project constants (row ranges, hex widths, id ceilings) living in Go instead of the dataset that owns them
|
||||||
@@ -0,0 +1,134 @@
|
|||||||
|
---
|
||||||
|
name: diagnosing-bugs
|
||||||
|
description: Diagnosis loop for hard bugs and performance regressions. Use when the user says "diagnose"/"debug this", or reports something broken/throwing/failing/slow.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Diagnosing Bugs
|
||||||
|
|
||||||
|
A discipline for hard bugs. Skip phases only when explicitly justified.
|
||||||
|
|
||||||
|
When exploring the codebase, read `CONTEXT.md` (if it exists) to get a clear mental model of the relevant modules, and check ADRs in the area you're touching.
|
||||||
|
|
||||||
|
## Phase 1 — Build a feedback loop
|
||||||
|
|
||||||
|
**This is the skill.** Everything else is mechanical. If you have a **tight** pass/fail signal for the bug — one that goes red on _this_ bug — you will find the cause; bisection, hypothesis-testing, and instrumentation all just consume it. If you don't have one, no amount of staring at code will save you.
|
||||||
|
|
||||||
|
Spend disproportionate effort here. **Be aggressive. Be creative. Refuse to give up.**
|
||||||
|
|
||||||
|
### Ways to construct one — try them in roughly this order
|
||||||
|
|
||||||
|
1. **Failing test** at whatever seam reaches the bug — unit, integration, e2e.
|
||||||
|
2. **Curl / HTTP script** against a running dev server.
|
||||||
|
3. **CLI invocation** with a fixture input, diffing stdout against a known-good snapshot.
|
||||||
|
4. **Headless browser script** (Playwright / Puppeteer) — drives the UI, asserts on DOM/console/network.
|
||||||
|
5. **Replay a captured trace.** Save a real network request / payload / event log to disk; replay it through the code path in isolation.
|
||||||
|
6. **Throwaway harness.** Spin up a minimal subset of the system (one service, mocked deps) that exercises the bug code path with a single function call.
|
||||||
|
7. **Property / fuzz loop.** If the bug is "sometimes wrong output", run 1000 random inputs and look for the failure mode.
|
||||||
|
8. **Bisection harness.** If the bug appeared between two known states (commit, dataset, version), automate "boot at state X, check, repeat" so you can `git bisect run` it.
|
||||||
|
9. **Differential loop.** Run the same input through old-version vs new-version (or two configs) and diff outputs.
|
||||||
|
10. **HITL bash script.** Last resort. If a human must click, drive _them_ with `scripts/hitl-loop.template.sh` so the loop is still structured. Captured output feeds back to you.
|
||||||
|
|
||||||
|
Build the right feedback loop, and the bug is 90% fixed.
|
||||||
|
|
||||||
|
### Tighten the loop
|
||||||
|
|
||||||
|
Treat the loop as a product. Once you have _a_ loop, **tighten** it:
|
||||||
|
|
||||||
|
- Can I make it faster? (Cache setup, skip unrelated init, narrow the test scope.)
|
||||||
|
- Can I make the signal sharper? (Assert on the specific symptom, not "didn't crash".)
|
||||||
|
- Can I make it more deterministic? (Pin time, seed RNG, isolate filesystem, freeze network.)
|
||||||
|
|
||||||
|
A 30-second flaky loop is barely better than no loop; a 2-second deterministic one is tight — a debugging superpower.
|
||||||
|
|
||||||
|
### Non-deterministic bugs
|
||||||
|
|
||||||
|
The goal is not a clean repro but a **higher reproduction rate**. Loop the trigger 100×, parallelise, add stress, narrow timing windows, inject sleeps. A 50%-flake bug is debuggable; 1% is not — keep raising the rate until it's debuggable.
|
||||||
|
|
||||||
|
### When you genuinely cannot build a loop
|
||||||
|
|
||||||
|
Stop and say so explicitly. List what you tried. Ask the user for: (a) access to whatever environment reproduces it, (b) a captured artifact (HAR file, log dump, core dump, screen recording with timestamps), or (c) permission to add temporary production instrumentation. Do **not** proceed to hypothesise without a loop.
|
||||||
|
|
||||||
|
### Completion criterion — a tight loop that goes red
|
||||||
|
|
||||||
|
Phase 1 is done when the loop is **tight** and **red-capable**: you can name **one command** — a script path, a test invocation, a curl — that you have **already run at least once** (paste the invocation and its output), and that is:
|
||||||
|
|
||||||
|
- [ ] **Red-capable** — it drives the actual bug code path and asserts the **user's exact symptom**, so it can go red on this bug and green once fixed. Not "runs without erroring" — it must be able to _catch this specific bug_.
|
||||||
|
- [ ] **Deterministic** — same verdict every run (flaky bugs: a pinned, high reproduction rate, per above).
|
||||||
|
- [ ] **Fast** — seconds, not minutes.
|
||||||
|
- [ ] **Agent-runnable** — you can run it unattended; a human in the loop only via `scripts/hitl-loop.template.sh`.
|
||||||
|
|
||||||
|
If you catch yourself reading code to build a theory before this command exists, **stop — jumping straight to a hypothesis is the exact failure this skill prevents.** No red-capable command, no Phase 2.
|
||||||
|
|
||||||
|
## Phase 2 — Reproduce + minimise
|
||||||
|
|
||||||
|
Run the loop. Watch it go red — the bug appears.
|
||||||
|
|
||||||
|
Confirm:
|
||||||
|
|
||||||
|
- [ ] The loop produces the failure mode the **user** described — not a different failure that happens to be nearby. Wrong bug = wrong fix.
|
||||||
|
- [ ] The failure is reproducible across multiple runs (or, for non-deterministic bugs, reproducible at a high enough rate to debug against).
|
||||||
|
- [ ] You have captured the exact symptom (error message, wrong output, slow timing) so later phases can verify the fix actually addresses it.
|
||||||
|
|
||||||
|
### Minimise
|
||||||
|
|
||||||
|
Once it's red, shrink the repro to the **smallest scenario that still goes red**. Cut inputs, callers, config, data, and steps **one at a time**, re-running the loop after each cut — keep only what's load-bearing for the failure.
|
||||||
|
|
||||||
|
Why bother: a minimal repro shrinks the hypothesis space in Phase 3 (fewer moving parts left to suspect) and becomes the clean regression test in Phase 5.
|
||||||
|
|
||||||
|
Done when **every remaining element is load-bearing** — removing any one of them makes the loop go green.
|
||||||
|
|
||||||
|
Do not proceed until you have reproduced **and** minimised.
|
||||||
|
|
||||||
|
## Phase 3 — Hypothesise
|
||||||
|
|
||||||
|
Generate **3–5 ranked hypotheses** before testing any of them. Single-hypothesis generation anchors on the first plausible idea.
|
||||||
|
|
||||||
|
Each hypothesis must be **falsifiable**: state the prediction it makes.
|
||||||
|
|
||||||
|
> Format: "If <X> is the cause, then <changing Y> will make the bug disappear / <changing Z> will make it worse."
|
||||||
|
|
||||||
|
If you cannot state the prediction, the hypothesis is a vibe — discard or sharpen it.
|
||||||
|
|
||||||
|
**Show the ranked list to the user before testing.** They often have domain knowledge that re-ranks instantly ("we just deployed a change to #3"), or know hypotheses they've already ruled out. Cheap checkpoint, big time saver. Don't block on it — proceed with your ranking if the user is AFK.
|
||||||
|
|
||||||
|
## Phase 4 — Instrument
|
||||||
|
|
||||||
|
Each probe must map to a specific prediction from Phase 3. **Change one variable at a time.**
|
||||||
|
|
||||||
|
Tool preference:
|
||||||
|
|
||||||
|
1. **Debugger / REPL inspection** if the env supports it. One breakpoint beats ten logs.
|
||||||
|
2. **Targeted logs** at the boundaries that distinguish hypotheses.
|
||||||
|
3. Never "log everything and grep".
|
||||||
|
|
||||||
|
**Tag every debug log** with a unique prefix, e.g. `[DEBUG-a4f2]`. Cleanup at the end becomes a single grep. Untagged logs survive; tagged logs die.
|
||||||
|
|
||||||
|
**Perf branch.** For performance regressions, logs are usually wrong. Instead: establish a baseline measurement (timing harness, `performance.now()`, profiler, query plan), then bisect. Measure first, fix second.
|
||||||
|
|
||||||
|
## Phase 5 — Fix + regression test
|
||||||
|
|
||||||
|
Write the regression test **before the fix** — but only if there is a **correct seam** for it.
|
||||||
|
|
||||||
|
A correct seam is one where the test exercises the **real bug pattern** as it occurs at the call site. If the only available seam is too shallow (single-caller test when the bug needs multiple callers, unit test that can't replicate the chain that triggered the bug), a regression test there gives false confidence.
|
||||||
|
|
||||||
|
**If no correct seam exists, that itself is the finding.** Note it. The codebase architecture is preventing the bug from being locked down. Flag this for the next phase.
|
||||||
|
|
||||||
|
If a correct seam exists:
|
||||||
|
|
||||||
|
1. Turn the minimised repro into a failing test at that seam.
|
||||||
|
2. Watch it fail.
|
||||||
|
3. Apply the fix.
|
||||||
|
4. Watch it pass.
|
||||||
|
5. Re-run the Phase 1 feedback loop against the original (un-minimised) scenario.
|
||||||
|
|
||||||
|
## Phase 6 — Cleanup + post-mortem
|
||||||
|
|
||||||
|
Required before declaring done:
|
||||||
|
|
||||||
|
- [ ] Original repro no longer reproduces (re-run the Phase 1 loop)
|
||||||
|
- [ ] Regression test passes (or absence of seam is documented)
|
||||||
|
- [ ] All `[DEBUG-...]` instrumentation removed (`grep` the prefix)
|
||||||
|
- [ ] Throwaway prototypes deleted (or moved to a clearly-marked debug location)
|
||||||
|
- [ ] The hypothesis that turned out correct is stated in the commit / PR message — so the next debugger learns
|
||||||
|
|
||||||
|
**Then ask: what would have prevented this bug?** If the answer involves architectural change (no good test seam, tangled callers, hidden coupling) hand off to the `/improve-codebase-architecture` skill with the specifics. Make the recommendation **after** the fix is in, not before — you have more information now than when you started.
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Human-in-the-loop reproduction loop.
|
||||||
|
# Copy this file, edit the steps below, and run it.
|
||||||
|
# The agent runs the script; the user follows prompts in their terminal.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# bash hitl-loop.template.sh
|
||||||
|
#
|
||||||
|
# Two helpers:
|
||||||
|
# step "<instruction>" → show instruction, wait for Enter
|
||||||
|
# capture VAR "<question>" → show question, read response into VAR
|
||||||
|
#
|
||||||
|
# At the end, captured values are printed as KEY=VALUE for the agent to parse.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
step() {
|
||||||
|
printf '\n>>> %s\n' "$1"
|
||||||
|
read -r -p " [Enter when done] " _
|
||||||
|
}
|
||||||
|
|
||||||
|
capture() {
|
||||||
|
local var="$1" question="$2" answer
|
||||||
|
printf '\n>>> %s\n' "$question"
|
||||||
|
read -r -p " > " answer
|
||||||
|
printf -v "$var" '%s' "$answer"
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- edit below ---------------------------------------------------------
|
||||||
|
|
||||||
|
step "Open the app at http://localhost:3000 and sign in."
|
||||||
|
|
||||||
|
capture ERRORED "Click the 'Export' button. Did it throw an error? (y/n)"
|
||||||
|
|
||||||
|
capture ERROR_MSG "Paste the error message (or 'none'):"
|
||||||
|
|
||||||
|
# --- edit above ---------------------------------------------------------
|
||||||
|
|
||||||
|
printf '\n--- Captured ---\n'
|
||||||
|
printf 'ERRORED=%s\n' "$ERRORED"
|
||||||
|
printf 'ERROR_MSG=%s\n' "$ERROR_MSG"
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
# ADR Format
|
||||||
|
|
||||||
|
ADRs live in `docs/adr/` and use sequential numbering: `0001-slug.md`, `0002-slug.md`, etc.
|
||||||
|
|
||||||
|
Create the `docs/adr/` directory lazily — only when the first ADR is needed.
|
||||||
|
|
||||||
|
## Template
|
||||||
|
|
||||||
|
```md
|
||||||
|
# {Short title of the decision}
|
||||||
|
|
||||||
|
{1-3 sentences: what's the context, what did we decide, and why.}
|
||||||
|
```
|
||||||
|
|
||||||
|
That's it. An ADR can be a single paragraph. The value is in recording *that* a decision was made and *why* — not in filling out sections.
|
||||||
|
|
||||||
|
## Optional sections
|
||||||
|
|
||||||
|
Only include these when they add genuine value. Most ADRs won't need them.
|
||||||
|
|
||||||
|
- **Status** frontmatter (`proposed | accepted | deprecated | superseded by ADR-NNNN`) — useful when decisions are revisited
|
||||||
|
- **Considered Options** — only when the rejected alternatives are worth remembering
|
||||||
|
- **Consequences** — only when non-obvious downstream effects need to be called out
|
||||||
|
|
||||||
|
## Numbering
|
||||||
|
|
||||||
|
Scan `docs/adr/` for the highest existing number and increment by one.
|
||||||
|
|
||||||
|
## When to offer an ADR
|
||||||
|
|
||||||
|
All three of these must be true:
|
||||||
|
|
||||||
|
1. **Hard to reverse** — the cost of changing your mind later is meaningful
|
||||||
|
2. **Surprising without context** — a future reader will look at the code and wonder "why on earth did they do it this way?"
|
||||||
|
3. **The result of a real trade-off** — there were genuine alternatives and you picked one for specific reasons
|
||||||
|
|
||||||
|
If a decision is easy to reverse, skip it — you'll just reverse it. If it's not surprising, nobody will wonder why. If there was no real alternative, there's nothing to record beyond "we did the obvious thing."
|
||||||
|
|
||||||
|
### What qualifies
|
||||||
|
|
||||||
|
- **Architectural shape.** "We're using a monorepo." "The write model is event-sourced, the read model is projected into Postgres."
|
||||||
|
- **Integration patterns between contexts.** "Ordering and Billing communicate via domain events, not synchronous HTTP."
|
||||||
|
- **Technology choices that carry lock-in.** Database, message bus, auth provider, deployment target. Not every library — just the ones that would take a quarter to swap out.
|
||||||
|
- **Boundary and scope decisions.** "Customer data is owned by the Customer context; other contexts reference it by ID only." The explicit no-s are as valuable as the yes-s.
|
||||||
|
- **Deliberate deviations from the obvious path.** "We're using manual SQL instead of an ORM because X." Anything where a reasonable reader would assume the opposite. These stop the next engineer from "fixing" something that was deliberate.
|
||||||
|
- **Constraints not visible in the code.** "We can't use AWS because of compliance requirements." "Response times must be under 200ms because of the partner API contract."
|
||||||
|
- **Rejected alternatives when the rejection is non-obvious.** If you considered GraphQL and picked REST for subtle reasons, record it — otherwise someone will suggest GraphQL again in six months.
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
# CONTEXT.md Format
|
||||||
|
|
||||||
|
## Structure
|
||||||
|
|
||||||
|
```md
|
||||||
|
# {Context Name}
|
||||||
|
|
||||||
|
{One or two sentence description of what this context is and why it exists.}
|
||||||
|
|
||||||
|
## Language
|
||||||
|
|
||||||
|
**Order**:
|
||||||
|
{A one or two sentence description of the term}
|
||||||
|
_Avoid_: Purchase, transaction
|
||||||
|
|
||||||
|
**Invoice**:
|
||||||
|
A request for payment sent to a customer after delivery.
|
||||||
|
_Avoid_: Bill, payment request
|
||||||
|
|
||||||
|
**Customer**:
|
||||||
|
A person or organization that places orders.
|
||||||
|
_Avoid_: Client, buyer, account
|
||||||
|
```
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
- **Be opinionated.** When multiple words exist for the same concept, pick the best one and list the others under `_Avoid_`.
|
||||||
|
- **Keep definitions tight.** One or two sentences max. Define what it IS, not what it does.
|
||||||
|
- **Only include terms specific to this project's context.** General programming concepts (timeouts, error types, utility patterns) don't belong even if the project uses them extensively. Before adding a term, ask: is this a concept unique to this context, or a general programming concept? Only the former belongs.
|
||||||
|
- **Group terms under subheadings** when natural clusters emerge. If all terms belong to a single cohesive area, a flat list is fine.
|
||||||
|
|
||||||
|
## Single vs multi-context repos
|
||||||
|
|
||||||
|
**Single context (most repos):** One `CONTEXT.md` at the repo root.
|
||||||
|
|
||||||
|
**Multiple contexts:** A `CONTEXT-MAP.md` at the repo root lists the contexts, where they live, and how they relate to each other:
|
||||||
|
|
||||||
|
```md
|
||||||
|
# Context Map
|
||||||
|
|
||||||
|
## Contexts
|
||||||
|
|
||||||
|
- [Ordering](./src/ordering/CONTEXT.md) — receives and tracks customer orders
|
||||||
|
- [Billing](./src/billing/CONTEXT.md) — generates invoices and processes payments
|
||||||
|
- [Fulfillment](./src/fulfillment/CONTEXT.md) — manages warehouse picking and shipping
|
||||||
|
|
||||||
|
## Relationships
|
||||||
|
|
||||||
|
- **Ordering → Fulfillment**: Ordering emits `OrderPlaced` events; Fulfillment consumes them to start picking
|
||||||
|
- **Fulfillment → Billing**: Fulfillment emits `ShipmentDispatched` events; Billing consumes them to generate invoices
|
||||||
|
- **Ordering ↔ Billing**: Shared types for `CustomerId` and `Money`
|
||||||
|
```
|
||||||
|
|
||||||
|
The skill infers which structure applies:
|
||||||
|
|
||||||
|
- If `CONTEXT-MAP.md` exists, read it to find contexts
|
||||||
|
- If only a root `CONTEXT.md` exists, single context
|
||||||
|
- If neither exists, create a root `CONTEXT.md` lazily when the first term is resolved
|
||||||
|
|
||||||
|
When multiple contexts exist, infer which one the current topic relates to. If unclear, ask.
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
---
|
||||||
|
name: domain-modeling
|
||||||
|
description: Build and sharpen a project's domain model. Use when the user wants to pin down domain terminology or a ubiquitous language, record an architectural decision, or when another skill needs to maintain the domain model.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Domain Modeling
|
||||||
|
|
||||||
|
Actively build and sharpen the project's domain model as you design. This is the *active* discipline — challenging terms, inventing edge-case scenarios, and writing the glossary and decisions down the moment they crystallise. (Merely *reading* `CONTEXT.md` for vocabulary is not this skill — that's a one-line habit any skill can do. This skill is for when you're changing the model, not just consuming it.)
|
||||||
|
|
||||||
|
## File structure
|
||||||
|
|
||||||
|
Most repos have a single context:
|
||||||
|
|
||||||
|
```
|
||||||
|
/
|
||||||
|
├── CONTEXT.md
|
||||||
|
├── docs/
|
||||||
|
│ └── adr/
|
||||||
|
│ ├── 0001-event-sourced-orders.md
|
||||||
|
│ └── 0002-postgres-for-write-model.md
|
||||||
|
└── src/
|
||||||
|
```
|
||||||
|
|
||||||
|
If a `CONTEXT-MAP.md` exists at the root, the repo has multiple contexts. The map points to where each one lives:
|
||||||
|
|
||||||
|
```
|
||||||
|
/
|
||||||
|
├── CONTEXT-MAP.md
|
||||||
|
├── docs/
|
||||||
|
│ └── adr/ ← system-wide decisions
|
||||||
|
├── src/
|
||||||
|
│ ├── ordering/
|
||||||
|
│ │ ├── CONTEXT.md
|
||||||
|
│ │ └── docs/adr/ ← context-specific decisions
|
||||||
|
│ └── billing/
|
||||||
|
│ ├── CONTEXT.md
|
||||||
|
│ └── docs/adr/
|
||||||
|
```
|
||||||
|
|
||||||
|
Create files lazily — only when you have something to write. If no `CONTEXT.md` exists, create one when the first term is resolved. If no `docs/adr/` exists, create it when the first ADR is needed.
|
||||||
|
|
||||||
|
## During the session
|
||||||
|
|
||||||
|
### Challenge against the glossary
|
||||||
|
|
||||||
|
When the user uses a term that conflicts with the existing language in `CONTEXT.md`, call it out immediately. "Your glossary defines 'cancellation' as X, but you seem to mean Y — which is it?"
|
||||||
|
|
||||||
|
### Sharpen fuzzy language
|
||||||
|
|
||||||
|
When the user uses vague or overloaded terms, propose a precise canonical term. "You're saying 'account' — do you mean the Customer or the User? Those are different things."
|
||||||
|
|
||||||
|
### Discuss concrete scenarios
|
||||||
|
|
||||||
|
When domain relationships are being discussed, stress-test them with specific scenarios. Invent scenarios that probe edge cases and force the user to be precise about the boundaries between concepts.
|
||||||
|
|
||||||
|
### Cross-reference with code
|
||||||
|
|
||||||
|
When the user states how something works, check whether the code agrees. If you find a contradiction, surface it: "Your code cancels entire Orders, but you just said partial cancellation is possible — which is right?"
|
||||||
|
|
||||||
|
### Update CONTEXT.md inline
|
||||||
|
|
||||||
|
When a term is resolved, update `CONTEXT.md` right there. Don't batch these up — capture them as they happen. Use the format in [CONTEXT-FORMAT.md](./CONTEXT-FORMAT.md).
|
||||||
|
|
||||||
|
`CONTEXT.md` should be totally devoid of implementation details. Do not treat `CONTEXT.md` as a spec, a scratch pad, or a repository for implementation decisions. It is a glossary and nothing else.
|
||||||
|
|
||||||
|
### Offer ADRs sparingly
|
||||||
|
|
||||||
|
Only offer to create an ADR when all three are true:
|
||||||
|
|
||||||
|
1. **Hard to reverse** — the cost of changing your mind later is meaningful
|
||||||
|
2. **Surprising without context** — a future reader will wonder "why did they do it this way?"
|
||||||
|
3. **The result of a real trade-off** — there were genuine alternatives and you picked one for specific reasons
|
||||||
|
|
||||||
|
If any of the three is missing, skip the ADR. Use the format in [ADR-FORMAT.md](./ADR-FORMAT.md).
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
---
|
||||||
|
name: grill-me
|
||||||
|
description: A relentless interview to sharpen a plan or design.
|
||||||
|
disable-model-invocation: true
|
||||||
|
---
|
||||||
|
|
||||||
|
Run a `/grilling` session.
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
---
|
||||||
|
name: grilling
|
||||||
|
description: Grill the user relentlessly about a plan or design. Use when the user wants to stress-test a plan before building, or uses any 'grill' trigger phrases.
|
||||||
|
---
|
||||||
|
|
||||||
|
Interview me relentlessly about every aspect of this plan until we reach a shared understanding. Walk down each branch of the design tree, resolving dependencies between decisions one-by-one. For each question, provide your recommended answer.
|
||||||
|
|
||||||
|
Ask the questions one at a time, waiting for feedback on each question before continuing. Asking multiple questions at once is bewildering.
|
||||||
|
|
||||||
|
If a question can be answered by exploring the codebase, explore the codebase instead.
|
||||||
|
|
||||||
|
Do not enact the plan until I confirm we have reached a shared understanding.
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
---
|
||||||
|
name: handoff
|
||||||
|
description: Compact the current conversation into a handoff document for another agent to pick up.
|
||||||
|
argument-hint: "What will the next session be used for?"
|
||||||
|
disable-model-invocation: true
|
||||||
|
---
|
||||||
|
|
||||||
|
Write a handoff document summarising the current conversation so a fresh agent can continue the work. Save to the temporary directory of the user's OS - not the current workspace.
|
||||||
|
|
||||||
|
Include a "suggested skills" section in the document, which suggests skills that the agent should invoke.
|
||||||
|
|
||||||
|
Do not duplicate content already captured in other artifacts (PRDs, plans, ADRs, issues, commits, diffs). Reference them by path or URL instead.
|
||||||
|
|
||||||
|
Redact any sensitive information, such as API keys, passwords, or personally identifiable information.
|
||||||
|
|
||||||
|
If the user passed arguments, treat them as a description of what the next session will focus on and tailor the doc accordingly.
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
---
|
||||||
|
name: implement
|
||||||
|
description: "Implement a piece of work based on a PRD or set of issues."
|
||||||
|
disable-model-invocation: true
|
||||||
|
---
|
||||||
|
|
||||||
|
Implement the work described by the user in the PRD or issues.
|
||||||
|
|
||||||
|
Use /tdd where possible, at pre-agreed seams.
|
||||||
|
|
||||||
|
Run typechecking regularly, single test files regularly, and the full test suite once at the end.
|
||||||
|
|
||||||
|
Once done, use /code-review to review the work.
|
||||||
|
|
||||||
|
Commit your work to the current branch.
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
---
|
||||||
|
name: research
|
||||||
|
description: Investigate a question against high-trust primary sources and capture the findings as a Markdown file in the repo. Use when the user wants a topic researched, docs or API facts gathered, or reading legwork delegated to a background agent.
|
||||||
|
---
|
||||||
|
|
||||||
|
Spin up a **background agent** to do the research, so you keep working while it reads.
|
||||||
|
|
||||||
|
Its job:
|
||||||
|
|
||||||
|
1. Investigate the question against **primary sources** — official docs, source code, specs, first-party APIs — not a secondary write-up of them. Follow every claim back to the source that owns it.
|
||||||
|
2. Write the findings to a single Markdown file, citing each claim's source.
|
||||||
|
3. Save it where the repo already keeps such notes; match the existing convention, and if there is none, put it somewhere sensible and say where.
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
---
|
||||||
|
name: tdd
|
||||||
|
description: Test-driven development. Use when the user wants to build features or fix bugs test-first, mentions "red-green-refactor", or wants integration tests.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Test-Driven Development
|
||||||
|
|
||||||
|
TDD is the red → green loop. This skill is the reference that makes that loop produce tests worth keeping: what a good test is, where tests go, the anti-patterns, and the rules of the loop. Every section applies on every cycle — consult them before and during the loop, not after.
|
||||||
|
|
||||||
|
When exploring the codebase, read `CONTEXT.md` (if it exists) so test names and interface vocabulary match the project's domain language, and respect ADRs in the area you're touching.
|
||||||
|
|
||||||
|
## What a good test is
|
||||||
|
|
||||||
|
Tests verify behavior through public interfaces, not implementation details. Code can change entirely; tests shouldn't. A good test reads like a specification — "user can checkout with valid cart" tells you exactly what capability exists — and survives refactors because it doesn't care about internal structure.
|
||||||
|
|
||||||
|
See [tests.md](tests.md) for examples and [mocking.md](mocking.md) for mocking guidelines.
|
||||||
|
|
||||||
|
## Seams — where tests go
|
||||||
|
|
||||||
|
A **seam** is the public boundary you test at: the interface where you observe behavior without reaching inside. Tests live at seams, never against internals.
|
||||||
|
|
||||||
|
**Test only at pre-agreed seams.** Before writing any test, write down the seams under test and confirm them with the user. No test is written at an unconfirmed seam. You can't test everything — agreeing the seams up front is how testing effort lands on the critical paths and complex logic instead of every edge case.
|
||||||
|
|
||||||
|
Ask: "What's the public interface, and which seams should we test?"
|
||||||
|
|
||||||
|
## Anti-patterns
|
||||||
|
|
||||||
|
- **Implementation-coupled** — mocks internal collaborators, tests private methods, or verifies through a side channel (querying the database instead of using the interface). The tell: the test breaks when you refactor but behavior hasn't changed.
|
||||||
|
- **Tautological** — the assertion recomputes the expected value the way the code does (`expect(add(a, b)).toBe(a + b)`, a snapshot derived by hand the same way, a constant asserted equal to itself), so it passes by construction and can never disagree with the code. Expected values must come from an independent source of truth — a known-good literal, a worked example, the spec.
|
||||||
|
- **Horizontal slicing** — writing all tests first, then all implementation. Bulk tests verify _imagined_ behavior: you test the _shape_ of things rather than user-facing behavior, the tests go insensitive to real changes, and you commit to test structure before understanding the implementation. Work in **vertical slices** instead — one test → one implementation → repeat, each test a **tracer bullet** that responds to what the last cycle taught you.
|
||||||
|
|
||||||
|
## Rules of the loop
|
||||||
|
|
||||||
|
- **Red before green.** Write the failing test first, then only enough code to pass it. Don't anticipate future tests or add speculative features.
|
||||||
|
- **One slice at a time.** One seam, one test, one minimal implementation per cycle.
|
||||||
|
- **Refactoring is not part of the loop.** It belongs to the review stage (see the `code-review` skill), not the red → green implementation cycle.
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
# When to Mock
|
||||||
|
|
||||||
|
Mock at **system boundaries** only:
|
||||||
|
|
||||||
|
- External APIs (payment, email, etc.)
|
||||||
|
- Databases (sometimes - prefer test DB)
|
||||||
|
- Time/randomness
|
||||||
|
- File system (sometimes)
|
||||||
|
|
||||||
|
Don't mock:
|
||||||
|
|
||||||
|
- Your own classes/modules
|
||||||
|
- Internal collaborators
|
||||||
|
- Anything you control
|
||||||
|
|
||||||
|
## Designing for Mockability
|
||||||
|
|
||||||
|
At system boundaries, design interfaces that are easy to mock:
|
||||||
|
|
||||||
|
**1. Use dependency injection**
|
||||||
|
|
||||||
|
Pass external dependencies in rather than creating them internally:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Easy to mock
|
||||||
|
function processPayment(order, paymentClient) {
|
||||||
|
return paymentClient.charge(order.total);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Hard to mock
|
||||||
|
function processPayment(order) {
|
||||||
|
const client = new StripeClient(process.env.STRIPE_KEY);
|
||||||
|
return client.charge(order.total);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**2. Prefer SDK-style interfaces over generic fetchers**
|
||||||
|
|
||||||
|
Create specific functions for each external operation instead of one generic function with conditional logic:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// GOOD: Each function is independently mockable
|
||||||
|
const api = {
|
||||||
|
getUser: (id) => fetch(`/users/${id}`),
|
||||||
|
getOrders: (userId) => fetch(`/users/${userId}/orders`),
|
||||||
|
createOrder: (data) => fetch('/orders', { method: 'POST', body: data }),
|
||||||
|
};
|
||||||
|
|
||||||
|
// BAD: Mocking requires conditional logic inside the mock
|
||||||
|
const api = {
|
||||||
|
fetch: (endpoint, options) => fetch(endpoint, options),
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
The SDK approach means:
|
||||||
|
- Each mock returns one specific shape
|
||||||
|
- No conditional logic in test setup
|
||||||
|
- Easier to see which endpoints a test exercises
|
||||||
|
- Type safety per endpoint
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
# Good and Bad Tests
|
||||||
|
|
||||||
|
## Good Tests
|
||||||
|
|
||||||
|
**Integration-style**: Test through real interfaces, not mocks of internal parts.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// GOOD: Tests observable behavior
|
||||||
|
test("user can checkout with valid cart", async () => {
|
||||||
|
const cart = createCart();
|
||||||
|
cart.add(product);
|
||||||
|
const result = await checkout(cart, paymentMethod);
|
||||||
|
expect(result.status).toBe("confirmed");
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
Characteristics:
|
||||||
|
|
||||||
|
- Tests behavior users/callers care about
|
||||||
|
- Uses public API only
|
||||||
|
- Survives internal refactors
|
||||||
|
- Describes WHAT, not HOW
|
||||||
|
- One logical assertion per test
|
||||||
|
|
||||||
|
## Bad Tests
|
||||||
|
|
||||||
|
**Implementation-detail tests**: Coupled to internal structure.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// BAD: Tests implementation details
|
||||||
|
test("checkout calls paymentService.process", async () => {
|
||||||
|
const mockPayment = jest.mock(paymentService);
|
||||||
|
await checkout(cart, payment);
|
||||||
|
expect(mockPayment.process).toHaveBeenCalledWith(cart.total);
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
Red flags:
|
||||||
|
|
||||||
|
- Mocking internal collaborators
|
||||||
|
- Testing private methods
|
||||||
|
- Asserting on call counts/order
|
||||||
|
- Test breaks when refactoring without behavior change
|
||||||
|
- Test name describes HOW not WHAT
|
||||||
|
- Verifying through external means instead of interface
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// BAD: Bypasses interface to verify
|
||||||
|
test("createUser saves to database", async () => {
|
||||||
|
await createUser({ name: "Alice" });
|
||||||
|
const row = await db.query("SELECT * FROM users WHERE name = ?", ["Alice"]);
|
||||||
|
expect(row).toBeDefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
// GOOD: Verifies through interface
|
||||||
|
test("createUser makes user retrievable", async () => {
|
||||||
|
const user = await createUser({ name: "Alice" });
|
||||||
|
const retrieved = await getUser(user.id);
|
||||||
|
expect(retrieved.name).toBe("Alice");
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
**Tautological tests**: Expected value restates the implementation, so the test passes by construction.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// BAD: Expected value is recomputed the way the code computes it
|
||||||
|
test("calculateTotal sums line items", () => {
|
||||||
|
const items = [{ price: 10 }, { price: 5 }];
|
||||||
|
const expected = items.reduce((sum, i) => sum + i.price, 0);
|
||||||
|
expect(calculateTotal(items)).toBe(expected);
|
||||||
|
});
|
||||||
|
|
||||||
|
// GOOD: Expected value is an independent, known literal
|
||||||
|
test("calculateTotal sums line items", () => {
|
||||||
|
expect(calculateTotal([{ price: 10 }, { price: 5 }])).toBe(15);
|
||||||
|
});
|
||||||
|
```
|
||||||
@@ -1,20 +1,22 @@
|
|||||||
# Cross-platform Crucible binaries (D7 trigger standard):
|
# Cross-platform Crucible release binaries: a v* tag builds all targets,
|
||||||
# PR / push main -> cross-build ALL targets to prove they compile (no publish)
|
# then uploads them, SHA256SUMS, and the canonical wrappers to its Gitea release.
|
||||||
# tag v* -> build all targets + SHA256SUMS, upload to the Gitea release
|
|
||||||
# Crucible is pure Go (CGO_ENABLED=0), so cross-compiling is a fast loop.
|
# Crucible is pure Go (CGO_ENABLED=0), so cross-compiling is a fast loop.
|
||||||
name: build-binaries
|
name: build-binaries
|
||||||
|
|
||||||
on:
|
on:
|
||||||
pull_request:
|
|
||||||
push:
|
push:
|
||||||
branches: [main]
|
|
||||||
tags: ['v*']
|
tags: ['v*']
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
code: read
|
||||||
|
releases: write
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
build-binaries:
|
build-binaries:
|
||||||
runs-on: nix-docker
|
runs-on: nix-docker
|
||||||
|
timeout-minutes: 30
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
|
||||||
with: { fetch-depth: 0 }
|
with: { fetch-depth: 0 }
|
||||||
|
|
||||||
- name: Cross-build all targets
|
- name: Cross-build all targets
|
||||||
@@ -40,9 +42,8 @@ jobs:
|
|||||||
'
|
'
|
||||||
|
|
||||||
- name: Upload to Gitea release
|
- name: Upload to Gitea release
|
||||||
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
|
|
||||||
env:
|
env:
|
||||||
TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
TOKEN: ${{ secrets.GITEA_TOKEN }}
|
||||||
TAG: ${{ github.ref_name }}
|
TAG: ${{ github.ref_name }}
|
||||||
REPO: ${{ github.repository }}
|
REPO: ${{ github.repository }}
|
||||||
SERVER: ${{ github.server_url }}
|
SERVER: ${{ github.server_url }}
|
||||||
@@ -64,3 +65,17 @@ jobs:
|
|||||||
"${api}/releases/${id}/assets?name=$(basename "$f")"
|
"${api}/releases/${id}/assets?name=$(basename "$f")"
|
||||||
done
|
done
|
||||||
'
|
'
|
||||||
|
|
||||||
|
# Gitea has no asset retention (#52): without this every tag keeps its
|
||||||
|
# ~57 MB binary set forever. Best-effort — a stale asset is cheaper than
|
||||||
|
# a blocked publish, so a failure here never fails the release.
|
||||||
|
- name: Prune assets of older releases
|
||||||
|
continue-on-error: true
|
||||||
|
env:
|
||||||
|
TOKEN: ${{ secrets.GITEA_TOKEN }}
|
||||||
|
REPO: ${{ github.repository }}
|
||||||
|
SERVER: ${{ github.server_url }}
|
||||||
|
run: |
|
||||||
|
nix develop --command bash -c '
|
||||||
|
API="${SERVER}/api/v1/repos/${REPO}" scripts/prune-release-assets.sh
|
||||||
|
'
|
||||||
|
|||||||
@@ -1,52 +0,0 @@
|
|||||||
# Publish the immutable crucible:<sha> image on version tags.
|
|
||||||
# PR/main test builds live in test-image.yml and never publish, so registry
|
|
||||||
# packages are only generated for releases we intend to use (not on every merge).
|
|
||||||
#
|
|
||||||
# Daemonless: the host-mode runner has no container runtime, so the image is
|
|
||||||
# built by Nix (`nix build .#image`, see flake.nix) and pushed with skopeo
|
|
||||||
# straight from the OCI tarball. No `docker build`/`docker login` involved.
|
|
||||||
name: build-image
|
|
||||||
|
|
||||||
on:
|
|
||||||
push:
|
|
||||||
tags: ['v*']
|
|
||||||
|
|
||||||
env:
|
|
||||||
REGISTRY: registry.westgate.pw
|
|
||||||
IMAGE: deployment/crucible
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
publish:
|
|
||||||
runs-on: nix-docker
|
|
||||||
steps:
|
|
||||||
- uses: actions/checkout@v4
|
|
||||||
with: { fetch-depth: 0 }
|
|
||||||
|
|
||||||
- name: Resolve tag
|
|
||||||
id: tag
|
|
||||||
run: echo "sha=$(git rev-parse --short=12 HEAD)" >> "$GITHUB_OUTPUT"
|
|
||||||
|
|
||||||
- name: Build OCI image (daemonless)
|
|
||||||
run: nix build .#image
|
|
||||||
|
|
||||||
# This workflow only runs on v* tags, so every run is a release publish.
|
|
||||||
- name: Publish image
|
|
||||||
env:
|
|
||||||
REGISTRY_USER: ${{ secrets.REGISTRY_USER }}
|
|
||||||
REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
|
|
||||||
run: |
|
|
||||||
nix shell nixpkgs#skopeo -c skopeo copy \
|
|
||||||
--dest-creds "${REGISTRY_USER}:${REGISTRY_PASSWORD}" \
|
|
||||||
docker-archive:result \
|
|
||||||
"docker://${REGISTRY}/${IMAGE}:${{ steps.tag.outputs.sha }}"
|
|
||||||
|
|
||||||
- name: Emit release fragment
|
|
||||||
run: |
|
|
||||||
FRAG_REPO=sow-tools \
|
|
||||||
FRAG_SHA=${{ steps.tag.outputs.sha }} \
|
|
||||||
FRAG_ARTIFACT="${REGISTRY}/${IMAGE}:${{ steps.tag.outputs.sha }}" \
|
|
||||||
FRAG_URL="${REGISTRY}/${IMAGE}" \
|
|
||||||
FRAG_RUN_ID=${{ gitea.run_id }} \
|
|
||||||
bash scripts/emit-release-fragment.sh
|
|
||||||
- uses: actions/upload-artifact@v3
|
|
||||||
with: { name: release-fragment, path: release-fragment.json }
|
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
# Pull-request validation for Crucible. Releases and wrapper synchronization
|
||||||
|
# have their own narrow workflows because they need tag/main events.
|
||||||
|
name: ci
|
||||||
|
|
||||||
|
on:
|
||||||
|
pull_request:
|
||||||
|
|
||||||
|
permissions: read-all
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
ci:
|
||||||
|
runs-on: nix-docker
|
||||||
|
timeout-minutes: 60
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
|
||||||
|
with:
|
||||||
|
fetch-depth: 0
|
||||||
|
|
||||||
|
- name: vet + test + lint
|
||||||
|
run: |
|
||||||
|
nix develop --command bash -c '
|
||||||
|
set -euo pipefail
|
||||||
|
go vet ./...
|
||||||
|
go test ./...
|
||||||
|
shellcheck scripts/*.sh
|
||||||
|
yamllint .gitea
|
||||||
|
'
|
||||||
|
|
||||||
|
- name: binary smoke (fail-closed contract)
|
||||||
|
run: nix develop --command make smoke
|
||||||
|
|
||||||
|
- name: Cross-build all targets
|
||||||
|
run: |
|
||||||
|
nix develop --command bash -c '
|
||||||
|
set -euo pipefail
|
||||||
|
sha="$(git rev-parse --short=12 HEAD)"
|
||||||
|
ldflags="-s -w -X git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo.Version=${sha}"
|
||||||
|
rm -rf dist && mkdir -p dist
|
||||||
|
export CGO_ENABLED=0
|
||||||
|
for target in linux/amd64 linux/arm64 darwin/amd64 darwin/arm64 windows/amd64 windows/arm64; do
|
||||||
|
os="${target%/*}"; arch="${target#*/}"
|
||||||
|
ext=""; [ "$os" = windows ] && ext=".exe"
|
||||||
|
echo "building crucible-${os}-${arch}${ext}"
|
||||||
|
GOOS="$os" GOARCH="$arch" go build -trimpath -ldflags "$ldflags" \
|
||||||
|
-o "dist/crucible-${os}-${arch}${ext}" ./cmd/crucible
|
||||||
|
done
|
||||||
|
'
|
||||||
@@ -1,49 +0,0 @@
|
|||||||
# Manual compatibility publisher for the Crucible image.
|
|
||||||
# Normal release publishing happens in build-image.yml on v* tags; this is the
|
|
||||||
# break-glass / re-publish path, triggered by hand.
|
|
||||||
#
|
|
||||||
# Daemonless: built by Nix (`nix build .#image`) and pushed with skopeo from the
|
|
||||||
# OCI tarball. No `docker build`/`docker login` involved.
|
|
||||||
name: publish-image
|
|
||||||
|
|
||||||
on:
|
|
||||||
workflow_dispatch:
|
|
||||||
|
|
||||||
env:
|
|
||||||
REGISTRY: registry.westgate.pw
|
|
||||||
IMAGE: deployment/crucible
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
publish:
|
|
||||||
runs-on: nix-docker
|
|
||||||
steps:
|
|
||||||
- uses: actions/checkout@v4
|
|
||||||
with: { fetch-depth: 0 }
|
|
||||||
|
|
||||||
- name: Resolve tag
|
|
||||||
id: tag
|
|
||||||
run: echo "sha=$(git rev-parse --short=12 HEAD)" >> "$GITHUB_OUTPUT"
|
|
||||||
|
|
||||||
- name: Build OCI image (daemonless)
|
|
||||||
run: nix build .#image
|
|
||||||
|
|
||||||
- name: Publish image
|
|
||||||
env:
|
|
||||||
REGISTRY_USER: ${{ secrets.REGISTRY_USER }}
|
|
||||||
REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
|
|
||||||
run: |
|
|
||||||
nix shell nixpkgs#skopeo -c skopeo copy \
|
|
||||||
--dest-creds "${REGISTRY_USER}:${REGISTRY_PASSWORD}" \
|
|
||||||
docker-archive:result \
|
|
||||||
"docker://${REGISTRY}/${IMAGE}:${{ steps.tag.outputs.sha }}"
|
|
||||||
|
|
||||||
- name: Emit release fragment
|
|
||||||
run: |
|
|
||||||
FRAG_REPO=sow-tools \
|
|
||||||
FRAG_SHA=${{ steps.tag.outputs.sha }} \
|
|
||||||
FRAG_ARTIFACT="${REGISTRY}/${IMAGE}:${{ steps.tag.outputs.sha }}" \
|
|
||||||
FRAG_URL="${REGISTRY}/${IMAGE}" \
|
|
||||||
FRAG_RUN_ID=${{ gitea.run_id }} \
|
|
||||||
bash scripts/emit-release-fragment.sh
|
|
||||||
- uses: actions/upload-artifact@v3
|
|
||||||
with: { name: release-fragment, path: release-fragment.json }
|
|
||||||
@@ -1,7 +1,10 @@
|
|||||||
# Auto-PR canonical wrapper updates to consumer repos when wrappers/ changes on
|
# Auto-PR canonical wrapper updates to consumer repos when wrappers/ changes on
|
||||||
# main. Maintenance automation (not artifact publishing), so it is allowed on a
|
# main. Maintenance automation (not artifact publishing), so it is allowed on a
|
||||||
# main push under the D7 trigger standard. Requires WRAPPER_SYNC_TOKEN: a bot
|
# main push under the D7 trigger standard. Requires BOT_TOKEN: the gitea-bot
|
||||||
# user's token with content+PR write to the consumer repos (never committed).
|
# org token (content+PR write to the consumer repos; never committed).
|
||||||
|
#
|
||||||
|
# Consumer drift checks run only after their sync PRs merge to main. They must
|
||||||
|
# not run on the sync PR itself, which can create recursive cross-repo checks.
|
||||||
name: sync-wrappers
|
name: sync-wrappers
|
||||||
|
|
||||||
on:
|
on:
|
||||||
@@ -11,20 +14,24 @@ on:
|
|||||||
- 'wrappers/crucible.sh'
|
- 'wrappers/crucible.sh'
|
||||||
- 'wrappers/crucible.ps1'
|
- 'wrappers/crucible.ps1'
|
||||||
|
|
||||||
|
permissions: read-all
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
sync:
|
sync:
|
||||||
runs-on: nix-docker
|
runs-on: nix-docker
|
||||||
|
timeout-minutes: 30
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
|
||||||
|
|
||||||
- name: Open sync PRs to consumers
|
- name: Open sync PRs to consumers
|
||||||
env:
|
env:
|
||||||
TOKEN: ${{ secrets.WRAPPER_SYNC_TOKEN }}
|
TOKEN: ${{ secrets.BOT_TOKEN }}
|
||||||
SERVER: ${{ github.server_url }}
|
SERVER: ${{ github.server_url }}
|
||||||
SRC_SHA: ${{ github.sha }}
|
SRC_SHA: ${{ github.sha }}
|
||||||
run: |
|
run: |
|
||||||
nix develop --command bash -c '
|
nix develop --command bash -c '
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
[ -n "${TOKEN}" ] || { echo "::error::BOT_TOKEN secret is empty — set the gitea-bot org token"; exit 1; }
|
||||||
host="$(echo "$SERVER" | sed -E "s#https?://##")"
|
host="$(echo "$SERVER" | sed -E "s#https?://##")"
|
||||||
branch="chore/sync-wrappers-$(echo "$SRC_SHA" | cut -c1-12)"
|
branch="chore/sync-wrappers-$(echo "$SRC_SHA" | cut -c1-12)"
|
||||||
grep -vE "^\s*#|^\s*$" wrappers/consumers.txt | while read -r target; do
|
grep -vE "^\s*#|^\s*$" wrappers/consumers.txt | while read -r target; do
|
||||||
@@ -40,10 +47,19 @@ jobs:
|
|||||||
git add crucible.sh crucible.ps1
|
git add crucible.sh crucible.ps1
|
||||||
git commit -m "chore: sync crucible wrappers from sow-tools@${SRC_SHA}"
|
git commit -m "chore: sync crucible wrappers from sow-tools@${SRC_SHA}"
|
||||||
git push -f origin "$branch"
|
git push -f origin "$branch"
|
||||||
curl -fsS -X POST \
|
# Capture HTTP status: 201=created, 422=PR already open for this
|
||||||
|
# branch (fine, the force-push above refreshed it). Anything else
|
||||||
|
# (401/404/...) is a real failure — fail loud, do not swallow it.
|
||||||
|
resp="$(mktemp)"
|
||||||
|
code="$(curl -sS -o "$resp" -w "%{http_code}" -X POST \
|
||||||
-H "Authorization: token ${TOKEN}" -H "Content-Type: application/json" \
|
-H "Authorization: token ${TOKEN}" -H "Content-Type: application/json" \
|
||||||
"${SERVER}/api/v1/repos/${target}/pulls" \
|
"${SERVER}/api/v1/repos/${target}/pulls" \
|
||||||
-d "{\"head\":\"${branch}\",\"base\":\"main\",\"title\":\"chore: sync crucible wrappers from sow-tools\"}" || true
|
-d "{\"head\":\"${branch}\",\"base\":\"main\",\"title\":\"chore: sync crucible wrappers from sow-tools\"}")"
|
||||||
|
case "$code" in
|
||||||
|
201) echo "opened sync PR for $target" ;;
|
||||||
|
422) echo "sync PR already open for $target; refreshed its branch" ;;
|
||||||
|
*) echo "::error::PR create failed for $target (HTTP $code)"; cat "$resp"; exit 1 ;;
|
||||||
|
esac
|
||||||
)
|
)
|
||||||
rm -rf "$work"
|
rm -rf "$work"
|
||||||
done
|
done
|
||||||
|
|||||||
@@ -1,19 +0,0 @@
|
|||||||
# Test-build the Crucible image on PRs and main. Proves `nix build .#image`
|
|
||||||
# still works (daemonless, Nix-built OCI tarball) but does NOT publish —
|
|
||||||
# release publishing happens in build-image.yml on v* tags.
|
|
||||||
name: test-image
|
|
||||||
|
|
||||||
on:
|
|
||||||
push:
|
|
||||||
branches: [main]
|
|
||||||
pull_request:
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
build-image:
|
|
||||||
runs-on: nix-docker
|
|
||||||
steps:
|
|
||||||
- uses: actions/checkout@v4
|
|
||||||
with: { fetch-depth: 0 }
|
|
||||||
|
|
||||||
- name: Build OCI image (daemonless, no publish)
|
|
||||||
run: nix build .#image
|
|
||||||
@@ -1,25 +0,0 @@
|
|||||||
# Lint + unit tests for the Crucible suite. PR-first: PRs + push to main (D7).
|
|
||||||
name: test
|
|
||||||
|
|
||||||
on:
|
|
||||||
push:
|
|
||||||
branches: [main]
|
|
||||||
pull_request:
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
test:
|
|
||||||
runs-on: nix-docker
|
|
||||||
steps:
|
|
||||||
- uses: actions/checkout@v4
|
|
||||||
with: { fetch-depth: 0 }
|
|
||||||
- name: vet + test + lint
|
|
||||||
run: |
|
|
||||||
nix develop --command bash -c '
|
|
||||||
set -euo pipefail
|
|
||||||
go vet ./...
|
|
||||||
go test ./...
|
|
||||||
shellcheck scripts/*.sh
|
|
||||||
yamllint .gitea
|
|
||||||
'
|
|
||||||
- name: binary smoke (fail-closed contract)
|
|
||||||
run: nix develop --command make smoke
|
|
||||||
+3
-2
@@ -6,8 +6,8 @@ nwn-tool
|
|||||||
sow-toolkit
|
sow-toolkit
|
||||||
|
|
||||||
# Go / build cache
|
# Go / build cache
|
||||||
.cache/
|
.cache/*
|
||||||
.cache/**
|
!.cache/.gitkeep
|
||||||
|
|
||||||
# nix build symlink
|
# nix build symlink
|
||||||
result
|
result
|
||||||
@@ -18,3 +18,4 @@ result-*
|
|||||||
*.swp
|
*.swp
|
||||||
.idea/
|
.idea/
|
||||||
.vscode/
|
.vscode/
|
||||||
|
.direnv/
|
||||||
|
|||||||
@@ -5,34 +5,67 @@ alwaysApply: true
|
|||||||
|
|
||||||
# sow-tools / Crucible Agent Guide
|
# sow-tools / Crucible Agent Guide
|
||||||
|
|
||||||
This repo owns the **builder logic** for the migration: one Go module
|
This repo owns the **builder logic:** one Go module
|
||||||
(`git.westgate.pw/ShadowsOverWestgate/sow-tools`) producing the `crucible`
|
(`git.westgate.pw/ShadowsOverWestgate/sow-tools`) producing the `crucible`
|
||||||
dispatcher and the `crucible-<name>` binaries (D11). Read
|
dispatcher and the `crucible-<name>` binaries.
|
||||||
`../AGENTS.md` (migration hub) and `../../KICKOFF_PROMPT.md` first.
|
|
||||||
|
## What it is / part it serves
|
||||||
|
|
||||||
|
Shadows Over Westgate (SoW) is a Neverwinter Nights: Enhanced Edition
|
||||||
|
persistent world, split into single-purpose repos. This repo is Crucible, the
|
||||||
|
Go build toolkit. The content repos hold game source (module areas, rules
|
||||||
|
data, binary assets) and call Crucible to turn that source into artifacts:
|
||||||
|
the `.mod` file, 2DA/TLK tables, HAKs, wiki pages. Crucible is the only place
|
||||||
|
builder logic lives; content repos only run it through thin wrapper scripts.
|
||||||
|
|
||||||
|
## Guidance map
|
||||||
|
|
||||||
|
| Where | What |
|
||||||
|
|-------|------|
|
||||||
|
| `cmd/crucible/`, `cmd/crucible-<name>/` | dispatcher + per-builder shims (thin `main.go` files) |
|
||||||
|
| `internal/dispatch/` | the command registry — single source of truth for the command surface |
|
||||||
|
| `internal/` (app, pipeline, project, erf, gff, topdata, changelog, validator, depot, menu, buildinfo) | the actual builder logic |
|
||||||
|
| `wrappers/` | canonical bootstrap wrappers (`crucible`, `crucible.ps1`) synced to consumer repos; `wrappers/consumers.txt` lists targets |
|
||||||
|
| `docs/command-surface.md` | every command, old `nwn-tool` name → new home |
|
||||||
|
| `docs/consumer-contract.md` | how consumer repos resolve/pin a Crucible binary |
|
||||||
|
| `docs/migration-from-nwn-tool.md` | migration status, what remains |
|
||||||
|
| `tests/`, `Makefile`, `flake.nix` | checks, targets, dev shell |
|
||||||
|
|
||||||
|
Task routing: adding/changing a command → read `docs/command-surface.md`
|
||||||
|
first, then `internal/dispatch`. Changing how consumers get binaries →
|
||||||
|
`docs/consumer-contract.md` + `wrappers/`. Release/CI questions → README "CI"
|
||||||
|
section and `.gitea/workflows/`.
|
||||||
|
|
||||||
|
## How it is used
|
||||||
|
|
||||||
|
- Dev loop: `nix develop`, then `make check` / `make build` / `make smoke`
|
||||||
|
(see Commands below).
|
||||||
|
- Release: push a `v*` tag. CI uploads cross-built binaries + wrappers to the
|
||||||
|
Gitea release. There is no container image; Crucible ships as binaries,
|
||||||
|
wrappers, and the Nix input.
|
||||||
|
- Consumers (they download released binaries via the wrapper; they never
|
||||||
|
vendor a toolkit):
|
||||||
|
- sow-module — https://git.westgate.pw/ShadowsOverWestgate/sow-module
|
||||||
|
- sow-topdata — https://git.westgate.pw/ShadowsOverWestgate/sow-topdata
|
||||||
|
- sow-assets-manifest — https://git.westgate.pw/ShadowsOverWestgate/sow-assets-manifest
|
||||||
|
- sow-platform (infra/deploy authority) —
|
||||||
|
https://git.westgate.pw/ShadowsOverWestgate/sow-platform
|
||||||
|
|
||||||
## What this repo owns / does not own
|
## What this repo owns / does not own
|
||||||
|
|
||||||
Owns: build/extract/validate/compare pipeline, ERF/HAK packing, topdata 2da/tlk
|
Owns: build/extract/validate/compare pipeline, ERF/HAK packing, topdata 2da/tlk
|
||||||
compilation, wiki rendering/deploy, depot blob verify, music conversion,
|
compilation, wiki rendering/deploy, depot blob verify, changelog. Does **not**
|
||||||
changelog. Does **not** own authored game content (that is `sow-module` /
|
own authored game content (that is `sow-module` /
|
||||||
`sow-topdata` / `sow-assets-manifest`) or any production deploy authority (that
|
`sow-topdata` / `sow-assets-manifest`) or any production deploy authority (that
|
||||||
is `sow-platform`).
|
is `sow-platform`).
|
||||||
|
|
||||||
## Rules
|
## Rules
|
||||||
|
|
||||||
1. **Migrated logic, not a fresh rewrite.** The `internal/` packages (`app`,
|
1. **Fail closed, never fake.** A builder with no migrated logic yet (`depot`)
|
||||||
`pipeline`, `project`, `erf`, `gff`, `topdata`, `music`, `changelog`,
|
|
||||||
`validator`) were folded in from `gitea/sow-tools` at cutover; wired builders
|
|
||||||
delegate to `internal/app`'s command surface. Keep them in step with upstream
|
|
||||||
fixes rather than diverging silently.
|
|
||||||
2. **Fail closed, never fake.** A builder with no migrated logic yet (`depot`)
|
|
||||||
exits `70`. Do not stub a builder to emit a placeholder artifact.
|
exits `70`. Do not stub a builder to emit a placeholder artifact.
|
||||||
3. **Binaries are not committed.** They are CI artifacts / image layers (D19).
|
2. **Binaries are not committed.** They are CI artifacts.
|
||||||
`/bin/`, `*.exe`, `nwn-tool`, `sow-toolkit` are gitignored.
|
`/bin/`, `*.exe`, `nwn-tool`, `sow-toolkit` are gitignored.
|
||||||
4. **No home-dir / `NWN_ROOT` guessing.** Builders take roots explicitly via
|
3. **The registry is the command surface.** `internal/dispatch.Registry` is the
|
||||||
flag or env (project resolution is CWD-based, never `$HOME`). See
|
|
||||||
[`docs/consumer-contract.md`](docs/consumer-contract.md).
|
|
||||||
5. **The registry is the command surface.** `internal/dispatch.Registry` is the
|
|
||||||
single source of truth; keep it in sync with `cmd/` and
|
single source of truth; keep it in sync with `cmd/` and
|
||||||
[`docs/command-surface.md`](docs/command-surface.md). Adding a builder = a
|
[`docs/command-surface.md`](docs/command-surface.md). Adding a builder = a
|
||||||
`cmd/crucible-<name>/main.go` shim + a `Registry` entry + a doc row.
|
`cmd/crucible-<name>/main.go` shim + a `Registry` entry + a doc row.
|
||||||
@@ -40,11 +73,8 @@ is `sow-platform`).
|
|||||||
## Wiring a builder
|
## Wiring a builder
|
||||||
|
|
||||||
1. Ensure the relevant `internal/` package(s) cover the work.
|
1. Ensure the relevant `internal/` package(s) cover the work.
|
||||||
2. Add the legacy command(s) to the builder's `Legacy`/`Extra` set and set
|
2. Add tests; keep outputs deterministic (same input → same bytes).
|
||||||
`Wired: true` in `internal/dispatch`; the dispatcher delegates to
|
3. `make check` must stay green; update `make smoke` to expect the wired exit.
|
||||||
`app.Run`. `depot` is the remaining unwired builder.
|
|
||||||
3. Add tests; keep outputs deterministic (same input → same bytes).
|
|
||||||
4. `make check` must stay green; update `make smoke` to expect the wired exit.
|
|
||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
@@ -52,9 +82,26 @@ is `sow-platform`).
|
|||||||
nix develop && make check # vet + test + shellcheck + yamllint
|
nix develop && make check # vet + test + shellcheck + yamllint
|
||||||
make build # cmd/* -> ./bin
|
make build # cmd/* -> ./bin
|
||||||
make smoke # assert fail-closed contract
|
make smoke # assert fail-closed contract
|
||||||
make image # crucible:<sha>
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Git
|
## Tests
|
||||||
|
|
||||||
Never commit, branch, or push. Suggest a commit message; let the operator do it.
|
Tests must survive harmless changes to constants, defaults, wording, ordering, fixture data, and internal implementation details. A test that fails merely because a basic value changed is usually a bad test. Only assert exact values when the value is part of a documented public contract, external protocol, compatibility requirement, security rule, migration, or business rule.
|
||||||
|
|
||||||
|
## Agent skills
|
||||||
|
|
||||||
|
### Issue tracker
|
||||||
|
|
||||||
|
Issues live in Gitea at git.westgate.pw (`ShadowsOverWestgate/sow-tools`), managed with the `tea` CLI. Issues follow ownership — file work in the repo that owns it, not the one you happen to be standing in. See `docs/agents/issue-tracker.md`.
|
||||||
|
|
||||||
|
### Triage labels
|
||||||
|
|
||||||
|
Default label vocabulary (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
|
||||||
|
|
||||||
|
### Domain docs
|
||||||
|
|
||||||
|
Single-context: `CONTEXT.md` at the repo root plus `docs/adr/`. See `docs/agents/domain.md`.
|
||||||
|
|
||||||
|
### Where work lives
|
||||||
|
|
||||||
|
Markdown here is **reference, law, or an ADR — nothing else** (`sow-codebase` ADR-0001). Live work -> wayfinder maps + Gitea issues (closeable, assignable, queryable). Settled decisions -> ADR files in `docs/adr/` (immutable, never closed, only superseded). Standing law -> `DOCTRINE.md` / `AGENTS.md` / `CONTEXT.md`. Current-state reference -> docs describing what the code does now. Everything else — plans, specs, concepts, handoffs, trackers — is process: it belongs in a Gitea issue, not a file. Harvest unfinished intent to an issue before deleting a process doc. `sow-docs` is deprecated and read-only.
|
||||||
|
|||||||
@@ -0,0 +1,674 @@
|
|||||||
|
GNU GENERAL PUBLIC LICENSE
|
||||||
|
Version 3, 29 June 2007
|
||||||
|
|
||||||
|
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
|
||||||
|
Everyone is permitted to copy and distribute verbatim copies
|
||||||
|
of this license document, but changing it is not allowed.
|
||||||
|
|
||||||
|
Preamble
|
||||||
|
|
||||||
|
The GNU General Public License is a free, copyleft license for
|
||||||
|
software and other kinds of works.
|
||||||
|
|
||||||
|
The licenses for most software and other practical works are designed
|
||||||
|
to take away your freedom to share and change the works. By contrast,
|
||||||
|
the GNU General Public License is intended to guarantee your freedom to
|
||||||
|
share and change all versions of a program--to make sure it remains free
|
||||||
|
software for all its users. We, the Free Software Foundation, use the
|
||||||
|
GNU General Public License for most of our software; it applies also to
|
||||||
|
any other work released this way by its authors. You can apply it to
|
||||||
|
your programs, too.
|
||||||
|
|
||||||
|
When we speak of free software, we are referring to freedom, not
|
||||||
|
price. Our General Public Licenses are designed to make sure that you
|
||||||
|
have the freedom to distribute copies of free software (and charge for
|
||||||
|
them if you wish), that you receive source code or can get it if you
|
||||||
|
want it, that you can change the software or use pieces of it in new
|
||||||
|
free programs, and that you know you can do these things.
|
||||||
|
|
||||||
|
To protect your rights, we need to prevent others from denying you
|
||||||
|
these rights or asking you to surrender the rights. Therefore, you have
|
||||||
|
certain responsibilities if you distribute copies of the software, or if
|
||||||
|
you modify it: responsibilities to respect the freedom of others.
|
||||||
|
|
||||||
|
For example, if you distribute copies of such a program, whether
|
||||||
|
gratis or for a fee, you must pass on to the recipients the same
|
||||||
|
freedoms that you received. You must make sure that they, too, receive
|
||||||
|
or can get the source code. And you must show them these terms so they
|
||||||
|
know their rights.
|
||||||
|
|
||||||
|
Developers that use the GNU GPL protect your rights with two steps:
|
||||||
|
(1) assert copyright on the software, and (2) offer you this License
|
||||||
|
giving you legal permission to copy, distribute and/or modify it.
|
||||||
|
|
||||||
|
For the developers' and authors' protection, the GPL clearly explains
|
||||||
|
that there is no warranty for this free software. For both users' and
|
||||||
|
authors' sake, the GPL requires that modified versions be marked as
|
||||||
|
changed, so that their problems will not be attributed erroneously to
|
||||||
|
authors of previous versions.
|
||||||
|
|
||||||
|
Some devices are designed to deny users access to install or run
|
||||||
|
modified versions of the software inside them, although the manufacturer
|
||||||
|
can do so. This is fundamentally incompatible with the aim of
|
||||||
|
protecting users' freedom to change the software. The systematic
|
||||||
|
pattern of such abuse occurs in the area of products for individuals to
|
||||||
|
use, which is precisely where it is most unacceptable. Therefore, we
|
||||||
|
have designed this version of the GPL to prohibit the practice for those
|
||||||
|
products. If such problems arise substantially in other domains, we
|
||||||
|
stand ready to extend this provision to those domains in future versions
|
||||||
|
of the GPL, as needed to protect the freedom of users.
|
||||||
|
|
||||||
|
Finally, every program is threatened constantly by software patents.
|
||||||
|
States should not allow patents to restrict development and use of
|
||||||
|
software on general-purpose computers, but in those that do, we wish to
|
||||||
|
avoid the special danger that patents applied to a free program could
|
||||||
|
make it effectively proprietary. To prevent this, the GPL assures that
|
||||||
|
patents cannot be used to render the program non-free.
|
||||||
|
|
||||||
|
The precise terms and conditions for copying, distribution and
|
||||||
|
modification follow.
|
||||||
|
|
||||||
|
TERMS AND CONDITIONS
|
||||||
|
|
||||||
|
0. Definitions.
|
||||||
|
|
||||||
|
"This License" refers to version 3 of the GNU General Public License.
|
||||||
|
|
||||||
|
"Copyright" also means copyright-like laws that apply to other kinds of
|
||||||
|
works, such as semiconductor masks.
|
||||||
|
|
||||||
|
"The Program" refers to any copyrightable work licensed under this
|
||||||
|
License. Each licensee is addressed as "you". "Licensees" and
|
||||||
|
"recipients" may be individuals or organizations.
|
||||||
|
|
||||||
|
To "modify" a work means to copy from or adapt all or part of the work
|
||||||
|
in a fashion requiring copyright permission, other than the making of an
|
||||||
|
exact copy. The resulting work is called a "modified version" of the
|
||||||
|
earlier work or a work "based on" the earlier work.
|
||||||
|
|
||||||
|
A "covered work" means either the unmodified Program or a work based
|
||||||
|
on the Program.
|
||||||
|
|
||||||
|
To "propagate" a work means to do anything with it that, without
|
||||||
|
permission, would make you directly or secondarily liable for
|
||||||
|
infringement under applicable copyright law, except executing it on a
|
||||||
|
computer or modifying a private copy. Propagation includes copying,
|
||||||
|
distribution (with or without modification), making available to the
|
||||||
|
public, and in some countries other activities as well.
|
||||||
|
|
||||||
|
To "convey" a work means any kind of propagation that enables other
|
||||||
|
parties to make or receive copies. Mere interaction with a user through
|
||||||
|
a computer network, with no transfer of a copy, is not conveying.
|
||||||
|
|
||||||
|
An interactive user interface displays "Appropriate Legal Notices"
|
||||||
|
to the extent that it includes a convenient and prominently visible
|
||||||
|
feature that (1) displays an appropriate copyright notice, and (2)
|
||||||
|
tells the user that there is no warranty for the work (except to the
|
||||||
|
extent that warranties are provided), that licensees may convey the
|
||||||
|
work under this License, and how to view a copy of this License. If
|
||||||
|
the interface presents a list of user commands or options, such as a
|
||||||
|
menu, a prominent item in the list meets this criterion.
|
||||||
|
|
||||||
|
1. Source Code.
|
||||||
|
|
||||||
|
The "source code" for a work means the preferred form of the work
|
||||||
|
for making modifications to it. "Object code" means any non-source
|
||||||
|
form of a work.
|
||||||
|
|
||||||
|
A "Standard Interface" means an interface that either is an official
|
||||||
|
standard defined by a recognized standards body, or, in the case of
|
||||||
|
interfaces specified for a particular programming language, one that
|
||||||
|
is widely used among developers working in that language.
|
||||||
|
|
||||||
|
The "System Libraries" of an executable work include anything, other
|
||||||
|
than the work as a whole, that (a) is included in the normal form of
|
||||||
|
packaging a Major Component, but which is not part of that Major
|
||||||
|
Component, and (b) serves only to enable use of the work with that
|
||||||
|
Major Component, or to implement a Standard Interface for which an
|
||||||
|
implementation is available to the public in source code form. A
|
||||||
|
"Major Component", in this context, means a major essential component
|
||||||
|
(kernel, window system, and so on) of the specific operating system
|
||||||
|
(if any) on which the executable work runs, or a compiler used to
|
||||||
|
produce the work, or an object code interpreter used to run it.
|
||||||
|
|
||||||
|
The "Corresponding Source" for a work in object code form means all
|
||||||
|
the source code needed to generate, install, and (for an executable
|
||||||
|
work) run the object code and to modify the work, including scripts to
|
||||||
|
control those activities. However, it does not include the work's
|
||||||
|
System Libraries, or general-purpose tools or generally available free
|
||||||
|
programs which are used unmodified in performing those activities but
|
||||||
|
which are not part of the work. For example, Corresponding Source
|
||||||
|
includes interface definition files associated with source files for
|
||||||
|
the work, and the source code for shared libraries and dynamically
|
||||||
|
linked subprograms that the work is specifically designed to require,
|
||||||
|
such as by intimate data communication or control flow between those
|
||||||
|
subprograms and other parts of the work.
|
||||||
|
|
||||||
|
The Corresponding Source need not include anything that users
|
||||||
|
can regenerate automatically from other parts of the Corresponding
|
||||||
|
Source.
|
||||||
|
|
||||||
|
The Corresponding Source for a work in source code form is that
|
||||||
|
same work.
|
||||||
|
|
||||||
|
2. Basic Permissions.
|
||||||
|
|
||||||
|
All rights granted under this License are granted for the term of
|
||||||
|
copyright on the Program, and are irrevocable provided the stated
|
||||||
|
conditions are met. This License explicitly affirms your unlimited
|
||||||
|
permission to run the unmodified Program. The output from running a
|
||||||
|
covered work is covered by this License only if the output, given its
|
||||||
|
content, constitutes a covered work. This License acknowledges your
|
||||||
|
rights of fair use or other equivalent, as provided by copyright law.
|
||||||
|
|
||||||
|
You may make, run and propagate covered works that you do not
|
||||||
|
convey, without conditions so long as your license otherwise remains
|
||||||
|
in force. You may convey covered works to others for the sole purpose
|
||||||
|
of having them make modifications exclusively for you, or provide you
|
||||||
|
with facilities for running those works, provided that you comply with
|
||||||
|
the terms of this License in conveying all material for which you do
|
||||||
|
not control copyright. Those thus making or running the covered works
|
||||||
|
for you must do so exclusively on your behalf, under your direction
|
||||||
|
and control, on terms that prohibit them from making any copies of
|
||||||
|
your copyrighted material outside their relationship with you.
|
||||||
|
|
||||||
|
Conveying under any other circumstances is permitted solely under
|
||||||
|
the conditions stated below. Sublicensing is not allowed; section 10
|
||||||
|
makes it unnecessary.
|
||||||
|
|
||||||
|
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
|
||||||
|
|
||||||
|
No covered work shall be deemed part of an effective technological
|
||||||
|
measure under any applicable law fulfilling obligations under article
|
||||||
|
11 of the WIPO copyright treaty adopted on 20 December 1996, or
|
||||||
|
similar laws prohibiting or restricting circumvention of such
|
||||||
|
measures.
|
||||||
|
|
||||||
|
When you convey a covered work, you waive any legal power to forbid
|
||||||
|
circumvention of technological measures to the extent such circumvention
|
||||||
|
is effected by exercising rights under this License with respect to
|
||||||
|
the covered work, and you disclaim any intention to limit operation or
|
||||||
|
modification of the work as a means of enforcing, against the work's
|
||||||
|
users, your or third parties' legal rights to forbid circumvention of
|
||||||
|
technological measures.
|
||||||
|
|
||||||
|
4. Conveying Verbatim Copies.
|
||||||
|
|
||||||
|
You may convey verbatim copies of the Program's source code as you
|
||||||
|
receive it, in any medium, provided that you conspicuously and
|
||||||
|
appropriately publish on each copy an appropriate copyright notice;
|
||||||
|
keep intact all notices stating that this License and any
|
||||||
|
non-permissive terms added in accord with section 7 apply to the code;
|
||||||
|
keep intact all notices of the absence of any warranty; and give all
|
||||||
|
recipients a copy of this License along with the Program.
|
||||||
|
|
||||||
|
You may charge any price or no price for each copy that you convey,
|
||||||
|
and you may offer support or warranty protection for a fee.
|
||||||
|
|
||||||
|
5. Conveying Modified Source Versions.
|
||||||
|
|
||||||
|
You may convey a work based on the Program, or the modifications to
|
||||||
|
produce it from the Program, in the form of source code under the
|
||||||
|
terms of section 4, provided that you also meet all of these conditions:
|
||||||
|
|
||||||
|
a) The work must carry prominent notices stating that you modified
|
||||||
|
it, and giving a relevant date.
|
||||||
|
|
||||||
|
b) The work must carry prominent notices stating that it is
|
||||||
|
released under this License and any conditions added under section
|
||||||
|
7. This requirement modifies the requirement in section 4 to
|
||||||
|
"keep intact all notices".
|
||||||
|
|
||||||
|
c) You must license the entire work, as a whole, under this
|
||||||
|
License to anyone who comes into possession of a copy. This
|
||||||
|
License will therefore apply, along with any applicable section 7
|
||||||
|
additional terms, to the whole of the work, and all its parts,
|
||||||
|
regardless of how they are packaged. This License gives no
|
||||||
|
permission to license the work in any other way, but it does not
|
||||||
|
invalidate such permission if you have separately received it.
|
||||||
|
|
||||||
|
d) If the work has interactive user interfaces, each must display
|
||||||
|
Appropriate Legal Notices; however, if the Program has interactive
|
||||||
|
interfaces that do not display Appropriate Legal Notices, your
|
||||||
|
work need not make them do so.
|
||||||
|
|
||||||
|
A compilation of a covered work with other separate and independent
|
||||||
|
works, which are not by their nature extensions of the covered work,
|
||||||
|
and which are not combined with it such as to form a larger program,
|
||||||
|
in or on a volume of a storage or distribution medium, is called an
|
||||||
|
"aggregate" if the compilation and its resulting copyright are not
|
||||||
|
used to limit the access or legal rights of the compilation's users
|
||||||
|
beyond what the individual works permit. Inclusion of a covered work
|
||||||
|
in an aggregate does not cause this License to apply to the other
|
||||||
|
parts of the aggregate.
|
||||||
|
|
||||||
|
6. Conveying Non-Source Forms.
|
||||||
|
|
||||||
|
You may convey a covered work in object code form under the terms
|
||||||
|
of sections 4 and 5, provided that you also convey the
|
||||||
|
machine-readable Corresponding Source under the terms of this License,
|
||||||
|
in one of these ways:
|
||||||
|
|
||||||
|
a) Convey the object code in, or embodied in, a physical product
|
||||||
|
(including a physical distribution medium), accompanied by the
|
||||||
|
Corresponding Source fixed on a durable physical medium
|
||||||
|
customarily used for software interchange.
|
||||||
|
|
||||||
|
b) Convey the object code in, or embodied in, a physical product
|
||||||
|
(including a physical distribution medium), accompanied by a
|
||||||
|
written offer, valid for at least three years and valid for as
|
||||||
|
long as you offer spare parts or customer support for that product
|
||||||
|
model, to give anyone who possesses the object code either (1) a
|
||||||
|
copy of the Corresponding Source for all the software in the
|
||||||
|
product that is covered by this License, on a durable physical
|
||||||
|
medium customarily used for software interchange, for a price no
|
||||||
|
more than your reasonable cost of physically performing this
|
||||||
|
conveying of source, or (2) access to copy the
|
||||||
|
Corresponding Source from a network server at no charge.
|
||||||
|
|
||||||
|
c) Convey individual copies of the object code with a copy of the
|
||||||
|
written offer to provide the Corresponding Source. This
|
||||||
|
alternative is allowed only occasionally and noncommercially, and
|
||||||
|
only if you received the object code with such an offer, in accord
|
||||||
|
with subsection 6b.
|
||||||
|
|
||||||
|
d) Convey the object code by offering access from a designated
|
||||||
|
place (gratis or for a charge), and offer equivalent access to the
|
||||||
|
Corresponding Source in the same way through the same place at no
|
||||||
|
further charge. You need not require recipients to copy the
|
||||||
|
Corresponding Source along with the object code. If the place to
|
||||||
|
copy the object code is a network server, the Corresponding Source
|
||||||
|
may be on a different server (operated by you or a third party)
|
||||||
|
that supports equivalent copying facilities, provided you maintain
|
||||||
|
clear directions next to the object code saying where to find the
|
||||||
|
Corresponding Source. Regardless of what server hosts the
|
||||||
|
Corresponding Source, you remain obligated to ensure that it is
|
||||||
|
available for as long as needed to satisfy these requirements.
|
||||||
|
|
||||||
|
e) Convey the object code using peer-to-peer transmission, provided
|
||||||
|
you inform other peers where the object code and Corresponding
|
||||||
|
Source of the work are being offered to the general public at no
|
||||||
|
charge under subsection 6d.
|
||||||
|
|
||||||
|
A separable portion of the object code, whose source code is excluded
|
||||||
|
from the Corresponding Source as a System Library, need not be
|
||||||
|
included in conveying the object code work.
|
||||||
|
|
||||||
|
A "User Product" is either (1) a "consumer product", which means any
|
||||||
|
tangible personal property which is normally used for personal, family,
|
||||||
|
or household purposes, or (2) anything designed or sold for incorporation
|
||||||
|
into a dwelling. In determining whether a product is a consumer product,
|
||||||
|
doubtful cases shall be resolved in favor of coverage. For a particular
|
||||||
|
product received by a particular user, "normally used" refers to a
|
||||||
|
typical or common use of that class of product, regardless of the status
|
||||||
|
of the particular user or of the way in which the particular user
|
||||||
|
actually uses, or expects or is expected to use, the product. A product
|
||||||
|
is a consumer product regardless of whether the product has substantial
|
||||||
|
commercial, industrial or non-consumer uses, unless such uses represent
|
||||||
|
the only significant mode of use of the product.
|
||||||
|
|
||||||
|
"Installation Information" for a User Product means any methods,
|
||||||
|
procedures, authorization keys, or other information required to install
|
||||||
|
and execute modified versions of a covered work in that User Product from
|
||||||
|
a modified version of its Corresponding Source. The information must
|
||||||
|
suffice to ensure that the continued functioning of the modified object
|
||||||
|
code is in no case prevented or interfered with solely because
|
||||||
|
modification has been made.
|
||||||
|
|
||||||
|
If you convey an object code work under this section in, or with, or
|
||||||
|
specifically for use in, a User Product, and the conveying occurs as
|
||||||
|
part of a transaction in which the right of possession and use of the
|
||||||
|
User Product is transferred to the recipient in perpetuity or for a
|
||||||
|
fixed term (regardless of how the transaction is characterized), the
|
||||||
|
Corresponding Source conveyed under this section must be accompanied
|
||||||
|
by the Installation Information. But this requirement does not apply
|
||||||
|
if neither you nor any third party retains the ability to install
|
||||||
|
modified object code on the User Product (for example, the work has
|
||||||
|
been installed in ROM).
|
||||||
|
|
||||||
|
The requirement to provide Installation Information does not include a
|
||||||
|
requirement to continue to provide support service, warranty, or updates
|
||||||
|
for a work that has been modified or installed by the recipient, or for
|
||||||
|
the User Product in which it has been modified or installed. Access to a
|
||||||
|
network may be denied when the modification itself materially and
|
||||||
|
adversely affects the operation of the network or violates the rules and
|
||||||
|
protocols for communication across the network.
|
||||||
|
|
||||||
|
Corresponding Source conveyed, and Installation Information provided,
|
||||||
|
in accord with this section must be in a format that is publicly
|
||||||
|
documented (and with an implementation available to the public in
|
||||||
|
source code form), and must require no special password or key for
|
||||||
|
unpacking, reading or copying.
|
||||||
|
|
||||||
|
7. Additional Terms.
|
||||||
|
|
||||||
|
"Additional permissions" are terms that supplement the terms of this
|
||||||
|
License by making exceptions from one or more of its conditions.
|
||||||
|
Additional permissions that are applicable to the entire Program shall
|
||||||
|
be treated as though they were included in this License, to the extent
|
||||||
|
that they are valid under applicable law. If additional permissions
|
||||||
|
apply only to part of the Program, that part may be used separately
|
||||||
|
under those permissions, but the entire Program remains governed by
|
||||||
|
this License without regard to the additional permissions.
|
||||||
|
|
||||||
|
When you convey a copy of a covered work, you may at your option
|
||||||
|
remove any additional permissions from that copy, or from any part of
|
||||||
|
it. (Additional permissions may be written to require their own
|
||||||
|
removal in certain cases when you modify the work.) You may place
|
||||||
|
additional permissions on material, added by you to a covered work,
|
||||||
|
for which you have or can give appropriate copyright permission.
|
||||||
|
|
||||||
|
Notwithstanding any other provision of this License, for material you
|
||||||
|
add to a covered work, you may (if authorized by the copyright holders of
|
||||||
|
that material) supplement the terms of this License with terms:
|
||||||
|
|
||||||
|
a) Disclaiming warranty or limiting liability differently from the
|
||||||
|
terms of sections 15 and 16 of this License; or
|
||||||
|
|
||||||
|
b) Requiring preservation of specified reasonable legal notices or
|
||||||
|
author attributions in that material or in the Appropriate Legal
|
||||||
|
Notices displayed by works containing it; or
|
||||||
|
|
||||||
|
c) Prohibiting misrepresentation of the origin of that material, or
|
||||||
|
requiring that modified versions of such material be marked in
|
||||||
|
reasonable ways as different from the original version; or
|
||||||
|
|
||||||
|
d) Limiting the use for publicity purposes of names of licensors or
|
||||||
|
authors of the material; or
|
||||||
|
|
||||||
|
e) Declining to grant rights under trademark law for use of some
|
||||||
|
trade names, trademarks, or service marks; or
|
||||||
|
|
||||||
|
f) Requiring indemnification of licensors and authors of that
|
||||||
|
material by anyone who conveys the material (or modified versions of
|
||||||
|
it) with contractual assumptions of liability to the recipient, for
|
||||||
|
any liability that these contractual assumptions directly impose on
|
||||||
|
those licensors and authors.
|
||||||
|
|
||||||
|
All other non-permissive additional terms are considered "further
|
||||||
|
restrictions" within the meaning of section 10. If the Program as you
|
||||||
|
received it, or any part of it, contains a notice stating that it is
|
||||||
|
governed by this License along with a term that is a further
|
||||||
|
restriction, you may remove that term. If a license document contains
|
||||||
|
a further restriction but permits relicensing or conveying under this
|
||||||
|
License, you may add to a covered work material governed by the terms
|
||||||
|
of that license document, provided that the further restriction does
|
||||||
|
not survive such relicensing or conveying.
|
||||||
|
|
||||||
|
If you add terms to a covered work in accord with this section, you
|
||||||
|
must place, in the relevant source files, a statement of the
|
||||||
|
additional terms that apply to those files, or a notice indicating
|
||||||
|
where to find the applicable terms.
|
||||||
|
|
||||||
|
Additional terms, permissive or non-permissive, may be stated in the
|
||||||
|
form of a separately written license, or stated as exceptions;
|
||||||
|
the above requirements apply either way.
|
||||||
|
|
||||||
|
8. Termination.
|
||||||
|
|
||||||
|
You may not propagate or modify a covered work except as expressly
|
||||||
|
provided under this License. Any attempt otherwise to propagate or
|
||||||
|
modify it is void, and will automatically terminate your rights under
|
||||||
|
this License (including any patent licenses granted under the third
|
||||||
|
paragraph of section 11).
|
||||||
|
|
||||||
|
However, if you cease all violation of this License, then your
|
||||||
|
license from a particular copyright holder is reinstated (a)
|
||||||
|
provisionally, unless and until the copyright holder explicitly and
|
||||||
|
finally terminates your license, and (b) permanently, if the copyright
|
||||||
|
holder fails to notify you of the violation by some reasonable means
|
||||||
|
prior to 60 days after the cessation.
|
||||||
|
|
||||||
|
Moreover, your license from a particular copyright holder is
|
||||||
|
reinstated permanently if the copyright holder notifies you of the
|
||||||
|
violation by some reasonable means, this is the first time you have
|
||||||
|
received notice of violation of this License (for any work) from that
|
||||||
|
copyright holder, and you cure the violation prior to 30 days after
|
||||||
|
your receipt of the notice.
|
||||||
|
|
||||||
|
Termination of your rights under this section does not terminate the
|
||||||
|
licenses of parties who have received copies or rights from you under
|
||||||
|
this License. If your rights have been terminated and not permanently
|
||||||
|
reinstated, you do not qualify to receive new licenses for the same
|
||||||
|
material under section 10.
|
||||||
|
|
||||||
|
9. Acceptance Not Required for Having Copies.
|
||||||
|
|
||||||
|
You are not required to accept this License in order to receive or
|
||||||
|
run a copy of the Program. Ancillary propagation of a covered work
|
||||||
|
occurring solely as a consequence of using peer-to-peer transmission
|
||||||
|
to receive a copy likewise does not require acceptance. However,
|
||||||
|
nothing other than this License grants you permission to propagate or
|
||||||
|
modify any covered work. These actions infringe copyright if you do
|
||||||
|
not accept this License. Therefore, by modifying or propagating a
|
||||||
|
covered work, you indicate your acceptance of this License to do so.
|
||||||
|
|
||||||
|
10. Automatic Licensing of Downstream Recipients.
|
||||||
|
|
||||||
|
Each time you convey a covered work, the recipient automatically
|
||||||
|
receives a license from the original licensors, to run, modify and
|
||||||
|
propagate that work, subject to this License. You are not responsible
|
||||||
|
for enforcing compliance by third parties with this License.
|
||||||
|
|
||||||
|
An "entity transaction" is a transaction transferring control of an
|
||||||
|
organization, or substantially all assets of one, or subdividing an
|
||||||
|
organization, or merging organizations. If propagation of a covered
|
||||||
|
work results from an entity transaction, each party to that
|
||||||
|
transaction who receives a copy of the work also receives whatever
|
||||||
|
licenses to the work the party's predecessor in interest had or could
|
||||||
|
give under the previous paragraph, plus a right to possession of the
|
||||||
|
Corresponding Source of the work from the predecessor in interest, if
|
||||||
|
the predecessor has it or can get it with reasonable efforts.
|
||||||
|
|
||||||
|
You may not impose any further restrictions on the exercise of the
|
||||||
|
rights granted or affirmed under this License. For example, you may
|
||||||
|
not impose a license fee, royalty, or other charge for exercise of
|
||||||
|
rights granted under this License, and you may not initiate litigation
|
||||||
|
(including a cross-claim or counterclaim in a lawsuit) alleging that
|
||||||
|
any patent claim is infringed by making, using, selling, offering for
|
||||||
|
sale, or importing the Program or any portion of it.
|
||||||
|
|
||||||
|
11. Patents.
|
||||||
|
|
||||||
|
A "contributor" is a copyright holder who authorizes use under this
|
||||||
|
License of the Program or a work on which the Program is based. The
|
||||||
|
work thus licensed is called the contributor's "contributor version".
|
||||||
|
|
||||||
|
A contributor's "essential patent claims" are all patent claims
|
||||||
|
owned or controlled by the contributor, whether already acquired or
|
||||||
|
hereafter acquired, that would be infringed by some manner, permitted
|
||||||
|
by this License, of making, using, or selling its contributor version,
|
||||||
|
but do not include claims that would be infringed only as a
|
||||||
|
consequence of further modification of the contributor version. For
|
||||||
|
purposes of this definition, "control" includes the right to grant
|
||||||
|
patent sublicenses in a manner consistent with the requirements of
|
||||||
|
this License.
|
||||||
|
|
||||||
|
Each contributor grants you a non-exclusive, worldwide, royalty-free
|
||||||
|
patent license under the contributor's essential patent claims, to
|
||||||
|
make, use, sell, offer for sale, import and otherwise run, modify and
|
||||||
|
propagate the contents of its contributor version.
|
||||||
|
|
||||||
|
In the following three paragraphs, a "patent license" is any express
|
||||||
|
agreement or commitment, however denominated, not to enforce a patent
|
||||||
|
(such as an express permission to practice a patent or covenant not to
|
||||||
|
sue for patent infringement). To "grant" such a patent license to a
|
||||||
|
party means to make such an agreement or commitment not to enforce a
|
||||||
|
patent against the party.
|
||||||
|
|
||||||
|
If you convey a covered work, knowingly relying on a patent license,
|
||||||
|
and the Corresponding Source of the work is not available for anyone
|
||||||
|
to copy, free of charge and under the terms of this License, through a
|
||||||
|
publicly available network server or other readily accessible means,
|
||||||
|
then you must either (1) cause the Corresponding Source to be so
|
||||||
|
available, or (2) arrange to deprive yourself of the benefit of the
|
||||||
|
patent license for this particular work, or (3) arrange, in a manner
|
||||||
|
consistent with the requirements of this License, to extend the patent
|
||||||
|
license to downstream recipients. "Knowingly relying" means you have
|
||||||
|
actual knowledge that, but for the patent license, your conveying the
|
||||||
|
covered work in a country, or your recipient's use of the covered work
|
||||||
|
in a country, would infringe one or more identifiable patents in that
|
||||||
|
country that you have reason to believe are valid.
|
||||||
|
|
||||||
|
If, pursuant to or in connection with a single transaction or
|
||||||
|
arrangement, you convey, or propagate by procuring conveyance of, a
|
||||||
|
covered work, and grant a patent license to some of the parties
|
||||||
|
receiving the covered work authorizing them to use, propagate, modify
|
||||||
|
or convey a specific copy of the covered work, then the patent license
|
||||||
|
you grant is automatically extended to all recipients of the covered
|
||||||
|
work and works based on it.
|
||||||
|
|
||||||
|
A patent license is "discriminatory" if it does not include within
|
||||||
|
the scope of its coverage, prohibits the exercise of, or is
|
||||||
|
conditioned on the non-exercise of one or more of the rights that are
|
||||||
|
specifically granted under this License. You may not convey a covered
|
||||||
|
work if you are a party to an arrangement with a third party that is
|
||||||
|
in the business of distributing software, under which you make payment
|
||||||
|
to the third party based on the extent of your activity of conveying
|
||||||
|
the work, and under which the third party grants, to any of the
|
||||||
|
parties who would receive the covered work from you, a discriminatory
|
||||||
|
patent license (a) in connection with copies of the covered work
|
||||||
|
conveyed by you (or copies made from those copies), or (b) primarily
|
||||||
|
for and in connection with specific products or compilations that
|
||||||
|
contain the covered work, unless you entered into that arrangement,
|
||||||
|
or that patent license was granted, prior to 28 March 2007.
|
||||||
|
|
||||||
|
Nothing in this License shall be construed as excluding or limiting
|
||||||
|
any implied license or other defenses to infringement that may
|
||||||
|
otherwise be available to you under applicable patent law.
|
||||||
|
|
||||||
|
12. No Surrender of Others' Freedom.
|
||||||
|
|
||||||
|
If conditions are imposed on you (whether by court order, agreement or
|
||||||
|
otherwise) that contradict the conditions of this License, they do not
|
||||||
|
excuse you from the conditions of this License. If you cannot convey a
|
||||||
|
covered work so as to satisfy simultaneously your obligations under this
|
||||||
|
License and any other pertinent obligations, then as a consequence you may
|
||||||
|
not convey it at all. For example, if you agree to terms that obligate you
|
||||||
|
to collect a royalty for further conveying from those to whom you convey
|
||||||
|
the Program, the only way you could satisfy both those terms and this
|
||||||
|
License would be to refrain entirely from conveying the Program.
|
||||||
|
|
||||||
|
13. Use with the GNU Affero General Public License.
|
||||||
|
|
||||||
|
Notwithstanding any other provision of this License, you have
|
||||||
|
permission to link or combine any covered work with a work licensed
|
||||||
|
under version 3 of the GNU Affero General Public License into a single
|
||||||
|
combined work, and to convey the resulting work. The terms of this
|
||||||
|
License will continue to apply to the part which is the covered work,
|
||||||
|
but the special requirements of the GNU Affero General Public License,
|
||||||
|
section 13, concerning interaction through a network will apply to the
|
||||||
|
combination as such.
|
||||||
|
|
||||||
|
14. Revised Versions of this License.
|
||||||
|
|
||||||
|
The Free Software Foundation may publish revised and/or new versions of
|
||||||
|
the GNU General Public License from time to time. Such new versions will
|
||||||
|
be similar in spirit to the present version, but may differ in detail to
|
||||||
|
address new problems or concerns.
|
||||||
|
|
||||||
|
Each version is given a distinguishing version number. If the
|
||||||
|
Program specifies that a certain numbered version of the GNU General
|
||||||
|
Public License "or any later version" applies to it, you have the
|
||||||
|
option of following the terms and conditions either of that numbered
|
||||||
|
version or of any later version published by the Free Software
|
||||||
|
Foundation. If the Program does not specify a version number of the
|
||||||
|
GNU General Public License, you may choose any version ever published
|
||||||
|
by the Free Software Foundation.
|
||||||
|
|
||||||
|
If the Program specifies that a proxy can decide which future
|
||||||
|
versions of the GNU General Public License can be used, that proxy's
|
||||||
|
public statement of acceptance of a version permanently authorizes you
|
||||||
|
to choose that version for the Program.
|
||||||
|
|
||||||
|
Later license versions may give you additional or different
|
||||||
|
permissions. However, no additional obligations are imposed on any
|
||||||
|
author or copyright holder as a result of your choosing to follow a
|
||||||
|
later version.
|
||||||
|
|
||||||
|
15. Disclaimer of Warranty.
|
||||||
|
|
||||||
|
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
|
||||||
|
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
|
||||||
|
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
|
||||||
|
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
|
||||||
|
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
||||||
|
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
|
||||||
|
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
|
||||||
|
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
||||||
|
|
||||||
|
16. Limitation of Liability.
|
||||||
|
|
||||||
|
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
||||||
|
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
|
||||||
|
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
|
||||||
|
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
|
||||||
|
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
|
||||||
|
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
|
||||||
|
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
|
||||||
|
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
|
||||||
|
SUCH DAMAGES.
|
||||||
|
|
||||||
|
17. Interpretation of Sections 15 and 16.
|
||||||
|
|
||||||
|
If the disclaimer of warranty and limitation of liability provided
|
||||||
|
above cannot be given local legal effect according to their terms,
|
||||||
|
reviewing courts shall apply local law that most closely approximates
|
||||||
|
an absolute waiver of all civil liability in connection with the
|
||||||
|
Program, unless a warranty or assumption of liability accompanies a
|
||||||
|
copy of the Program in return for a fee.
|
||||||
|
|
||||||
|
END OF TERMS AND CONDITIONS
|
||||||
|
|
||||||
|
How to Apply These Terms to Your New Programs
|
||||||
|
|
||||||
|
If you develop a new program, and you want it to be of the greatest
|
||||||
|
possible use to the public, the best way to achieve this is to make it
|
||||||
|
free software which everyone can redistribute and change under these terms.
|
||||||
|
|
||||||
|
To do so, attach the following notices to the program. It is safest
|
||||||
|
to attach them to the start of each source file to most effectively
|
||||||
|
state the exclusion of warranty; and each file should have at least
|
||||||
|
the "copyright" line and a pointer to where the full notice is found.
|
||||||
|
|
||||||
|
<one line to give the program's name and a brief idea of what it does.>
|
||||||
|
Copyright (C) <year> <name of author>
|
||||||
|
|
||||||
|
This program is free software: you can redistribute it and/or modify
|
||||||
|
it under the terms of the GNU General Public License as published by
|
||||||
|
the Free Software Foundation, either version 3 of the License, or
|
||||||
|
(at your option) any later version.
|
||||||
|
|
||||||
|
This program is distributed in the hope that it will be useful,
|
||||||
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
GNU General Public License for more details.
|
||||||
|
|
||||||
|
You should have received a copy of the GNU General Public License
|
||||||
|
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
Also add information on how to contact you by electronic and paper mail.
|
||||||
|
|
||||||
|
If the program does terminal interaction, make it output a short
|
||||||
|
notice like this when it starts in an interactive mode:
|
||||||
|
|
||||||
|
<program> Copyright (C) <year> <name of author>
|
||||||
|
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
|
||||||
|
This is free software, and you are welcome to redistribute it
|
||||||
|
under certain conditions; type `show c' for details.
|
||||||
|
|
||||||
|
The hypothetical commands `show w' and `show c' should show the appropriate
|
||||||
|
parts of the General Public License. Of course, your program's commands
|
||||||
|
might be different; for a GUI interface, you would use an "about box".
|
||||||
|
|
||||||
|
You should also get your employer (if you work as a programmer) or school,
|
||||||
|
if any, to sign a "copyright disclaimer" for the program, if necessary.
|
||||||
|
For more information on this, and how to apply and follow the GNU GPL, see
|
||||||
|
<https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
The GNU General Public License does not permit incorporating your program
|
||||||
|
into proprietary programs. If your program is a subroutine library, you
|
||||||
|
may consider it more useful to permit linking proprietary applications with
|
||||||
|
the library. If this is what you want to do, use the GNU Lesser General
|
||||||
|
Public License instead of this License. But first, please read
|
||||||
|
<https://www.gnu.org/licenses/why-not-lgpl.html>.
|
||||||
@@ -1,11 +1,13 @@
|
|||||||
SHELL := bash
|
SHELL := bash
|
||||||
.ONESHELL:
|
.ONESHELL:
|
||||||
.PHONY: check test vet build smoke fmt image
|
.PHONY: check test vet build smoke fmt
|
||||||
|
|
||||||
# Lint + unit tests. Mirrors the `test` CI job; runs green inside `nix develop`.
|
# Lint + unit tests. Mirrors the `test` CI job; runs green inside `nix develop`.
|
||||||
check: vet test
|
check: vet test
|
||||||
shellcheck scripts/*.sh
|
shellcheck scripts/*.sh
|
||||||
yamllint .gitea
|
yamllint .gitea
|
||||||
|
bash tests/workflow-contract.sh
|
||||||
|
bash tests/prune-release-assets.sh
|
||||||
|
|
||||||
vet:
|
vet:
|
||||||
go vet ./...
|
go vet ./...
|
||||||
@@ -22,9 +24,3 @@ smoke: build
|
|||||||
|
|
||||||
fmt:
|
fmt:
|
||||||
gofmt -l -w cmd internal
|
gofmt -l -w cmd internal
|
||||||
|
|
||||||
# Build the Crucible image locally. Requires docker; mirrors build-image CI.
|
|
||||||
IMAGE ?= crucible
|
|
||||||
GIT_SHA ?= $(shell git rev-parse --short=12 HEAD 2>/dev/null || echo unknown)
|
|
||||||
image:
|
|
||||||
docker build --build-arg GIT_SHA=$(GIT_SHA) -f docker/Dockerfile -t $(IMAGE):$(GIT_SHA) .
|
|
||||||
|
|||||||
@@ -19,6 +19,7 @@ Crucible is how the artifact repos turn source into artifacts.
|
|||||||
| `crucible-depot` | `crucible depot` | content-addressed depot blob verify/move |
|
| `crucible-depot` | `crucible depot` | content-addressed depot blob verify/move |
|
||||||
| `crucible-hak` | `crucible hak` | ERF/HAK pack/unpack + hak manifests |
|
| `crucible-hak` | `crucible hak` | ERF/HAK pack/unpack + hak manifests |
|
||||||
| `crucible-module` | `crucible module` | build/extract/validate/compare the `.mod` |
|
| `crucible-module` | `crucible module` | build/extract/validate/compare the `.mod` |
|
||||||
|
| `crucible-nwsync` | `crucible nwsync` | NWSync blob emit + manifest assemble + verify |
|
||||||
| `crucible-topdata` | `crucible topdata` | compile 2da/tlk topdata + packages |
|
| `crucible-topdata` | `crucible topdata` | compile 2da/tlk topdata + packages |
|
||||||
| `crucible-wiki` | `crucible wiki` | render + deploy mechanical wiki pages |
|
| `crucible-wiki` | `crucible wiki` | render + deploy mechanical wiki pages |
|
||||||
|
|
||||||
@@ -29,7 +30,7 @@ command lands is mapped in [`docs/command-surface.md`](docs/command-surface.md).
|
|||||||
|
|
||||||
## Status (cutover performed)
|
## Status (cutover performed)
|
||||||
|
|
||||||
The internal `app`/`pipeline`/`project`/`erf`/`gff`/`topdata`/`music`/`changelog`/
|
The internal `app`/`pipeline`/`project`/`erf`/`gff`/`topdata`/`changelog`/
|
||||||
`validator` packages from `gitea/sow-tools` have been migrated into this tree, and
|
`validator` packages from `gitea/sow-tools` have been migrated into this tree, and
|
||||||
the `module`, `topdata`, `hak`, and `wiki` builders now **delegate to the migrated
|
the `module`, `topdata`, `hak`, and `wiki` builders now **delegate to the migrated
|
||||||
`nwn-tool` command surface** (mapped in
|
`nwn-tool` command surface** (mapped in
|
||||||
@@ -49,7 +50,7 @@ which downloads the latest released `crucible` for your OS and runs it:
|
|||||||
```bash
|
```bash
|
||||||
./crucible # interactive menu (pick a command)
|
./crucible # interactive menu (pick a command)
|
||||||
./crucible module build
|
./crucible module build
|
||||||
./crucible topdata validate-topdata
|
./crucible topdata validate
|
||||||
```
|
```
|
||||||
|
|
||||||
Windows (PowerShell):
|
Windows (PowerShell):
|
||||||
@@ -62,40 +63,38 @@ The binary is cached under `~/.cache/crucible/<version>/` (`%LOCALAPPDATA%\cruci
|
|||||||
on Windows); `--repo-local` caches inside the repo instead. Private releases:
|
on Windows); `--repo-local` caches inside the repo instead. Private releases:
|
||||||
set `CRUCIBLE_TOKEN` or write the token to `~/.config/crucible/token`.
|
set `CRUCIBLE_TOKEN` or write the token to `~/.config/crucible/token`.
|
||||||
|
|
||||||
Only the **music** builder needs `ffmpeg`; everything else has zero dependencies.
|
|
||||||
If you run a music command without it, Crucible prints a per-OS install hint.
|
|
||||||
|
|
||||||
## Develop
|
## Develop
|
||||||
|
|
||||||
Self-contained (D8) — a host with only Nix can run everything:
|
Self-contained (D8) — a host with only Nix can run everything:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
nix develop # Go + ffmpeg + shellcheck + yamllint + make
|
nix develop # Go + shellcheck + yamllint + make
|
||||||
make check # go vet + go test + shellcheck + yamllint
|
make check # go vet + go test + shellcheck + yamllint
|
||||||
make build # build every cmd/* into ./bin (gitignored)
|
make build # build every cmd/* into ./bin (gitignored)
|
||||||
make smoke # build + assert the fail-closed contract
|
make smoke # build + assert the fail-closed contract
|
||||||
make image # docker build -> crucible:<sha>
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Binaries are **never committed** — they are CI artifacts / image layers (D19).
|
Binaries are **never committed** — they are CI artifacts (D19).
|
||||||
This retires the old habit of checking in `nwn-tool` / `sow-toolkit`.
|
This retires the old habit of checking in `nwn-tool` / `sow-toolkit`.
|
||||||
|
|
||||||
## CI
|
## CI
|
||||||
|
|
||||||
PR-first (D7): checks run on pull requests and on push to `main`; the only
|
PR-first (D7): checks run once on pull requests; the only publish event is a
|
||||||
publish event is a `v*` tag (see `sow-docs/runbooks/ci-trigger-standard.md`).
|
`v*` tag (see `runbooks/ci-trigger-standard.md` in sow-docs,
|
||||||
|
https://git.westgate.pw/ShadowsOverWestgate/sow-docs).
|
||||||
|
|
||||||
- `test.yml` — vet, test, shellcheck, yamllint, binary smoke (PR + main).
|
- `ci.yml` — vet, test, shellcheck, yamllint, binary smoke, and cross-build all
|
||||||
- `test-image.yml` — build the OCI image to prove it compiles (PR + main, no push).
|
targets once per pull request.
|
||||||
- `build-binaries.yml` — cross-build all targets (PR + main); on a `v*` tag, upload
|
- `build-binaries.yml` — on a `v*` tag, cross-build and upload the binaries,
|
||||||
the binaries, `SHA256SUMS`, and the wrappers to the Gitea release.
|
`SHA256SUMS`, and the wrappers to the Gitea release, then delete the assets
|
||||||
- `build-image.yml` — on a `v*` tag, build and publish
|
of every release except the newest two — Gitea keeps them forever otherwise,
|
||||||
`registry.westgate.pw/deployment/crucible:<sha>`.
|
and every binary is reproducible from its tag.
|
||||||
- `publish-image.yml` — manual `workflow_dispatch` break-glass republish.
|
|
||||||
- `sync-wrappers.yml` — on a `main` push that touches `wrappers/`, auto-PR the
|
- `sync-wrappers.yml` — on a `main` push that touches `wrappers/`, auto-PR the
|
||||||
canonical wrappers to the consumer repos in `wrappers/consumers.txt`.
|
canonical wrappers to the consumer repos in `wrappers/consumers.txt`.
|
||||||
|
Consumer drift checks run after those PRs merge to `main`, not on the PRs
|
||||||
|
themselves, to avoid recursive cross-repo checks.
|
||||||
|
|
||||||
## Consumers
|
## Consumers
|
||||||
|
|
||||||
How the artifact repos resolve a Crucible binary (and the `NWN_ROOT` rule) is
|
How the artifact repos resolve a Crucible binary is
|
||||||
documented in [`docs/consumer-contract.md`](docs/consumer-contract.md).
|
documented in [`docs/consumer-contract.md`](docs/consumer-contract.md).
|
||||||
|
|||||||
@@ -0,0 +1,11 @@
|
|||||||
|
// Command crucible-assets is the standalone assets builder (equivalent to
|
||||||
|
// `crucible assets`). A single-token binary keeps consumer wrapper scripts simple.
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
|
||||||
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/dispatch"
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() { os.Exit(dispatch.RunBuilder("assets", os.Args[1:])) }
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
// Command crucible-nwsync is the standalone nwsync builder (equivalent to
|
||||||
|
// `crucible nwsync`). A single-token binary keeps consumer wrapper scripts simple.
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
|
||||||
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/dispatch"
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() { os.Exit(dispatch.RunBuilder("nwsync", os.Args[1:])) }
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
# syntax=docker/dockerfile:1
|
|
||||||
#
|
|
||||||
# Crucible toolchain image: registry.westgate.pw/deployment/crucible:<git-sha>
|
|
||||||
#
|
|
||||||
# Reproducible multi-stage build, no on-VPS build. Produces every cmd/* binary
|
|
||||||
# and ships them on a static base. The `crucible` dispatcher is the entrypoint;
|
|
||||||
# consumer CI can also call the standalone crucible-<name> binaries by path.
|
|
||||||
#
|
|
||||||
# NOTE: the internal pipeline is migrated and the music conversion path is wired,
|
|
||||||
# so the runtime stage is debian-slim with ffmpeg on PATH (BMU encode/decode).
|
|
||||||
# See docs/migration-from-nwn-tool.md.
|
|
||||||
|
|
||||||
FROM golang:1.26-alpine AS build
|
|
||||||
WORKDIR /src
|
|
||||||
RUN apk add --no-cache git
|
|
||||||
# go.sum is committed now that the migrated packages pull golang.org/x/text and
|
|
||||||
# gopkg.in/yaml.v3; the glob keeps the build working if it is ever absent.
|
|
||||||
COPY go.mod go.sum* ./
|
|
||||||
RUN go mod download
|
|
||||||
COPY . .
|
|
||||||
ARG GIT_SHA=unknown
|
|
||||||
ENV CGO_ENABLED=0
|
|
||||||
RUN set -eux; \
|
|
||||||
mkdir -p /out; \
|
|
||||||
for dir in ./cmd/*/; do \
|
|
||||||
name="$(basename "${dir}")"; \
|
|
||||||
go build -trimpath \
|
|
||||||
-ldflags "-s -w -X git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo.Version=${GIT_SHA}" \
|
|
||||||
-o "/out/${name}" "${dir}"; \
|
|
||||||
done
|
|
||||||
|
|
||||||
FROM debian:12-slim
|
|
||||||
# ffmpeg: the migrated music pipeline shells out to it for BMU conversion.
|
|
||||||
# ca-certificates: builders fetch published manifests over HTTPS.
|
|
||||||
RUN set -eux; \
|
|
||||||
apt-get update; \
|
|
||||||
apt-get install -y --no-install-recommends ffmpeg ca-certificates; \
|
|
||||||
rm -rf /var/lib/apt/lists/*; \
|
|
||||||
useradd --system --create-home --uid 65532 nonroot
|
|
||||||
COPY --from=build /out/ /usr/local/bin/
|
|
||||||
USER nonroot
|
|
||||||
ENTRYPOINT ["/usr/local/bin/crucible"]
|
|
||||||
CMD ["help"]
|
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
# ADR-0001: Markdown in this repo is reference, law, or an ADR — nothing else
|
||||||
|
|
||||||
|
- **Status:** Accepted
|
||||||
|
- **Date:** 2026-07-24
|
||||||
|
- **Origin:** Adopted workspace-wide from `sow-codebase` ADR-0001, which records
|
||||||
|
the full context (a `docs/` tree that had grown to 434 files, ~27MB, most of
|
||||||
|
it dead process artifacts). This repo adopts the same law so the rule is
|
||||||
|
local, not a cross-repo reference.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Markdown that *claims to describe current state or a plan* rots, because
|
||||||
|
reality diverges and nobody edits the file. Markdown that claims neither — a
|
||||||
|
dated, immutable decision record, or a standing rule — does not rot.
|
||||||
|
|
||||||
|
Two different things get conflated:
|
||||||
|
|
||||||
|
- **Rationale** — why a system is shaped the way it is. Worth keeping.
|
||||||
|
- **Process artifacts** — plans, specs, concepts, handoffs, trackers. A record
|
||||||
|
of how work happened, not what is true now.
|
||||||
|
|
||||||
|
Left unchecked, every completed effort leaves its planning behind and the repo
|
||||||
|
accumulates a "historical" pile that agents and humans must route around to
|
||||||
|
find the few live docs.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Markdown may exist in this repo only as one of three genres:
|
||||||
|
|
||||||
|
1. **Current-state reference** — describes what the code does now, kept honest
|
||||||
|
by code review touching it.
|
||||||
|
2. **Standing law** — `DOCTRINE.md`, root and folder-scoped `AGENTS.md`,
|
||||||
|
`CONTEXT.md`. States rules and vocabulary, not plans.
|
||||||
|
3. **Immutable decision record** — ADRs under `docs/adr/`. Append-only; never
|
||||||
|
edited, only superseded by a later ADR that points back.
|
||||||
|
|
||||||
|
Anything else — implementation plans, design specs, concepts, handoffs,
|
||||||
|
progress trackers, scratch — is **process** and does not live in the repo. It
|
||||||
|
lives in Gitea issues and wayfinder maps, where it can be assigned, closed, and
|
||||||
|
superseded. Small tasks need no written plan at all.
|
||||||
|
|
||||||
|
Deleting a process document is not destroying history: `git log -- <path>`
|
||||||
|
recovers it. The exception is *unfinished intent* (a design never built, an
|
||||||
|
open question still wanted) — that is live, not history, and must be harvested
|
||||||
|
to a Gitea ticket before its file is deleted. A citation from a living doc or
|
||||||
|
from code must be resolved before the cited file is deleted.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- The documentation map lists only current truth; there is no "historical" pile
|
||||||
|
to route around.
|
||||||
|
- No agent, under any plugin or skill, writes process-markdown into the repo.
|
||||||
|
The rule is plugin-agnostic on purpose — it binds the agent, not a named tool.
|
||||||
|
- History of deleted process docs is in git; unfinished intent is in the issue
|
||||||
|
tracker; rationale is in ADRs and law. Each thing has exactly one home.
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
# ADR-0002: `sow-tools` becomes the Crucible suite
|
||||||
|
|
||||||
|
- **Status:** Accepted
|
||||||
|
- **Date:** 2026-06-11
|
||||||
|
- **Migrated from:** `sow-docs` `07-decisions-log.md` entry **D11**, on the
|
||||||
|
retirement of `sow-docs`. Content is the original decision, unchanged.
|
||||||
|
|
||||||
|
- **Decision:** Rename the future toolchain surface to Crucible: one Go module,
|
||||||
|
multiple `cmd/` binaries, plus a dispatcher so wrapper commands can stay
|
||||||
|
stable.
|
||||||
|
- **Rationale:** Depot, HAK, module, topdata, and wiki tooling share internals
|
||||||
|
but have different command surfaces. One module avoids premature repo splits.
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# ADR-0003: Crucible binary contract: standalone shims, fail-closed scaffold, no committed binaries
|
||||||
|
|
||||||
|
- **Status:** Accepted
|
||||||
|
- **Date:** 2026-06-11
|
||||||
|
- **Migrated from:** `sow-docs` `07-decisions-log.md` entry **D19**, on the
|
||||||
|
retirement of `sow-docs`. Content is the original decision, unchanged.
|
||||||
|
|
||||||
|
- **Decision:** Phase 5 builds `migration/sow-tools` (Crucible, D11) as a
|
||||||
|
multi-binary Go scaffold:
|
||||||
|
- One `crucible` dispatcher plus standalone `crucible-{depot,hak,module,topdata,wiki}`
|
||||||
|
binaries sharing one registry (`internal/dispatch`). The dispatcher is for
|
||||||
|
humans/CI; the **standalone shims are canonical for consumers** so wrapper
|
||||||
|
scripts resolve a single-token command and `"$builder" args` quoting stays
|
||||||
|
correct.
|
||||||
|
- Consumer resolution order: explicit env override
|
||||||
|
(`$SOW_MODULE_BUILD`/`$SOW_TOPDATA_BUILD`/`$CRUCIBLE`) → `crucible-<name>` →
|
||||||
|
legacy `sow-<name>-build`. CI runs inside the pinned `crucible:<sha>` image.
|
||||||
|
- Every builder is **unwired** in the scaffold and fails closed (exit 70);
|
||||||
|
it never fakes an artifact. The `internal/` logic is migrated from
|
||||||
|
`gitea/sow-tools` by the operator at cutover (hard rule: no source
|
||||||
|
transplant by tooling).
|
||||||
|
- **Binaries are never committed** — they are CI artifacts / image layers.
|
||||||
|
Retires the committed `nwn-tool` / `tools/sow-toolkit`.
|
||||||
|
- Builders take `NWN_ROOT` only via explicit env/flag; no `$HOME` defaulting.
|
||||||
|
- **Rationale:** Locks the command surface, container, and CI shape so migrated
|
||||||
|
logic drops into a stable frame; aligns the three artifact repos onto one
|
||||||
|
Crucible contract while keeping back-compat with the pre-D11 skeletons.
|
||||||
|
- **Image name:** `registry.westgate.pw/deployment/crucible:<git-sha>` (the bare
|
||||||
|
`crucible:<sha>` is the local/dev tag).
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
# ADR-0004: Nix-built OCI images start server-side with Crucible; local Anvil images deferred
|
||||||
|
|
||||||
|
- **Status:** Accepted
|
||||||
|
- **Date:** 2026-06-12
|
||||||
|
- **Migrated from:** `sow-docs` `07-decisions-log.md` entry **D29**, on the
|
||||||
|
retirement of `sow-docs`. Content is the original decision, unchanged.
|
||||||
|
|
||||||
|
- **Decision:** Introduce Nix-built OCI images as an artifact-production option,
|
||||||
|
starting with `sow-tools`/Crucible in server-side CI. `sow-tools` will grow a
|
||||||
|
Nix-built `crucible` package plus `crucible-image`; PR CI builds/smokes it and
|
||||||
|
`main` publishes the normal pinned OCI image
|
||||||
|
`registry.westgate.pw/deployment/crucible:<sha>`. `docker/Dockerfile` stays during
|
||||||
|
parity and as the client-compatible fallback.
|
||||||
|
- **Local rule:** Nix users may get optional wrappers that provide tools and
|
||||||
|
call the existing Dockerfiles for local Anvil/NWServer testing. A true
|
||||||
|
`nix build .#nwserver-test-image` is explicitly deferred until real
|
||||||
|
`sow-codebase/src` exists and the Dockerfile image is proven.
|
||||||
|
- **Rationale:** Server-side image build shape is harder to change once deploy
|
||||||
|
promotion and registry gates are active, so prove Nix image builds before
|
||||||
|
cutover. Crucible is the lowest-risk pilot because it is a Go toolchain image;
|
||||||
|
NodeBB and Anvil/NWServer depend more heavily on upstream container filesystem
|
||||||
|
semantics.
|
||||||
|
- **Non-goals:** no source builds in `sow-platform`; no `nix-sidecar` or
|
||||||
|
Kubernetes-style runtime cache layer; no removal of client Dockerfiles.
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
# Domain Docs
|
||||||
|
|
||||||
|
How the engineering skills should consume this repo's domain documentation when exploring the codebase.
|
||||||
|
|
||||||
|
## Before exploring, read these
|
||||||
|
|
||||||
|
- **`CONTEXT.md`** at the repo root, or
|
||||||
|
- **`CONTEXT-MAP.md`** at the repo root if it exists — it points at one `CONTEXT.md` per context. Read each one relevant to the topic.
|
||||||
|
- **`docs/adr/`** — read ADRs that touch the area you're about to work in. In multi-context repos, also check `src/<context>/docs/adr/` for context-scoped decisions.
|
||||||
|
|
||||||
|
If any of these files don't exist, **proceed silently**. Don't flag their absence; don't suggest creating them upfront. The `/domain-modeling` skill (reached via `/grill-with-docs` and `/improve-codebase-architecture`) creates them lazily when terms or decisions actually get resolved.
|
||||||
|
|
||||||
|
## File structure
|
||||||
|
|
||||||
|
Single-context repo (most repos):
|
||||||
|
|
||||||
|
```
|
||||||
|
/
|
||||||
|
├── CONTEXT.md
|
||||||
|
├── docs/adr/
|
||||||
|
│ ├── 0001-event-sourced-orders.md
|
||||||
|
│ └── 0002-postgres-for-write-model.md
|
||||||
|
└── src/
|
||||||
|
```
|
||||||
|
|
||||||
|
Multi-context repo (presence of `CONTEXT-MAP.md` at the root):
|
||||||
|
|
||||||
|
```
|
||||||
|
/
|
||||||
|
├── CONTEXT-MAP.md
|
||||||
|
├── docs/adr/ ← system-wide decisions
|
||||||
|
└── src/
|
||||||
|
├── ordering/
|
||||||
|
│ ├── CONTEXT.md
|
||||||
|
│ └── docs/adr/ ← context-specific decisions
|
||||||
|
└── billing/
|
||||||
|
├── CONTEXT.md
|
||||||
|
└── docs/adr/
|
||||||
|
```
|
||||||
|
|
||||||
|
## Use the glossary's vocabulary
|
||||||
|
|
||||||
|
When your output names a domain concept (in an issue title, a refactor proposal, a hypothesis, a test name), use the term as defined in `CONTEXT.md`. Don't drift to synonyms the glossary explicitly avoids.
|
||||||
|
|
||||||
|
If the concept you need isn't in the glossary yet, that's a signal — either you're inventing language the project doesn't use (reconsider) or there's a real gap (note it for `/domain-modeling`).
|
||||||
|
|
||||||
|
## Flag ADR conflicts
|
||||||
|
|
||||||
|
If your output contradicts an existing ADR, surface it explicitly rather than silently overriding:
|
||||||
|
|
||||||
|
> _Contradicts ADR-0007 (event-sourced orders) — but worth reopening because…_
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
# Issue tracker: Gitea (via tea)
|
||||||
|
|
||||||
|
Issues for this repo live in Gitea at `git.westgate.pw`, repo
|
||||||
|
`ShadowsOverWestgate/sow-tools`. Use the `tea` CLI for all operations —
|
||||||
|
`gh` does not work here. For anything `tea` lacks a subcommand for, use
|
||||||
|
`tea api <endpoint>` (Gitea's API mirrors GitHub's closely).
|
||||||
|
|
||||||
|
Authenticate with your own `tea` login (`tea login add`); never commit tokens
|
||||||
|
or tea config into this repo. Note Gitea blocks self-review, so approving a PR
|
||||||
|
needs a different account than the one that opened it.
|
||||||
|
|
||||||
|
## Where work lives
|
||||||
|
|
||||||
|
Markdown in this repo is **reference, law, or an ADR — nothing else**
|
||||||
|
(`sow-codebase` ADR-0001). Four homes, no overlap:
|
||||||
|
|
||||||
|
- **Live work** → wayfinder maps + Gitea issues. Closeable, assignable,
|
||||||
|
queryable. Never a markdown file.
|
||||||
|
- **Settled decisions** → ADR files in `docs/adr/`. Immutable, findable, never
|
||||||
|
closed, never edited — only superseded by a later ADR pointing back. When an
|
||||||
|
issue ends in a durable decision, write the ADR, then close the issue
|
||||||
|
pointing at it. Decisions spanning repos go to `sow-platform/docs/adr/`.
|
||||||
|
- **Standing law** → `DOCTRINE.md`, `AGENTS.md`, `CONTEXT.md`. Rules that are
|
||||||
|
always true and vocabulary everyone shares — not the record of one decision.
|
||||||
|
- **Current-state reference** → docs describing what the code does now, kept
|
||||||
|
honest by review touching them.
|
||||||
|
|
||||||
|
Anything else — implementation plans, design specs, concepts, handoffs,
|
||||||
|
progress trackers, scratch — is **process**. It does not live in this repo. It
|
||||||
|
lives in a Gitea issue or a wayfinder map, where it can be assigned, closed,
|
||||||
|
and superseded. Small tasks need no written plan at all.
|
||||||
|
|
||||||
|
Deleting a process doc is not destroying history — `git log -- <path>` recovers
|
||||||
|
it. But *unfinished intent* (a design never built, an open question still
|
||||||
|
wanted) is live, not history: harvest it to a Gitea issue before deleting.
|
||||||
|
|
||||||
|
`sow-docs` is deprecated and read-only. Never add to it, never send work there.
|
||||||
|
|
||||||
|
## Which repo gets the issue
|
||||||
|
|
||||||
|
Issues follow ownership. File the issue in the repo that **owns the work** —
|
||||||
|
see the Repo/Owns/Produces table in the workspace root `AGENTS.md`. Standing
|
||||||
|
in one repo is not a reason to file there.
|
||||||
|
|
||||||
|
If work spans repos, file it in the repo that owns the *outcome* and reference
|
||||||
|
the others from it. A wrong-repo issue is a routing bug, not a filing
|
||||||
|
preference — move it.
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
- **Create an issue**: `tea issues create --title "..." --description "..."`
|
||||||
|
- **Read an issue**: `tea issues <number>` and
|
||||||
|
`tea api repos/ShadowsOverWestgate/sow-tools/issues/<number>/comments` for comments.
|
||||||
|
- **List issues**: `tea issues list --state open` (add `--labels ...` to filter).
|
||||||
|
- **Comment**: `tea comment <number> "..." </dev/null`
|
||||||
|
Always redirect stdin. `tea` reads stdin to EOF and appends it to the body,
|
||||||
|
so any non-interactive shell (every agent) hangs forever without
|
||||||
|
`</dev/null`. Same trap on `tea issues create --description` and
|
||||||
|
`tea pr create`.
|
||||||
|
- **Apply / remove labels**: `tea api --method PATCH` on the issue, or
|
||||||
|
`tea api repos/ShadowsOverWestgate/sow-tools/issues/<number>/labels` endpoints.
|
||||||
|
- **Close**: `tea issues close <number>`
|
||||||
|
|
||||||
|
`tea` infers the repo from the git remote when run inside the clone.
|
||||||
|
Gitea shares one number space across issues and PRs.
|
||||||
|
|
||||||
|
## Pull requests as a triage surface
|
||||||
|
|
||||||
|
**PRs as a request surface: no.**
|
||||||
|
|
||||||
|
## When a skill says "publish to the issue tracker"
|
||||||
|
|
||||||
|
Create a Gitea issue with `tea issues create`.
|
||||||
|
|
||||||
|
## When a skill says "fetch the relevant ticket"
|
||||||
|
|
||||||
|
Run `tea issues <number>` plus the comments API call above.
|
||||||
|
|
||||||
|
## Wayfinding operations
|
||||||
|
|
||||||
|
Used by `/wayfinder`. The **map** is a single issue with **child** issues as tickets.
|
||||||
|
|
||||||
|
- **Map**: a single issue labelled `wayfinder:map`, holding the Notes / Decisions-so-far / Fog body. `tea issues create --title "..." --description "..." --labels wayfinder:map`.
|
||||||
|
- **Child ticket**: this Gitea instance (v1.27) has no native sub-issue hierarchy, so a child issue carries `Part of #<map>` at the top of its description, and is also added to a task list in the map body. Labels: `wayfinder:<type>` (`research`/`prototype`/`grilling`/`task`). Once claimed, the ticket is assigned to the driving dev.
|
||||||
|
- **Blocking**: Gitea's **native dependencies API** — the canonical, UI-visible representation (shows as "Depends on" / "Blocks" on the issue page). Add an edge with `tea api -X POST repos/ShadowsOverWestgate/sow-tools/issues/<child>/dependencies -f owner=ShadowsOverWestgate -f repo=sow-tools -F index=<blocker>`, where `<blocker>` is the blocker's issue **index** (its `#number` — Gitea's dependency API takes the index directly, unlike GitHub's numeric database id). Check status with `tea api repos/ShadowsOverWestgate/sow-tools/issues/<child>/dependencies` (GET) — a ticket is unblocked when every returned issue's `state` is `closed`.
|
||||||
|
- **Frontier query**: list the map's open children (`tea issues list --state open`, keep the ones whose description contains `Part of #<map>`), drop any with an open dependency (per the GET above) or an assignee; first in map order wins.
|
||||||
|
- **Claim**: `tea issues edit <n> --add-assignees <username>` — the session's first write.
|
||||||
|
- **Resolve**: `tea comment <n> "<answer>" </dev/null`, then `tea issues close <n>`, then append a context pointer (gist + link) to the map's Decisions-so-far.
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
# Triage Labels
|
||||||
|
|
||||||
|
The skills speak in terms of five canonical triage roles. This file maps those roles to the actual label strings used in this repo's issue tracker (Gitea — see `issue-tracker.md` for how to apply labels with `tea`).
|
||||||
|
|
||||||
|
| Label in mattpocock/skills | Label in our tracker | Meaning |
|
||||||
|
| -------------------------- | -------------------- | ---------------------------------------- |
|
||||||
|
| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue |
|
||||||
|
| `needs-info` | `needs-info` | Waiting on reporter for more information |
|
||||||
|
| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent |
|
||||||
|
| `ready-for-human` | `ready-for-human` | Requires human implementation |
|
||||||
|
| `wontfix` | `wontfix` | Will not be actioned |
|
||||||
|
|
||||||
|
When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the corresponding label string from this table.
|
||||||
|
|
||||||
|
Edit the right-hand column to match whatever vocabulary you actually use.
|
||||||
+179
-35
@@ -1,51 +1,195 @@
|
|||||||
# Crucible command surface
|
# Crucible command surface
|
||||||
|
|
||||||
Crucible re-homes the single `nwn-tool` (a.k.a. `sow-toolkit`) command surface
|
`internal/dispatch.Registry` is the single source of truth for Crucible's
|
||||||
into five builders plus a dispatcher (D11). This table maps every legacy command
|
builders, visible commands, descriptions, help, and hidden compatibility
|
||||||
to its Crucible home. It is the migration contract — keep it in sync with
|
aliases.
|
||||||
`internal/dispatch.Registry`.
|
|
||||||
|
|
||||||
## Builders
|
## Visible commands
|
||||||
|
|
||||||
| Builder | Legacy `nwn-tool` commands subsumed |
|
| Builder | Command | Purpose |
|
||||||
| ------------------ | ------------------------------------------------------------------------------------- |
|
| --- | --- | --- |
|
||||||
| `crucible-module` | `build`, `build-module`, `extract`, `validate`, `compare`, `apply-hak-manifest`† |
|
| `hak` | `build` | Build configured HAK archives and their manifest. |
|
||||||
| `crucible-topdata` | `validate-topdata`, `build-topdata`, `build-top-package`, `compare-topdata`, `convert-topdata` |
|
| `hak` | `manifest` | Apply a generated HAK list to module source. |
|
||||||
| `crucible-hak` | `build-haks`, `apply-hak-manifest` |
|
| `module` | `build` | Build the module and configured project outputs. |
|
||||||
| `crucible-wiki` | `build-wiki`, `deploy-wiki` |
|
| `module` | `extract` | Extract built archives into configured source trees. |
|
||||||
| `crucible-depot` | (new) depot blob verify/move — was shell `mc`/S3 in `sow-assets-manifest` |
|
| `module` | `validate` | Validate project layout, config, and source inventory. |
|
||||||
|
| `module` | `compare` | Compare source content with built archives. |
|
||||||
|
| `module` | `manifest` | Apply a generated HAK list to module source. |
|
||||||
|
| `topdata` | `validate` | Validate topdata source and canonical JSON compatibility. |
|
||||||
|
| `topdata` | `build` | Compile topdata and refresh the configured package. |
|
||||||
|
| `topdata` | `package` | Package cached outputs into HAK and TLK files. |
|
||||||
|
| `topdata` | `compare` | Compare current output with a fresh native build. |
|
||||||
|
| `topdata` | `convert` | Convert between 2DA and native JSON/module formats. |
|
||||||
|
| `wiki` | `build` | Render wiki page drafts from compiled topdata. |
|
||||||
|
| `wiki` | `deploy` | Deploy generated wiki pages to NodeBB. |
|
||||||
|
| `assets` | `compile` | Compile ASCII `.mdl` models to binary in place. |
|
||||||
|
| `assets` | `convert` | Convert textures to/from NWN DDS (flips vertically). |
|
||||||
|
| `assets` | `upscale` | Upscale textures through an installed backend. |
|
||||||
|
| `assets` | `check-mdl` | Report uncompiled ASCII `.mdl` and model-name mismatches. |
|
||||||
|
| `assets` | `fix-mdl` | Lowercase `.mdl` names and rewrite model identity to match. |
|
||||||
|
| `assets` | `check-dupes` | Report runtime-name (basename) collisions across dirs. |
|
||||||
|
| `assets` | `clean-dupes` | Delete clean-tree files whose basename collides with primary. |
|
||||||
|
| `depot` | `status` | Report referenced-vs-present drift against a backend. |
|
||||||
|
| `depot` | `push` | Upload referenced-but-absent blobs from a local depot. |
|
||||||
|
| `depot` | `verify` | Existence sweep plus sampled download re-hash. |
|
||||||
|
| `depot` | `get` | Fetch one blob with sha re-verify. |
|
||||||
|
| `depot` | `pull` | Incremental verified pull of every referenced blob. |
|
||||||
|
| `nwsync` | `emit` | Explode one artifact into NWSync blobs plus its own NSYM manifest. |
|
||||||
|
| `nwsync` | `assemble` | Merge per-artifact NSYM manifests into one merged manifest. |
|
||||||
|
| `nwsync` | `verify` | Decompress and hash a published manifest's blobs through the pull zone. |
|
||||||
|
|
||||||
† `apply-hak-manifest` is HAK-list maintenance on a module; it is reachable from
|
`depot status` and `depot get` pick their backend either with `--out DIR`, a
|
||||||
both `crucible-module` and `crucible-hak`. Canonical home is `crucible-hak`.
|
depot tree on disk, or with `--target bunny|cdn`, a remote backend. The two
|
||||||
|
flags are mutually exclusive.
|
||||||
|
|
||||||
## Folded / global commands
|
`nwsync emit` runs where an artifact is born (a `.hak`/`.erf`, or a loose file
|
||||||
|
such as the TLK); `nwsync assemble` runs at module release and reads only the
|
||||||
|
small per-artifact indexes. Both take **depot keys**: an artifact's index lives
|
||||||
|
beside the artifact itself with the extension replaced, so `emit` and
|
||||||
|
`assemble` agree on where it is without being told.
|
||||||
|
|
||||||
These legacy commands are **not** top-level builders:
|
```
|
||||||
|
nwsync emit [--as NAME] [--out DIR] [--jobs N] [--verify] <artifact-key> <file>
|
||||||
|
nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
|
||||||
|
nwsync verify [--sample N] [--base URL] [--jobs N] <manifest-sha1>
|
||||||
|
```
|
||||||
|
|
||||||
- `music *` (`list-datasets`, `scan`, `build`, `credits`, `validate`, `manifest`,
|
`emit` is latency-bound, not CPU-bound: every blob costs an existence probe
|
||||||
`normalize`) → folds into the **module/hak build pipeline** (music BMUs are
|
plus an upload, and a measured backfill spent 26 seconds of CPU across 9.5
|
||||||
packed into HAKs at build time). Exposed as `crucible-module music ...` /
|
minutes of wall clock. `--jobs N` (default 16) sets how many resources are in
|
||||||
`crucible-hak music ...`. Needs `ffmpeg` at runtime (in the image).
|
flight at once. The manifest is byte-identical at any value — the number of
|
||||||
- `config *` (`validate`, `effective`, `inspect`, `explain`, `sources`) → a
|
workers is never observable in the output. Peak memory is `N` times the
|
||||||
**global** concern shared by every builder; exposed as the global
|
per-resource limit of 15 MB plus its compressed copy, so raising `N` far past
|
||||||
`crucible config ...`, not its own binary.
|
the default costs real memory for little gain: the transport keeps 16 idle
|
||||||
- `build-changelog` → release tooling; exposed as the global `crucible changelog`
|
connections per host, and past that a worker pays a fresh TLS handshake.
|
||||||
rather than a content builder.
|
|
||||||
|
|
||||||
## Global commands (implemented)
|
Both verbs upload by default; nothing bulky is ever written to the runner's
|
||||||
|
disk. `--out DIR` writes a local repository tree instead, which is the
|
||||||
|
conformance path against upstream `nwn_nwsync_write`. The zone comes from
|
||||||
|
`NWSYNC_STORAGE_ZONE` and `NWSYNC_STORAGE_PASSWORD`, with the host from
|
||||||
|
`BUNNY_STORAGE_HOST` — NWSync data is a separate zone from the asset depot.
|
||||||
|
|
||||||
|
`assemble`'s artifact keys are in `Mod_HakList` order, highest priority first: a
|
||||||
|
resref in more than one artifact resolves to the earliest one, the way the game
|
||||||
|
resolves it. `--tlk-key` has its own slot because the TLK shadows nothing.
|
||||||
|
`--group-id` is per channel — 1 is current, 2 is testing, and 0 leaves the field
|
||||||
|
out of the sidecar.
|
||||||
|
|
||||||
|
`nwsync verify` is the only check on a published blob upstream of a player's
|
||||||
|
client. It reads the **pull zone**, not the storage API, and needs no
|
||||||
|
credential: what matters is the bytes a client is served, edge behaviour
|
||||||
|
included. Every blob is decompressed and hashed, and the zstd frame is asserted
|
||||||
|
to declare its content size. Neither half is optional — a `Content-Length` check
|
||||||
|
passes a byte-correct-looking object whose contents are short, and a round-trip
|
||||||
|
check alone passes a frame the game client cannot decode but Go's decoder can.
|
||||||
|
Failures are reported per blob as missing, malformed framing, size mismatch or
|
||||||
|
hash mismatch, and the exit code is 1.
|
||||||
|
|
||||||
|
A full sweep of the live manifest is roughly 69,000 blobs and 15 GB, so
|
||||||
|
`--sample N` exists to make verifying routine; the default is a full sweep.
|
||||||
|
`--base URL` (or `NWSYNC_PULL_BASE`) overrides the public host.
|
||||||
|
|
||||||
|
`emit --verify` applies the same check where `emit` would otherwise skip. `emit`
|
||||||
|
normally reads a blob's presence as proof of its contents, decided by a 1-byte
|
||||||
|
range GET, so an object written truncated — or written by an emitter since found
|
||||||
|
broken — is skipped by every later run forever and no backfill repairs it. With
|
||||||
|
`--verify` the stored copy is read back, unwrapped, hashed against its own name,
|
||||||
|
and replaced when it does not match. It costs a full GET per existing blob, so
|
||||||
|
it is a repair pass, not the default.
|
||||||
|
|
||||||
|
`emit` uploads blobs first and the index last, so the presence of an index is
|
||||||
|
the publication marker: an artifact whose emit died halfway leaves real blobs in
|
||||||
|
the zone and no index. Blob names are content hashes, so re-running skips
|
||||||
|
whatever already landed, and `assemble` fails closed on an artifact with no
|
||||||
|
index rather than publishing a manifest that is missing a hak.
|
||||||
|
|
||||||
|
### What an emitted tree looks like
|
||||||
|
|
||||||
|
`--out DIR` produces the same tree `emit` would upload, which makes it the way
|
||||||
|
to check a zone by hand without touching one:
|
||||||
|
|
||||||
|
```
|
||||||
|
<artifact-sha>.nsym binary index
|
||||||
|
<artifact-sha>.nsym.json the same index, readable
|
||||||
|
data/sha1/a7/4a/a74aa84a... one blob per resource, two-level fanout
|
||||||
|
```
|
||||||
|
|
||||||
|
A blob's name is the SHA-1 of the resource's **original** bytes, but the file on
|
||||||
|
disk is not those bytes: each blob is wrapped in NWCompressedBuffer framing, a
|
||||||
|
24-byte `NSYC` header followed by a zstd frame. Hashing the file directly will
|
||||||
|
not match its name, which is the obvious
|
||||||
|
first thing to try and the obvious first thing to be confused by. Strip the
|
||||||
|
header first:
|
||||||
|
|
||||||
|
```
|
||||||
|
tail -c +25 <blob> | zstd -dc | sha1sum # == the blob's filename
|
||||||
|
```
|
||||||
|
|
||||||
|
The header carries the uncompressed length as a little-endian `uint32` at offset
|
||||||
|
12, so the decompressed size is checkable without decompressing. Compression is
|
||||||
|
worth roughly a 4:1
|
||||||
|
saving on hak content: a 250 MB hak emitted 2296 blobs totalling 59 MB on disk
|
||||||
|
against 249 MB of resources, as recorded in the sidecar's `on_disk_bytes` and
|
||||||
|
`total_bytes`.
|
||||||
|
|
||||||
|
The zstd frame always declares its `Frame_Content_Size`. The game client sizes
|
||||||
|
its output buffer from that field and cannot decode a frame without one, but the
|
||||||
|
Go encoder omits it below 256 bytes, so `emit` re-headers those frames into the
|
||||||
|
shape reference libzstd emits: `Single_Segment_flag` set, `Window_Descriptor`
|
||||||
|
dropped, and a one-byte content size in its place. `zstd -l <frame>` must print a
|
||||||
|
decompressed size; a blank column there is the fault, and it is invisible to any
|
||||||
|
check that only decompresses, because both `zstd -dc` and Go's decoder stream
|
||||||
|
such a frame happily. This is what the sidecar's `emitter_version` counts:
|
||||||
|
version 1 omitted the field and no client could sync past such a blob, version 2
|
||||||
|
declares it. `assemble` refuses to merge indexes that disagree.
|
||||||
|
|
||||||
|
## Hidden compatibility aliases
|
||||||
|
|
||||||
|
Existing scripts may continue using these names indefinitely. They are accepted
|
||||||
|
but omitted from routine help and the interactive menu:
|
||||||
|
|
||||||
|
| Builder | Hidden alias | Implementation |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `hak` | `build-haks` | `build-haks` |
|
||||||
|
| `hak` | `apply-hak-manifest` | `apply-hak-manifest` |
|
||||||
|
| `module` | `build-module` | module-only `build-module` |
|
||||||
|
| `module` | `apply-hak-manifest` | `apply-hak-manifest` |
|
||||||
|
| `topdata` | `validate-topdata` | `validate-topdata` |
|
||||||
|
| `topdata` | `build-topdata` | `build-topdata` |
|
||||||
|
| `topdata` | `build-top-package` | `build-top-package` |
|
||||||
|
| `topdata` | `compare-topdata` | `compare-topdata` |
|
||||||
|
| `topdata` | `convert-topdata` | `convert-topdata` |
|
||||||
|
| `depot` | `--target local` | `--out $DEPOT_DIR` on `status`/`get` |
|
||||||
|
| `wiki` | `build-wiki` | `build-wiki` |
|
||||||
|
| `wiki` | `deploy-wiki` | `deploy-wiki` |
|
||||||
|
|
||||||
|
`module build-module` deliberately retains its narrower module-only behavior;
|
||||||
|
it is not redirected to the broader `module build`.
|
||||||
|
|
||||||
|
## Global commands
|
||||||
|
|
||||||
```text
|
```text
|
||||||
crucible help | -h usage + builder list
|
crucible help | -h usage + builder list
|
||||||
crucible version | -V build version (git sha via -ldflags)
|
crucible version | -V build version
|
||||||
crucible list builders, one per line: name<TAB>bin<TAB>summary
|
crucible list name<TAB>binary<TAB>summary
|
||||||
crucible config [args] -> legacy `config` command group
|
crucible config [args] inspect and validate effective configuration
|
||||||
crucible changelog [args] -> legacy `build-changelog`
|
crucible changelog [args] generate a release changelog
|
||||||
```
|
```
|
||||||
|
|
||||||
`crucible list` is machine-readable so CI can enumerate builders without parsing
|
## HAK build options
|
||||||
help text.
|
|
||||||
|
|
||||||
## Global flags (planned, at wiring time)
|
```text
|
||||||
|
crucible hak build
|
||||||
|
[--hak <hak-name> ...]
|
||||||
|
[--archive <archive-name> ...]
|
||||||
|
[--source-manifest <path>]
|
||||||
|
[--content-addressed-root <path>]
|
||||||
|
[--plan-only]
|
||||||
|
[--quiet|--verbose|--debug]
|
||||||
|
```
|
||||||
|
|
||||||
`--quiet`, `--verbose`, `--debug` (the legacy verbosity model), passed after the
|
When a source manifest contains `asset_sources`, pass
|
||||||
builder name, e.g. `crucible topdata build-topdata --verbose`.
|
`--content-addressed-root <directory>`. Crucible resolves each declared SHA-256
|
||||||
|
under `sha256/<first-two>/<next-two>/<full-sha256>` and verifies streamed bytes
|
||||||
|
while writing selected HAK archives. Authored `.bmu` files are ordinary assets
|
||||||
|
and require no conversion pipeline.
|
||||||
|
|||||||
+24
-17
@@ -22,30 +22,37 @@ If none resolve, the wrapper exits non-zero with a Phase 5 message. The
|
|||||||
wrappers prefer the single-token `crucible-<name>` shim so `"$builder" args`
|
wrappers prefer the single-token `crucible-<name>` shim so `"$builder" args`
|
||||||
quoting stays correct.
|
quoting stays correct.
|
||||||
|
|
||||||
## In CI (preferred)
|
## Parity contract — wrappers are not authoritative for build inputs
|
||||||
|
|
||||||
**OPERATOR NOTE: Stop. Do not do it this way.**
|
A consumer build wrapper may only:
|
||||||
**Use the pinned `prod.yml` version. That's what it's there for.**
|
|
||||||
|
|
||||||
~~Run the consumer job inside the pinned image and the binaries are on `PATH`:~~
|
1. **validate** — a pre-build gate that does not change output;
|
||||||
|
2. **publish** — a post-build step on a separate artifact;
|
||||||
|
3. **resolve the crucible binary** — the bootstrap above.
|
||||||
|
|
||||||
```yaml
|
A wrapper MUST NOT fetch, resolve, generate, or stage any input Crucible reads to
|
||||||
container:
|
produce the artifact. If Crucible needs an input, Crucible acquires it. This makes
|
||||||
image: registry.westgate.pw/deployment/crucible:<sha> # pinned, immutable
|
the "no local-machine assumptions" + "same inputs → identical output" rules
|
||||||
```
|
explicit: local `crucible <build>` and CI `crucible <build>` against the same
|
||||||
|
committed config must produce identical bytes, with no CI-only pre-steps.
|
||||||
|
|
||||||
~~No host install, no `$HOME` layout, no developer machine assumptions.~~
|
Builders read every input from the project tree. No NWN install is required. A
|
||||||
|
builder that one day needs NWN game data auto-detects the install (the NWN home
|
||||||
|
and Steam path are reliably discoverable), with an optional explicit override for
|
||||||
|
non-standard layouts — never a required hand-set env.
|
||||||
|
|
||||||
## `NWN_ROOT` rule
|
## In CI
|
||||||
|
|
||||||
Crucible **never guesses `NWN_ROOT` from `$HOME`** (this was a deploy-notes
|
Builds run the Crucible **binary** — from the `crucible.sh` / `crucible.ps1`
|
||||||
pain point). The NWN install/data root is passed explicitly:
|
bootstrap (anonymous download) or, for Nix users, from the flake devshell pinned
|
||||||
|
by `flake.lock`. CI runs the _same_ `crucible <build>` as local dev, inside the
|
||||||
|
nix devshell (binary-cache fast). No build container, no token, no CI-only
|
||||||
|
pre-steps.
|
||||||
|
|
||||||
- env `NWN_ROOT=/path/to/nwn`, or
|
The `registry.westgate.pw/deployment/crucible:<sha>` image is **retired**
|
||||||
- flag `--nwn-root /path/to/nwn`.
|
(2026-07-17): nothing consumed it, `prod.yml` no longer reserves a slot for
|
||||||
|
it, and CI no longer builds or publishes it. Binaries, wrappers, and the Nix
|
||||||
A builder that needs `NWN_ROOT` and receives neither must fail closed with a
|
input are the only supported ways to run Crucible.
|
||||||
clear message, not fall back to a home-directory default.
|
|
||||||
|
|
||||||
## Determinism
|
## Determinism
|
||||||
|
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ checklist is kept as the record of what was done.
|
|||||||
## Cutover steps
|
## Cutover steps
|
||||||
|
|
||||||
1. **Move the internal packages.** Bring `internal/{app,pipeline,project,erf,
|
1. **Move the internal packages.** Bring `internal/{app,pipeline,project,erf,
|
||||||
gff,topdata,music,changelog,validator}` from `gitea/sow-tools` into this repo.
|
gff,topdata,changelog,validator}` from `gitea/sow-tools` into this repo.
|
||||||
Re-home `internal/app` command wiring behind `internal/dispatch` instead of
|
Re-home `internal/app` command wiring behind `internal/dispatch` instead of
|
||||||
the old flag parser in `cmd/nwn-tool`.
|
the old flag parser in `cmd/nwn-tool`.
|
||||||
2. **Restore dependencies.** Add `golang.org/x/text` and `gopkg.in/yaml.v3` back
|
2. **Restore dependencies.** Add `golang.org/x/text` and `gopkg.in/yaml.v3` back
|
||||||
@@ -43,10 +43,7 @@ checklist is kept as the record of what was done.
|
|||||||
artifacts and image layers only.
|
artifacts and image layers only.
|
||||||
6. **Carry the determinism test.** Port
|
6. **Carry the determinism test.** Port
|
||||||
`TestBuildNativeKeepsGeneratedIDsStableAcrossRepeatedBuilds` and keep it green.
|
`TestBuildNativeKeepsGeneratedIDsStableAcrossRepeatedBuilds` and keep it green.
|
||||||
7. **Runtime base for music.** If music conversion is wired, switch the final
|
7. **Update `make smoke`.** Once a builder is wired, change the smoke
|
||||||
Docker stage from `distroless/static` to a debian-slim base that includes
|
|
||||||
`ffmpeg`.
|
|
||||||
8. **Update `make smoke`.** Once a builder is wired, change the smoke
|
|
||||||
expectation for it from exit `70` to a real success path.
|
expectation for it from exit `70` to a real success path.
|
||||||
|
|
||||||
## Wrapper contract carried over
|
## Wrapper contract carried over
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,346 @@
|
|||||||
|
# Crucible Command Surface Cleanup Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use
|
||||||
|
> `superpowers:subagent-driven-development` or
|
||||||
|
> `superpowers:executing-plans` to implement this plan task-by-task. Steps use
|
||||||
|
> checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** Replace repetitive Crucible command names with a descriptive,
|
||||||
|
registry-driven public surface, retain long names as hidden aliases, and remove
|
||||||
|
the unused music conversion pipeline without affecting authored `.bmu` assets.
|
||||||
|
|
||||||
|
**Architecture:** `internal/dispatch.Registry` gains explicit command records
|
||||||
|
that drive lookup, help, and the interactive menu while translating to unchanged
|
||||||
|
`internal/app` implementation names. The unused music subsystem is removed
|
||||||
|
through its app, pipeline, project-config, validator, packaging, and consumer
|
||||||
|
boundaries. The two active `sow-assets-manifest` call sites are decoupled in the
|
||||||
|
same worktree session.
|
||||||
|
|
||||||
|
**Tech Stack:** Go 1.26, Go tests, Nix flakes, Dockerfile, Bash/Bats.
|
||||||
|
|
||||||
|
## Global Constraints
|
||||||
|
|
||||||
|
- Never commit or push.
|
||||||
|
- Do not touch secrets or `.sops` data.
|
||||||
|
- Keep `internal/dispatch.Registry` synchronized with
|
||||||
|
`docs/command-surface.md`.
|
||||||
|
- Preserve deterministic artifact behavior.
|
||||||
|
- Preserve every existing long command as a hidden alias except all music
|
||||||
|
commands, which are removed.
|
||||||
|
- Preserve `.bmu` as an ordinary HAK asset.
|
||||||
|
- Do not retain generated binaries or build output.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: Registry-driven canonical commands and hidden aliases
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `internal/dispatch/dispatch.go`
|
||||||
|
- Modify: `internal/dispatch/dispatch_test.go`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
|
||||||
|
- Produces a `Command` record owned by each `Builder`.
|
||||||
|
- Produces lookup that returns the command record and exact app target.
|
||||||
|
- Preserves `Builder.Legacy` only if required by migration tests; otherwise
|
||||||
|
replaces it with command-derived compatibility metadata.
|
||||||
|
|
||||||
|
- [ ] Add failing tests covering the visible command table:
|
||||||
|
`hak build/manifest`, `module build/extract/validate/compare/manifest`,
|
||||||
|
`topdata validate/build/package/compare/convert`, and `wiki build/deploy`.
|
||||||
|
- [ ] Add failing tests proving long aliases resolve to the same app target and
|
||||||
|
that `module build-module` resolves to `build-module`, not `build`.
|
||||||
|
- [ ] Add a failing test proving all music commands are rejected.
|
||||||
|
- [ ] Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command go test ./internal/dispatch -run 'TestCanonical|TestHidden|TestMusic' -v
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: failures caused by the old registry shape and exposed music
|
||||||
|
commands.
|
||||||
|
|
||||||
|
- [ ] Introduce command records with fields equivalent to:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type Command struct {
|
||||||
|
Name string
|
||||||
|
Summary string
|
||||||
|
AppCommand string
|
||||||
|
Usage string
|
||||||
|
Options []string
|
||||||
|
HiddenAlias []string
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Use the minimum final field names that keep lookup, menu, and help readable.
|
||||||
|
|
||||||
|
- [ ] Replace builder-level subcommand acceptance with command lookup. Translate
|
||||||
|
canonical and hidden aliases to `AppCommand` before calling `app.Run`.
|
||||||
|
- [ ] Define the approved command table and remove dispatcher music entries.
|
||||||
|
- [ ] Re-run the focused dispatch tests and the full dispatch package:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command go test ./internal/dispatch -v
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: pass.
|
||||||
|
|
||||||
|
### Task 2: Descriptive menu and command-specific help
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `internal/menu/menu.go`
|
||||||
|
- Modify: `internal/menu/menu_test.go`
|
||||||
|
- Modify: `internal/dispatch/dispatch.go`
|
||||||
|
- Modify: `internal/dispatch/dispatch_test.go`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
|
||||||
|
- `menu.Item` carries display label, summary, dispatcher args, usage, and option
|
||||||
|
lines.
|
||||||
|
- `menu.Select` returns selected dispatcher args plus entered arguments.
|
||||||
|
- Dispatcher command help uses the same registry guidance as the menu.
|
||||||
|
|
||||||
|
- [ ] Add failing menu tests proving descriptions start at one aligned column
|
||||||
|
for labels of different lengths and hidden aliases never enter `menuItems()`.
|
||||||
|
- [ ] Add a failing test selecting a command and asserting output contains its
|
||||||
|
usage, option guidance, and
|
||||||
|
`arguments (press Enter to use defaults):`.
|
||||||
|
- [ ] Add failing dispatcher tests proving
|
||||||
|
`crucible topdata build --help` returns `0`, prints canonical usage/options,
|
||||||
|
and does not delegate into project loading.
|
||||||
|
- [ ] Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command go test ./internal/menu ./internal/dispatch -run 'Test.*Menu|Test.*Help' -v
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: fail against the current fixed-width menu and builder-only help.
|
||||||
|
|
||||||
|
- [ ] Compute menu label width from visible items and render descriptions with
|
||||||
|
that width.
|
||||||
|
- [ ] Print command guidance immediately after a selection and before reading
|
||||||
|
optional arguments.
|
||||||
|
- [ ] Add command-level help handling before app delegation.
|
||||||
|
- [ ] Ensure builder help lists only canonical names and their summaries.
|
||||||
|
- [ ] Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command go test ./internal/menu ./internal/dispatch -v
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: pass.
|
||||||
|
|
||||||
|
### Task 3: Remove music from HAK app and pipeline
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Delete: `internal/pipeline/music.go`
|
||||||
|
- Modify: `internal/pipeline/build.go`
|
||||||
|
- Modify: `internal/pipeline/pipeline_test.go`
|
||||||
|
- Modify: `internal/app/app.go`
|
||||||
|
- Modify: `internal/app/app_test.go`
|
||||||
|
- Delete: `internal/music/config.go`
|
||||||
|
- Delete: `internal/music/credits.go`
|
||||||
|
- Delete: `internal/music/metadata.go`
|
||||||
|
- Delete: `internal/music/music.go`
|
||||||
|
- Delete: `internal/music/music_test.go`
|
||||||
|
- Delete: `internal/music/naming.go`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
|
||||||
|
- `pipeline.BuildHAKOptions` retains only HAK selection, source-manifest,
|
||||||
|
content-addressed-root, and progress controls.
|
||||||
|
- `pipeline.BuildResult` retains artifact fields unrelated to generated music
|
||||||
|
credits.
|
||||||
|
- `build-haks` accepts no music-specific flags.
|
||||||
|
|
||||||
|
- [ ] Replace music pipeline tests with a failing contract test that builds a
|
||||||
|
HAK containing a pre-authored `.bmu` and proves the file appears in the
|
||||||
|
manifest/archive without conversion setup.
|
||||||
|
- [ ] Add failing parser tests proving `--skip-music` and `--music-dataset`
|
||||||
|
return unknown-argument errors and help does not mention them.
|
||||||
|
- [ ] Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command go test ./internal/app ./internal/pipeline -run 'Test.*Music|Test.*BMU|TestParseBuildHAK' -v
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: at least the removed-flag assertions fail.
|
||||||
|
|
||||||
|
- [ ] Remove the app `music` command, argument parser, emitters, music HAK
|
||||||
|
options, and credits console output.
|
||||||
|
- [ ] Remove music fields from `BuildHAKOptions` and `BuildResult`.
|
||||||
|
- [ ] Remove preparation/cleanup calls from HAK builds so asset collection reads
|
||||||
|
authored resources directly.
|
||||||
|
- [ ] Delete the music implementation packages and obsolete conversion tests.
|
||||||
|
- [ ] Keep `.bmu` handling in ordinary ERF/resource and asset collection paths.
|
||||||
|
- [ ] Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command go test ./internal/app ./internal/pipeline -v
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: pass.
|
||||||
|
|
||||||
|
### Task 4: Remove music configuration and validation
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `internal/project/project.go`
|
||||||
|
- Modify: `internal/project/effective.go`
|
||||||
|
- Modify: `internal/project/project_test.go`
|
||||||
|
- Modify: `internal/validator/validator.go`
|
||||||
|
- Modify: `internal/validator/validator_test.go`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
|
||||||
|
- `project.Config` and `project.EffectiveConfig` contain no music section.
|
||||||
|
- Project scanning uses only configured inventory extensions.
|
||||||
|
- `.bmu` and `.wav` remain default asset extensions; `.mp3` and `.ogg` do not.
|
||||||
|
|
||||||
|
- [ ] Add or adjust failing project tests asserting default extensions include
|
||||||
|
`.bmu` and `.wav` but exclude `.mp3` and `.ogg`.
|
||||||
|
- [ ] Add a failing strict-decoder test proving a top-level `music:` field is
|
||||||
|
rejected as unknown.
|
||||||
|
- [ ] Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command go test ./internal/project ./internal/validator -run 'Test.*Music|Test.*AssetExtension|Test.*BMU' -v
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: fail while music config is still accepted.
|
||||||
|
|
||||||
|
- [ ] Remove music config types, defaults, normalization, validation, accessors,
|
||||||
|
effective config, provenance defaults, and environment overrides.
|
||||||
|
- [ ] Simplify `Project.Scan` to accept assets solely from
|
||||||
|
`effective.Inventory.AssetExtensions`.
|
||||||
|
- [ ] Remove validator logic that exempts conversion source files.
|
||||||
|
- [ ] Remove conversion-only `.mp3` and `.ogg` from default asset extensions;
|
||||||
|
retain `.bmu` and `.wav`.
|
||||||
|
- [ ] Remove obsolete music-focused project and validator tests.
|
||||||
|
- [ ] Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command go test ./internal/project ./internal/validator -v
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: pass.
|
||||||
|
|
||||||
|
### Task 5: Remove ffmpeg packaging and stale documentation
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `flake.nix`
|
||||||
|
- Modify: `docker/Dockerfile`
|
||||||
|
- Modify: `wrappers/crucible.sh`
|
||||||
|
- Modify: `README.md`
|
||||||
|
- Modify: `AGENTS.md`
|
||||||
|
- Modify: `docs/command-surface.md`
|
||||||
|
- Modify: `docs/migration-from-nwn-tool.md`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
|
||||||
|
- Nix development shell and image no longer contain ffmpeg.
|
||||||
|
- Docker runtime contains only certificates and the non-root runtime support
|
||||||
|
needed by current builders.
|
||||||
|
- User docs advertise canonical short commands.
|
||||||
|
|
||||||
|
- [ ] Add a shell verification that fails while active packaging still mentions
|
||||||
|
ffmpeg or removed command names:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
! rg -n 'ffmpeg|ffprobe|skip-music|music-dataset|module music|hak music' \
|
||||||
|
flake.nix docker wrappers README.md AGENTS.md docs/command-surface.md
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] Remove `pkgs.ffmpeg-headless` and `ffmpeg` from Nix image/dev-shell
|
||||||
|
contents and update comments.
|
||||||
|
- [ ] Return the Docker runtime to a minimal base without ffmpeg while retaining
|
||||||
|
CA certificates and non-root execution.
|
||||||
|
- [ ] Remove the wrapper's `ffmpeg-free` wording.
|
||||||
|
- [ ] Rewrite command-surface documentation around canonical names and a
|
||||||
|
concise hidden-alias compatibility section.
|
||||||
|
- [ ] Update README examples and ownership/status prose.
|
||||||
|
- [ ] Re-run the search. Expected: no matches in active files.
|
||||||
|
|
||||||
|
### Task 6: Decouple active consumer scripts
|
||||||
|
|
||||||
|
**Files in `../sow-assets-manifest`:**
|
||||||
|
|
||||||
|
- Modify: `scripts/pack-haks.sh`
|
||||||
|
- Modify: `scripts/build-local-haks.sh`
|
||||||
|
- Modify: `tests/pack-haks.bats`
|
||||||
|
- Modify only directly related active comments/tests if a focused test proves
|
||||||
|
they require adjustment.
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
|
||||||
|
- Both scripts call the hidden-compatible `hak build-haks` or canonical
|
||||||
|
`hak build` without `--skip-music`.
|
||||||
|
|
||||||
|
- [ ] Read `../sow-assets-manifest/AGENTS.md` before editing.
|
||||||
|
- [ ] Add or adjust a focused test to reject `--skip-music` in generated
|
||||||
|
Crucible invocations.
|
||||||
|
- [ ] Run the focused test and confirm it fails while the flag is present.
|
||||||
|
- [ ] Remove `--skip-music` and its lockstep comments from both scripts.
|
||||||
|
- [ ] Update the HAK parser test invocation to omit the removed flag.
|
||||||
|
- [ ] Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ../sow-assets-manifest
|
||||||
|
nix develop --command bats tests/pack-haks.bats
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: pass.
|
||||||
|
|
||||||
|
### Task 7: Full verification and scratchpad cleanup
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `../SCRATCHPAD.md`
|
||||||
|
- Do not retain: `bin/`, `.cache/go-build/`, result symlinks, image tarballs, or
|
||||||
|
other generated outputs.
|
||||||
|
|
||||||
|
- [ ] Run formatting:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command gofmt -w internal/app internal/dispatch internal/menu internal/pipeline internal/project internal/validator
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] Run full repository checks:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command make check
|
||||||
|
make build
|
||||||
|
make smoke
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] Run focused consumer verification from Task 6.
|
||||||
|
- [ ] Verify deleted pipeline identifiers are absent from active code:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'BuildMusic|CreditsSummary|MusicDataset|skip-music|music-dataset|ffmpeg|ffprobe' \
|
||||||
|
--glob '!docs/superpowers/**' .
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: no matches.
|
||||||
|
|
||||||
|
- [ ] Verify active command docs/examples do not advertise long names:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'topdata (validate-topdata|build-topdata|build-top-package|compare-topdata|convert-topdata)|wiki (build-wiki|deploy-wiki)|hak (build-haks|apply-hak-manifest)' \
|
||||||
|
README.md docs/command-surface.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: matches only inside the explicitly labeled hidden-alias
|
||||||
|
compatibility section.
|
||||||
|
|
||||||
|
- [ ] Inspect `git diff --check`, `git status --short`, and both repository
|
||||||
|
diffs. Remove only generated files created by this work.
|
||||||
|
- [ ] Remove the completed “Simplify crucible commands” to-do and the full
|
||||||
|
“Crucible commands issue” section from `../SCRATCHPAD.md`; do not archive it.
|
||||||
|
- [ ] Record exact verification commands and outcomes in the final handoff.
|
||||||
|
|
||||||
@@ -0,0 +1,858 @@
|
|||||||
|
# Topdata JSON Validation Tightening Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use
|
||||||
|
> `superpowers:subagent-driven-development` (recommended) or
|
||||||
|
> `superpowers:executing-plans` to implement this plan task-by-task. Steps use
|
||||||
|
> checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** Reject unsupported properties in established canonical topdata JSON
|
||||||
|
containers, add the non-empty `any_present` global-injection condition, and keep
|
||||||
|
standalone validation and direct native builds fail-closed and consistent.
|
||||||
|
|
||||||
|
**Architecture:** Add one focused `internal/topdata/json_contract.go` unit that
|
||||||
|
owns deterministic closed-key checks plus parsing and evaluation of global
|
||||||
|
injection condition groups. Existing validation in `topdata.go` uses that unit
|
||||||
|
to collect actionable diagnostics, while `native.go` uses the same parsed
|
||||||
|
conditions during ordered injection evaluation. Dataset payload rows continue
|
||||||
|
through the existing column-aware canonicalization paths and are not assigned a
|
||||||
|
global schema.
|
||||||
|
|
||||||
|
**Tech Stack:** Go 1.26, standard-library `encoding/json`, Go tests, Nix flakes,
|
||||||
|
Make, authored JSON in the sibling `sow-topdata` checkout.
|
||||||
|
|
||||||
|
## Global Constraints
|
||||||
|
|
||||||
|
- Work on a fresh non-`main` branch and open all changes as a pull request.
|
||||||
|
- Never commit or push secrets, `.sops` data, generated binaries, `.cache/`,
|
||||||
|
`generated/`, HAK/TLK artifacts, or result symlinks.
|
||||||
|
- Add no JSON Schema framework and no new validation dependency.
|
||||||
|
- Close only the canonical base/plain rows root, canonical module root,
|
||||||
|
`global.json` root, global injection, global condition, global default rule,
|
||||||
|
global default match object, and global default format object.
|
||||||
|
- Preserve specialized parser-owned JSON dialects outside those canonical
|
||||||
|
containers.
|
||||||
|
- Preserve dataset-specific columns and documented row controls inside `rows`,
|
||||||
|
`entries`, `overrides`, injection `row`, and default `values`.
|
||||||
|
- Preserve current-row, ordered-injection behavior: an earlier injection may
|
||||||
|
satisfy or block a later injection.
|
||||||
|
- Preserve empty-list behavior for `require_present` and `unless_present`;
|
||||||
|
reject an empty `any_present`.
|
||||||
|
- Do not add a compatibility alias for `when_present`.
|
||||||
|
- Use deterministic property ordering in diagnostics.
|
||||||
|
- Inventory evidence requires `compare_reference` to remain valid on
|
||||||
|
base/plain rows roots: `internal/topdata/native.go` reads it,
|
||||||
|
`internal/topdata/feat_migrate.go` emits it, and existing tests exercise it.
|
||||||
|
This is an established canonical property covered by the design's moderate
|
||||||
|
strictness rule even though the initial property table omitted it.
|
||||||
|
- The active `sow-topdata` branch
|
||||||
|
`fix-always-present-metamagic` already replaces the known `when_present` with
|
||||||
|
`require_present` in commit `ed139f5`; preserve that change and use the
|
||||||
|
checkout for integration validation rather than duplicating or rewriting it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: Introduce the shared closed-object and condition contract
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Create: `internal/topdata/json_contract.go`
|
||||||
|
- Create: `internal/topdata/json_contract_test.go`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
|
||||||
|
- Produces:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type globalCondition struct {
|
||||||
|
Field string
|
||||||
|
ID string
|
||||||
|
}
|
||||||
|
|
||||||
|
type globalConditionGroups struct {
|
||||||
|
RequirePresent []globalCondition
|
||||||
|
AnyPresent []globalCondition
|
||||||
|
UnlessPresent []globalCondition
|
||||||
|
}
|
||||||
|
|
||||||
|
func unsupportedObjectKeys(obj map[string]any, supported ...string) []string
|
||||||
|
func unsupportedObjectKeysError(label string, obj map[string]any, supported ...string) error
|
||||||
|
func parseGlobalConditionGroups(injection map[string]any) (globalConditionGroups, error)
|
||||||
|
func globalConditionGroupsMatch(groups globalConditionGroups, rows []map[string]any) bool
|
||||||
|
```
|
||||||
|
|
||||||
|
- `unsupportedObjectKeys` returns unknown keys in lexical order.
|
||||||
|
- `parseGlobalConditionGroups` accepts absent groups, permits empty
|
||||||
|
`require_present` and `unless_present`, rejects empty `any_present`, validates
|
||||||
|
condition keys as exactly `field` and `id`, and returns the first deterministic
|
||||||
|
error.
|
||||||
|
- `globalConditionGroupsMatch` implements all/any/none semantics without
|
||||||
|
reparsing JSON.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Create the feature branch after checking it was not previously
|
||||||
|
merged and deleted**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git fetch origin
|
||||||
|
git branch --show-current
|
||||||
|
git ls-remote --heads origin feat/topdata-json-validation
|
||||||
|
git log --all --oneline --decorate --grep='topdata json validation'
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: current branch is not `main`. If
|
||||||
|
`feat/topdata-json-validation` has never been used, create it from the approved
|
||||||
|
design commit:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git switch -c feat/topdata-json-validation
|
||||||
|
```
|
||||||
|
|
||||||
|
If that exact branch name has prior remote history, choose a new descriptive
|
||||||
|
name and record it in the PR; do not reuse the old branch.
|
||||||
|
|
||||||
|
- [ ] **Step 2: Re-run the canonical-format inventory before writing the
|
||||||
|
allowlists**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
for f in $(find ../sow-topdata/data -type f -name 'base.json' | sort); do
|
||||||
|
jq -r --arg f "${f#../sow-topdata/}" \
|
||||||
|
'[$f, (keys | sort | join(","))] | @tsv' "$f"
|
||||||
|
done
|
||||||
|
|
||||||
|
for f in $(find ../sow-topdata/data -type f -name 'global.json' | sort); do
|
||||||
|
jq -r --arg f "${f#../sow-topdata/}" \
|
||||||
|
'[$f, (keys | sort | join(","))] | @tsv' "$f"
|
||||||
|
done
|
||||||
|
|
||||||
|
find ../sow-topdata/data -type f -path '*/modules/*.json' -print0 |
|
||||||
|
while IFS= read -r -d '' f; do
|
||||||
|
jq -r --arg f "${f#../sow-topdata/}" \
|
||||||
|
'[$f, (keys | sort | join(","))] | @tsv' "$f"
|
||||||
|
done | sort
|
||||||
|
|
||||||
|
rg -n 'compare_reference' internal/topdata ../sow-topdata/data
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: active base roots use `columns`, `key`, `output`, and `rows`;
|
||||||
|
active modules use `columns`, `entries`, `overrides`, or the legitimate
|
||||||
|
`entries`+`overrides` combination; active globals use the approved global
|
||||||
|
keys. `sow-tools` additionally confirms established `compare_reference`
|
||||||
|
support.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Write failing unit tests for deterministic unsupported-key
|
||||||
|
diagnostics**
|
||||||
|
|
||||||
|
Add table-driven tests equivalent to:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func TestUnsupportedObjectKeysAreSorted(t *testing.T) {
|
||||||
|
obj := map[string]any{
|
||||||
|
"row": map[string]any{},
|
||||||
|
"when_present": []any{},
|
||||||
|
"aaa": true,
|
||||||
|
}
|
||||||
|
|
||||||
|
got := unsupportedObjectKeys(
|
||||||
|
obj,
|
||||||
|
"row",
|
||||||
|
"require_present",
|
||||||
|
"any_present",
|
||||||
|
"unless_present",
|
||||||
|
)
|
||||||
|
want := []string{"aaa", "when_present"}
|
||||||
|
if !slices.Equal(got, want) {
|
||||||
|
t.Fatalf("unsupportedObjectKeys() = %v, want %v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUnsupportedObjectKeysErrorNamesObjectAndContract(t *testing.T) {
|
||||||
|
err := unsupportedObjectKeysError(
|
||||||
|
"global injection 10",
|
||||||
|
map[string]any{"row": map[string]any{}, "when_present": []any{}},
|
||||||
|
"row",
|
||||||
|
"require_present",
|
||||||
|
"any_present",
|
||||||
|
"unless_present",
|
||||||
|
)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected unsupported-property error")
|
||||||
|
}
|
||||||
|
for _, want := range []string{
|
||||||
|
"global injection 10",
|
||||||
|
`"when_present"`,
|
||||||
|
"row",
|
||||||
|
"require_present",
|
||||||
|
"any_present",
|
||||||
|
"unless_present",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(err.Error(), want) {
|
||||||
|
t.Fatalf("error %q does not contain %q", err, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Write failing parser tests for all condition-group contracts**
|
||||||
|
|
||||||
|
Cover these exact cases in `json_contract_test.go`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
injection map[string]any
|
||||||
|
want globalConditionGroups
|
||||||
|
wantError string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "all groups",
|
||||||
|
injection: map[string]any{
|
||||||
|
"require_present": []any{
|
||||||
|
map[string]any{"field": "FeatIndex", "id": "feat:required"},
|
||||||
|
},
|
||||||
|
"any_present": []any{
|
||||||
|
map[string]any{"field": "FeatIndex", "id": "feat:first"},
|
||||||
|
map[string]any{"field": "FeatIndex", "id": "feat:second"},
|
||||||
|
},
|
||||||
|
"unless_present": []any{
|
||||||
|
map[string]any{"field": "FeatIndex", "id": "feat:blocked"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
want: globalConditionGroups{
|
||||||
|
RequirePresent: []globalCondition{
|
||||||
|
{Field: "FeatIndex", ID: "feat:required"},
|
||||||
|
},
|
||||||
|
AnyPresent: []globalCondition{
|
||||||
|
{Field: "FeatIndex", ID: "feat:first"},
|
||||||
|
{Field: "FeatIndex", ID: "feat:second"},
|
||||||
|
},
|
||||||
|
UnlessPresent: []globalCondition{
|
||||||
|
{Field: "FeatIndex", ID: "feat:blocked"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "empty existing groups remain valid",
|
||||||
|
injection: map[string]any{
|
||||||
|
"require_present": []any{},
|
||||||
|
"unless_present": []any{},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "any present must be array",
|
||||||
|
injection: map[string]any{
|
||||||
|
"any_present": "feat:first",
|
||||||
|
},
|
||||||
|
wantError: "any_present must be an array",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "any present must not be empty",
|
||||||
|
injection: map[string]any{
|
||||||
|
"any_present": []any{},
|
||||||
|
},
|
||||||
|
wantError: "any_present must contain at least one condition",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "condition is closed",
|
||||||
|
injection: map[string]any{
|
||||||
|
"require_present": []any{
|
||||||
|
map[string]any{
|
||||||
|
"field": "FeatIndex",
|
||||||
|
"id": "feat:required",
|
||||||
|
"typo": true,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
wantError: `require_present[0] contains unsupported key "typo"`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "field is required",
|
||||||
|
injection: map[string]any{
|
||||||
|
"require_present": []any{
|
||||||
|
map[string]any{"id": "feat:required"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
wantError: "require_present[0].field is required",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "id is required",
|
||||||
|
injection: map[string]any{
|
||||||
|
"require_present": []any{
|
||||||
|
map[string]any{"field": "FeatIndex"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
wantError: "require_present[0].id is required",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 5: Write failing evaluator tests for all/any/none and group
|
||||||
|
conjunction**
|
||||||
|
|
||||||
|
Build rows with `FeatIndex` references for `feat:required`,
|
||||||
|
`feat:second`, and `feat:blocked`, then assert:
|
||||||
|
|
||||||
|
```go
|
||||||
|
if !globalConditionGroupsMatch(globalConditionGroups{
|
||||||
|
RequirePresent: []globalCondition{
|
||||||
|
{Field: "FeatIndex", ID: "feat:required"},
|
||||||
|
},
|
||||||
|
AnyPresent: []globalCondition{
|
||||||
|
{Field: "FeatIndex", ID: "feat:first"},
|
||||||
|
{Field: "FeatIndex", ID: "feat:second"},
|
||||||
|
},
|
||||||
|
}, rowsWithoutBlocked) {
|
||||||
|
t.Fatal("expected required + later any_present alternative to match")
|
||||||
|
}
|
||||||
|
|
||||||
|
if globalConditionGroupsMatch(globalConditionGroups{
|
||||||
|
AnyPresent: []globalCondition{
|
||||||
|
{Field: "FeatIndex", ID: "feat:missing"},
|
||||||
|
},
|
||||||
|
}, rowsWithoutBlocked) {
|
||||||
|
t.Fatal("expected missing any_present alternatives to reject")
|
||||||
|
}
|
||||||
|
|
||||||
|
if globalConditionGroupsMatch(globalConditionGroups{
|
||||||
|
UnlessPresent: []globalCondition{
|
||||||
|
{Field: "FeatIndex", ID: "feat:blocked"},
|
||||||
|
},
|
||||||
|
}, rowsWithBlocked) {
|
||||||
|
t.Fatal("expected unless_present match to reject")
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 6: Run the focused tests and confirm they fail**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command go test ./internal/topdata \
|
||||||
|
-run 'TestUnsupportedObjectKeys|TestParseGlobalConditionGroups|TestGlobalConditionGroupsMatch' \
|
||||||
|
-v
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: build failure because the shared contract types and functions do
|
||||||
|
not exist.
|
||||||
|
|
||||||
|
- [ ] **Step 7: Implement the minimal shared contract**
|
||||||
|
|
||||||
|
In `json_contract.go`, define fixed key sets and implement:
|
||||||
|
|
||||||
|
```go
|
||||||
|
var (
|
||||||
|
baseOrPlainRootKeys = []string{
|
||||||
|
"output", "key", "columns", "rows", "compare_reference",
|
||||||
|
}
|
||||||
|
canonicalModuleRootKeys = []string{
|
||||||
|
"output", "key", "columns", "entries", "overrides", "rows",
|
||||||
|
}
|
||||||
|
globalRootKeys = []string{
|
||||||
|
"columns", "entries", "overrides", "defaults", "position", "injections",
|
||||||
|
}
|
||||||
|
globalInjectionKeys = []string{
|
||||||
|
"row", "require_present", "any_present", "unless_present",
|
||||||
|
}
|
||||||
|
globalConditionKeys = []string{"field", "id"}
|
||||||
|
globalDefaultRuleKeys = []string{"match", "values"}
|
||||||
|
globalDefaultMatchKeys = []string{"source"}
|
||||||
|
globalDefaultFormatKeys = []string{"format"}
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
`unsupportedObjectKeysError` must report one deterministic error per call,
|
||||||
|
using the first lexically sorted unsupported key and a supported-key list in
|
||||||
|
declaration order. Implement condition parsing through one private helper:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func parseGlobalConditionList(
|
||||||
|
name string,
|
||||||
|
raw any,
|
||||||
|
requireNonEmpty bool,
|
||||||
|
) ([]globalCondition, error)
|
||||||
|
```
|
||||||
|
|
||||||
|
It must reject non-arrays, reject non-object elements, run the closed-key
|
||||||
|
check before required-field checks, trim `field` and `id`, and enforce the
|
||||||
|
`any_present` non-empty rule.
|
||||||
|
|
||||||
|
`globalConditionGroupsMatch` must call the existing
|
||||||
|
`globalReferencePresent` helper and implement:
|
||||||
|
|
||||||
|
```text
|
||||||
|
require_present: every condition is found
|
||||||
|
any_present: at least one condition is found
|
||||||
|
unless_present: no condition is found
|
||||||
|
all groups: every present group passes
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 8: Run the focused and package tests**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command go test ./internal/topdata \
|
||||||
|
-run 'TestUnsupportedObjectKeys|TestParseGlobalConditionGroups|TestGlobalConditionGroupsMatch' \
|
||||||
|
-v
|
||||||
|
nix develop --command go test ./internal/topdata
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: pass.
|
||||||
|
|
||||||
|
- [ ] **Step 9: Commit the shared contract**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add internal/topdata/json_contract.go internal/topdata/json_contract_test.go
|
||||||
|
git commit -m "feat(topdata): define canonical json contracts"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 2: Close canonical structural objects during project validation
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `internal/topdata/topdata.go`
|
||||||
|
- Modify: `internal/topdata/json_contract_test.go`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
|
||||||
|
- Consumes the key sets and condition parser from Task 1.
|
||||||
|
- `validateDataObject` selects exactly one root contract from file context:
|
||||||
|
base/plain rows root, canonical module root, or `global.json` root.
|
||||||
|
- A module validates every present supported container; it does not return
|
||||||
|
after the first of `entries`, `overrides`, `rows`, or `columns`.
|
||||||
|
- Specialized paths continue to return to their existing validators before
|
||||||
|
canonical root closure.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Add failing validation tests for unsupported canonical root
|
||||||
|
properties**
|
||||||
|
|
||||||
|
Add table-driven temporary-project cases covering:
|
||||||
|
- base root with `"typo": true`;
|
||||||
|
- loose plain rows root with `"typo": true`;
|
||||||
|
- module root with `"typo": true`;
|
||||||
|
- `global.json` root with `"typo": true`.
|
||||||
|
|
||||||
|
For every case, call `ValidateProject`, require errors, and assert only stable
|
||||||
|
semantic fragments: the file/object label, `typo`, and `unsupported`.
|
||||||
|
|
||||||
|
- [ ] **Step 2: Add failing validation tests for closed nested global objects**
|
||||||
|
|
||||||
|
Add cases for:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"injections":[{"row":{},"when_present":[]}]}
|
||||||
|
{"injections":[{"row":{},"require_present":[{"field":"FeatIndex","id":"feat:x","typo":true}]}]}
|
||||||
|
{"defaults":[{"match":"all","values":{},"typo":true}]}
|
||||||
|
{"defaults":[{"match":{"source":"entries","typo":true},"values":{}}]}
|
||||||
|
{"defaults":[{"match":"all","values":{"ImpactScript":{"format":"ss_{id}","typo":true}}}]}
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected diagnostics name `global injection 0`, the condition label,
|
||||||
|
`global default 0`, `global default 0 match`, or
|
||||||
|
`global default 0 values.ImpactScript` respectively.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Add failing validation tests for `any_present` shape**
|
||||||
|
|
||||||
|
Test one accepted non-empty list and rejected string/empty-list cases. The
|
||||||
|
accepted case must use a declared dataset column in the injection row so this
|
||||||
|
also protects payload flexibility.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Add regression tests for canonical combinations and payload
|
||||||
|
flexibility**
|
||||||
|
|
||||||
|
Build one table-driven test that validates without errors:
|
||||||
|
- a columns-only module;
|
||||||
|
- an entries-only module;
|
||||||
|
- an overrides-only module;
|
||||||
|
- a module containing both entries and overrides;
|
||||||
|
- a rows module;
|
||||||
|
- a current base root;
|
||||||
|
- a loose plain rows root;
|
||||||
|
- a base/plain root with `compare_reference: false`;
|
||||||
|
- dataset columns plus `id`, `key`, `inherit`, `_tlk`, and `meta` in row-like
|
||||||
|
payloads where their existing contracts allow them;
|
||||||
|
- an injection row using a declared dataset column.
|
||||||
|
|
||||||
|
The combined entries+overrides case must contain an independently invalid
|
||||||
|
override value first, prove validation catches it, then fix only that value
|
||||||
|
and prove the module passes. This specifically prevents the current
|
||||||
|
first-container early return.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Run the focused tests and confirm failures**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command go test ./internal/topdata \
|
||||||
|
-run 'TestValidateProject(RejectsUnsupportedCanonicalJSON|RejectsUnsupportedGlobalJSON|ValidatesEveryCanonicalModuleContainer|AcceptsCanonicalJSONContracts|AcceptsAnyPresent|RejectsInvalidAnyPresent)' \
|
||||||
|
-v
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: unknown keys are currently ignored, `any_present` is not validated,
|
||||||
|
and the combined module does not validate all containers.
|
||||||
|
|
||||||
|
- [ ] **Step 6: Add a validation adapter for shared contract errors**
|
||||||
|
|
||||||
|
In `topdata.go`, add:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func reportTopdataContractError(path string, err error, report *ValidationReport) {
|
||||||
|
if err == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
report.Diagnostics = append(report.Diagnostics, Diagnostic{
|
||||||
|
Severity: SeverityError,
|
||||||
|
Path: path,
|
||||||
|
Message: err.Error(),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Use `unsupportedObjectKeysError` for canonical roots and nested structural
|
||||||
|
objects. Keep row payload validation in existing `validateRowCollection`,
|
||||||
|
`validateEntriesFile`, and `validateOverridesFile` paths.
|
||||||
|
|
||||||
|
- [ ] **Step 7: Refactor canonical module dispatch to validate all containers**
|
||||||
|
|
||||||
|
After specialized-path checks, classify module files by directory context.
|
||||||
|
For recognized canonical modules:
|
||||||
|
1. check `canonicalModuleRootKeys`;
|
||||||
|
2. call `validateColumnsFile` when `columns` is present;
|
||||||
|
3. call `validateEntriesFile` when `entries` is present;
|
||||||
|
4. call `validateOverridesFile` when `overrides` is present;
|
||||||
|
5. call `validateRowsFile(path, obj, report, false)` when `rows` is present;
|
||||||
|
6. emit the existing warning only when none of those containers is present.
|
||||||
|
|
||||||
|
Avoid duplicate `validateColumnsFile` calls by removing the nested column
|
||||||
|
calls from this module dispatch path or by adding a caller-controlled flag;
|
||||||
|
retain current behavior for callers outside canonical module dispatch.
|
||||||
|
|
||||||
|
- [ ] **Step 8: Close global roots, injections, defaults, matches, and format
|
||||||
|
objects**
|
||||||
|
- Run the global-root key check before validating present containers.
|
||||||
|
- Run the injection key check before validating `row` and condition groups.
|
||||||
|
- Replace calls to `validateGlobalConditionList` with
|
||||||
|
`parseGlobalConditionGroups`; report its error through
|
||||||
|
`reportTopdataContractError`.
|
||||||
|
- Run default-rule and match-object key checks before existing type/value
|
||||||
|
checks.
|
||||||
|
- Treat an object containing `format` as a format object, reject every key
|
||||||
|
other than `format`, and preserve non-format object literals as payload
|
||||||
|
values.
|
||||||
|
|
||||||
|
- [ ] **Step 9: Run focused and full package tests**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command go test ./internal/topdata \
|
||||||
|
-run 'TestValidateProject(RejectsUnsupportedCanonicalJSON|RejectsUnsupportedGlobalJSON|ValidatesEveryCanonicalModuleContainer|AcceptsCanonicalJSONContracts|AcceptsAnyPresent|RejectsInvalidAnyPresent)' \
|
||||||
|
-v
|
||||||
|
nix develop --command go test ./internal/topdata
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: pass.
|
||||||
|
|
||||||
|
- [ ] **Step 10: Commit validation tightening**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add internal/topdata/topdata.go internal/topdata/json_contract_test.go
|
||||||
|
git commit -m "feat(topdata): reject unsupported canonical json properties"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 3: Use shared condition parsing and `any_present` in native builds
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `internal/topdata/native.go`
|
||||||
|
- Modify: `internal/topdata/json_contract_test.go`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
|
||||||
|
- Consumes `parseGlobalConditionGroups` and
|
||||||
|
`globalConditionGroupsMatch` from Task 1.
|
||||||
|
- Removes the old boolean-mode
|
||||||
|
`globalConditionListMatches(name, raw, rows, required)` path.
|
||||||
|
- Direct `BuildNativeWithOptions` calls fail before output mutation when
|
||||||
|
canonical control syntax is invalid because the existing build entry point
|
||||||
|
runs `ValidateProject` first and native evaluation uses the same parser.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Add failing native-build tests for `any_present` alternatives**
|
||||||
|
|
||||||
|
Use one compact plain-dataset fixture with:
|
||||||
|
- an authored row satisfying the first alternative;
|
||||||
|
- a second subtest satisfying only the later alternative;
|
||||||
|
- a third subtest satisfying neither alternative.
|
||||||
|
|
||||||
|
Assert the injected row exists in the first two outputs and is absent in the
|
||||||
|
third. Inspect the built 2DA content rather than internal slices.
|
||||||
|
|
||||||
|
- [ ] **Step 2: Add failing native-build tests for condition-group
|
||||||
|
conjunction**
|
||||||
|
|
||||||
|
Add subtests proving:
|
||||||
|
- `require_present` + `any_present` injects only when both pass;
|
||||||
|
- `any_present` + `unless_present` injects only when both pass;
|
||||||
|
- existing `require_present` all semantics remain unchanged;
|
||||||
|
- existing `unless_present` none semantics remain unchanged.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Add a failing ordered-injection test**
|
||||||
|
|
||||||
|
Author injection 0 so it adds the row referenced by injection 1's
|
||||||
|
`any_present`. Assert injection 1 applies. This protects evaluation against
|
||||||
|
the same current-row view and ordered mutation used today.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Add direct-build fail-closed tests**
|
||||||
|
|
||||||
|
Call `BuildNativeWithOptions` without a prior explicit `ValidateProject` call
|
||||||
|
for fixtures containing:
|
||||||
|
- `when_present` on an injection;
|
||||||
|
- an unknown condition key;
|
||||||
|
- an empty `any_present`;
|
||||||
|
- an unknown canonical root key.
|
||||||
|
|
||||||
|
Require an error containing the stable object/property fragments and assert
|
||||||
|
the expected output 2DA does not exist.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Run focused tests and confirm failures**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command go test ./internal/topdata \
|
||||||
|
-run 'TestBuildNative(AnyPresent|CombinesGlobalConditionGroups|UsesCurrentRowsForAnyPresent|RejectsUnsupportedGlobalControlSyntax|RejectsUnsupportedCanonicalRoot)' \
|
||||||
|
-v
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: `any_present` is ignored by current native evaluation and the new
|
||||||
|
parser/evaluator is not wired.
|
||||||
|
|
||||||
|
- [ ] **Step 6: Replace native condition evaluation with the shared contract**
|
||||||
|
|
||||||
|
Replace `globalInjectionConditionsMatch` and
|
||||||
|
`globalConditionListMatches` with:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func globalInjectionConditionsMatch(
|
||||||
|
injection map[string]any,
|
||||||
|
rows []map[string]any,
|
||||||
|
) (bool, error) {
|
||||||
|
groups, err := parseGlobalConditionGroups(injection)
|
||||||
|
if err != nil {
|
||||||
|
return false, err
|
||||||
|
}
|
||||||
|
return globalConditionGroupsMatch(groups, rows), nil
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Keep `applyPlainDatasetGlobalInjections` calling this function immediately
|
||||||
|
before deduplication/canonicalization, using `currentRows()` for each
|
||||||
|
injection.
|
||||||
|
|
||||||
|
- [ ] **Step 7: Run focused, package, and repository tests**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command go test ./internal/topdata \
|
||||||
|
-run 'TestBuildNative(AnyPresent|CombinesGlobalConditionGroups|UsesCurrentRowsForAnyPresent|RejectsUnsupportedGlobalControlSyntax|RejectsUnsupportedCanonicalRoot)' \
|
||||||
|
-v
|
||||||
|
nix develop --command go test ./internal/topdata
|
||||||
|
nix develop --command go test ./...
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: pass.
|
||||||
|
|
||||||
|
- [ ] **Step 8: Commit native `any_present` support**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add internal/topdata/native.go internal/topdata/json_contract_test.go
|
||||||
|
git commit -m "feat(topdata): add any-present injection conditions"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 4: Document the global injection grammar
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
|
||||||
|
- Documents the exact injection keys and all/any/none condition semantics.
|
||||||
|
- States that all present groups must pass and that conditions observe current
|
||||||
|
rows in injection order.
|
||||||
|
- States that `any_present` must be a non-empty array and `when_present` is
|
||||||
|
invalid.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Add a documentation contract check**
|
||||||
|
|
||||||
|
Run this before editing:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'any_present|when_present|all.*any.*none|require_present.*unless_present' \
|
||||||
|
internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: no complete description of `any_present` and group conjunction.
|
||||||
|
|
||||||
|
- [ ] **Step 2: Update the manifest shape and semantics**
|
||||||
|
|
||||||
|
Add an example containing all three groups:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"row": {
|
||||||
|
"FeatIndex": { "id": "feat:example" }
|
||||||
|
},
|
||||||
|
"require_present": [{ "field": "FeatIndex", "id": "feat:required" }],
|
||||||
|
"any_present": [
|
||||||
|
{ "field": "FeatIndex", "id": "masterfeats:metamagic" },
|
||||||
|
{ "field": "FeatIndex", "id": "masterfeats:combat" }
|
||||||
|
],
|
||||||
|
"unless_present": [{ "field": "FeatIndex", "id": "feat:example" }]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Document:
|
||||||
|
- `require_present`: all listed references must exist;
|
||||||
|
- `any_present`: at least one listed reference must exist and the list cannot
|
||||||
|
be empty;
|
||||||
|
- `unless_present`: none of the listed references may exist;
|
||||||
|
- every present group must pass;
|
||||||
|
- earlier injections affect later conditions;
|
||||||
|
- supported injection keys are `row`, `require_present`, `any_present`, and
|
||||||
|
`unless_present`;
|
||||||
|
- `when_present` is rejected rather than aliased.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Verify the documentation contains the complete contract**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'require_present|any_present|unless_present|when_present|earlier injection|every present group' \
|
||||||
|
internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: each semantic point appears in the contract.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Commit documentation**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md
|
||||||
|
git commit -m "docs(topdata): define global injection conditions"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 5: Validate the active topdata checkout and complete repository checks
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- No authored files should change in `sow-tools`.
|
||||||
|
- Preserve existing sibling change:
|
||||||
|
`../sow-topdata/data/classes/feats/global.json`.
|
||||||
|
- Do not retain: `bin/`, `.cache/`, `generated/`, `result`, HAK/TLK files, or
|
||||||
|
temporary Crucible binaries.
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
|
||||||
|
- The built Crucible validates the active `sow-topdata` checkout.
|
||||||
|
- The active checkout's single-condition correction uses `require_present`;
|
||||||
|
`any_present` remains available for future multi-alternative authoring.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Format and run the full `sow-tools` checks**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command gofmt -w \
|
||||||
|
internal/topdata/json_contract.go \
|
||||||
|
internal/topdata/json_contract_test.go \
|
||||||
|
internal/topdata/topdata.go \
|
||||||
|
internal/topdata/native.go
|
||||||
|
nix develop --command make check
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: vet, Go tests, shellcheck, and yamllint pass.
|
||||||
|
|
||||||
|
- [ ] **Step 2: Build a temporary Crucible binary without tracking it**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --command go build \
|
||||||
|
-o /tmp/crucible-topdata-json-validation \
|
||||||
|
./cmd/crucible
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: `/tmp/crucible-topdata-json-validation` exists outside the
|
||||||
|
repository.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Confirm the active consumer syntax and repository state**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git -C ../sow-topdata status --short --branch
|
||||||
|
rg -n 'when_present|require_present|any_present|unless_present' \
|
||||||
|
../sow-topdata/data/classes/feats/global.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: the existing consumer branch contains `require_present` and no
|
||||||
|
`when_present`. Do not alter unrelated consumer work.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Validate and build active `sow-topdata` with the changed
|
||||||
|
Crucible**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
(
|
||||||
|
cd ../sow-topdata
|
||||||
|
nix develop --command /tmp/crucible-topdata-json-validation topdata validate
|
||||||
|
nix develop --command /tmp/crucible-topdata-json-validation topdata build
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: validation and build pass, global injections execute, and generated
|
||||||
|
outputs remain ignored.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Verify no generated files became tracked**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git status --short
|
||||||
|
git -C ../sow-topdata status --short
|
||||||
|
git ls-files --error-unmatch bin generated .cache result 2>/dev/null
|
||||||
|
git -C ../sow-topdata ls-files --error-unmatch generated .cache result 2>/dev/null
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: the first two commands show only intentional source/doc changes;
|
||||||
|
the `git ls-files --error-unmatch` commands find no newly tracked generated
|
||||||
|
output. Remove only untracked artifacts created by this plan.
|
||||||
|
|
||||||
|
- [ ] **Step 6: Run final semantic searches and diff checks**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'when_present' internal/topdata --glob '!docs/superpowers/**'
|
||||||
|
rg -n 'any_present' \
|
||||||
|
internal/topdata/json_contract.go \
|
||||||
|
internal/topdata/json_contract_test.go \
|
||||||
|
internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md
|
||||||
|
git diff --check
|
||||||
|
git status --short --branch
|
||||||
|
git log --oneline origin/main..HEAD
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: `when_present` appears only in rejection tests/documentation;
|
||||||
|
implementation, tests, and contract document `any_present`; diff check is
|
||||||
|
clean.
|
||||||
|
|
||||||
|
- [ ] **Step 7: Remove the temporary binary**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rm -f /tmp/crucible-topdata-json-validation
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 8: Push the feature branch and open the required pull request**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git push -u origin HEAD
|
||||||
|
tea pr create \
|
||||||
|
--title "feat: tighten topdata JSON validation" \
|
||||||
|
--description $'## Summary\n- reject unsupported canonical topdata JSON properties\n- add non-empty any_present global injection conditions\n- keep validation and direct builds on one shared contract\n\n## Verification\n- nix develop --command make check\n- active sow-topdata validate and build with the changed Crucible'
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: a PR targeting `main`; no production workflow or release tag is
|
||||||
|
triggered.
|
||||||
|
|
||||||
|
- [ ] **Step 9: Record the handoff**
|
||||||
|
|
||||||
|
Report:
|
||||||
|
- the `sow-tools` branch and PR URL;
|
||||||
|
- the existing `sow-topdata` consumer branch/commit used for integration;
|
||||||
|
- exact focused and full verification commands with outcomes;
|
||||||
|
- confirmation that no generated files were tracked;
|
||||||
|
- any diagnostics wording intentionally asserted as public semantic
|
||||||
|
contract.
|
||||||
@@ -0,0 +1,330 @@
|
|||||||
|
# Crucible Depot Core (Increment 1) Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** Implement `crucible depot {status,push,verify,get,pull}` with local/cdn/bunny backends in sow-tools, then cut sow-assets-manifest over to it and delete the bash it replaces.
|
||||||
|
|
||||||
|
**Architecture:** New stdlib-only `internal/depot` package with its own `Run(args) int` entrypoint (exit contract 0/1/2/70 is richer than `internal/app.Run`'s 0/1, so dispatch special-cases depot instead of routing through `app.Run`). Backends implement a small interface; a sweep engine does parallel range-GET probes with serial re-confirm. Cutover repo work happens in sow-assets-manifest on its own branch.
|
||||||
|
|
||||||
|
**Tech Stack:** Go 1.26, stdlib `net/http`/`crypto/sha256`/`sync`, `gopkg.in/yaml.v3` (already a dep — add nothing to go.mod, vendorHash must not change).
|
||||||
|
|
||||||
|
**Spec:** `docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md` — read it before any task; it is authoritative.
|
||||||
|
|
||||||
|
## Global Constraints
|
||||||
|
|
||||||
|
- **No new Go dependencies.** stdlib + existing `gopkg.in/yaml.v3` only.
|
||||||
|
- **HEAD is banned** for existence checks. All probes: IPv4, `GET` with `Range: bytes=0-0`, expect `206` (accept any 2xx). A test must fail if HEAD is introduced.
|
||||||
|
- **Never prompt, never read stdin** for credentials — read path included. Missing key → clear error on stderr, exit `70`.
|
||||||
|
- **No cache of "what I uploaded"** — presence is always probed against the real target backend.
|
||||||
|
- `unconfirmed` is a distinct state from `missing`; exit `2` when unconfirmed-only.
|
||||||
|
- Exit codes: `0` clean, `1` drift (referenced-but-absent), `2` unconfirmed-only, `64` usage, `70` internal/backend error.
|
||||||
|
- Blob key: `sha256/<sha[0:2]>/<sha[2:4]>/<sha>`; sha must match `^[0-9a-f]{64}$`.
|
||||||
|
- Env names (unchanged from bash): `DEPOT_CDN_BASE`/`BUNNY_CDN_BASE` (cdn read base, default `https://cdn-a7f3k9.westgate.pw`), `BUNNY_STORAGE_HOST` (**required for bunny, no default** — spec overrides bash), `BUNNY_STORAGE_ZONE` (default `sow-assets-depot`), `BUNNY_STORAGE_READ_PASSWORD` (falls back to `BUNNY_STORAGE_PASSWORD`) for reads/probes, `BUNNY_STORAGE_PASSWORD` for PUT, `DEPOT_PROBE_JOBS` (default 16), `DEPOT_JOBS` (default 16), `DEPOT_CONNECT_TIMEOUT` (default 10s), `DEPOT_PROBE_MAX_TIME` (default 30s), `DEPOT_CONFIRM_RETRIES` (default 3), `DEPOT_CONFIRM_MAX` (default 200, 0 = unlimited).
|
||||||
|
- Upload: `PUT` with `AccessKey: <write key>` and `Checksum: <UPPERCASE sha>` headers; each sha uploaded at most once per run; probe-before-put.
|
||||||
|
- Every downloaded blob is re-hashed; deleted on mismatch.
|
||||||
|
- `cdn` backend writes fail closed (error, never fake success).
|
||||||
|
- Verify with `make vet test` in sow-tools (run inside `nix develop` if go missing from PATH).
|
||||||
|
- Commit after each task. **Never commit to main.** sow-tools branch: `depot-core-increment-1`. sow-assets-manifest branch: `crucible-depot-cutover`. sow-tools has an unrelated modified spec file on main — leave it out of commits unless the task says otherwise.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: Package skeleton — config + manifest parsing
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `internal/depot/config.go`, `internal/depot/config_test.go`
|
||||||
|
- Create: `internal/depot/manifest.go`, `internal/depot/manifest_test.go`
|
||||||
|
|
||||||
|
**Interfaces (Produces):**
|
||||||
|
|
||||||
|
```go
|
||||||
|
package depot
|
||||||
|
|
||||||
|
// config.go
|
||||||
|
type Config struct {
|
||||||
|
CDNBase string // DEPOT_CDN_BASE || BUNNY_CDN_BASE || default
|
||||||
|
StorageHost string // BUNNY_STORAGE_HOST, no default
|
||||||
|
StorageZone string // BUNNY_STORAGE_ZONE, default "sow-assets-depot"
|
||||||
|
ReadKey string // BUNNY_STORAGE_READ_PASSWORD || BUNNY_STORAGE_PASSWORD
|
||||||
|
WriteKey string // BUNNY_STORAGE_PASSWORD
|
||||||
|
ProbeJobs int // DEPOT_PROBE_JOBS, default 16
|
||||||
|
Jobs int // DEPOT_JOBS, default 16
|
||||||
|
ConnectTimeout time.Duration // DEPOT_CONNECT_TIMEOUT seconds, default 10s
|
||||||
|
ProbeMaxTime time.Duration // DEPOT_PROBE_MAX_TIME seconds, default 30s
|
||||||
|
ConfirmRetries int // DEPOT_CONFIRM_RETRIES, default 3
|
||||||
|
ConfirmMax int // DEPOT_CONFIRM_MAX, default 200 (0 = unlimited)
|
||||||
|
}
|
||||||
|
func LoadConfig(getenv func(string) string) Config
|
||||||
|
|
||||||
|
// manifest.go
|
||||||
|
// ReferencedSHAs parses every *.yml in dir (schema: assets[].{path,sha256,size,restype,hak})
|
||||||
|
// and returns the deduplicated sha set plus per-sha size (for pull).
|
||||||
|
func ReferencedSHAs(dir string) (map[string]int64, error)
|
||||||
|
func ValidSHA(s string) bool // ^[0-9a-f]{64}$
|
||||||
|
func BlobKey(sha string) string // "sha256/ab/cd/<sha>"
|
||||||
|
```
|
||||||
|
|
||||||
|
`LoadConfig` takes `getenv` so tests inject env without `t.Setenv` races. Invalid ints/durations in env → fall back to default (do not error). `ReferencedSHAs` errors on: unreadable dir, no `*.yml` files, YAML parse failure, or an entry whose `sha256` fails `ValidSHA` (report file + path). Manifest struct:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type manifestFile struct {
|
||||||
|
Assets []struct {
|
||||||
|
Path string `yaml:"path"`
|
||||||
|
SHA256 string `yaml:"sha256"`
|
||||||
|
Size int64 `yaml:"size"`
|
||||||
|
} `yaml:"assets"`
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 1:** `git -C sow-tools switch -c depot-core-increment-1` (from main).
|
||||||
|
- [ ] **Step 2:** Write failing tests: `TestLoadConfigDefaults`, `TestLoadConfigReadKeyFallback` (READ unset → WriteKey value; READ set → its own), `TestLoadConfigBadIntFallsBack`, `TestBlobKey` (`BlobKey("0b1d…")=="sha256/0b/1d/0b1d…"`), `TestValidSHA` (rejects uppercase, short, non-hex), `TestReferencedSHAs` (temp dir with two yml files sharing one sha → dedup; bad sha → error naming the file; empty dir → error).
|
||||||
|
- [ ] **Step 3:** `go test ./internal/depot/` — verify FAIL (compile error is fine).
|
||||||
|
- [ ] **Step 4:** Implement `config.go` + `manifest.go` minimally.
|
||||||
|
- [ ] **Step 5:** `go test ./internal/depot/` PASS; `gofmt -l -w internal/depot`.
|
||||||
|
- [ ] **Step 6:** Commit: `feat(depot): config resolution and manifest sha parsing`.
|
||||||
|
|
||||||
|
### Task 2: Local backend
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `internal/depot/backend.go`, `internal/depot/local.go`, `internal/depot/local_test.go`
|
||||||
|
|
||||||
|
**Interfaces (Produces):**
|
||||||
|
|
||||||
|
```go
|
||||||
|
// backend.go
|
||||||
|
type ProbeState int
|
||||||
|
const (
|
||||||
|
Present ProbeState = iota
|
||||||
|
Absent
|
||||||
|
Unconfirmed
|
||||||
|
)
|
||||||
|
type Backend interface {
|
||||||
|
Name() string
|
||||||
|
// Probe returns the existence state of one sha. transient=true means a
|
||||||
|
// retry might change the answer (feeds the serial confirm loop).
|
||||||
|
Probe(ctx context.Context, sha string) (state ProbeState, transient bool, err error)
|
||||||
|
// Get fetches sha into dest (temp file + rename), re-hashes, deletes on mismatch.
|
||||||
|
Get(ctx context.Context, sha, dest string) error
|
||||||
|
// Put uploads bytes from src for sha. Read-only backends return an error.
|
||||||
|
Put(ctx context.Context, sha, src string) error
|
||||||
|
}
|
||||||
|
|
||||||
|
// local.go
|
||||||
|
type LocalBackend struct{ Root string } // implements Backend; Name()=="local"
|
||||||
|
```
|
||||||
|
|
||||||
|
Local semantics: `Probe` = `os.Stat(Root/BlobKey(sha))`, never transient. `Put` = copy to temp file in dest dir then `os.Rename` (MkdirAll parents). `Get` = copy out + re-hash (hash the source bytes while copying via `io.TeeReader` into `sha256.New()`), delete dest and error on mismatch. `err` from Probe only for real I/O errors (permission), not absence.
|
||||||
|
|
||||||
|
- [ ] **Step 1:** Failing tests: `TestLocalRoundTrip` (Put→Probe Present→Get→bytes equal), `TestLocalProbeAbsent`, `TestLocalGetHashMismatch` (hand-write a corrupt blob file at the keyed path, Get returns error and dest does not exist), `TestLocalPutIdempotent`.
|
||||||
|
- [ ] **Step 2:** Verify FAIL. **Step 3:** Implement. **Step 4:** PASS + gofmt.
|
||||||
|
- [ ] **Step 5:** Commit: `feat(depot): backend interface and local backend`.
|
||||||
|
|
||||||
|
### Task 3: Remote backends (cdn, bunny) with fake-Bunny tests
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `internal/depot/remote.go`, `internal/depot/remote_test.go`
|
||||||
|
|
||||||
|
**Interfaces (Consumes):** `Backend`, `ProbeState`, `Config`, `BlobKey`.
|
||||||
|
**Produces:**
|
||||||
|
|
||||||
|
```go
|
||||||
|
// NewBackend returns the backend for name ("local"|"cdn"|"bunny").
|
||||||
|
// localRoot is used only for "local". Fails closed: "bunny" with empty
|
||||||
|
// StorageHost or empty ReadKey returns an error (never prompts);
|
||||||
|
// unknown name returns an error.
|
||||||
|
func NewBackend(name, localRoot string, cfg Config) (Backend, error)
|
||||||
|
|
||||||
|
type httpBackend struct { ... } // shared by cdn and bunny
|
||||||
|
```
|
||||||
|
|
||||||
|
Implementation requirements:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// IPv4-only transport — REQUIRED, spec bans IPv6/HEAD probe hazards.
|
||||||
|
transport := &http.Transport{
|
||||||
|
DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
|
||||||
|
d := net.Dialer{Timeout: cfg.ConnectTimeout}
|
||||||
|
return d.DialContext(ctx, "tcp4", addr)
|
||||||
|
},
|
||||||
|
MaxIdleConnsPerHost: cfg.ProbeJobs,
|
||||||
|
}
|
||||||
|
client := &http.Client{Transport: transport, Timeout: cfg.ProbeMaxTime}
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Probe** (both remotes): `GET <base>/<BlobKey(sha)>` with header `Range: bytes=0-0`; drain/close body. 2xx → `Present`; 404/410 → `Absent`; anything else (incl. transport error) → `Unconfirmed, transient=true`. **Never http.Head / MethodHead.**
|
||||||
|
- cdn: base = `cfg.CDNBase`, no auth headers, `Put` returns `errors.New("cdn backend is read-only")`.
|
||||||
|
- bunny: probe/read against storage `https://<StorageHost>/<StorageZone>/<BlobKey(sha)>` with `AccessKey: <ReadKey>`. `Get`: try CDN base first, on non-2xx fall back to storage URL with ReadKey (mirrors bash `_depot_bunny_get`); re-hash, delete on mismatch. `Put`: probe first, skip if Present; else `PUT` storage URL, headers `AccessKey: <WriteKey>`, `Checksum: <strings.ToUpper(sha)>`, body streamed from src file; non-2xx → error. `Put` with empty WriteKey → error before any I/O.
|
||||||
|
- Both `Get`s write via temp file + rename and re-hash exactly like local.
|
||||||
|
|
||||||
|
httptest note: `httptest.NewServer` listens on `127.0.0.1`, which the tcp4 dialer reaches — no test shim needed.
|
||||||
|
|
||||||
|
- [ ] **Step 1:** Failing tests against an `httptest` fake Bunny (a handler recording method+path+headers, serving a blob map):
|
||||||
|
- `TestProbeUsesRangeGetNotHead` — probe a present sha; assert recorded request has `Method=="GET"` and `Range=="bytes=0-0"`; **also** assert the depot source contains no HEAD probe: the handler returns 500 for any HEAD request, and the test asserts zero HEAD requests were received across the whole test run (this is the anti-HEAD regression tripwire from field note 3).
|
||||||
|
- `TestProbeStates` — 200/206→Present, 404→Absent, 428→Unconfirmed+transient, 503→Unconfirmed+transient.
|
||||||
|
- `TestBunnyPutSequence` — Put of an absent sha: recorded sequence is [GET range probe, PUT]; PUT has `Checksum` = uppercase sha and `AccessKey` = write key; Put of a present sha: probe only, no PUT.
|
||||||
|
- `TestReadWriteKeySplit` — probe uses ReadKey, PUT uses WriteKey (distinct values asserted).
|
||||||
|
- `TestBunnyNoReadKeyFailsClosed` — `NewBackend("bunny",…)` with empty read key returns error **without reading stdin**: replace `os.Stdin` with the read end of an `os.Pipe()` for the test; after the call, write+close and confirm the pipe is undrained (or simply assert error occurs with no stdin interaction because the code path has no stdin reference — grep-based assertion acceptable: test fails if `internal/depot` sources mention `os.Stdin` or `/dev/tty`).
|
||||||
|
- `TestBunnyNoStorageHostFailsClosed`.
|
||||||
|
- `TestGetHashMismatchDeletes` — server returns wrong bytes; Get errors, dest removed.
|
||||||
|
- `TestCDNPutReadOnly`.
|
||||||
|
- [ ] **Step 2:** Verify FAIL. **Step 3:** Implement `remote.go`. **Step 4:** PASS + gofmt + `go vet ./...`.
|
||||||
|
- [ ] **Step 5:** Commit: `feat(depot): cdn and bunny HTTP backends with fake-Bunny tests`.
|
||||||
|
|
||||||
|
### Task 4: Sweep engine — parallel probe + serial confirm
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `internal/depot/sweep.go`, `internal/depot/sweep_test.go`
|
||||||
|
|
||||||
|
**Interfaces (Consumes):** `Backend`, `Config`, `ProbeState`.
|
||||||
|
**Produces:**
|
||||||
|
|
||||||
|
```go
|
||||||
|
type SweepResult struct {
|
||||||
|
Present []string
|
||||||
|
Missing []string // confirmed absent (404/410 after retries)
|
||||||
|
Unconfirmed []string // probe budget exhausted, state unknown
|
||||||
|
}
|
||||||
|
// Sweep probes every sha against b: one parallel pass (cfg.ProbeJobs workers,
|
||||||
|
// sync.WaitGroup + buffered-channel work queue), then any non-Present result
|
||||||
|
// is re-probed serially with linear backoff (attempt t sleeps t seconds,
|
||||||
|
// cfg.ConfirmRetries attempts) — unless the non-Present count exceeds
|
||||||
|
// cfg.ConfirmMax (>0), in which case the remainder is reported Unconfirmed
|
||||||
|
// without serial re-probe (never collapsed into Missing).
|
||||||
|
// A transient result that stays non-2xx after retries is Unconfirmed;
|
||||||
|
// a 404/410 is Missing. Backend errors (non-transient) abort with error.
|
||||||
|
func Sweep(ctx context.Context, b Backend, shas []string, cfg Config, progress io.Writer) (SweepResult, error)
|
||||||
|
```
|
||||||
|
|
||||||
|
Sort `shas` before sweeping (deterministic output). Progress: print `probed X/N` roughly every 1000 blobs to `progress` (may be `io.Discard`). For tests, make the backoff sleeper injectable: package-level `var confirmSleep = time.Sleep`, overridden in tests.
|
||||||
|
|
||||||
|
- [ ] **Step 1:** Failing tests using a stub Backend (function fields):
|
||||||
|
- `TestSweepAllPresent`, `TestSweepMissing` (404 → Missing, no retries beyond first serial confirm).
|
||||||
|
- `TestSweepThrottledUnconfirmed` — stub always returns Unconfirmed/transient → after retries lands in `Unconfirmed`, never `Missing`; assert `confirmSleep` called with 1s then 2s.
|
||||||
|
- `TestSweepTransientRecovers` — first probe 428, serial re-probe 206 → Present.
|
||||||
|
- `TestSweepConfirmMaxCap` — 5 non-present, ConfirmMax=2 → all 5 Unconfirmed, zero serial re-probes.
|
||||||
|
- `TestSweepParallelBounded` — 100 shas, ProbeJobs=4; stub counts concurrent in-flight probes (atomic), assert max ≤ 4.
|
||||||
|
- [ ] **Step 2:** FAIL. **Step 3:** Implement. **Step 4:** PASS + gofmt.
|
||||||
|
- [ ] **Step 5:** Commit: `feat(depot): parallel probe sweep with serial confirm and unconfirmed state`.
|
||||||
|
|
||||||
|
### Task 5: Commands + Run entrypoint + integration test
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `internal/depot/run.go`, `internal/depot/run_test.go`, `internal/depot/integration_test.go`
|
||||||
|
|
||||||
|
**Interfaces (Consumes):** everything above.
|
||||||
|
**Produces:**
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Run executes a depot subcommand. args[0] is the subcommand
|
||||||
|
// (status|push|verify|get|pull); returns the process exit code.
|
||||||
|
// Exit: 0 clean, 1 drift, 2 unconfirmed-only, 64 usage, 70 internal.
|
||||||
|
func Run(args []string, stdout, stderr io.Writer, getenv func(string) string) int
|
||||||
|
```
|
||||||
|
|
||||||
|
Flag parsing: one `flag.FlagSet` per subcommand (stdlib; `internal/depot` is a fresh package, the hand-rolled loop in `internal/app` is not a constraint here). Flags per spec:
|
||||||
|
|
||||||
|
```
|
||||||
|
status [--manifests DIR] [--source LOCAL_DEPOT] --target bunny|cdn|local
|
||||||
|
push [--manifests DIR] --source LOCAL_DEPOT --target bunny
|
||||||
|
verify [--manifests DIR] --target bunny|cdn [--sample N]
|
||||||
|
get <sha> <dest> --target cdn|bunny|local
|
||||||
|
pull [--manifests DIR] --dest DIR --target cdn|bunny
|
||||||
|
```
|
||||||
|
|
||||||
|
`--manifests` default `assets`. `--target local` uses `--source` as root for status, and `--target`-side root comes from `DEPOT_DIR` env when target is local (matches bash `DEPOT_DIR`); document in usage string. Missing required flag / unknown subcommand / no args → usage on stderr, exit 64.
|
||||||
|
|
||||||
|
Command semantics (all build shas via `ReferencedSHAs`, backend via `NewBackend`, sweep via `Sweep`):
|
||||||
|
|
||||||
|
- **status**: sweep; print `referenced=<n> present=<n> missing=<n> unconfirmed=<n>` to stdout, then each missing sha on its own line prefixed `missing ` and each unconfirmed prefixed `unconfirmed `. Exit 1 if missing>0; else 2 if unconfirmed>0; else 0.
|
||||||
|
- **push**: target must be `bunny` (enforce). Sweep; treat missing∪unconfirmed as to-upload (unconfirmed→re-upload is safe/idempotent per spec). For each, read bytes from `--source` local root; a sha absent from source → report and count as failed. Upload with `cfg.Jobs` parallel workers, each sha at most once. Exit 70 on upload error, 1 if any sha was missing from source, else 0. Print `uploaded=<n> failed=<n>`.
|
||||||
|
- **verify**: sweep (exit 1/2 as status) **plus**, when clean, download `--sample` N (default 3, 0=skip) random blobs (`math/rand` seeded — pick via deterministic-enough `rand.New(rand.NewSource(time.Now().UnixNano()))`) to temp files via `Get` (which re-hashes); any failure → exit 1.
|
||||||
|
- **get**: `Get(sha, dest)`; validate sha; exit 0/70.
|
||||||
|
- **pull**: for every referenced sha, dest path `--dest/BlobKey(sha)`. Incremental: if dest file exists, re-hash it; matching → skip, mismatch → re-download. Download absent/mismatched with `cfg.Jobs` workers via `Get`. Print `pulled=<n> present=<n>`; any failed download → exit 70 after finishing the batch (report each failure). Non-zero missing at the source (404) → exit 1 listing them.
|
||||||
|
|
||||||
|
Wire the binary: **Modify** `internal/dispatch/dispatch.go` — in `runBuilder`, before the `Wired` check/`delegateLegacy` routing, special-case:
|
||||||
|
|
||||||
|
```go
|
||||||
|
if b.Name == "depot" && b.Wired {
|
||||||
|
return depot.Run(args[1:], os.Stdout, errw, os.Getenv)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
(Exact insertion point: the implementer must read `runBuilder` and place this after subcommand resolution is bypassed — depot does its own subcommand parsing, so pass the raw args through, i.e. `args[1:]` where `args[0]=="depot"` at the dispatcher level, or the full arg slice in `RunBuilder("depot", …)` context. Verify both `crucible depot status …` and `crucible-depot status …` paths hit `depot.Run`.)
|
||||||
|
|
||||||
|
- [ ] **Step 1:** Failing unit tests (`run_test.go`): `TestRunUsage` (no args→64, unknown cmd→64, push --target cdn→64), `TestStatusExitCodes` (drift→1, unconfirmed-only→2 using a bunny backend pointed at an httptest server that 428s), `TestGetInvalidSHA` (→64 or 70, pick 64), `TestNoKeyStatusBunnyFailsClosedNoStdin` (getenv returns empty keys → exit 70, stderr mentions the env var name).
|
||||||
|
- [ ] **Step 2:** Failing integration test (`integration_test.go`), pure local→local in temp dirs: build manifests yml referencing 3 blobs; source depot has all 3; target depot has 1. Then: `status --target local` → exit 1, missing=2 → `push` (local→local: for the test, allow `push --target local` **only** via an unexported hook? No — keep the spec surface: instead run the integration through bunny backend backed by httptest fake-Bunny with an in-memory blob map, which exercises the real push path) → `status` → 0 → `push` again → 0 uploads (idempotent, assert fake recorded no second PUT). Also `pull` into a pre-populated dest: present-and-hash-ok skipped (fake records no GET for it), corrupt pre-existing file re-downloaded.
|
||||||
|
- [ ] **Step 3:** FAIL. **Step 4:** Implement `run.go`. **Step 5:** `make vet test` PASS + gofmt.
|
||||||
|
- [ ] **Step 6:** Commit: `feat(depot): status/push/verify/get/pull commands with exit-code contract`.
|
||||||
|
|
||||||
|
### Task 6: Registry wiring + docs
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `internal/dispatch/dispatch.go` (depot Registry entry)
|
||||||
|
- Modify: `internal/dispatch/dispatch_test.go` (if it asserts wired/unwired sets)
|
||||||
|
|
||||||
|
Flip the depot entry:
|
||||||
|
|
||||||
|
```go
|
||||||
|
{
|
||||||
|
Name: "depot",
|
||||||
|
Bin: "crucible-depot",
|
||||||
|
Summary: "content-addressed asset depot (local/cdn/bunny)",
|
||||||
|
Commands: []Command{
|
||||||
|
{Name: "status", Summary: "report referenced-vs-present drift against a backend", Usage: "crucible depot status [--manifests DIR] [--source DIR] --target bunny|cdn|local"},
|
||||||
|
{Name: "push", Summary: "upload referenced-but-absent blobs from a local depot", Usage: "crucible depot push [--manifests DIR] --source DIR --target bunny"},
|
||||||
|
{Name: "verify", Summary: "existence sweep plus sampled download re-hash", Usage: "crucible depot verify [--manifests DIR] --target bunny|cdn [--sample N]"},
|
||||||
|
{Name: "get", Summary: "fetch one blob with sha re-verify", Usage: "crucible depot get <sha> <dest> --target cdn|bunny|local"},
|
||||||
|
{Name: "pull", Summary: "incremental verified pull of every referenced blob", Usage: "crucible depot pull [--manifests DIR] --dest DIR --target cdn|bunny"},
|
||||||
|
},
|
||||||
|
Wired: true,
|
||||||
|
},
|
||||||
|
```
|
||||||
|
|
||||||
|
The stale "(SeaweedFS)" wording must not survive anywhere: `grep -ri seaweedfs` across the repo and fix hits.
|
||||||
|
|
||||||
|
- [ ] **Step 1:** Update registry; run `go test ./internal/dispatch/` — fix any test that pinned depot as unwired; add/extend a test asserting `crucible depot` routes to `depot.Run` (e.g. `RunBuilder("depot", []string{"status"})` with no manifests dir returns 70/64, not the unwired 70 message — assert on stderr text).
|
||||||
|
- [ ] **Step 2:** `make check` (vet+test+shellcheck+yamllint) PASS. Also `make build && ./bin/crucible depot` prints depot usage; `./bin/crucible` interactive menu shows depot entries (manual eyeball via `printf 'q\n' | ./bin/crucible` or equivalent).
|
||||||
|
- [ ] **Step 3:** Commit: `feat(depot): wire depot into dispatch registry and menu`.
|
||||||
|
|
||||||
|
### Task 7: sow-assets-manifest cutover
|
||||||
|
|
||||||
|
**Repo:** `/home/vicky/Projects/westgate/repositories/migration/sow-assets-manifest`, new branch `crucible-depot-cutover` from main.
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Delete: `scripts/mirror.sh`
|
||||||
|
- Modify: `scripts/build-haks.sh` (the `--full` depot-verify block, ~L138-192), `Makefile` (`mirror` target, `haks` target), `.gitea/workflows/release.yml`, `.gitea/workflows/publish-haks.yml`, `.gitea/workflows/build-haks.yml`, `scripts/lib.sh` (header comments only), `AGENTS.md`/`README.md` (author-facing docs)
|
||||||
|
|
||||||
|
**Consumes:** the `crucible depot` CLI from Tasks 1–6. NOTE: CI resolves crucible via the flake input tracking sow-tools' default branch — the sow-tools branch must be merged before this repo's CI can pass. Local verification uses a locally built binary (`sow-tools/bin/crucible` on PATH). Do not run `nix flake update` here; note in the PR description that a `nix flake update sow-tools` (or lock bump) must land with/after the sow-tools merge.
|
||||||
|
|
||||||
|
Changes:
|
||||||
|
|
||||||
|
1. **`release.yml`** "Warm depot cache (verified full pull)" step body (keep name, job env unchanged; drop the ticker/compgen bash):
|
||||||
|
```yaml
|
||||||
|
- name: Warm depot cache (verified full pull)
|
||||||
|
env:
|
||||||
|
BUNNY_STORAGE_READ_PASSWORD: ${{ secrets.BUNNY_STORAGE_READ_PASSWORD }}
|
||||||
|
BUNNY_STORAGE_HOST: storage.bunnycdn.com
|
||||||
|
BUNNY_STORAGE_ZONE: sow-assets-depot
|
||||||
|
run: |
|
||||||
|
nix develop --command bash -c '
|
||||||
|
set -euo pipefail
|
||||||
|
crucible depot status --manifests assets --target bunny
|
||||||
|
crucible depot pull --manifests assets --dest "$DEPOT_CACHE_DIR" --target cdn
|
||||||
|
'
|
||||||
|
```
|
||||||
|
(If the `BUNNY_STORAGE_READ_PASSWORD` secret does not exist in the repo, use `secrets.BUNNY_STORAGE_PASSWORD` and say so in the PR body. Check what other steps use.)
|
||||||
|
2. **`publish-haks.yml`** — same swap of the identical step.
|
||||||
|
3. **`build-haks.yml`** — the PR-check step keeps `build-haks.sh --dry-run --no-verify --base "$base"` (structural check) and **adds** `crucible depot verify --manifests assets --target cdn --sample 3` as the cdn-backend verification.
|
||||||
|
4. **`scripts/build-haks.sh`** — delete the depot existence-sweep + sample-download block inside `--dry-run` (the `depot_has_many`/`depot_get` verify at ~L138-185); replace with a comment pointing at `crucible depot verify`. Remove now-unused locals. `shellcheck scripts/build-haks.sh` must pass.
|
||||||
|
5. **`Makefile`** — `mirror:` target now runs `crucible depot pull --manifests assets --dest "$(DEPOT_DIR)" --target cdn`. `haks:` gains a trailing non-fatal drift report: `-crucible depot status --manifests assets --target cdn` (leading `-` so drift never fails a local build) and an echo pointing at `crucible depot push --source workspace/depot --target bunny` as the standard post-edit step.
|
||||||
|
6. **`scripts/lib.sh`** — no function deletions (caller analysis: every depot function used by mirror.sh/build-haks.sh retains Increment 2–3 callers). Add a header comment block at the top of the depot section: `# NOTE: this bash depot layer is being retired. Increment 1 moved mirror/verify to 'crucible depot' (sow-tools). import/sync go in Increment 2; export/pack/publish/artifact/cdn_purge in Increment 3. Do not add new callers.` **Do not touch the backend-keyed stat-cache in import.sh** (spec forbids it).
|
||||||
|
7. **Docs** — in README.md (or AGENTS.md where depot workflow is documented): document `crucible depot status --manifests assets --target cdn` as the canonical credential-free "is my content live?" author check, and `crucible depot push --source workspace/depot --target bunny` as the standard post-edit publish step.
|
||||||
|
|
||||||
|
- [ ] **Step 1:** Branch; make all edits above.
|
||||||
|
- [ ] **Step 2:** Verify: `shellcheck scripts/*.sh`; `yamllint .gitea`; `grep -rn "mirror.sh" . --exclude-dir=.git` returns nothing; with sow-tools' `bin` on PATH run `crucible depot status --manifests assets --target cdn` for real (credential-free) and confirm it produces a sane report (exit 0 or 2 acceptable; exit 1 means real drift — report it, don't hide it).
|
||||||
|
- [ ] **Step 3:** Commit: `feat: cut depot mirror/verify over to crucible depot, retire mirror.sh`.
|
||||||
|
|
||||||
|
## Self-Review Notes
|
||||||
|
|
||||||
|
- Spec coverage: command surface (T5), drift model + unconfirmed/exit 2 (T4/T5), backends + env + no-prompt (T1/T3), IPv4 range probe + HEAD ban test (T3), Checksum PUT (T3), pull incremental (T5), cutover of all three workflows + Makefile + lib.sh headers + author docs (T7), registry/SeaweedFS fix + menu (T6), testing section requirements all mapped (T1–T5), interim contract `make haks` drift report (T7.5).
|
||||||
|
- Deliberate deviations from spec, justified: `status --target local` root via `DEPOT_DIR`; push treats source-missing blobs as exit 1; pull download failure exits 70 (backend error) but source-404 exits 1 (drift).
|
||||||
|
- Out of scope, per spec non-goals: import/sync, prune/gc/export, mdl checks, rich TUI, cdn_purge.
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,357 @@
|
|||||||
|
# Crucible build parity — local == CI, zero-config reproducible
|
||||||
|
|
||||||
|
Date: 2026-06-21
|
||||||
|
Status: approved (design); implementation pending
|
||||||
|
Repos touched: `sow-tools` (Crucible + contract doc + image notes),
|
||||||
|
`sow-topdata` (config, flake, wrapper, CI), `sow-assets-manifest` (flake pin),
|
||||||
|
and the consumer wrapper/flake pattern shared by every Crucible repo.
|
||||||
|
|
||||||
|
Supersedes the resolver-script approach in
|
||||||
|
`2026-06-21-topdata-vfx-autogen-channel-design.md` (the channel→tag→`vfxs.yml`
|
||||||
|
resolution moves out of a bash wrapper and into Crucible).
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
`crucible topdata build-topdata` run by hand does **not** reproduce the CI
|
||||||
|
build. CI runs `scripts/build-topdata.sh`, which resolves the accessory-VFX
|
||||||
|
model list with `scripts/resolve-accessory-vfx.sh` and writes
|
||||||
|
`.cache/sow-accessory-vfx-manifest.json` **before** Crucible runs; Crucible only
|
||||||
|
reads that pre-staged file (`nwn-tool.yaml: manifest_file:`). Run Crucible
|
||||||
|
directly and the file never exists, the `optional` consumer skips, and
|
||||||
|
`visualeffects.2da` ships **zero accessory rows** — exactly the local build the
|
||||||
|
operator observed (637 rows in CI, 0 locally).
|
||||||
|
|
||||||
|
Root cause: a **build-output-affecting input is resolved outside Crucible**, in
|
||||||
|
a wrapper script. The wrapper, not Crucible, is authoritative for that input.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Crucible is the single authority for everything that lands in a build artifact.
|
||||||
|
A clone of any consumer repo, on any system, builds the same bytes CI builds —
|
||||||
|
with no secret beyond what cloning already required and no hand-written
|
||||||
|
configuration.
|
||||||
|
|
||||||
|
### Zero-config reproducibility requirements
|
||||||
|
|
||||||
|
- **R1 — No extra secrets.** All build-input reads are anonymous: the asset
|
||||||
|
channel pointer (`releases/haks/channels.json`) and `vfxs.yml` are public CDN
|
||||||
|
text. The crucible binary is fetchable anonymously — **verified**: the
|
||||||
|
sow-tools release assets return HTTP 200 without auth, and the bootstrap tries
|
||||||
|
anon before any token, so `CRUCIBLE_TOKEN` in consumer CI is **vestigial and is
|
||||||
|
removed**. Git-LFS asset bytes use the **same** credentials the clone already
|
||||||
|
used — never a separate key.
|
||||||
|
- **R2 — No hand-written config.** Every default needed to build is baked into
|
||||||
|
committed `nwn-tool.yaml` and Crucible defaults: CDN base, channel selection,
|
||||||
|
source paths. The user types `crucible topdata build-topdata` and nothing else.
|
||||||
|
- **R3 — No host data assumptions, no `NWN_ROOT`.** Builders read every input
|
||||||
|
from the project tree (`nwn-tool.yaml` + committed `data/**/base.json`); no NWN
|
||||||
|
install is required. `NWN_ROOT` is **dead** today — nothing in Crucible reads it
|
||||||
|
(see *NWN_ROOT cleanup* below). The accessory build reads
|
||||||
|
`data/visualeffects/base.json`. (Enforced by the CI parity guard, which runs in
|
||||||
|
a bare checkout with no `NWN_ROOT`.)
|
||||||
|
- **R4 — Nix is convenience, not a requirement.** The flake devshell provides
|
||||||
|
`crucible`, `git-lfs`, `jq`, `yq`, etc. on `PATH` for Nix users. Non-Nix users
|
||||||
|
(incl. Windows) get the same Crucible via the `crucible.sh`/`crucible.ps1`
|
||||||
|
bootstrap and a host `git-lfs`; the build path through Crucible is identical.
|
||||||
|
- **R5 — Same Crucible, both worlds.** Local dev and CI invoke the same Crucible
|
||||||
|
subcommand against the same committed config. No CI-only pre-steps that change
|
||||||
|
output.
|
||||||
|
|
||||||
|
## Parity contract (prevents recurrence)
|
||||||
|
|
||||||
|
Added to `sow-tools/docs/consumer-contract.md`. A consumer build wrapper may only:
|
||||||
|
|
||||||
|
1. **validate** — a pre-build gate that does not change output;
|
||||||
|
2. **publish** — post-build, on a separate artifact;
|
||||||
|
3. **resolve the crucible binary** — the bootstrap in `consumer-contract.md`.
|
||||||
|
|
||||||
|
A wrapper MUST NOT fetch, resolve, generate, or stage any input Crucible reads
|
||||||
|
to produce the artifact. If Crucible needs an input, Crucible acquires it. This
|
||||||
|
is the existing "no local-machine assumptions" + "same inputs → identical
|
||||||
|
output" rule, made explicit: **the wrapper is not allowed to be authoritative
|
||||||
|
for build inputs.**
|
||||||
|
|
||||||
|
## Design
|
||||||
|
|
||||||
|
### A. Accessory-VFX resolution moves into Crucible
|
||||||
|
|
||||||
|
**Config (`sow-topdata/nwn-tool.yaml`).** Drop `manifest_file:`; declare the
|
||||||
|
source on the consumer. The 4-group + `mdl` filter is **not** re-declared — it
|
||||||
|
is derived from the keys already present under `accessory_visualeffects.groups`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
autogen:
|
||||||
|
consumers:
|
||||||
|
- id: accessory_visualeffects
|
||||||
|
producer: accessory_visualeffects
|
||||||
|
dataset: visualeffects
|
||||||
|
mode: accessory_visualeffects
|
||||||
|
optional: true
|
||||||
|
source:
|
||||||
|
kind: cdn_channel
|
||||||
|
cdn_base_env: BUNNY_CDN_BASE # default https://depot.westgate.pw
|
||||||
|
channels_path: releases/haks/channels.json
|
||||||
|
manifest_path: releases/haks/{tag}/vfxs.yml
|
||||||
|
release_marker_path: releases/haks/{tag}/haks.json # the broken-release guard probe
|
||||||
|
channel_env: SOW_TOPDATA_ASSET_CHANNEL # else derive from git tag
|
||||||
|
offline_override_env: SOW_VFXS_MANIFEST # dev/air-gapped vfxs.yml path
|
||||||
|
accessory_visualeffects:
|
||||||
|
groups: { chest_accessories: {...}, head_accessories: {...},
|
||||||
|
head_decorations: {...}, head_features: {...} }
|
||||||
|
# ... rest of the policy block unchanged
|
||||||
|
```
|
||||||
|
|
||||||
|
New struct field on `AutogenConsumerConfig` (sow-tools
|
||||||
|
`internal/project/project.go`): `Source AutogenSourceConfig` with the keys
|
||||||
|
above. Validation: when `source.kind == cdn_channel`, `channels_path` and
|
||||||
|
`manifest_path` are required and `manifest_path` must contain `{tag}`.
|
||||||
|
|
||||||
|
**Resolution (`internal/topdata/autogen.go`,
|
||||||
|
`resolveAutogenConsumerManifest`).** A new branch for `source.kind ==
|
||||||
|
cdn_channel`, reusing the existing `fetchJSON` HTTP client and the project YAML
|
||||||
|
parser:
|
||||||
|
|
||||||
|
1. **Offline override.** If `offline_override_env` names a readable file (or a
|
||||||
|
manifest-repo checkout root containing `assets/vfxs.yml`), parse it and skip
|
||||||
|
the network — the dev/air-gapped path.
|
||||||
|
2. **Channel.** `channel_env` if set; else from the git tag / `GITHUB_REF_NAME`
|
||||||
|
(`v*-*` → `testing`, `v*` → `current`, otherwise `current`).
|
||||||
|
3. **CDN base.** `cdn_base_env` value, else the baked default.
|
||||||
|
4. `GET {cdn}/{channels_path}` → JSON → `tag = channels[channel]`.
|
||||||
|
5. `GET {cdn}/{manifest_path with {tag}}` → parse YAML `assets[]`.
|
||||||
|
6. **Filter** to `restype == mdl` whose `path` is under `vfxs/<g>/` for `<g>` in
|
||||||
|
the `accessory_visualeffects.groups` keys. Derive each entry's `source`
|
||||||
|
(leading `vfxs/` stripped), `group`, `subgroup`, `model_stem`. Sort by
|
||||||
|
`source`. Hand the entries to the existing accessory augmentor unchanged.
|
||||||
|
|
||||||
|
**Fail policy (ported verbatim from the resolver — this is the v0.1.4-bug
|
||||||
|
guard).** Distinguish *unreachable* (fail open) from *broken* (hard fail):
|
||||||
|
|
||||||
|
- channels.json unreachable / not JSON / channel absent → **fail open**: the
|
||||||
|
`optional` consumer contributes no rows and `data/visualeffects/lock.json` IDs
|
||||||
|
are preserved.
|
||||||
|
- vfxs.yml network error, or 404 **and** the `release_marker_path` (haks.json)
|
||||||
|
is **absent** for that tag (no published release) → **fail open**.
|
||||||
|
- vfxs.yml 404 **and** haks.json **present** → **HARD fail** ("release `<tag>`
|
||||||
|
has haks.json but no vfxs.yml — accessory rows would silently vanish").
|
||||||
|
- vfxs.yml present but malformed / no `assets` array → **HARD fail**.
|
||||||
|
|
||||||
|
Fail-open writes nothing and lets the optional consumer skip; hard-fail aborts
|
||||||
|
the build with the message above.
|
||||||
|
|
||||||
|
**Deleted:** `sow-topdata/scripts/resolve-accessory-vfx.sh`,
|
||||||
|
`tests/resolve-accessory-vfx-contract.sh`, the `manifest_file:` key, and the
|
||||||
|
resolver pre-step at `build-topdata.sh:19`. The four bash contract cases
|
||||||
|
(offline override, unreachable fail-open, broken-release hard-fail, no-release
|
||||||
|
fail-open) are re-expressed as Go tests in `internal/topdata`.
|
||||||
|
|
||||||
|
### B. Git-LFS materialization moves into Crucible
|
||||||
|
|
||||||
|
So a bare clone + `crucible` packs real bytes, not pointer stubs (today only
|
||||||
|
`build-topdata.sh` pulls LFS, so direct Crucible would ship stubs in CI's
|
||||||
|
non-smudging checkout).
|
||||||
|
|
||||||
|
Before packing assets into a hak, Crucible: detects git-LFS pointer stubs under
|
||||||
|
the asset tree; if any, runs `git lfs install --local` then `git lfs pull` in
|
||||||
|
the repo; re-checks and **hard-fails** if stubs remain ("refusing to build from
|
||||||
|
pointer stubs"). Uses the clone's own credentials (R1). Requires `git` +
|
||||||
|
`git-lfs` on `PATH` — provided by the Nix devshell, a standard host install
|
||||||
|
otherwise (a clone without git-lfs already yields stubs, so this is the existing
|
||||||
|
host expectation, now enforced loudly by Crucible instead of a wrapper).
|
||||||
|
|
||||||
|
A skip control replaces `TOPDATA_SKIP_LFS` for the maintain-tree flow, which
|
||||||
|
regenerates the `data/` tree and **discards** the package: a Crucible flag/env
|
||||||
|
(e.g. `--skip-lfs` / `CRUCIBLE_SKIP_LFS`) so that one job can opt out. Default
|
||||||
|
is fail-closed (pull).
|
||||||
|
|
||||||
|
### C. Wrappers thin out
|
||||||
|
|
||||||
|
`scripts/build-topdata.sh` (and the shared pattern in every Crucible repo)
|
||||||
|
reduces to: validate (gate) → resolve crucible → `crucible <build>`. No input
|
||||||
|
resolution, no LFS pull. `sow-module/scripts/package-module.sh` already matches
|
||||||
|
this shape (validate → resolve → build); it stays the reference.
|
||||||
|
|
||||||
|
### D. CI parity guard (catches the next regression)
|
||||||
|
|
||||||
|
A `sow-tools` integration test builds the topdata package via Crucible **alone**
|
||||||
|
in a bare checkout (no wrapper, clean `.cache`, no `NWN_ROOT`, anonymous CDN)
|
||||||
|
and asserts `visualeffects.2da` contains accessory rows. This is the test that
|
||||||
|
would have caught today's bug: it fails if any build-output-affecting step ever
|
||||||
|
drifts back out of Crucible into a wrapper, and it enforces R2/R3/R5 mechanically.
|
||||||
|
|
||||||
|
The contract rule in `consumer-contract.md` is the written half; this test is
|
||||||
|
the enforced half.
|
||||||
|
|
||||||
|
### E. NWN_ROOT cleanup (stale, contradicts the goal)
|
||||||
|
|
||||||
|
`NWN_ROOT` is unused — no `os.Getenv`, no `--nwn-root` flag, no consumer. The
|
||||||
|
only traces are documentation/help-text describing a rule for a thing that does
|
||||||
|
not exist: the `## NWN_ROOT rule` section in `consumer-contract.md`, the help
|
||||||
|
text in `internal/dispatch/dispatch.go` ("if a builder ever needs an NWN
|
||||||
|
root… crucible never guesses from $HOME"), and the `README.md` pointer. That
|
||||||
|
old rule — *never guess, require an explicit env* — directly contradicts the
|
||||||
|
zero-config goal (R2): it would make a future builder demand a hand-set path.
|
||||||
|
|
||||||
|
- **Remove** the `NWN_ROOT` references: the `consumer-contract.md` section, the
|
||||||
|
dispatch help-text lines, the README pointer. Leave the
|
||||||
|
`migration-from-nwn-tool.md` bullet as historical record.
|
||||||
|
- **Replace the forward rule.** Should a future builder ever need NWN game data,
|
||||||
|
it **auto-detects** it (the NWN home — containing `modules/`, `hak/`, `tlk/` —
|
||||||
|
and the Steam install path are reliably discoverable), with an optional
|
||||||
|
explicit override for non-standard layouts. Auto-detect first, never a required
|
||||||
|
hand-set env. This keeps "clone → run → Just Work" intact even for a builder
|
||||||
|
that one day reads game data. None do today.
|
||||||
|
|
||||||
|
### F. Toolchain convergence — one Crucible, three modes
|
||||||
|
|
||||||
|
Parity needs everyone to run the *same* Crucible. Today three mechanisms
|
||||||
|
diverge: non-Nix uses the `crucible.sh` bootstrap download; sow-assets-manifest's
|
||||||
|
flake pins `crucible v0.2.5` via `fetchurl`+hash; sow-topdata's flake ships no
|
||||||
|
crucible at all (falls back to the bootstrap, with a needless `CRUCIBLE_TOKEN`).
|
||||||
|
The pin is scattered (a hardcoded flake version, none in another flake, an image
|
||||||
|
sha in `prod.yml`, a HEAD-floating CI binary build).
|
||||||
|
|
||||||
|
Converge on **one binary, one pin, three modes — all anonymous, all identical**:
|
||||||
|
|
||||||
|
1. **Non-Nix (local + CI fallback).** `crucible.sh` / `crucible.ps1` bootstrap
|
||||||
|
downloads the release binary anonymously. `CRUCIBLE_TOKEN` is dropped from
|
||||||
|
consumer CI (R1).
|
||||||
|
2. **Nix (local + CI optimized).** Every consumer flake provides `crucible` in
|
||||||
|
its devshell from a **sow-tools flake input**, pinned by that repo's
|
||||||
|
`flake.lock`. Idiomatic, reproducible, no manual hashes; bump with
|
||||||
|
`nix flake update`. This replaces sow-assets-manifest's manual `fetchurl`
|
||||||
|
pin and **adds crucible to sow-topdata's devshell** (today it has none).
|
||||||
|
3. **CI** runs the *same* `crucible <build>` as local, inside the nix devshell
|
||||||
|
(binary-cache fast). No build container, no token, no CI-only pre-steps.
|
||||||
|
|
||||||
|
**Pin source of truth = `flake.lock` per repo**, bumped deliberately in a PR. We
|
||||||
|
**reject floating "latest"**: it would let local drift from CI between releases
|
||||||
|
and break reproducibility. The non-Nix bootstrap, which resolves "latest" by
|
||||||
|
default, is the convenience path; reproducible/parity-critical builds pin (nix
|
||||||
|
devshell, or `CRUCIBLE_*` pin env for the bootstrap).
|
||||||
|
|
||||||
|
### G. The crucible container is deployment-only
|
||||||
|
|
||||||
|
`sow-tools` builds and publishes `registry.westgate.pw/deployment/crucible:<sha>`
|
||||||
|
on `v*` tags (`nix build .#image` → `skopeo`). It exists for **deployment** —
|
||||||
|
`prod.yml` pins it so tools travel with the runtime host (`nwn.enable`), and it
|
||||||
|
is a **disabled placeholder** until the NWN stack lands. No build/CI job consumes
|
||||||
|
it, and the host-mode runners have no container runtime to run it. It is **not a
|
||||||
|
build tool.**
|
||||||
|
|
||||||
|
- Keep the image build (cheap, `v*`-gated) for its deployment purpose.
|
||||||
|
- **Fix `consumer-contract.md`:** remove the crossed-out "run CI inside the
|
||||||
|
pinned image" guidance and its OPERATOR NOTE; state plainly that builds use the
|
||||||
|
binary (bootstrap or nix devshell) and the image is deployment-only. This kills
|
||||||
|
the "why do we build a container we don't use?" confusion.
|
||||||
|
|
||||||
|
### H. Autogen lock identity — model-anchored, immutable
|
||||||
|
|
||||||
|
**General lock contract (hard rule, all datasets).** A dataset lock maps a stable
|
||||||
|
key to a row id. The build MUST: reuse the existing id for a key that still
|
||||||
|
exists; allocate the first free id (lexicographic 0→1→2…) for a new key; and free
|
||||||
|
an id only when its key is gone everywhere. An id, once assigned to a live key,
|
||||||
|
never changes. (Already implemented for the accessory consumer via
|
||||||
|
`historicalLockData` reuse + `nextAvailableAutogenID`; this section makes it the
|
||||||
|
written rule and fixes the autogen-specific anchor below.)
|
||||||
|
|
||||||
|
**The autogen problem.** Autogen rows describe models that do **not** exist in
|
||||||
|
`sow-topdata/data/` — they live in the asset `vfxs.yml` manifest. Today the lock
|
||||||
|
key is the *config-derived presentation key*:
|
||||||
|
`KeyFormat = {dataset}:{group}{delimiter}{category_segment}{stem}`. Group token,
|
||||||
|
category, delimiter, case, and prefix-stripping all come from `nwn-tool.yaml`, so
|
||||||
|
a **config-only** change (`key_format`, `delimiter`, `case`, a group rename,
|
||||||
|
`category_from`, `strip_model_prefixes`) changes the key → the lock no longer
|
||||||
|
matches → a new id is allocated → **every row shifts**. That is a config edit
|
||||||
|
silently reshuffling ids for models that never changed.
|
||||||
|
|
||||||
|
**Fix — anchor the lock identity on the dataset + model, not the presentation.**
|
||||||
|
The lock key for an autogen row becomes `{dataset}:{model_source_path}` — the
|
||||||
|
stable `dataset` namespace prefix (kept as the 2da/namespace linker developers use
|
||||||
|
instead of raw ids) plus the **model source path** from `vfxs.yml`, e.g.
|
||||||
|
`visualeffects:head_accessories/hat/hfx_bandana.mdl` (leading `vfxs/` stripped —
|
||||||
|
the path is the value already on `entry.Source`). Dropped from the key are only
|
||||||
|
the config-derived parts (group token, `category_segment`, `delimiter`, case,
|
||||||
|
prefix-stripping). The row's `key`/`label` *columns* still derive from config via
|
||||||
|
`KeyFormat`/`LabelFormat` — only the **lock identity** is decoupled and
|
||||||
|
model-anchored. Consequences:
|
||||||
|
|
||||||
|
- A config-only change keeps every id (the path is unchanged).
|
||||||
|
- A model re-export (same path, new `sha256`) keeps its id — a content change is
|
||||||
|
not an identity change. (Audit trail comes free from `git diff` on the committed
|
||||||
|
`vfxs.yml`, which carries `path` + `sha256`; the lock stays a flat `string→int`
|
||||||
|
map — no schema change, no fork of the shared lock format. A future `id`+`sha`
|
||||||
|
object form is a clean later add if observability ever needs it, without
|
||||||
|
changing id semantics.)
|
||||||
|
- A renamed/moved/removed model is a genuinely different (or absent) identity.
|
||||||
|
|
||||||
|
**One-time cutover remap.** On the first build under `{dataset}:{path}` keys, for
|
||||||
|
each current model compute its *old* config-key (with the current policy) and its
|
||||||
|
new `{dataset}:{path}` key; if the new key is absent but the old key carries an id
|
||||||
|
in the lock, adopt that id under the new key. So ids carry over with zero shift on
|
||||||
|
cutover; the now-unreferenced old keys fall stale and are pruned (below).
|
||||||
|
|
||||||
|
**Stale pruning.** A lock key absent from the resolved entry set is dropped and
|
||||||
|
its id freed for first-free reuse — but **only on a successful, non-empty
|
||||||
|
resolution**. The fail-open paths (section A) contribute no entries and skip the
|
||||||
|
augmentor entirely, so an unreachable/missing manifest can never mass-prune the
|
||||||
|
lock. This realizes the general "free a fully-stale id" rule without risking the
|
||||||
|
lock on a transient source outage.
|
||||||
|
|
||||||
|
**Testing.** Go tests in `internal/topdata`: config-only change (rename a group
|
||||||
|
token, change `key_format`/`delimiter`/`case`) → ids unchanged; model re-export
|
||||||
|
(same path, different sha) → id unchanged; model rename/remove → old id freed and
|
||||||
|
reusable; cutover remap → existing ids preserved across the key-scheme switch;
|
||||||
|
fail-open → no pruning, lock untouched.
|
||||||
|
|
||||||
|
## Determinism vs. channel mutability (decision)
|
||||||
|
|
||||||
|
The committed topdata commit pins everything **except** the asset channel
|
||||||
|
pointer. `current` resolves through `channels.json`, which is intentionally
|
||||||
|
mutable, so "reproducible" here means **same commit + same channel state →
|
||||||
|
identical bytes**, not "this commit is frozen forever." This is deliberate: the
|
||||||
|
accessory rows must track the asset HAKs shipped beside `sow_top.hak`, which is
|
||||||
|
exactly what the channel pointer expresses. A given asset release **tag** is
|
||||||
|
immutable; the pointer moving is a real input change, and `lock.json` preserves
|
||||||
|
accessory IDs across it (no reshuffle). Builds against an explicit tag
|
||||||
|
(`SOW_TOPDATA_ASSET_CHANNEL` set, or a `v*` ref) are fully pinned.
|
||||||
|
|
||||||
|
This satisfies R1–R5: no secret, no config, deterministic for a fixed input
|
||||||
|
set, on any system.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
- **Go unit tests (`sow-tools/internal/topdata`)** for the `cdn_channel`
|
||||||
|
resolution: offline-override path; unreachable channels.json → fail open;
|
||||||
|
channel absent → fail open; vfxs.yml 404 + no haks.json → fail open; vfxs.yml
|
||||||
|
404 + haks.json present → hard fail (asserts message); malformed vfxs.yml →
|
||||||
|
hard fail; happy path → correct filtered/sorted entries (mdl-only, 4-groups-
|
||||||
|
only, `vfxs/` stripped, group/subgroup/stem). Mirrors the deleted bash
|
||||||
|
contract cases.
|
||||||
|
- **Go config validation test** for the new `source` block.
|
||||||
|
- **CI parity guard** (section D) in the topdata build workflow.
|
||||||
|
- Existing accessory augmentor / lock-preserve tests unchanged.
|
||||||
|
|
||||||
|
## Rollout
|
||||||
|
|
||||||
|
1. sow-tools: add `AutogenSourceConfig`, the `cdn_channel` resolution branch,
|
||||||
|
LFS materialization, the model-anchored lock identity + cutover remap + stale
|
||||||
|
pruning (section H), the parity-guard test; update `consumer-contract.md`
|
||||||
|
(add the parity contract, remove the `NWN_ROOT` rule per section E, replace the
|
||||||
|
image guidance with deployment-only per section G).
|
||||||
|
2. Release Crucible (assets already anon-downloadable — R1).
|
||||||
|
3. Toolchain convergence (section F): expose `crucible` from a sow-tools flake
|
||||||
|
input in each consumer flake — add it to sow-topdata's devshell, switch
|
||||||
|
sow-assets-manifest off its manual `fetchurl` pin; drop `CRUCIBLE_TOKEN` from
|
||||||
|
consumer CI; CI builds inside the nix devshell.
|
||||||
|
4. sow-topdata: swap `manifest_file:` → `source:` in `nwn-tool.yaml`; delete the
|
||||||
|
resolver script + bash contract test; thin `build-topdata.sh` (validate →
|
||||||
|
resolve crucible → build); pin Crucible via `flake.lock`.
|
||||||
|
5. Verify: bare clone → `crucible topdata build-topdata` → 637 accessory rows,
|
||||||
|
no env, no key; and the same command inside `nix develop`.
|
||||||
|
|
||||||
|
## Follow-ups (not this work)
|
||||||
|
|
||||||
|
- Parts `2da` / `cachedmodels` reinstatement (separate spec) can reuse the same
|
||||||
|
`cdn_channel` source type.
|
||||||
|
- Roll the thinned-wrapper pattern review across `sow-module` / `sow-codebase`.
|
||||||
@@ -0,0 +1,147 @@
|
|||||||
|
# Crucible Command Surface Cleanup Design
|
||||||
|
|
||||||
|
**Date:** 2026-06-25
|
||||||
|
**Status:** Approved
|
||||||
|
**Scope:** `sow-tools`, with the two active `sow-assets-manifest` call sites
|
||||||
|
updated in lockstep
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
Crucible exposes migrated `nwn-tool` implementation names directly. Several
|
||||||
|
public commands repeat their builder name (`topdata build-topdata`,
|
||||||
|
`wiki build-wiki`), the interactive menu repeats builder-level descriptions
|
||||||
|
instead of explaining each action, and selected commands only prompt for
|
||||||
|
unexplained "extra args".
|
||||||
|
|
||||||
|
The registry also exposes a migrated music-conversion subsystem that is not part
|
||||||
|
of the current project workflow. Live HAK builds explicitly bypass it with
|
||||||
|
`--skip-music`; authored `.bmu` files are already packed as ordinary assets.
|
||||||
|
|
||||||
|
## Decisions
|
||||||
|
|
||||||
|
### Canonical command names
|
||||||
|
|
||||||
|
The visible command surface is:
|
||||||
|
|
||||||
|
| Builder | Visible commands |
|
||||||
|
| --- | --- |
|
||||||
|
| `hak` | `build`, `manifest` |
|
||||||
|
| `module` | `build`, `extract`, `validate`, `compare`, `manifest` |
|
||||||
|
| `topdata` | `validate`, `build`, `package`, `compare`, `convert` |
|
||||||
|
| `wiki` | `build`, `deploy` |
|
||||||
|
|
||||||
|
The dispatcher translates these names to the existing `internal/app`
|
||||||
|
implementation commands. The app implementation names do not need to be
|
||||||
|
renamed.
|
||||||
|
|
||||||
|
### Hidden compatibility aliases
|
||||||
|
|
||||||
|
Existing long names remain callable indefinitely but are omitted from the
|
||||||
|
interactive menu and routine builder help:
|
||||||
|
|
||||||
|
- `hak build-haks`
|
||||||
|
- `hak apply-hak-manifest`
|
||||||
|
- `module build-module`
|
||||||
|
- `module apply-hak-manifest`
|
||||||
|
- `topdata validate-topdata`
|
||||||
|
- `topdata build-topdata`
|
||||||
|
- `topdata build-top-package`
|
||||||
|
- `topdata compare-topdata`
|
||||||
|
- `topdata convert-topdata`
|
||||||
|
- `wiki build-wiki`
|
||||||
|
- `wiki deploy-wiki`
|
||||||
|
|
||||||
|
Aliases preserve their existing implementation target. In particular,
|
||||||
|
`module build-module` continues to run the module-only implementation; it must
|
||||||
|
not silently become the broader `module build`.
|
||||||
|
|
||||||
|
### Registry model
|
||||||
|
|
||||||
|
`internal/dispatch.Registry` remains the single source of truth. Each wired
|
||||||
|
builder owns command records containing:
|
||||||
|
|
||||||
|
- visible command name;
|
||||||
|
- action-oriented summary;
|
||||||
|
- underlying `internal/app` command;
|
||||||
|
- command usage and option guidance;
|
||||||
|
- hidden compatibility aliases, where applicable.
|
||||||
|
|
||||||
|
Execution, menu items, and builder help are generated from these records.
|
||||||
|
Unknown commands continue to fail with exit `64`.
|
||||||
|
|
||||||
|
### Interactive behavior
|
||||||
|
|
||||||
|
The main menu:
|
||||||
|
|
||||||
|
- lists only visible commands;
|
||||||
|
- aligns the description column from the longest visible label;
|
||||||
|
- shows a distinct description for every command.
|
||||||
|
|
||||||
|
After selection, it prints the selected command's usage and available options,
|
||||||
|
then prompts:
|
||||||
|
|
||||||
|
```text
|
||||||
|
arguments (press Enter to use defaults):
|
||||||
|
```
|
||||||
|
|
||||||
|
Arguments continue to use shell-like whitespace splitting, matching current
|
||||||
|
behavior. Quoted argument parsing is out of scope.
|
||||||
|
|
||||||
|
`crucible <builder> <command> --help` prints the same command-specific guidance,
|
||||||
|
returns success, and performs no project loading or build work.
|
||||||
|
|
||||||
|
### Remove the unused music pipeline
|
||||||
|
|
||||||
|
The music conversion subsystem is deleted rather than hidden:
|
||||||
|
|
||||||
|
- remove the `music` app and dispatcher commands;
|
||||||
|
- remove `internal/music` and `internal/pipeline/music.go`;
|
||||||
|
- remove music config, effective config, validation, environment overrides, and
|
||||||
|
project accessors;
|
||||||
|
- remove HAK music preparation, conversion, credits, generated-manifest result
|
||||||
|
fields, `--skip-music`, and `--music-dataset`;
|
||||||
|
- remove ffmpeg/ffprobe runtime and development dependencies;
|
||||||
|
- remove music-pipeline documentation and wrapper wording.
|
||||||
|
|
||||||
|
`.bmu` remains a supported ordinary asset extension and continues through the
|
||||||
|
same content-addressed HAK packing path as other authored files. Actual game
|
||||||
|
content, category names, or prose that happens to use the word "music" is not
|
||||||
|
part of this deletion.
|
||||||
|
|
||||||
|
The default asset-extension list drops conversion-only `.mp3` and `.ogg`.
|
||||||
|
`.wav` remains because it is an NWN resource type independent of the deleted
|
||||||
|
conversion pipeline.
|
||||||
|
|
||||||
|
Because `sow-assets-manifest` currently passes `--skip-music`, its
|
||||||
|
`scripts/pack-haks.sh` and `scripts/build-local-haks.sh` call sites are updated
|
||||||
|
in the same change before verification.
|
||||||
|
|
||||||
|
## Error handling
|
||||||
|
|
||||||
|
- Canonical and hidden alias lookup is deterministic within one builder.
|
||||||
|
- Registry tests reject duplicate visible names or aliases.
|
||||||
|
- Command help never delegates to `internal/app`.
|
||||||
|
- Removed music commands and flags fail as unknown usage.
|
||||||
|
- Existing project files containing a top-level `music:` configuration become
|
||||||
|
invalid under strict YAML decoding. No active consumer configuration contains
|
||||||
|
that field.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
Tests assert public contracts rather than exact incidental formatting:
|
||||||
|
|
||||||
|
- visible command names and hidden alias translation;
|
||||||
|
- preservation of the distinct `module build-module` target;
|
||||||
|
- hidden aliases absent from menu and normal help;
|
||||||
|
- command-specific help exits successfully without project setup;
|
||||||
|
- menu columns align and selected-command guidance is displayed;
|
||||||
|
- `.bmu` remains accepted as an asset;
|
||||||
|
- `.mp3`/`.ogg` are no longer default packable assets;
|
||||||
|
- HAK builds no longer expose or execute music preparation;
|
||||||
|
- ffmpeg is absent from package/image definitions;
|
||||||
|
- both consumer scripts no longer pass `--skip-music`.
|
||||||
|
|
||||||
|
Final verification is `nix develop --command make check`, `make build`,
|
||||||
|
`make smoke`, focused consumer tests, and repository-wide searches for deleted
|
||||||
|
pipeline identifiers.
|
||||||
|
|
||||||
@@ -0,0 +1,269 @@
|
|||||||
|
# Topdata JSON Validation Tightening Design
|
||||||
|
|
||||||
|
**Date:** 2026-06-25
|
||||||
|
**Status:** Implemented and reviewed
|
||||||
|
**Scope:** `sow-tools`
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
Topdata JSON validation checks many required properties and value types, but it
|
||||||
|
often ignores unknown structural properties. This allows plausible typos to
|
||||||
|
pass validation and then be ignored by the builder.
|
||||||
|
|
||||||
|
The immediate failure occurred in
|
||||||
|
`topdata/data/classes/feats/global.json`: an injection used
|
||||||
|
`when_present` even though the implemented condition is `require_present`.
|
||||||
|
Validation succeeded, the builder ignored `when_present`, and the injection was
|
||||||
|
therefore applied without the intended condition.
|
||||||
|
|
||||||
|
This is a fail-open authoring contract. A misspelled build-control property must
|
||||||
|
not silently change generated game data.
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
- Reject unsupported structural properties in canonical topdata JSON files.
|
||||||
|
- Add an `any_present` global injection condition.
|
||||||
|
- Keep standalone validation and direct native builds consistent.
|
||||||
|
- Preserve dataset-specific row flexibility.
|
||||||
|
- Tighten only established canonical formats; do not turn this work into a
|
||||||
|
repository-wide schema rewrite.
|
||||||
|
- Produce diagnostics that identify the file, object, and unsupported property.
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
- No schema migration or formatting change to authored topdata.
|
||||||
|
- No generic JSON Schema framework or new validation dependency.
|
||||||
|
- No blanket closure of every specialized JSON dialect.
|
||||||
|
- No change to generated feat family, registry, spellbook, TLK, or other
|
||||||
|
specialized formats unless they use the canonical containers covered here.
|
||||||
|
- No restriction of row values beyond existing column, metadata, inheritance,
|
||||||
|
reference, TLK, and authoring-sugar contracts.
|
||||||
|
- No compatibility alias for `when_present`; it is invalid syntax.
|
||||||
|
|
||||||
|
## Design
|
||||||
|
|
||||||
|
### Validation boundary
|
||||||
|
|
||||||
|
Structural objects use closed property sets. Dataset payload objects remain
|
||||||
|
dataset-aware.
|
||||||
|
|
||||||
|
Closed structural objects include:
|
||||||
|
|
||||||
|
| Object | Supported properties |
|
||||||
|
| --- | --- |
|
||||||
|
| Base or plain rows-file root | `output`, `key`, `columns`, `rows` |
|
||||||
|
| Canonical module root | `output`, `key`, `columns`, `entries`, `overrides`, `rows` |
|
||||||
|
| `global.json` root | `columns`, `entries`, `overrides`, `defaults`, `position`, `injections` |
|
||||||
|
| Global injection | `row`, `require_present`, `any_present`, `unless_present` |
|
||||||
|
| Global condition | `field`, `id` |
|
||||||
|
| Global default rule | `match`, `values` |
|
||||||
|
| Global default match object | `source` |
|
||||||
|
| Global default format object | `format` |
|
||||||
|
|
||||||
|
The allowed root set is selected from the file's existing canonical context;
|
||||||
|
properties from unrelated formats are not accepted merely because another
|
||||||
|
topdata file type supports them.
|
||||||
|
|
||||||
|
When a canonical root contains more than one supported container, each present
|
||||||
|
container must be validated. For example, a module containing both `entries`
|
||||||
|
and `overrides` must not stop validation after recognizing `entries`.
|
||||||
|
|
||||||
|
The following remain payload objects rather than global allowlists:
|
||||||
|
|
||||||
|
- objects inside `rows`;
|
||||||
|
- values inside `entries`;
|
||||||
|
- row-like objects inside `overrides`;
|
||||||
|
- an injection's `row`;
|
||||||
|
- a default rule's `values`.
|
||||||
|
|
||||||
|
Those objects continue to accept declared dataset columns plus documented row
|
||||||
|
control and metadata fields. The builder already rejects unknown row and entry
|
||||||
|
columns when it canonicalizes a dataset. This work must reuse or align with
|
||||||
|
that knowledge where practical, but must not invent a single fixed row schema
|
||||||
|
across all datasets.
|
||||||
|
|
||||||
|
### Moderate strictness
|
||||||
|
|
||||||
|
This change closes only canonical containers whose grammar is already
|
||||||
|
established by the parser and current authored data.
|
||||||
|
|
||||||
|
Before adding a closed root check, implementation must inventory active
|
||||||
|
`sow-topdata` files and existing `sow-tools` fixtures for legitimate property
|
||||||
|
combinations. A property used by an established canonical format must be added
|
||||||
|
to that format's explicit set rather than removed from authored data merely to
|
||||||
|
satisfy the validator.
|
||||||
|
|
||||||
|
Specialized parsers remain responsible for their own object shapes. Extending
|
||||||
|
closed-property checks to those formats is separate work.
|
||||||
|
|
||||||
|
### Global injection conditions
|
||||||
|
|
||||||
|
Global injections support three condition groups:
|
||||||
|
|
||||||
|
- `require_present`: every listed condition must match a current row.
|
||||||
|
- `any_present`: at least one listed condition must match a current row.
|
||||||
|
- `unless_present`: no listed condition may match a current row.
|
||||||
|
|
||||||
|
When more than one group is present, every group must pass. Conditions continue
|
||||||
|
to use the existing shape:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"field": "FeatIndex",
|
||||||
|
"id": "feat:example"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"row": {
|
||||||
|
"FeatIndex": {
|
||||||
|
"id": "feat:example"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"require_present": [
|
||||||
|
{
|
||||||
|
"field": "FeatIndex",
|
||||||
|
"id": "feat:base_requirement"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"any_present": [
|
||||||
|
{
|
||||||
|
"field": "FeatIndex",
|
||||||
|
"id": "masterfeats:metamagic"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"field": "FeatIndex",
|
||||||
|
"id": "masterfeats:combat"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"unless_present": [
|
||||||
|
{
|
||||||
|
"field": "FeatIndex",
|
||||||
|
"id": "feat:example"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
This injection applies only when the required feat exists, either listed
|
||||||
|
masterfeat exists, and the injected feat does not already exist.
|
||||||
|
|
||||||
|
`any_present` must contain at least one condition. An empty list is an
|
||||||
|
authoring error because it cannot express a useful successful condition.
|
||||||
|
Existing empty-list behavior for `require_present` and `unless_present` is not
|
||||||
|
changed by this work.
|
||||||
|
|
||||||
|
Conditions are evaluated against the same current-row view and in the same
|
||||||
|
injection order used today. This preserves the existing behavior where an
|
||||||
|
earlier injection can affect a later injection's conditions.
|
||||||
|
|
||||||
|
### Validation and build consistency
|
||||||
|
|
||||||
|
`ValidateProject` must report unsupported structural properties before a build.
|
||||||
|
The native builder must also fail closed when invoked directly with the same
|
||||||
|
invalid control syntax.
|
||||||
|
|
||||||
|
The implementation should centralize:
|
||||||
|
|
||||||
|
- deterministic unsupported-property checks;
|
||||||
|
- global condition parsing and validation;
|
||||||
|
- condition group evaluation.
|
||||||
|
|
||||||
|
Validation may collect multiple diagnostics while building returns the first
|
||||||
|
blocking error. Both paths must accept and reject the same property names and
|
||||||
|
condition shapes.
|
||||||
|
|
||||||
|
### Diagnostics
|
||||||
|
|
||||||
|
Diagnostics must name the containing object and unsupported property. Property
|
||||||
|
lists must be deterministic.
|
||||||
|
|
||||||
|
For the original error, an acceptable diagnostic is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
global injection 10 contains unsupported key "when_present";
|
||||||
|
supported keys are row, require_present, any_present, and unless_present
|
||||||
|
```
|
||||||
|
|
||||||
|
Equivalent concise wording is acceptable if tests assert the stable semantic
|
||||||
|
parts rather than the entire sentence.
|
||||||
|
|
||||||
|
## Compatibility
|
||||||
|
|
||||||
|
Current canonical `sow-topdata` roots fit the property sets above. The
|
||||||
|
implementation must rerun the inventory immediately before tightening checks
|
||||||
|
and run validation against the active `sow-topdata` checkout.
|
||||||
|
|
||||||
|
Unknown properties are not retained as compatibility behavior. They currently
|
||||||
|
have no defined effect, and silently accepting them is the bug being fixed.
|
||||||
|
Existing documented properties and dataset-specific row fields remain valid.
|
||||||
|
|
||||||
|
`require_present` and `unless_present` retain their current semantics.
|
||||||
|
`any_present` is additive.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
Tests must follow red-green development and cover validation and direct-build
|
||||||
|
paths.
|
||||||
|
|
||||||
|
### Unsupported properties
|
||||||
|
|
||||||
|
- Reject an unknown `global.json` root property.
|
||||||
|
- Reject `when_present` and other unknown injection properties.
|
||||||
|
- Reject an unknown condition property.
|
||||||
|
- Reject unknown properties in default rules, match objects, and format
|
||||||
|
objects.
|
||||||
|
- Reject unknown canonical base/plain rows-file root properties.
|
||||||
|
- Reject unknown canonical module root properties.
|
||||||
|
- Continue accepting legitimate combinations such as:
|
||||||
|
- a columns-only module;
|
||||||
|
- entries-only and overrides-only modules;
|
||||||
|
- a module containing both entries and overrides;
|
||||||
|
- current base and plain rows-file roots.
|
||||||
|
- Continue accepting dataset columns and documented row control fields inside
|
||||||
|
rows, entries, overrides, and injection rows.
|
||||||
|
|
||||||
|
### `any_present`
|
||||||
|
|
||||||
|
- Validation accepts a non-empty `any_present` condition list.
|
||||||
|
- Validation rejects a non-array or empty `any_present`.
|
||||||
|
- A build injects when the first alternative is present.
|
||||||
|
- A build injects when a later alternative is present.
|
||||||
|
- A build skips the injection when no alternative is present.
|
||||||
|
- `any_present` combines correctly with `require_present`.
|
||||||
|
- `any_present` combines correctly with `unless_present`.
|
||||||
|
- Existing `require_present` and `unless_present` behavior remains unchanged.
|
||||||
|
- Direct building rejects unsupported condition syntax even when project
|
||||||
|
validation was not called first.
|
||||||
|
|
||||||
|
### Integration
|
||||||
|
|
||||||
|
- Run focused `internal/topdata` tests.
|
||||||
|
- Run `nix develop --command make check`.
|
||||||
|
- Validate the active `sow-topdata` checkout with the changed Crucible.
|
||||||
|
- Build topdata far enough to exercise global injections and confirm no
|
||||||
|
generated files are tracked.
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
Update the class feat global injection contract to document `require_present`,
|
||||||
|
`any_present`, and `unless_present`, including their all/any/none semantics and
|
||||||
|
how multiple groups combine.
|
||||||
|
|
||||||
|
No broad validation guide is added unless implementation reveals an existing
|
||||||
|
operator document that would otherwise become stale.
|
||||||
|
|
||||||
|
## Acceptance criteria
|
||||||
|
|
||||||
|
- `when_present` in a global injection fails validation and direct building.
|
||||||
|
- `any_present` is valid only as a non-empty list of valid conditions.
|
||||||
|
- `any_present` applies an injection when one or more alternatives exist and
|
||||||
|
skips it when none exist.
|
||||||
|
- Unknown canonical structural properties fail with actionable diagnostics.
|
||||||
|
- Dataset-specific row fields remain valid when declared by the dataset.
|
||||||
|
- Existing active `sow-topdata` validates and builds after replacing invalid
|
||||||
|
syntax with the documented grammar.
|
||||||
|
- No specialized topdata format is tightened outside the stated scope.
|
||||||
@@ -0,0 +1,375 @@
|
|||||||
|
# Crucible Depot Core Design (Increment 1)
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Give Crucible real ownership of the content-addressed asset depot. Today
|
||||||
|
`crucible-depot` is a registered-but-unwired stub (fails closed, exit `70`), and
|
||||||
|
the actual depot logic lives as ~2000 lines of bash embedded in
|
||||||
|
`sow-assets-manifest/scripts` — a direct violation of the Crucible principle that
|
||||||
|
*artifact repos invoke Crucible through wrappers and never embed a toolkit*.
|
||||||
|
|
||||||
|
Increment 1 wires the depot **core** — blob backends plus `status` / `push` /
|
||||||
|
`verify` / `get` — into `crucible depot`, and cuts the manifest repo over to call
|
||||||
|
it directly. The bash depot layer is **retired, not wrapped**: one tool owns
|
||||||
|
depot byte-motion for both local workflows and CI/CD, with complete parity.
|
||||||
|
|
||||||
|
## Why now (root cause that triggered this)
|
||||||
|
|
||||||
|
Release run 222 (`v0.1.5`) failed in 45s at the "Warm depot cache" step. PR #27's
|
||||||
|
mdl-name normalization recompiled the corpus, producing **5903 net-new sha256s**
|
||||||
|
in `assets/*.yml`. Sampling proved **15/15 net-new blobs return HTTP 404 on the
|
||||||
|
CDN** while **10/10 old blobs return 200** — the CDN works; the new content was
|
||||||
|
never uploaded. The blobs exist in the local depot (`workspace/depot`, 69,473
|
||||||
|
blobs) but never reached Bunny. `mirror.sh` then 404s and dies — the release is
|
||||||
|
correctly failing closed on a depot missing data.
|
||||||
|
|
||||||
|
Root cause of the *missing upload*: `import.sh`'s stat-cache
|
||||||
|
(`.cache/import/<cat>.tsv`) records `(path,size,mtime,sha)` after **any** import
|
||||||
|
and excludes "unchanged" files from the only upload call (`depot_put_many`) —
|
||||||
|
**without tracking which backend those bytes actually landed in**. A prior
|
||||||
|
`local` import poisons the cache so a later `bunny` import silently skips the
|
||||||
|
upload. Combined with `depot_require_write` only normalizing `cdn`/unset→`bunny`
|
||||||
|
(never `local`→anything), it is very easy to leave the manifests referencing
|
||||||
|
blobs that exist only in the local depot.
|
||||||
|
|
||||||
|
The design principle that kills this bug class: **presence is always checked
|
||||||
|
against the real target backend, never a cache.**
|
||||||
|
|
||||||
|
### Update 2026-07-04 — interim bash fix landed + corrected drift measurement
|
||||||
|
|
||||||
|
The bug above was fixed in bash ahead of this rewrite (this spec's Increment 2
|
||||||
|
retires the stat-cache concept entirely, making the fix structural):
|
||||||
|
[`sow-assets-manifest#29`](https://git.westgate.pw/ShadowsOverWestgate/sow-assets-manifest/pulls/29)
|
||||||
|
keys the stat-cache dir by write backend + target
|
||||||
|
(`.cache/import/<target>/`, `scripts/import.sh`), so a `local` import
|
||||||
|
(`make haks`) can no longer mark a blob "done" in a cache the next `bunny` sync
|
||||||
|
trusts. A cross-backend regression test guards it. When Increment 1/2 delete the
|
||||||
|
bash depot layer, this becomes moot — but until then, **do not "simplify" the
|
||||||
|
stat-cache back to a single shared dir**; that reintroduces the exact orphaning
|
||||||
|
that broke run 222.
|
||||||
|
|
||||||
|
**Corrected drift numbers.** The "15/15 sampled net-new blobs 404" reading in the
|
||||||
|
section above is a *sampling artifact*, not the real drift. The CDN edge
|
||||||
|
(`cdn-a7f3k9.westgate.pw`) is IPv6-reachable but resets HTTP/2 from some hosts,
|
||||||
|
and it throttles concurrent HEADs with `428`/`000` — which produce **both**
|
||||||
|
false-404s and false-200s. A reliable sweep (force `curl -4`, 1-byte range GET
|
||||||
|
`-r 0-0` → expect `206`, low concurrency, then serial re-confirm any non-2xx)
|
||||||
|
shows the true missing set was **exactly 2 net-new blobs**, not ~5903:
|
||||||
|
|
||||||
|
- `item/_shared_aenea_weaponvfx/wblpl_fxshblk.mdl` (`0b1d3f…`)
|
||||||
|
- `item/_shared_aenea_weaponvfx/wxdbsc_fxshpur.mdl` (`4e8d4f…`)
|
||||||
|
|
||||||
|
A 600-blob random sample of the pre-existing corpus was 100% present. `mirror.sh`
|
||||||
|
still correctly failed the release — it dies on the first missing blob — but the
|
||||||
|
gap was two files a local `make haks` had primed in the shared cache, not a
|
||||||
|
wholesale upload failure. Both blobs were re-uploaded to Bunny from
|
||||||
|
`workspace/depot` (present in the local depot, sha-verified), unblocking the
|
||||||
|
`v0.1.5` re-run.
|
||||||
|
|
||||||
|
Takeaway for `crucible depot status`/`verify`: probe over IPv4, treat a single
|
||||||
|
concurrent-probe non-2xx as *unconfirmed* (re-check serially) before reporting a
|
||||||
|
blob missing, so the tool doesn't over-report drift the way an ad-hoc HEAD sweep
|
||||||
|
did here.
|
||||||
|
|
||||||
|
### Field notes 2026-07-04 (content-side session) — issues the migration must fix
|
||||||
|
|
||||||
|
Surfaced while verifying a content edit (`over/ravenloft_potm_blood`, 18 blobs) had
|
||||||
|
actually reached the CDN. All corroborate the design above; #1 is a new gap to close in
|
||||||
|
Increment 1.
|
||||||
|
|
||||||
|
1. **`push` must be non-interactive — no tty prompt (new gap).** Today the only write
|
||||||
|
path, `depot_require_write` (`lib.sh`), *prompts* for `BUNNY_STORAGE_PASSWORD` on a tty
|
||||||
|
when it is unset. Headless agents and CI-without-a-pre-exported-secret cannot answer
|
||||||
|
that prompt, so there is no way to push except with a human at a terminal — which is
|
||||||
|
exactly why the current operational workaround is "an agent/human manually pushes each
|
||||||
|
fixed asset." `crucible depot push` must take the write key **from env only**, fail
|
||||||
|
closed with a clear message when it is absent, and **never prompt**. The Backends
|
||||||
|
section lists the env names but does not state the no-prompt requirement — make it
|
||||||
|
explicit and add a test that a missing write key exits non-zero without reading stdin.
|
||||||
|
|
||||||
|
2. **`status --target cdn` is the everyday author check, not only the release gate.** The
|
||||||
|
spec frames `status` as the release pre-flight (correct), but the same command,
|
||||||
|
credential-free against `cdn`, is the "did my edit actually reach the CDN?" check
|
||||||
|
content authors need routinely. This session answered that question by hand-probing 18
|
||||||
|
sha256s with `curl` — precisely the toil `crucible depot status --manifests assets
|
||||||
|
--target cdn` should replace. Document it as the canonical author-facing "is my content
|
||||||
|
live?" command so people stop ad-hoc probing.
|
||||||
|
|
||||||
|
3. **Probe reliability confirmed again — bake it into the tests.** Reproduced the HEAD
|
||||||
|
hazard directly: a `curl -sI` HEAD sweep of the 18 blobs returned all-`200`; the
|
||||||
|
prescribed reliable method (`curl -4`, 1-byte range GET `-r 0-0` → `206`, serial) also
|
||||||
|
returned all-`206` with the fake-sha control at `404`. HEAD did not false-negative this
|
||||||
|
time, but it stays the wrong tool. Make the IPv4 range-GET probe a hard requirement and
|
||||||
|
add a test that fails if a HEAD-based existence check is introduced.
|
||||||
|
|
||||||
|
4. **Drift is recurring, not the single v0.1.5 incident.** Operationally, local→CDN sync is
|
||||||
|
treated as "basically broken": every `make haks` / local import can re-orphan blobs from
|
||||||
|
Bunny, so authors push by hand after edits. That raises Increment 1's priority and argues
|
||||||
|
for making `crucible depot push` a standard post-edit step (or an on-edit CI hook), not
|
||||||
|
only a release-time gate.
|
||||||
|
|
||||||
|
## Ownership principle (from the maintainer)
|
||||||
|
|
||||||
|
- Crucible **owns** the depot tools. No wrapper scripts preserved for their own
|
||||||
|
sake.
|
||||||
|
- Every consumer repo already has `crucible` in its flake; that is the only
|
||||||
|
dependency. Consumers call `crucible depot …` directly (Makefile targets and
|
||||||
|
CI workflow steps alike).
|
||||||
|
- **Complete parity**: the same `crucible depot` commands serve local workflows
|
||||||
|
and CI/CD. No divergent code paths, no "CI-only" or "local-only" logic.
|
||||||
|
- Total clean-out over legacy preservation: bash depot code is **deleted** as its
|
||||||
|
Go replacement lands, not kept as a fallback.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This spec covers **Increment 1** only. Roadmap for context:
|
||||||
|
|
||||||
|
- **Increment 1 (this spec):** `crucible depot {status, push, verify, get, pull}`
|
||||||
|
+ blob backends (`local` / `cdn` / `bunny`). Cut over `mirror`/verify/release
|
||||||
|
gate. Delete the `lib.sh` depot functions that lose their last caller (see
|
||||||
|
Cutover for what actually remains until Increments 2–3).
|
||||||
|
- **Increment 2:** `crucible depot import` / `sync` (edit-tree → manifest+depot),
|
||||||
|
replacing `import.sh` / `sync-assets.sh`. Retire the stat-cache concept.
|
||||||
|
- **Increment 3:** mdl integrity checks, `mirror`/`pull` ergonomics, `prune`,
|
||||||
|
`gc`, `export`. Delete remaining bash.
|
||||||
|
|
||||||
|
Each increment ships independently and deletes the bash it replaces.
|
||||||
|
|
||||||
|
## Command surface (Increment 1)
|
||||||
|
|
||||||
|
```
|
||||||
|
crucible depot status [--manifests DIR] [--source LOCAL_DEPOT] --target bunny|cdn|local
|
||||||
|
crucible depot push [--manifests DIR] --source LOCAL_DEPOT --target bunny
|
||||||
|
crucible depot verify [--manifests DIR] --target bunny|cdn [--sample N]
|
||||||
|
crucible depot get <sha> <dest> --target cdn|bunny|local
|
||||||
|
crucible depot pull [--manifests DIR] --dest DIR --target cdn|bunny
|
||||||
|
```
|
||||||
|
|
||||||
|
- `pull` is the bulk fetch that replaces `mirror.sh`: every referenced sha into
|
||||||
|
`--dest/sha256/ab/cd/<sha>`, **incremental** (skip blobs already present with a
|
||||||
|
matching hash — the release warm step re-runs against a persistent
|
||||||
|
`/var/cache` dir holding ~69k blobs; a non-incremental pull would re-download
|
||||||
|
the corpus every release), sha re-verified on download, parallel with the same
|
||||||
|
probe/transfer bounds as `status`. Without `pull`, Increment 1 cannot delete
|
||||||
|
`mirror.sh` (single-blob `get` is not a replacement).
|
||||||
|
|
||||||
|
- `--manifests` defaults to `assets/` (the repo's manifest dir). Confirmed
|
||||||
|
decision: `crucible depot` parses `assets/*.yml` directly (same coupling model
|
||||||
|
as `crucible-hak` reading hak manifests); the caller does not pre-extract sha
|
||||||
|
lists.
|
||||||
|
- `--source` is the local content-addressed store used as the byte source for
|
||||||
|
`push` (e.g. `workspace/depot`).
|
||||||
|
- Config/auth is env-first with flags as local aliases (see Backends).
|
||||||
|
|
||||||
|
## Drift model (the fix)
|
||||||
|
|
||||||
|
The manifest is the source of truth for **what** must exist; the target backend
|
||||||
|
is the source of truth for **what does** exist; `push` reconciles them. There is
|
||||||
|
no cache of "what I uploaded."
|
||||||
|
|
||||||
|
- **`status`** — collect every `sha256` referenced by `--manifests/*.yml`, run
|
||||||
|
one parallel existence sweep against `--target`, report
|
||||||
|
`{referenced, present, missing, unconfirmed}`. Non-zero exit if any referenced
|
||||||
|
blob is missing. This is the release pre-flight gate.
|
||||||
|
|
||||||
|
**`unconfirmed` is a distinct state, not `missing`.** Any non-2xx from the
|
||||||
|
parallel sweep is re-probed serially with backoff (the bash
|
||||||
|
`depot_confirm_absent` behavior). If the serial re-probe budget is exhausted
|
||||||
|
(bash caps at 200; keep a cap, `DEPOT_CONFIRM_MAX`), the remainder is reported
|
||||||
|
as `unconfirmed` — **not** collapsed into `missing`. For `push` the
|
||||||
|
distinction is harmless (unconfirmed → treat as absent → re-upload; wasteful
|
||||||
|
but idempotent and safe). For `status` as the release gate it is critical: a
|
||||||
|
throttled CDN edge must produce "N unconfirmed, retry" (distinct exit code,
|
||||||
|
proposed `2`), never a false "thousands missing" hard-fail — exactly the
|
||||||
|
over-reporting the corrected v0.1.5 drift measurement documented above.
|
||||||
|
|
||||||
|
**Target authority split:** `bunny` probes storage — the origin, authoritative,
|
||||||
|
needs the read key; this is what the release gate uses. `cdn` probes the edge —
|
||||||
|
credential-free, what players actually fetch, the author-facing "is my content
|
||||||
|
live?" check; the edge cache can briefly lag storage (bash carries `cdn_purge`
|
||||||
|
for this reason). A blob `present` on bunny but `missing` on cdn is
|
||||||
|
propagation lag, not drift.
|
||||||
|
- **`push`** — run `status`, then for each missing sha read the bytes from
|
||||||
|
`--source` and upload to `--target`. Stateless and idempotent; re-running does
|
||||||
|
nothing once clean. **Cannot silently skip an upload** — "should I upload this?"
|
||||||
|
is answered by probing the target, not by trusting a local record.
|
||||||
|
- **`verify`** — existence sweep plus a bounded random sample downloaded and
|
||||||
|
re-hashed (the current `--full` tag-build check), decoupled from packing.
|
||||||
|
- **`get`** — single-blob fetch with sha re-verify (backfill / debugging).
|
||||||
|
|
||||||
|
## Backends & config
|
||||||
|
|
||||||
|
Faithful port of `lib.sh` semantics — this is a re-home, not a redesign of the
|
||||||
|
wire protocol.
|
||||||
|
|
||||||
|
| Backend | Read | Write | Role |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `local` | filesystem `sha256/ab/cd/<sha>` | filesystem | dev/CI; byte source for `push` |
|
||||||
|
| `cdn` | CDN GET + sha re-verify | ✗ read-only, fails closed | credential-free public reads |
|
||||||
|
| `bunny` | CDN GET, storage fallback | Storage `PUT` | the only writable remote |
|
||||||
|
|
||||||
|
Layout: `blob_key(sha) = sha256/<sha[0:2]>/<sha[2:4]>/<sha>`.
|
||||||
|
|
||||||
|
Config (env names unchanged so existing CI secrets and `release.yml` keep working
|
||||||
|
untouched; flags are local aliases):
|
||||||
|
|
||||||
|
- `DEPOT_CDN_BASE` / `BUNNY_CDN_BASE` — CDN read base.
|
||||||
|
- `BUNNY_STORAGE_HOST`, `BUNNY_STORAGE_ZONE` — Bunny storage endpoint.
|
||||||
|
- **Read/write key split (preserved):** `BUNNY_STORAGE_READ_PASSWORD` (falls back
|
||||||
|
to `BUNNY_STORAGE_PASSWORD`) for probes/reads; `BUNNY_STORAGE_PASSWORD` for
|
||||||
|
`PUT`. Note for the `release.yml` cutover: `status --target bunny` as the gate
|
||||||
|
step needs at least the read key exported in that step's env.
|
||||||
|
- **No prompting, read path included.** The no-prompt rule from field note 1
|
||||||
|
applies to every credentialed operation, not just `push`: `status`/`verify`/
|
||||||
|
`get` against `bunny` with no read key fail closed with a clear message —
|
||||||
|
never a hang, never a tty read.
|
||||||
|
- `BUNNY_STORAGE_HOST` stays **env-required, no baked default** — the bash never
|
||||||
|
defaulted it (`depot_require_write` defaults only the zone
|
||||||
|
`sow-assets-depot` and the CDN base); resolving the earlier open question.
|
||||||
|
- Concurrency: `DEPOT_PROBE_JOBS` (read probes), `DEPOT_JOBS` (uploads);
|
||||||
|
`DEPOT_CONNECT_TIMEOUT`, `DEPOT_PROBE_MAX_TIME` per-transfer bounds.
|
||||||
|
|
||||||
|
Invariants carried over exactly:
|
||||||
|
|
||||||
|
- **Existence probe** is an IPv4 1-byte range request (no body download) — for
|
||||||
|
**all** backends. This is a deliberate deviation from `lib.sh`, whose `cdn`
|
||||||
|
backend probes with HEAD (`_depot_cdn_probe_code`); the "faithful port"
|
||||||
|
clause above does not extend to the probe method. HEAD is banned (see field
|
||||||
|
note 3 and the test requirement there).
|
||||||
|
- **Upload** is `PUT` with `Checksum: <UPPER-sha>` so Bunny rejects corrupt
|
||||||
|
writes server-side; each sha uploaded at most once per run.
|
||||||
|
- **Fail-safe:** a blob with no confirmed-present reply is treated as absent and
|
||||||
|
(re)uploaded — never silently skipped. This is the invariant the stat-cache
|
||||||
|
violated; here it is structural, not cached.
|
||||||
|
- **`get`** re-hashes every downloaded blob and deletes on mismatch.
|
||||||
|
- `cdn` write operations fail closed (never fake a write).
|
||||||
|
|
||||||
|
Stale-wording note to reconcile in code/docs: the dispatch registry summarizes
|
||||||
|
depot as *"(SeaweedFS)"*, but the real depot is Bunny-CDN + local
|
||||||
|
(`sow-assets-manifest/AGENTS.md`: "No S3, no SeaweedFS"). The Go implementation
|
||||||
|
and the registry summary adopt the Bunny/local/cdn reality.
|
||||||
|
|
||||||
|
## Data flow
|
||||||
|
|
||||||
|
```
|
||||||
|
assets/*.yml ──parse──▶ referenced sha set ─┐
|
||||||
|
├─▶ target.HasMany() ─▶ missing set
|
||||||
|
--target (bunny) ───┘ │
|
||||||
|
▼
|
||||||
|
--source (workspace/depot) ──read bytes──▶ target.PutMany()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cutover / clean-out
|
||||||
|
|
||||||
|
Increment 1 deletes the bash it replaces (no wrappers left behind):
|
||||||
|
|
||||||
|
- `scripts/mirror.sh` → `crucible depot pull`; script removed.
|
||||||
|
- `build-haks.sh`'s `--full` depot-verify block → `crucible depot verify`.
|
||||||
|
- **All three depot-touching workflows switch in the same change** (not just
|
||||||
|
`release.yml`):
|
||||||
|
- `release.yml` "Warm depot cache (verified full pull)" →
|
||||||
|
`crucible depot status --target bunny` (gate) then
|
||||||
|
`crucible depot pull --dest $DEPOT_CACHE_DIR`; the inline bash
|
||||||
|
ticker/compgen logic goes away.
|
||||||
|
- `publish-haks.yml` mirrors the same warm+verify+pack flow — same swap.
|
||||||
|
- `build-haks.yml` (PR check, cdn-backend verification) →
|
||||||
|
`crucible depot verify --target cdn`.
|
||||||
|
- Registry wiring: the `depot` entry in `internal/dispatch` flips to
|
||||||
|
`Wired: true` with its `Commands` list populated (status/push/verify/get/pull)
|
||||||
|
and the stale "(SeaweedFS)" summary corrected. This automatically puts depot
|
||||||
|
in the interactive `crucible` menu alongside the other wired tools — no
|
||||||
|
menu-specific code needed.
|
||||||
|
|
||||||
|
**What `lib.sh` deletion actually looks like.** The depot layer cannot be fully
|
||||||
|
removed in Increment 1 — it is still referenced by scripts outside this
|
||||||
|
increment's scope:
|
||||||
|
|
||||||
|
- `import.sh` / `sync-assets.sh` (`make haks`, `make import`, `make sync`) —
|
||||||
|
Increment 2.
|
||||||
|
- `export.sh` (`make pull`), `pack-haks.sh` (release-time `DEPOT_BACKEND=bunny`
|
||||||
|
packing), `publish-release.sh` + `release_put`, and the path-addressed
|
||||||
|
`artifact_has/get/put` + `cdn_purge` helpers — Increment 3 (these were
|
||||||
|
previously unlisted; they are now explicitly Increment 3 scope).
|
||||||
|
|
||||||
|
Increment 1 deletes the functions whose last caller it removes (the
|
||||||
|
mirror/verify probe-and-get paths) and leaves the rest with a header comment
|
||||||
|
marking them scheduled for deletion in Increments 2–3. No compatibility shims,
|
||||||
|
no new wrappers — but no pretending `lib.sh` dies before its last caller does.
|
||||||
|
|
||||||
|
Consumers call `crucible depot …` directly. Nothing new is wrapped.
|
||||||
|
|
||||||
|
### Interim contract for local builds (until Increment 2)
|
||||||
|
|
||||||
|
`make haks` keeps running bash `sync-assets.sh` with `DEPOT_BACKEND=local` until
|
||||||
|
Increment 2 replaces import/sync. The gap this leaves — local imports creating
|
||||||
|
blobs Bunny never sees — is held closed by two things:
|
||||||
|
|
||||||
|
- The backend-keyed stat-cache
|
||||||
|
([`sow-assets-manifest#29`](https://git.westgate.pw/ShadowsOverWestgate/sow-assets-manifest/pulls/29))
|
||||||
|
stays exactly as it is; do not "simplify" it (see the 2026-07-04 update above).
|
||||||
|
- `make haks` gains a trailing `crucible depot status --manifests assets
|
||||||
|
--target cdn` (credential-free, report-only, non-fatal) so drift is visible
|
||||||
|
the moment it is created instead of at release time; and
|
||||||
|
`crucible depot push --source workspace/depot --target bunny` is documented
|
||||||
|
as the standard post-edit step (field note 4's ask, landed here). Upload when
|
||||||
|
you want the content live; build local-only when you don't — but either way
|
||||||
|
the drift is *reported*, never masked.
|
||||||
|
|
||||||
|
## Exit-code contract
|
||||||
|
|
||||||
|
Matches Crucible's fail-closed convention:
|
||||||
|
|
||||||
|
- `0` — clean / success.
|
||||||
|
- `1` — **drift found** (`status`/`push` saw referenced-but-absent blobs), so
|
||||||
|
`release.yml` can gate on `status` before packing.
|
||||||
|
- `2` — **unconfirmed only**: no confirmed-missing blobs, but the probe budget
|
||||||
|
ran out before every blob got a confirmed reply (throttled edge). The gate
|
||||||
|
should retry, not report drift.
|
||||||
|
- `70` — internal/backend error; never a faked result.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
- **Go unit tests** (`internal/depot`): `local` backend round-trip against a temp
|
||||||
|
dir; manifest parsing; drift computation (referenced − present = missing) as
|
||||||
|
table tests; backend selection + config resolution (incl. read/write key
|
||||||
|
fallback).
|
||||||
|
- **httptest fake-Bunny**: assert the range-probe-then-PUT sequence, the
|
||||||
|
`Checksum: <UPPER-sha>` header, and the read/write key split — no real Bunny.
|
||||||
|
- **Integration (local→local)**: `status` shows drift → `push` clears it →
|
||||||
|
`status` clean → `push` again is a no-op (idempotency).
|
||||||
|
- **New-surface tests**: fake-Bunny throttling (`428`) → blobs land in
|
||||||
|
`unconfirmed`, exit `2`, never `missing`; `pull` into a pre-populated dest
|
||||||
|
skips present-and-hash-ok blobs (incremental); credentialed op with no key
|
||||||
|
exits non-zero without reading stdin (read path and write path both).
|
||||||
|
|
||||||
|
## Non-goals (Increment 1)
|
||||||
|
|
||||||
|
- `import` / `sync` / edit-tree ingestion (Increment 2).
|
||||||
|
- mdl integrity checks, `prune`, `gc`, `export`, `mirror` ergonomics
|
||||||
|
(Increment 3).
|
||||||
|
- Any change to the depot wire protocol, blob layout, or Bunny account config.
|
||||||
|
- A depot-specific rich TUI. **Direction note:** Crucible is headed toward being
|
||||||
|
a TUI overall — the existing interactive dispatcher menu is the first step,
|
||||||
|
and depot joins it for free via the registry wiring above. What Increment 1
|
||||||
|
does not build is anything richer than that menu (progress UIs, pickers,
|
||||||
|
dashboards); those come with the Crucible-wide TUI work, not per-tool.
|
||||||
|
|
||||||
|
## Immediate operational payoff
|
||||||
|
|
||||||
|
`crucible depot push --source workspace/depot --target bunny` is exactly the
|
||||||
|
command that unblocks the stuck `v0.1.5` release: it probes the target and
|
||||||
|
uploads whatever is actually 404 while it re-hydrates from the local depot.
|
||||||
|
(The `5903` figure below is superseded — see the 2026-07-04 update above: the
|
||||||
|
real drift was **2** blobs, already re-uploaded; `push` would have been the
|
||||||
|
clean way to do it and is idempotent once the depot is whole.) Building
|
||||||
|
Increment 1 first both fixes the tooling and clears the current outage.
|
||||||
|
|
||||||
|
## Risks / open questions
|
||||||
|
|
||||||
|
- ~~`BUNNY_STORAGE_HOST` default~~ — resolved: env-required, no baked default
|
||||||
|
(see Backends & config).
|
||||||
|
- Manifest schema coupling: `crucible depot` now depends on the `assets/*.yml`
|
||||||
|
shape (`assets[].{path,sha256,size,restype,hak}`). Acceptable and intentional,
|
||||||
|
but a schema change now touches Crucible.
|
||||||
|
- Cutover ordering: `release.yml` and Makefile must switch to `crucible depot` in
|
||||||
|
the same change that removes the bash, to avoid a window where both exist.
|
||||||
@@ -0,0 +1,343 @@
|
|||||||
|
# Crucible `assets` builder — design
|
||||||
|
|
||||||
|
Date: 2026-07-12
|
||||||
|
Status: approved, pre-implementation
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Fold the NWN:EE asset tools out of two standalone toolkits into Crucible as a
|
||||||
|
new self-contained builder, `crucible assets`:
|
||||||
|
|
||||||
|
- from the Python toolkit (`nwnee-asset-processing`) — model compile, texture
|
||||||
|
convert, texture upscale. Those scripts are references only; broken as-is and
|
||||||
|
carrying a heavy config/staging/report layer that we drop.
|
||||||
|
- from `sow-assets-manifest/scripts/` — the model-name and duplicate-name
|
||||||
|
integrity tools (`check-ascii-mdl.sh`, `fix-mdl-model-names.sh`,
|
||||||
|
`check-duplicate-names.sh`, `clean-duplicate-names.sh`). These are ported into
|
||||||
|
Go and the shell drivers removed; `sow-assets-manifest` calls
|
||||||
|
`./crucible.sh assets …` through its bootstrap wrapper instead.
|
||||||
|
|
||||||
|
Seven commands:
|
||||||
|
|
||||||
|
- `crucible assets compile` — compile every ASCII `.mdl` in a directory to
|
||||||
|
binary, in place.
|
||||||
|
- `crucible assets convert` — convert every texture in a directory to NWN:EE
|
||||||
|
DDS (or back to PNG/TGA), flipping vertically **every time** so a DDS is
|
||||||
|
always upside-down relative to its source.
|
||||||
|
- `crucible assets upscale` — upscale every texture in a directory through
|
||||||
|
whatever upscaling backend is installed.
|
||||||
|
- `crucible assets check-mdl` — report uncompiled ASCII `.mdl` files and
|
||||||
|
model-name mismatches (read-only).
|
||||||
|
- `crucible assets fix-mdl` — lowercase `.mdl` filenames and rewrite ASCII
|
||||||
|
model/root names to match each file's stem.
|
||||||
|
- `crucible assets check-dupes` — report runtime-name (basename) collisions
|
||||||
|
across directories.
|
||||||
|
- `crucible assets clean-dupes` — delete files from a "clean" tree whose
|
||||||
|
basename collides with anything in a "primary" tree.
|
||||||
|
|
||||||
|
## Principles
|
||||||
|
|
||||||
|
- **No configuration files. Arguments only.** No `processing/` staging dirs, no
|
||||||
|
discards, no YAML, no reports. Operates in place on the target directory.
|
||||||
|
- **Discover, don't configure.** External tools (the NWN engine, ImageMagick, an
|
||||||
|
upscaler) are found on `PATH` and at standard install locations. Each has a
|
||||||
|
single override flag when discovery is not enough.
|
||||||
|
- **Fail closed, never fake** (Crucible rule #1). A missing backend is a clear
|
||||||
|
error naming what to install and a non-zero exit — never a placeholder
|
||||||
|
artifact.
|
||||||
|
- **Simplify.** Drop the reference tools' padding / power-of-two / `nwn-safe` /
|
||||||
|
explicit-format knobs (convert auto-picks DXT1 vs DXT5) and the small-texture
|
||||||
|
staging (upscale). These can return later if wanted.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
A new self-contained builder that follows the existing `depot` pattern exactly
|
||||||
|
(delegates straight to its own internal package, bypassing the legacy
|
||||||
|
`internal/app` surface):
|
||||||
|
|
||||||
|
- `internal/assets/run.go` — `func Run(args []string, stdout, stderr io.Writer,
|
||||||
|
getenv func(string) string) int`. Parses the subcommand
|
||||||
|
(`compile|convert|upscale|check-mdl|fix-mdl|check-dupes|clean-dupes`) and
|
||||||
|
dispatches. Returns a sysexits-style code.
|
||||||
|
- `internal/assets/` — one file per command plus small shared helpers
|
||||||
|
(recursion/file-selection, external-command runner, backend discovery).
|
||||||
|
- `cmd/crucible-assets/main.go` — one-line shim:
|
||||||
|
`func main() { os.Exit(dispatch.RunBuilder("assets", os.Args[1:])) }`.
|
||||||
|
- `internal/dispatch`:
|
||||||
|
- one `Registry` entry for `assets` (`Wired: true`) listing the seven
|
||||||
|
commands with `Usage`/`Options`;
|
||||||
|
- extend the direct-delegate branch that today reads
|
||||||
|
`if b.Name == "depot" && b.Wired` so `assets` also routes to
|
||||||
|
`assets.Run(...)` instead of the legacy `app.Run`.
|
||||||
|
- `docs/command-surface.md` — seven visible-command rows.
|
||||||
|
- `wrappers/consumers.txt` / wrappers are unchanged (no new consumer; the
|
||||||
|
consumers listed already vendor the wrapper).
|
||||||
|
|
||||||
|
### Shared helpers (`internal/assets`)
|
||||||
|
|
||||||
|
- `walk(roots, exts)` — collect files under each root (recursive by default)
|
||||||
|
whose extension is in `exts`, case-insensitively. Reject nothing fancy; these
|
||||||
|
are arguments the caller chose.
|
||||||
|
- `run(cmd, args...)` — thin `exec.Command` wrapper capturing combined output,
|
||||||
|
returning `(output, error)`. A single indirection point so tests can observe
|
||||||
|
invocations. cwd and env overrides passed as needed.
|
||||||
|
- `look(names...)` — return the first name found via `exec.LookPath`, or the
|
||||||
|
first path that exists from a supplied list of candidates. Used by every
|
||||||
|
backend discovery.
|
||||||
|
|
||||||
|
### Shared MDL name module (`internal/assets/mdl`)
|
||||||
|
|
||||||
|
A pure-Go port of `mdl-name-lib.sh` + `mdl-scan.awk`, no external tools. One
|
||||||
|
place three commands share (`compile`'s pre-check, `check-mdl`, `fix-mdl`):
|
||||||
|
|
||||||
|
- `IsASCII(path)` — NUL in the first 256 bytes → binary; else the leading
|
||||||
|
keyword must be `#`/`newmodel`/`node`/`setsupermodel`. Mirrors `mdl_is_ascii`.
|
||||||
|
- `ExpectedName(path)` — the file stem (sans `.mdl`).
|
||||||
|
- `CheckNames(path) []Mismatch` — for an ASCII model, report header-token
|
||||||
|
mismatches (`newmodel`, `setsupermodel`, `beginmodelgeom`, `endmodelgeom`,
|
||||||
|
`donemodel`, `newanim`, `doneanim` — case-insensitive vs the stem) and the
|
||||||
|
geometry **base node** mismatch (the node inside `beginmodelgeom` whose parent
|
||||||
|
is `null`; empty if none). Mirrors `mdl_check_model_name` + `mdl_base_name`.
|
||||||
|
- `FixNames(in) (out []byte, changed bool)` — rewrite only the model identity:
|
||||||
|
the header tokens above, plus the base node's declaration / any `parent` /
|
||||||
|
`animroot` pointing at the *old* base name → the expected name. Animation bone
|
||||||
|
names and supermodel animroots are left untouched so inheritance keeps working.
|
||||||
|
Mirrors `mdl_fix_model_name` byte-for-byte in intent.
|
||||||
|
|
||||||
|
Detection must stay identical to the awk classifier so behavior does not drift
|
||||||
|
while `import.sh` still runs the awk version (see Consumer migration).
|
||||||
|
|
||||||
|
## `crucible assets compile <dir>…`
|
||||||
|
|
||||||
|
Usage: `crucible assets compile [--nwn <install>] [--non-recursive] <dir>…`
|
||||||
|
|
||||||
|
Recursively find ASCII `.mdl` files under each `<dir>` and compile each to a
|
||||||
|
binary `.mdl` in place (the compiled result replaces the source).
|
||||||
|
|
||||||
|
The NWN:EE engine is the only real ASCII→binary MDL compiler, so this drives it
|
||||||
|
exactly as the reference does:
|
||||||
|
|
||||||
|
- **Discovery.** User data dir is fixed on Linux at
|
||||||
|
`~/.local/share/Neverwinter Nights` (holds the flat `development/` and
|
||||||
|
`modelcompiler/` folders the engine uses). The `nwmain` binary is found by
|
||||||
|
probing standard install roots, first hit wins:
|
||||||
|
- `~/.local/share/Steam/steamapps/common/Neverwinter Nights/bin/linux-x86/nwmain-linux`
|
||||||
|
- GOG: `~/GOG Games/Neverwinter Nights Enhanced Edition/.../nwmain-linux`
|
||||||
|
- Beamdog install dirs.
|
||||||
|
- `--nwn <install>` overrides (path to the install root or directly to the
|
||||||
|
binary).
|
||||||
|
- **Headless.** The engine is a GUI binary. Require `xvfb-run` and wrap the call
|
||||||
|
regardless of the caller's `DISPLAY` so compilation never opens the client
|
||||||
|
UI; fail before invoking the engine when `xvfb-run` is unavailable:
|
||||||
|
`xvfb-run -a --server-args=-screen 0 1024x768x24 nwmain-linux compilemodel <stem>`.
|
||||||
|
- **Per model (one at a time — the engine's `development/` and `modelcompiler/`
|
||||||
|
folders are flat and single-slot):**
|
||||||
|
1. Skip if the file is not ASCII (binary/compiled `.mdl`) — reported, not an
|
||||||
|
error.
|
||||||
|
2. Copy source into `development/<name>`.
|
||||||
|
3. Run `nwmain-linux compilemodel <stem>` with cwd = the binary dir.
|
||||||
|
4. Collect the compiled artifact from `modelcompiler/` matched
|
||||||
|
case-insensitively by stem.
|
||||||
|
5. Move it back over the source path, lowercased.
|
||||||
|
6. Clean the temp files created in `development/` and `modelcompiler/`.
|
||||||
|
- **Collisions.** A pre-existing `development/<name>` or `modelcompiler/<stem>`
|
||||||
|
aborts that model before any mutation.
|
||||||
|
|
||||||
|
Exit: non-zero if any model fails; each failure prints a short excerpt of the
|
||||||
|
engine log (`~/.local/share/Neverwinter Nights/logs/nwengineLog.txt`) as
|
||||||
|
advisory diagnostics.
|
||||||
|
|
||||||
|
Dropped vs reference: internal-name-mismatch pre-checks and the
|
||||||
|
`Model Names Differ` fail rule are kept (cheap, prevents silent wrong output),
|
||||||
|
using the shared `internal/assets/mdl` module — `compile` aborts a model whose
|
||||||
|
names differ and tells the user to run `crucible assets fix-mdl`. It does **not**
|
||||||
|
auto-fix (kept separate, by decision). The discard-manifest, dry-run/report
|
||||||
|
layer, and `processing/in|out` staging are dropped.
|
||||||
|
|
||||||
|
## `crucible assets convert <dir>…`
|
||||||
|
|
||||||
|
Usage: `crucible assets convert [--to dds|png|tga] [--backend <path>]
|
||||||
|
[--non-recursive] <dir>…`
|
||||||
|
|
||||||
|
Recursively convert textures in place. `--to` defaults to `dds`. The source set
|
||||||
|
is every texture under the roots whose format is not already the target, among
|
||||||
|
`{.dds, .png, .tga}`.
|
||||||
|
|
||||||
|
**Every conversion flips vertically exactly once.** Converting to DDS produces
|
||||||
|
an upside-down DDS (NWN's convention); converting a DDS back to PNG/TGA flips it
|
||||||
|
upright again. The flip is unconditional — it is the defining behavior, not a
|
||||||
|
policy.
|
||||||
|
|
||||||
|
Backend: ImageMagick (`magick`), one tool for decode + flip + encode across all
|
||||||
|
three formats:
|
||||||
|
|
||||||
|
- to DDS: `magick <src> -flip -define dds:mipmaps=<n> -define
|
||||||
|
dds:compression=<dxt1|dxt5> <dst>.dds`, choosing DXT1 when the source is
|
||||||
|
opaque and DXT5 when it has alpha (detected with `magick identify`).
|
||||||
|
- from DDS: `magick <src>.dds -flip <dst>.<png|tga>`.
|
||||||
|
- `--backend <path>` points at a different ImageMagick-compatible binary or a
|
||||||
|
dedicated encoder for higher-quality DXT (e.g. compressonator); default is
|
||||||
|
`magick`.
|
||||||
|
|
||||||
|
The original file is replaced in place — canonical DDS is the point. Exit
|
||||||
|
non-zero if any file fails.
|
||||||
|
|
||||||
|
Dropped vs reference: `--pad-color`, POT/legacy-safe/`nwn-safe`/`--strict`
|
||||||
|
sizing, explicit `--format`, alpha-detect toggles, discard staging. Auto DXT
|
||||||
|
selection replaces the format knobs.
|
||||||
|
|
||||||
|
## `crucible assets upscale <dir>…`
|
||||||
|
|
||||||
|
Usage: `crucible assets upscale [--scale N] [--backend <path>]
|
||||||
|
[--non-recursive] <dir>…`
|
||||||
|
|
||||||
|
Recursively upscale textures in place.
|
||||||
|
|
||||||
|
- **Backend discovery**, first found wins: `upscayl-bin`, `upscayl`,
|
||||||
|
`realesrgan-ncnn-vulkan`, `waifu2x-ncnn-vulkan`. These share a compatible
|
||||||
|
ncnn-vulkan CLI shape (`-i <in> -o <out> -s <scale>`). `--backend <path>`
|
||||||
|
overrides.
|
||||||
|
- `--scale N` default 4.
|
||||||
|
- Backends operate on PNG; real NWN textures are DDS, so for a `.dds` input the
|
||||||
|
command reuses the convert helpers: dds→png (flip), run the backend, png→dds
|
||||||
|
(flip back) — yielding a correctly-flipped upscaled DDS. PNG/TGA inputs are
|
||||||
|
upscaled directly in place.
|
||||||
|
- No backend found → fail closed with a message naming the supported backends.
|
||||||
|
|
||||||
|
Dropped vs reference: min-dimension small-texture staging, the separate
|
||||||
|
`--dds-backend` plumbing (it reuses convert), configured backend-path table.
|
||||||
|
|
||||||
|
## `crucible assets check-mdl <path>…`
|
||||||
|
|
||||||
|
Port of `check-ascii-mdl.sh`. Read-only. Each `<path>` is a file or a directory
|
||||||
|
(recursed for `*.mdl`). For every model, using `internal/assets/mdl`:
|
||||||
|
|
||||||
|
- report an **uncompiled ASCII** `.mdl` (a binary/compiled one is fine, skipped);
|
||||||
|
- report every model-name mismatch (header tokens and geometry base node) with
|
||||||
|
file, line, the offending token, and the expected stem.
|
||||||
|
|
||||||
|
Exit non-zero if any ASCII model or mismatch is found; zero and an `OK` line
|
||||||
|
otherwise. No mutation.
|
||||||
|
|
||||||
|
## `crucible assets fix-mdl [--dry-run] <path>…`
|
||||||
|
|
||||||
|
Port of `fix-mdl-model-names.sh`. For each `*.mdl` under the paths:
|
||||||
|
|
||||||
|
1. **Lowercase the filename** if it is not already lowercase (`mv`; abort that
|
||||||
|
file on a case-collision with an existing target).
|
||||||
|
2. For ASCII models with a name mismatch, rewrite the model identity in place
|
||||||
|
via `mdl.FixNames` (binary models are skipped — the engine owns those).
|
||||||
|
|
||||||
|
`--dry-run` reports every "would lowercase" / "would fix" without touching disk.
|
||||||
|
Prints a per-file log and a final `no broken mdl model names found` when clean.
|
||||||
|
|
||||||
|
Selection optimization from the reference is preserved: a file is a fix
|
||||||
|
candidate only if it needs a content fix (name mismatch) or a filename
|
||||||
|
lowercase, so the expensive per-file read/rewrite runs only on the few that
|
||||||
|
matter, not the whole tree.
|
||||||
|
|
||||||
|
## `crucible assets check-dupes <dir>…`
|
||||||
|
|
||||||
|
Port of `check-duplicate-names.sh`'s **directory mode only**. Read-only. NWN
|
||||||
|
packs hak entries by basename, not path, so files at different paths with the
|
||||||
|
same basename (case-insensitively) silently clobber on pack. Across all `<dir>`
|
||||||
|
args, report any two files whose lowercased basename matches.
|
||||||
|
|
||||||
|
Exit non-zero if any collision is found.
|
||||||
|
|
||||||
|
The script's `--manifests` mode (hak-scoped collisions read from `assets/*.yml`)
|
||||||
|
is **not** ported here — it inspects manifest structure, not files on disk, so
|
||||||
|
it belongs with the manifest/depot tooling. It moves in the follow-up spec (see
|
||||||
|
Out of scope); until then `sow-assets-manifest` keeps
|
||||||
|
`check-duplicate-names.sh` solely for that mode.
|
||||||
|
|
||||||
|
## `crucible assets clean-dupes [--dry-run] <primary> <clean>`
|
||||||
|
|
||||||
|
Port of `clean-duplicate-names.sh`. Deletes files from the `<clean>` tree whose
|
||||||
|
basename collides (case-insensitively) with any file in the `<primary>` tree.
|
||||||
|
Mutates only `<clean>`. Same rule as `check-dupes` directory mode, but resolves
|
||||||
|
the collision by removal instead of reporting.
|
||||||
|
|
||||||
|
- Refuses to run if `<primary>` and `<clean>` overlap (either contains the
|
||||||
|
other), matching the script's guard.
|
||||||
|
- `--dry-run` reports every "would delete" without removing.
|
||||||
|
|
||||||
|
Exit zero on success with a count of removed (or would-remove) files.
|
||||||
|
|
||||||
|
## Consumer migration (`sow-assets-manifest`)
|
||||||
|
|
||||||
|
`sow-assets-manifest` already ships the `crucible` bootstrap wrapper
|
||||||
|
(`crucible.sh`) and uses `crucible depot` from its Makefile, so it resolves a
|
||||||
|
released `crucible` with zero extra setup.
|
||||||
|
|
||||||
|
- **Delete** the three fully-folded driver scripts: `scripts/check-ascii-mdl.sh`,
|
||||||
|
`scripts/fix-mdl-model-names.sh`, `scripts/clean-duplicate-names.sh`.
|
||||||
|
- **Repoint** the Makefile targets to the wrapper:
|
||||||
|
- `check-mdl` → `./crucible.sh assets check-mdl …`
|
||||||
|
- `fix-mdl` → `./crucible.sh assets fix-mdl [--dry-run] …`
|
||||||
|
- `check-dupes` — its directory scan → `./crucible.sh assets check-dupes …`;
|
||||||
|
the `check-duplicate-names.sh --manifests` line stays as-is.
|
||||||
|
- `clean-dupes` → `./crucible.sh assets clean-dupes [--dry-run] <PRIMARY> <CLEAN>`
|
||||||
|
- **Keep for now:**
|
||||||
|
- `scripts/check-duplicate-names.sh` — kept **only** for its `--manifests`
|
||||||
|
mode, still called by the `check` and `haks` targets. Its directory mode is
|
||||||
|
superseded by `crucible assets check-dupes`. Ported and deleted in the
|
||||||
|
follow-up.
|
||||||
|
- `scripts/mdl-scan.awk` and `scripts/mdl-name-lib.sh` — still sourced by
|
||||||
|
`import.sh` for its batched pre-import gate (~28× faster than per-file, on a
|
||||||
|
hot path). They retire in the follow-up too. Because both the awk classifier
|
||||||
|
and the Go `mdl` module must agree, the port keeps detection byte-for-byte
|
||||||
|
identical (a shared fixture set checks this).
|
||||||
|
|
||||||
|
Nothing in `sow-assets-manifest` is committed by this project's plan except the
|
||||||
|
script deletions and Makefile edits; that repo's change lands as its own commit
|
||||||
|
once the new `crucible` release with `assets` is available.
|
||||||
|
|
||||||
|
## Out of scope — follow-up spec
|
||||||
|
|
||||||
|
Moving the **depot-interacting pipeline** into `crucible depot` is a separate
|
||||||
|
project with its own spec
|
||||||
|
(`2026-07-12-crucible-depot-pipeline-migration-design.md`). It covers
|
||||||
|
`import.sh`, `sync-assets.sh`, `export.sh`, the inline mdl gate, and the
|
||||||
|
`check-duplicate-names.sh --manifests` manifest-collision check. When that lands:
|
||||||
|
its gate uses the in-process `internal/assets/mdl` module, and
|
||||||
|
`scripts/mdl-scan.awk`, `scripts/mdl-name-lib.sh`, and
|
||||||
|
`scripts/check-duplicate-names.sh` are all deleted. This spec deliberately stops
|
||||||
|
at the standalone, file-on-disk integrity tools to keep one focused plan.
|
||||||
|
|
||||||
|
## Error handling
|
||||||
|
|
||||||
|
- Missing external tool (engine, `magick`, upscaler) → non-zero exit, message
|
||||||
|
names the tool and how to get it. Never a faked artifact. The pure-Go
|
||||||
|
integrity commands (`check-mdl`, `fix-mdl`, `check-dupes`, `clean-dupes`) need
|
||||||
|
no external tool.
|
||||||
|
- Per-file failures are collected and printed as a summary line at the end; the
|
||||||
|
command exits non-zero if any file failed but still processes the rest.
|
||||||
|
- Report commands (`check-mdl`, `check-dupes`) exit non-zero when they *find*
|
||||||
|
problems — that is their contract as CI/pre-import gates, not a tool error.
|
||||||
|
- Unknown subcommand / bad flags → usage error, exit 64.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
Go tests in `internal/assets`:
|
||||||
|
|
||||||
|
- flag/subcommand parsing and usage errors;
|
||||||
|
- backend discovery against a fabricated `PATH` (temp dir with stub
|
||||||
|
executables) — asserts first-match ordering and the `--backend` override;
|
||||||
|
- `walk` recursion and extension filtering, including `--non-recursive`;
|
||||||
|
- one real convert round-trip using `magick` (present in the dev shell):
|
||||||
|
encode a small PNG to DDS and back, asserting the pixels are vertically
|
||||||
|
flipped after a single conversion and restored after the round trip.
|
||||||
|
- `internal/assets/mdl`: table-driven fixtures — ASCII vs binary detection,
|
||||||
|
each header-token mismatch, the base-node mismatch, and `FixNames` producing
|
||||||
|
a model that then passes `CheckNames`. A parity fixture set shared in intent
|
||||||
|
with `mdl-scan.awk` so the Go port and the still-present awk agree.
|
||||||
|
- `check-dupes`/`clean-dupes`: temp trees asserting basename-collision
|
||||||
|
detection (case-insensitive), the overlap guard, and `--dry-run` mutating
|
||||||
|
nothing.
|
||||||
|
|
||||||
|
External engine/upscaler calls are exercised through the `run` indirection with
|
||||||
|
a stub in tests; they are not invoked for real in CI.
|
||||||
|
|
||||||
|
`make check` must stay green; add an `assets` expectation to `make smoke` for
|
||||||
|
the wired exit.
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
# Crucible `depot` pipeline migration — design (charter / draft)
|
||||||
|
|
||||||
|
Date: 2026-07-12
|
||||||
|
Status: **draft — needs its own brainstorming pass before implementation.**
|
||||||
|
Depends on: `2026-07-12-crucible-assets-builder-design.md` (the `assets` builder
|
||||||
|
and `internal/assets/mdl` module) landing first.
|
||||||
|
|
||||||
|
This is a scoping charter, not an implementation-ready design. The scripts it
|
||||||
|
covers are nontrivial and lean heavily on `sow-assets-manifest/scripts/lib.sh`
|
||||||
|
(~35 KB). A full design requires studying `lib.sh` and the manifest format in
|
||||||
|
depth; that work happens in this spec's own brainstorming before any code.
|
||||||
|
|
||||||
|
## Why this exists
|
||||||
|
|
||||||
|
`sow-assets-manifest` is the last consumer still carrying real builder logic in
|
||||||
|
shell. The `assets` spec folded its standalone integrity tools into Crucible but
|
||||||
|
deliberately left the **depot-interacting pipeline** behind, because those
|
||||||
|
scripts read/write the content-addressed depot and the asset manifests — depot
|
||||||
|
domain, and a bigger lift. This charter completes the migration so
|
||||||
|
`sow-assets-manifest` becomes what the other consumers already are: a thin
|
||||||
|
caller of released `crucible` binaries with no vendored toolkit logic.
|
||||||
|
|
||||||
|
## What moves into `crucible depot`
|
||||||
|
|
||||||
|
The depot builder already owns `status/push/verify/get/pull`. This adds the
|
||||||
|
authoring side:
|
||||||
|
|
||||||
|
| Script | Proposed command | What it does today |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `import.sh` (318 lines) | `crucible depot import <edit-dir> [prefix…]` | Import an edit tree into the depot + manifests: sticky per-path member assignment, new paths to the tail member, per-category `max_bytes` rebalance (spill overflow forward, no full repack), a local stat-cache to skip unchanged files, batched hashing + parallel `depot_put_many`. |
|
||||||
|
| `sync-assets.sh` (61 lines) | `crucible depot sync <tree> [prefix…]` | Full reconcile = import + prune manifest entries whose path no longer exists (scoped by prefix when given). Delegates to `import.sh`. |
|
||||||
|
| `export.sh` (96 lines) | `crucible depot export [glob] [--hak m] [--restype ext] --to <dir>` | Materialize a manifest-selected subset out of the depot into a clean edit tree (category paths preserved, no metadata). |
|
||||||
|
| `check-duplicate-names.sh --manifests` | `crucible depot check-dupes` (name TBD) | Hak-scoped basename collisions read from `assets/*.yml` (`.assets[].path` grouped by `.assets[].hak`). Manifest-structure check, not files on disk. |
|
||||||
|
|
||||||
|
The inline **pre-import mdl gate** in `import.sh` (batched `mdl-scan.awk` pass
|
||||||
|
that aborts on an H/B model-name mismatch before any blob is hashed/uploaded)
|
||||||
|
is reimplemented with the in-process `internal/assets/mdl` module from the
|
||||||
|
`assets` spec — same H/B contract, no subprocess.
|
||||||
|
|
||||||
|
## What retires when this lands
|
||||||
|
|
||||||
|
Deleted from `sow-assets-manifest/scripts/` once the commands above are wired
|
||||||
|
and the Makefile is repointed:
|
||||||
|
|
||||||
|
- `import.sh`, `sync-assets.sh`, `export.sh`
|
||||||
|
- `mdl-scan.awk`, `mdl-name-lib.sh` (only remaining consumer was `import.sh`)
|
||||||
|
- `check-duplicate-names.sh` (its directory mode already superseded by
|
||||||
|
`crucible assets check-dupes`; the `--manifests` mode moves here)
|
||||||
|
- whatever else in `lib.sh` becomes dead once the above are gone (assess during
|
||||||
|
design — `lib.sh` also serves `build-haks.sh`, `pack-haks.sh`, `promote.sh`,
|
||||||
|
etc., which are **not** in scope here, so `lib.sh` likely shrinks, not dies)
|
||||||
|
|
||||||
|
Makefile targets `import`, `sync`, `haks` (its sync/import/dupe steps), `export`
|
||||||
|
repoint to `./crucible.sh depot …`.
|
||||||
|
|
||||||
|
## Known hard parts (resolve in brainstorming)
|
||||||
|
|
||||||
|
- **Manifest read/write parity.** Byte-stable output, member `max_bytes`
|
||||||
|
rebalance semantics, sticky path→member assignment. Must match the current
|
||||||
|
awk/`lib.sh` behavior or diff-review of manifests becomes noise. Crucible
|
||||||
|
already parses these manifests (`internal/pipeline`, hak build) — reuse, don't
|
||||||
|
reinvent.
|
||||||
|
- **Depot backends.** `import`/`export` drive bunny/cdn/local through
|
||||||
|
`lib.sh`'s `depot_put_many`/`depot_require_write` (incl. the
|
||||||
|
`BUNNY_STORAGE_PASSWORD` prompt and cdn→bunny write normalization). Crucible's
|
||||||
|
`internal/depot` already speaks these backends for pull/push — the authoring
|
||||||
|
side should share that client, not a second implementation.
|
||||||
|
- **Stat-cache.** `import.sh`'s `.cache/import/<target>/<cat>.tsv` skip-unchanged
|
||||||
|
optimization is the dominant re-import speedup. Decide whether Crucible
|
||||||
|
reproduces it, replaces it (content-addressing already dedupes uploads), or
|
||||||
|
drops it with a measured justification.
|
||||||
|
- **Throughput.** The shell versions are already batched to avoid per-file
|
||||||
|
forks; a Go port should be at least as fast (in-process, no forks) — verify,
|
||||||
|
don't assume.
|
||||||
|
- **Scoping semantics.** Prefix-scoped import/prune (segment-boundary match) and
|
||||||
|
export glob matching (`**`/`*`/literal dots, no shell expansion) must port
|
||||||
|
exactly.
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
The HAK build/pack/promote/publish scripts (`build-haks.sh`, `pack-haks.sh`,
|
||||||
|
`promote.sh`, `publish-release*.sh`, `hak-artifact-record.sh`) and their
|
||||||
|
`lib.sh` support. `crucible hak build` already exists; whether these thin
|
||||||
|
release wrappers also migrate is a separate question, not this charter.
|
||||||
|
|
||||||
|
## Next step
|
||||||
|
|
||||||
|
Brainstorm this charter into a full design: read `lib.sh` and the manifest
|
||||||
|
format, settle the hard parts above, then write the implementation-ready spec
|
||||||
|
and plan. Do not implement from this document.
|
||||||
@@ -10,12 +10,11 @@
|
|||||||
in
|
in
|
||||||
{
|
{
|
||||||
# Self-contained (D8): a host with ONLY nix can enter this shell and run
|
# Self-contained (D8): a host with ONLY nix can enter this shell and run
|
||||||
# `make check`. ffmpeg is present because the migrated music pipeline needs
|
# `make check`.
|
||||||
# it; the scaffold itself is pure stdlib Go.
|
|
||||||
# Daemonless image build (D8): `nix build .#image` produces an OCI tarball
|
# Daemonless image build (D8): `nix build .#image` produces an OCI tarball
|
||||||
# with no docker/podman daemon. CI loads/pushes it with skopeo. This is the
|
# with no docker/podman daemon. CI loads/pushes it with skopeo. This is the
|
||||||
# Nix-native replacement for `docker build` on the host-mode runner, which
|
# Nix-native replacement for `docker build` on the host-mode runner, which
|
||||||
# has no container runtime. ffmpeg-headless ships the BMU codec path.
|
# has no container runtime.
|
||||||
packages = forAllSystems (pkgs:
|
packages = forAllSystems (pkgs:
|
||||||
let
|
let
|
||||||
version = self.shortRev or self.dirtyShortRev or "unknown";
|
version = self.shortRev or self.dirtyShortRev or "unknown";
|
||||||
@@ -23,12 +22,13 @@
|
|||||||
pname = "crucible";
|
pname = "crucible";
|
||||||
inherit version;
|
inherit version;
|
||||||
src = ./.;
|
src = ./.;
|
||||||
vendorHash = "sha256-hm6mrNAtXv0LidzHUfz4eukTFZouizGtxkZ8gKJFUVI=";
|
vendorHash = "sha256-0I8j7On9YGD2GK9xbj/KkgBrlkMJ6Y6XQv+KCLTgBBU=";
|
||||||
subPackages = [
|
subPackages = [
|
||||||
"cmd/crucible"
|
"cmd/crucible"
|
||||||
"cmd/crucible-depot"
|
"cmd/crucible-depot"
|
||||||
"cmd/crucible-hak"
|
"cmd/crucible-hak"
|
||||||
"cmd/crucible-module"
|
"cmd/crucible-module"
|
||||||
|
"cmd/crucible-nwsync"
|
||||||
"cmd/crucible-topdata"
|
"cmd/crucible-topdata"
|
||||||
"cmd/crucible-wiki"
|
"cmd/crucible-wiki"
|
||||||
];
|
];
|
||||||
@@ -49,7 +49,7 @@
|
|||||||
image = pkgs.dockerTools.buildLayeredImage {
|
image = pkgs.dockerTools.buildLayeredImage {
|
||||||
name = "registry.westgate.pw/deployment/crucible";
|
name = "registry.westgate.pw/deployment/crucible";
|
||||||
tag = version;
|
tag = version;
|
||||||
contents = [ crucible pkgs.cacert pkgs.ffmpeg-headless pkgs.fakeNss ];
|
contents = [ crucible pkgs.cacert pkgs.fakeNss ];
|
||||||
config = {
|
config = {
|
||||||
Entrypoint = [ "/bin/crucible" ];
|
Entrypoint = [ "/bin/crucible" ];
|
||||||
Cmd = [ "help" ];
|
Cmd = [ "help" ];
|
||||||
@@ -69,8 +69,8 @@
|
|||||||
go
|
go
|
||||||
gopls
|
gopls
|
||||||
gotools
|
gotools
|
||||||
ffmpeg
|
|
||||||
git
|
git
|
||||||
|
tea
|
||||||
gnumake
|
gnumake
|
||||||
bashInteractive
|
bashInteractive
|
||||||
shellcheck
|
shellcheck
|
||||||
|
|||||||
@@ -6,3 +6,5 @@ require (
|
|||||||
golang.org/x/text v0.35.0
|
golang.org/x/text v0.35.0
|
||||||
gopkg.in/yaml.v3 v3.0.1
|
gopkg.in/yaml.v3 v3.0.1
|
||||||
)
|
)
|
||||||
|
|
||||||
|
require github.com/klauspost/compress v1.19.1
|
||||||
|
|||||||
@@ -1,3 +1,5 @@
|
|||||||
|
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
|
||||||
|
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||||
golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8=
|
golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8=
|
||||||
golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA=
|
golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA=
|
||||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||||
|
|||||||
+40
-214
@@ -9,7 +9,6 @@ import (
|
|||||||
"os/exec"
|
"os/exec"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"runtime"
|
"runtime"
|
||||||
"sort"
|
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
@@ -184,11 +183,6 @@ var commands = []command{
|
|||||||
description: "Inspect, explain, and validate the resolved project configuration.",
|
description: "Inspect, explain, and validate the resolved project configuration.",
|
||||||
run: runConfig,
|
run: runConfig,
|
||||||
},
|
},
|
||||||
{
|
|
||||||
name: "music",
|
|
||||||
description: "List, scan, build, and validate configured music datasets.",
|
|
||||||
run: runMusic,
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
name: "compare",
|
name: "compare",
|
||||||
description: "Compare current source content against the built archives.",
|
description: "Compare current source content against the built archives.",
|
||||||
@@ -391,7 +385,7 @@ func refreshBuildModuleManifest(ctx context, p *project.Project, progress func(s
|
|||||||
}
|
}
|
||||||
|
|
||||||
progress("Refreshing hak list from the latest published sow-assets manifest...")
|
progress("Refreshing hak list from the latest published sow-assets manifest...")
|
||||||
if err := runProjectScript(ctx, p, []string{"scripts", "fetch-hak-manifest"}, manifestPath); err != nil {
|
if err := runProjectScript(ctx, p, []string{"scripts", "fetch-upstream-manifests"}, manifestPath); err != nil {
|
||||||
return "", "", err
|
return "", "", err
|
||||||
}
|
}
|
||||||
if _, err := pipeline.ApplyHAKManifest(p, manifestPath); err != nil {
|
if _, err := pipeline.ApplyHAKManifest(p, manifestPath); err != nil {
|
||||||
@@ -469,8 +463,7 @@ type buildHAKOptions struct {
|
|||||||
filteredHAKs []string
|
filteredHAKs []string
|
||||||
filteredArchives []string
|
filteredArchives []string
|
||||||
sourceManifest string
|
sourceManifest string
|
||||||
musicDatasets []string
|
contentAddressedRoot string
|
||||||
skipMusic bool
|
|
||||||
planOnly bool
|
planOnly bool
|
||||||
logLevel logLevel
|
logLevel logLevel
|
||||||
}
|
}
|
||||||
@@ -720,46 +713,11 @@ func (c *buildHAKConsole) emitResult(result pipeline.BuildResult) {
|
|||||||
spin.linebreak()
|
spin.linebreak()
|
||||||
fmt.Fprintln(c.stdout, "Build HAKs ----------")
|
fmt.Fprintln(c.stdout, "Build HAKs ----------")
|
||||||
if c.level != logLevelQuiet {
|
if c.level != logLevelQuiet {
|
||||||
c.emitCreditsSummary(result)
|
|
||||||
c.emitCoreSteps(result)
|
c.emitCoreSteps(result)
|
||||||
}
|
}
|
||||||
c.emitFinalSummary(result)
|
c.emitFinalSummary(result)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *buildHAKConsole) emitCreditsSummary(result pipeline.BuildResult) {
|
|
||||||
summary := result.CreditsSummary
|
|
||||||
if len(summary.ArtifactPaths) == 0 {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
fmt.Fprintln(c.stdout, "Refreshing credits cache")
|
|
||||||
if len(summary.ScannedDirs) > 0 {
|
|
||||||
fmt.Fprintf(c.stdout, " scanned: %s\n", strings.Join(summary.ScannedDirs, ", "))
|
|
||||||
}
|
|
||||||
fmt.Fprintf(c.stdout, " tracks: %d\n", summary.TrackCount)
|
|
||||||
if len(summary.GeneratedPaths) > 0 {
|
|
||||||
for _, path := range summary.GeneratedPaths {
|
|
||||||
fmt.Fprintf(c.stdout, " output: %s\n", c.relPath(path))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
switch {
|
|
||||||
case summary.ChangedFiles == 0:
|
|
||||||
fmt.Fprintln(c.stdout, " status: unchanged")
|
|
||||||
default:
|
|
||||||
fmt.Fprintf(c.stdout, " updated: %d artifact(s)\n", summary.ChangedFiles)
|
|
||||||
}
|
|
||||||
if summary.TrackCount > 0 {
|
|
||||||
if c.level >= logLevelVerbose {
|
|
||||||
fmt.Fprintln(c.stdout, " mappings:")
|
|
||||||
for _, mapping := range summary.Mappings {
|
|
||||||
fmt.Fprintf(c.stdout, " %s -> %s\n", mapping.Source, mapping.Output)
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
fmt.Fprintf(c.stdout, " mapped: %d music file(s); use --verbose to list mappings\n", summary.TrackCount)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
fmt.Fprintln(c.stdout)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (c *buildHAKConsole) emitCoreSteps(result pipeline.BuildResult) {
|
func (c *buildHAKConsole) emitCoreSteps(result pipeline.BuildResult) {
|
||||||
fmt.Fprintln(c.stdout, "Validating project")
|
fmt.Fprintln(c.stdout, "Validating project")
|
||||||
fmt.Fprintln(c.stdout, "Collecting asset resources")
|
fmt.Fprintln(c.stdout, "Collecting asset resources")
|
||||||
@@ -807,9 +765,6 @@ func (c *buildHAKConsole) emitFinalSummary(result pipeline.BuildResult) {
|
|||||||
if result.Manifest != "" {
|
if result.Manifest != "" {
|
||||||
fmt.Fprintf(c.stdout, "manifest: %s\n", c.relPath(result.Manifest))
|
fmt.Fprintf(c.stdout, "manifest: %s\n", c.relPath(result.Manifest))
|
||||||
}
|
}
|
||||||
if len(result.CreditsArtifactPaths) > 0 {
|
|
||||||
fmt.Fprintf(c.stdout, "credits artifacts: %d\n", len(result.CreditsArtifactPaths))
|
|
||||||
}
|
|
||||||
if len(result.AutogenManifestPaths) > 0 {
|
if len(result.AutogenManifestPaths) > 0 {
|
||||||
fmt.Fprintf(c.stdout, "autogen manifests: %d\n", len(result.AutogenManifestPaths))
|
fmt.Fprintf(c.stdout, "autogen manifests: %d\n", len(result.AutogenManifestPaths))
|
||||||
}
|
}
|
||||||
@@ -1048,8 +1003,7 @@ func runBuildHAKs(ctx context) error {
|
|||||||
Progress: console.progress,
|
Progress: console.progress,
|
||||||
ArchiveNames: opts.filteredArchives,
|
ArchiveNames: opts.filteredArchives,
|
||||||
SourceManifestPath: opts.sourceManifest,
|
SourceManifestPath: opts.sourceManifest,
|
||||||
SkipMusic: opts.skipMusic,
|
ContentAddressedRoot: opts.contentAddressedRoot,
|
||||||
MusicDatasetIDs: opts.musicDatasets,
|
|
||||||
}
|
}
|
||||||
if opts.planOnly {
|
if opts.planOnly {
|
||||||
result, err = pipeline.PlanHAKsWithOptions(p, pipelineOpts)
|
result, err = pipeline.PlanHAKsWithOptions(p, pipelineOpts)
|
||||||
@@ -1074,7 +1028,7 @@ func parseBuildHAKArgs(args []string) (buildHAKOptions, error) {
|
|||||||
arg := args[index]
|
arg := args[index]
|
||||||
switch arg {
|
switch arg {
|
||||||
case "-h", "--help":
|
case "-h", "--help":
|
||||||
return opts, errors.New("usage: build-haks [--hak <hak-name> ...] [--archive <archive-name> ...] [--source-manifest <path>] [--plan-only] [--skip-music] [--music-dataset <id> ...] [--quiet|--verbose|--debug]")
|
return opts, errors.New("usage: build-haks [--hak <hak-name> ...] [--archive <archive-name> ...] [--source-manifest <path>] [--content-addressed-root <path>] [--plan-only] [--quiet|--verbose|--debug]")
|
||||||
case "--hak":
|
case "--hak":
|
||||||
index++
|
index++
|
||||||
if index >= len(args) {
|
if index >= len(args) {
|
||||||
@@ -1089,14 +1043,6 @@ func parseBuildHAKArgs(args []string) (buildHAKOptions, error) {
|
|||||||
opts.filteredArchives = append(opts.filteredArchives, args[index])
|
opts.filteredArchives = append(opts.filteredArchives, args[index])
|
||||||
case "--plan-only":
|
case "--plan-only":
|
||||||
opts.planOnly = true
|
opts.planOnly = true
|
||||||
case "--skip-music":
|
|
||||||
opts.skipMusic = true
|
|
||||||
case "--music-dataset":
|
|
||||||
index++
|
|
||||||
if index >= len(args) {
|
|
||||||
return opts, errors.New("--music-dataset requires a value")
|
|
||||||
}
|
|
||||||
opts.musicDatasets = append(opts.musicDatasets, args[index])
|
|
||||||
case "--quiet":
|
case "--quiet":
|
||||||
opts.logLevel = logLevelQuiet
|
opts.logLevel = logLevelQuiet
|
||||||
case "--verbose":
|
case "--verbose":
|
||||||
@@ -1109,6 +1055,12 @@ func parseBuildHAKArgs(args []string) (buildHAKOptions, error) {
|
|||||||
return opts, errors.New("--source-manifest requires a value")
|
return opts, errors.New("--source-manifest requires a value")
|
||||||
}
|
}
|
||||||
opts.sourceManifest = args[index]
|
opts.sourceManifest = args[index]
|
||||||
|
case "--content-addressed-root":
|
||||||
|
index++
|
||||||
|
if index >= len(args) {
|
||||||
|
return opts, errors.New("--content-addressed-root requires a value")
|
||||||
|
}
|
||||||
|
opts.contentAddressedRoot = args[index]
|
||||||
default:
|
default:
|
||||||
if value, ok, err := requireInlineFlagValue(arg, "--hak"); ok || err != nil {
|
if value, ok, err := requireInlineFlagValue(arg, "--hak"); ok || err != nil {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1131,11 +1083,11 @@ func parseBuildHAKArgs(args []string) (buildHAKOptions, error) {
|
|||||||
opts.sourceManifest = value
|
opts.sourceManifest = value
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if value, ok, err := requireInlineFlagValue(arg, "--music-dataset"); ok || err != nil {
|
if value, ok, err := requireInlineFlagValue(arg, "--content-addressed-root"); ok || err != nil {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return opts, err
|
return opts, err
|
||||||
}
|
}
|
||||||
opts.musicDatasets = append(opts.musicDatasets, value)
|
opts.contentAddressedRoot = value
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if arg == "--quiet=true" {
|
if arg == "--quiet=true" {
|
||||||
@@ -1167,149 +1119,6 @@ func requireInlineFlagValue(arg, name string) (string, bool, error) {
|
|||||||
return value, true, nil
|
return value, true, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type musicCommandOptions struct {
|
|
||||||
datasets []string
|
|
||||||
json bool
|
|
||||||
dryRun bool
|
|
||||||
check bool
|
|
||||||
force bool
|
|
||||||
}
|
|
||||||
|
|
||||||
func runMusic(ctx context) error {
|
|
||||||
if len(ctx.args) < 2 {
|
|
||||||
return errors.New("usage: music <list-datasets|scan|build|credits|validate|manifest|normalize> [--dataset <id>] [--json] [--dry-run] [--check] [--force]")
|
|
||||||
}
|
|
||||||
p, err := loadProject(ctx)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if err := p.Scan(); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
subcommand := ctx.args[1]
|
|
||||||
opts, err := parseMusicCommandArgs(ctx.args[2:])
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
switch subcommand {
|
|
||||||
case "list-datasets":
|
|
||||||
return emitMusicDatasets(ctx, p, opts)
|
|
||||||
case "scan":
|
|
||||||
result, err := pipeline.BuildMusic(p, pipeline.MusicBuildOptions{DatasetIDs: opts.datasets, Write: false})
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
return emitMusicResult(ctx, "scan", result, opts)
|
|
||||||
case "build", "credits":
|
|
||||||
write := true
|
|
||||||
if subcommand == "build" && opts.dryRun {
|
|
||||||
write = false
|
|
||||||
}
|
|
||||||
if subcommand == "credits" && opts.check {
|
|
||||||
write = false
|
|
||||||
}
|
|
||||||
result, err := pipeline.BuildMusic(p, pipeline.MusicBuildOptions{DatasetIDs: opts.datasets, Write: write})
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
return emitMusicResult(ctx, subcommand, result, opts)
|
|
||||||
case "manifest", "normalize":
|
|
||||||
result, err := pipeline.BuildMusic(p, pipeline.MusicBuildOptions{DatasetIDs: opts.datasets, Write: false})
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
return emitMusicResult(ctx, subcommand, result, opts)
|
|
||||||
case "validate":
|
|
||||||
if err := p.ValidateLayout(); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
result, err := pipeline.BuildMusic(p, pipeline.MusicBuildOptions{DatasetIDs: opts.datasets, Write: false})
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
return emitMusicResult(ctx, "validate", result, opts)
|
|
||||||
default:
|
|
||||||
return fmt.Errorf("unknown music subcommand %q", subcommand)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func parseMusicCommandArgs(args []string) (musicCommandOptions, error) {
|
|
||||||
opts := musicCommandOptions{}
|
|
||||||
for index := 0; index < len(args); index++ {
|
|
||||||
arg := args[index]
|
|
||||||
switch arg {
|
|
||||||
case "--dataset":
|
|
||||||
index++
|
|
||||||
if index >= len(args) || strings.TrimSpace(args[index]) == "" {
|
|
||||||
return opts, errors.New("--dataset requires a value")
|
|
||||||
}
|
|
||||||
opts.datasets = append(opts.datasets, args[index])
|
|
||||||
case "--json":
|
|
||||||
opts.json = true
|
|
||||||
case "--dry-run":
|
|
||||||
opts.dryRun = true
|
|
||||||
case "--check":
|
|
||||||
opts.check = true
|
|
||||||
case "--force":
|
|
||||||
opts.force = true
|
|
||||||
default:
|
|
||||||
if value, ok, err := requireInlineFlagValue(arg, "--dataset"); ok || err != nil {
|
|
||||||
if err != nil {
|
|
||||||
return opts, err
|
|
||||||
}
|
|
||||||
opts.datasets = append(opts.datasets, value)
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
return opts, fmt.Errorf("unknown music argument %q", arg)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return opts, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func emitMusicDatasets(ctx context, p *project.Project, opts musicCommandOptions) error {
|
|
||||||
effective := p.EffectiveConfig()
|
|
||||||
if opts.json {
|
|
||||||
return emitConfigValue(ctx.stdout, effective.Music.Datasets, "json")
|
|
||||||
}
|
|
||||||
if len(effective.Music.Datasets) == 0 {
|
|
||||||
fmt.Fprintln(ctx.stdout, "music datasets: none")
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
fmt.Fprintln(ctx.stdout, "music datasets:")
|
|
||||||
ids := make([]string, 0, len(effective.Music.Datasets))
|
|
||||||
for id := range effective.Music.Datasets {
|
|
||||||
ids = append(ids, id)
|
|
||||||
}
|
|
||||||
sort.Strings(ids)
|
|
||||||
for _, id := range ids {
|
|
||||||
ds := effective.Music.Datasets[id]
|
|
||||||
fmt.Fprintf(ctx.stdout, " %s: source=%s output=%s prefix=%s\n", id, ds.Source, ds.Output, ds.Prefix)
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func emitMusicResult(ctx context, action string, result pipeline.BuildResult, opts musicCommandOptions) error {
|
|
||||||
if opts.json {
|
|
||||||
return emitConfigValue(ctx.stdout, result.CreditsSummary, "json")
|
|
||||||
}
|
|
||||||
fmt.Fprintf(ctx.stdout, "music %s\n", action)
|
|
||||||
if len(result.CreditsSummary.ScannedDirs) > 0 {
|
|
||||||
fmt.Fprintf(ctx.stdout, "scanned: %s\n", strings.Join(result.CreditsSummary.ScannedDirs, ", "))
|
|
||||||
}
|
|
||||||
fmt.Fprintf(ctx.stdout, "tracks: %d\n", result.CreditsSummary.TrackCount)
|
|
||||||
if action != "scan" && len(result.CreditsArtifactPaths) > 0 {
|
|
||||||
fmt.Fprintf(ctx.stdout, "credits artifacts: %d\n", len(result.CreditsArtifactPaths))
|
|
||||||
}
|
|
||||||
if len(result.CreditsSummary.Mappings) > 0 && ctx.logLevel >= logLevelVerbose {
|
|
||||||
fmt.Fprintln(ctx.stdout, "mappings:")
|
|
||||||
for _, mapping := range result.CreditsSummary.Mappings {
|
|
||||||
fmt.Fprintf(ctx.stdout, " %s -> %s\n", mapping.Source, mapping.Output)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
fmt.Fprintln(ctx.stdout, "status: ok")
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func runExtract(ctx context) error {
|
func runExtract(ctx context) error {
|
||||||
p, err := loadProject(ctx)
|
p, err := loadProject(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1605,16 +1414,20 @@ func runBuildTopData(ctx context) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
opts, err := parseBuildTopDataArgs("build-topdata", ctx.args[1:])
|
parsed, err := parseBuildTopDataArgs("build-topdata", ctx.args[1:])
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if parsed.skipLFS {
|
||||||
|
os.Setenv("CRUCIBLE_SKIP_LFS", "1") //nolint:errcheck
|
||||||
|
}
|
||||||
|
|
||||||
console := newTopdataConsole(ctx, p, "build-topdata")
|
console := newTopdataConsole(ctx, p, "build-topdata")
|
||||||
spin.configure(ctx.stderr, console.spinnerEnabled)
|
spin.configure(ctx.stderr, console.spinnerEnabled)
|
||||||
spin.start("Build Topdata: starting")
|
spin.start("Build Topdata: starting")
|
||||||
defer spin.stop()
|
defer spin.stop()
|
||||||
result, err := topdata.BuildAndPackageWithOptions(p, opts, console.progress)
|
result, err := topdata.BuildAndPackageWithOptions(p, parsed.opts, console.progress)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -1629,15 +1442,19 @@ func runBuildTopPackage(ctx context) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
opts, err := parseBuildTopDataArgs("build-top-package", ctx.args[1:])
|
parsed, err := parseBuildTopDataArgs("build-top-package", ctx.args[1:])
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
if opts.BuildWiki {
|
if parsed.opts.BuildWiki {
|
||||||
return fmt.Errorf("--wiki is not supported with build-top-package; use build-topdata when wiki generation is required")
|
return fmt.Errorf("--wiki is not supported with build-top-package; use build-topdata when wiki generation is required")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if parsed.skipLFS {
|
||||||
|
os.Setenv("CRUCIBLE_SKIP_LFS", "1") //nolint:errcheck
|
||||||
|
}
|
||||||
|
|
||||||
console := newTopdataConsole(ctx, p, "build-top-package")
|
console := newTopdataConsole(ctx, p, "build-top-package")
|
||||||
spin.configure(ctx.stderr, console.spinnerEnabled)
|
spin.configure(ctx.stderr, console.spinnerEnabled)
|
||||||
spin.start("Build Top Package: starting")
|
spin.start("Build Top Package: starting")
|
||||||
@@ -1651,21 +1468,30 @@ func runBuildTopPackage(ctx context) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func parseBuildTopDataArgs(commandName string, args []string) (topdata.BuildAndPackageOptions, error) {
|
type buildTopDataArgs struct {
|
||||||
opts := topdata.BuildAndPackageOptions{}
|
opts topdata.BuildAndPackageOptions
|
||||||
|
skipLFS bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseBuildTopDataArgs(commandName string, args []string) (buildTopDataArgs, error) {
|
||||||
|
var parsed buildTopDataArgs
|
||||||
for _, arg := range args {
|
for _, arg := range args {
|
||||||
switch arg {
|
switch arg {
|
||||||
case "--force":
|
case "--force":
|
||||||
opts.Force = true
|
parsed.opts.Force = true
|
||||||
case "--wiki":
|
case "--wiki":
|
||||||
opts.BuildWiki = true
|
parsed.opts.BuildWiki = true
|
||||||
|
case "--skip-lfs":
|
||||||
|
// ponytail: CRUCIBLE_SKIP_LFS env is the single skip mechanism; set it at
|
||||||
|
// the CLI entry point rather than carrying a dead field through BuildAndPackageOptions.
|
||||||
|
parsed.skipLFS = true
|
||||||
case "-h", "--help":
|
case "-h", "--help":
|
||||||
return opts, fmt.Errorf("usage: %s [--force] [--wiki]", commandName)
|
return parsed, fmt.Errorf("usage: %s [--force] [--wiki] [--skip-lfs]", commandName)
|
||||||
default:
|
default:
|
||||||
return opts, fmt.Errorf("unknown %s argument %q", commandName, arg)
|
return parsed, fmt.Errorf("unknown %s argument %q", commandName, arg)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return opts, nil
|
return parsed, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func runCompareTopData(ctx context) error {
|
func runCompareTopData(ctx context) error {
|
||||||
|
|||||||
+92
-207
@@ -2,6 +2,7 @@ package app
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -11,6 +12,70 @@ import (
|
|||||||
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/pipeline"
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/pipeline"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
func TestParseBuildHAKArgsContentAddressedRoot(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
args []string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "separated",
|
||||||
|
args: []string{"--content-addressed-root", "/var/cache/blobs"},
|
||||||
|
want: "/var/cache/blobs",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "inline",
|
||||||
|
args: []string{"--content-addressed-root=/var/cache/blobs"},
|
||||||
|
want: "/var/cache/blobs",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
opts, err := parseBuildHAKArgs(tt.args)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse build-haks args: %v", err)
|
||||||
|
}
|
||||||
|
if opts.contentAddressedRoot != tt.want {
|
||||||
|
t.Fatalf("content-addressed root = %q, want %q", opts.contentAddressedRoot, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseBuildHAKArgsHelpListsContentAddressedRoot(t *testing.T) {
|
||||||
|
_, err := parseBuildHAKArgs([]string{"--help"})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected help usage error")
|
||||||
|
}
|
||||||
|
for _, flag := range []string{
|
||||||
|
"--hak",
|
||||||
|
"--archive",
|
||||||
|
"--source-manifest",
|
||||||
|
"--content-addressed-root",
|
||||||
|
"--plan-only",
|
||||||
|
"--quiet",
|
||||||
|
"--verbose",
|
||||||
|
"--debug",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(err.Error(), flag) {
|
||||||
|
t.Errorf("help usage missing documented flag %q: %v", flag, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseBuildHAKArgsRejectsRemovedMusicFlags(t *testing.T) {
|
||||||
|
for _, args := range [][]string{
|
||||||
|
{"--skip-music"},
|
||||||
|
{"--music-dataset", "westgate"},
|
||||||
|
{"--music-dataset=westgate"},
|
||||||
|
} {
|
||||||
|
if _, err := parseBuildHAKArgs(args); err == nil {
|
||||||
|
t.Errorf("expected removed music arguments %v to fail", args)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunBuildTopPackageUsesCachedCompiledOutputs(t *testing.T) {
|
func TestRunBuildTopPackageUsesCachedCompiledOutputs(t *testing.T) {
|
||||||
root := t.TempDir()
|
root := t.TempDir()
|
||||||
mkdirAll(t, filepath.Join(root, "build"))
|
mkdirAll(t, filepath.Join(root, "build"))
|
||||||
@@ -43,9 +108,8 @@ topdata:
|
|||||||
setFileTime(t, filepath.Join(root, ".cache", "2da", "repadjust.2da"), outputTime)
|
setFileTime(t, filepath.Join(root, ".cache", "2da", "repadjust.2da"), outputTime)
|
||||||
setFileTime(t, filepath.Join(root, "build", "sow_tlk.tlk"), outputTime)
|
setFileTime(t, filepath.Join(root, "build", "sow_tlk.tlk"), outputTime)
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
|
||||||
ctx := context{
|
ctx := context{
|
||||||
stdout: &stdout,
|
stdout: &bytes.Buffer{},
|
||||||
stderr: &bytes.Buffer{},
|
stderr: &bytes.Buffer{},
|
||||||
cwd: root,
|
cwd: root,
|
||||||
args: []string{"build-top-package"},
|
args: []string{"build-top-package"},
|
||||||
@@ -54,19 +118,12 @@ topdata:
|
|||||||
if err := runBuildTopPackage(ctx); err != nil {
|
if err := runBuildTopPackage(ctx); err != nil {
|
||||||
t.Fatalf("runBuildTopPackage failed: %v", err)
|
t.Fatalf("runBuildTopPackage failed: %v", err)
|
||||||
}
|
}
|
||||||
output := stdout.String()
|
|
||||||
if !strings.Contains(output, "top package hak: build/sow_top.hak") {
|
|
||||||
t.Fatalf("expected build-top-package output, got %q", output)
|
|
||||||
}
|
|
||||||
if strings.Contains(output, "[build-top-package]") {
|
|
||||||
t.Fatalf("did not expect raw progress lines in normal output, got %q", output)
|
|
||||||
}
|
|
||||||
if _, err := os.Stat(filepath.Join(root, "build", "sow_top.hak")); err != nil {
|
if _, err := os.Stat(filepath.Join(root, "build", "sow_top.hak")); err != nil {
|
||||||
t.Fatalf("expected packaged hak output: %v", err)
|
t.Fatalf("expected packaged hak output: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunBuildHAKsEmitsCompactSummary(t *testing.T) {
|
func TestRunBuildHAKsPacksAuthoredBMU(t *testing.T) {
|
||||||
root := t.TempDir()
|
root := t.TempDir()
|
||||||
mkdirAll(t, filepath.Join(root, "assets", "envi", "music", "westgate"))
|
mkdirAll(t, filepath.Join(root, "assets", "envi", "music", "westgate"))
|
||||||
mkdirAll(t, filepath.Join(root, "build"))
|
mkdirAll(t, filepath.Join(root, "build"))
|
||||||
@@ -88,23 +145,7 @@ haks:
|
|||||||
include:
|
include:
|
||||||
- envi/**
|
- envi/**
|
||||||
`)
|
`)
|
||||||
writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "AleandAnecdotes.mp3"), "source-mp3")
|
writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "westgate_theme.bmu"), "authored-bmu")
|
||||||
writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "CREDITS.md"), "# NWN Music Pack Credits\n\n| Artist | Title | Output File | Original File | Album | Date | License / Copyright | Notes |\n|---|---|---|---|---|---|---|---|\n| Darren Curtis | Ale and Anecdotes | `mus_wg_andnc.bmu` | `AleandAnecdotes.mp3` | Darren's Commercially Free Music | | Darren Curtis Music | |\n")
|
|
||||||
|
|
||||||
ffprobePath := filepath.Join(root, "ffprobe")
|
|
||||||
writeFile(t, ffprobePath, "#!/bin/sh\nprintf '%s\\n' '{\"format\":{\"tags\":{\"artist\":\"Unknown\",\"title\":\"Broken Metadata\"}}}'\n")
|
|
||||||
if err := os.Chmod(ffprobePath, 0o755); err != nil {
|
|
||||||
t.Fatalf("chmod ffprobe: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
ffmpegPath := filepath.Join(root, "ffmpeg")
|
|
||||||
writeFile(t, ffmpegPath, "#!/bin/sh\nfor last; do :; done\nprintf 'bmu-data' > \"$last\"\n")
|
|
||||||
if err := os.Chmod(ffmpegPath, 0o755); err != nil {
|
|
||||||
t.Fatalf("chmod ffmpeg: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Setenv("SOW_FFMPEG", ffmpegPath)
|
|
||||||
t.Setenv("SOW_FFPROBE", ffprobePath)
|
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
var stdout bytes.Buffer
|
||||||
ctx := context{
|
ctx := context{
|
||||||
@@ -118,18 +159,12 @@ haks:
|
|||||||
t.Fatalf("runBuildHAKs failed: %v", err)
|
t.Fatalf("runBuildHAKs failed: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
output := stdout.String()
|
manifest, err := os.ReadFile(filepath.Join(root, "build", "haks.json"))
|
||||||
if !strings.Contains(output, "Build HAKs ----------") {
|
if err != nil {
|
||||||
t.Fatalf("expected build header, got %q", output)
|
t.Fatalf("read HAK manifest: %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(output, "mapped: 1 music file(s); use --verbose to list mappings") {
|
if !strings.Contains(string(manifest), "envi/music/westgate/westgate_theme.bmu") {
|
||||||
t.Fatalf("expected compact mapping summary, got %q", output)
|
t.Fatalf("authored BMU missing from HAK manifest:\n%s", manifest)
|
||||||
}
|
|
||||||
if strings.Contains(output, "AleandAnecdotes.mp3 -> mus_wg_andnc.bmu") {
|
|
||||||
t.Fatalf("did not expect verbose mapping in normal mode, got %q", output)
|
|
||||||
}
|
|
||||||
if !strings.Contains(output, "manifest: build/haks.json") {
|
|
||||||
t.Fatalf("expected relative manifest path, got %q", output)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -146,70 +181,6 @@ func setTreeTime(t *testing.T, root string, modTime time.Time) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunBuildHAKsVerboseListsMappings(t *testing.T) {
|
|
||||||
root := t.TempDir()
|
|
||||||
mkdirAll(t, filepath.Join(root, "assets", "envi", "music", "westgate"))
|
|
||||||
mkdirAll(t, filepath.Join(root, "build"))
|
|
||||||
mkdirAll(t, filepath.Join(root, "src"))
|
|
||||||
|
|
||||||
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
|
|
||||||
module:
|
|
||||||
name: Test Module
|
|
||||||
resref: testmod
|
|
||||||
paths:
|
|
||||||
source: src
|
|
||||||
assets: assets
|
|
||||||
build: build
|
|
||||||
haks:
|
|
||||||
- name: envi
|
|
||||||
priority: 1
|
|
||||||
max_bytes: 1048576
|
|
||||||
split: false
|
|
||||||
include:
|
|
||||||
- envi/**
|
|
||||||
`)
|
|
||||||
writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "AleandAnecdotes.mp3"), "source-mp3")
|
|
||||||
writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "CREDITS.md"), "# NWN Music Pack Credits\n\n| Artist | Title | Output File | Original File | Album | Date | License / Copyright | Notes |\n|---|---|---|---|---|---|---|---|\n| Darren Curtis | Ale and Anecdotes | `mus_wg_andnc.bmu` | `AleandAnecdotes.mp3` | Darren's Commercially Free Music | | Darren Curtis Music | |\n")
|
|
||||||
|
|
||||||
ffprobePath := filepath.Join(root, "ffprobe")
|
|
||||||
writeFile(t, ffprobePath, "#!/bin/sh\nprintf '%s\\n' '{\"format\":{\"tags\":{\"artist\":\"Unknown\",\"title\":\"Broken Metadata\"}}}'\n")
|
|
||||||
if err := os.Chmod(ffprobePath, 0o755); err != nil {
|
|
||||||
t.Fatalf("chmod ffprobe: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
ffmpegPath := filepath.Join(root, "ffmpeg")
|
|
||||||
writeFile(t, ffmpegPath, "#!/bin/sh\nfor last; do :; done\nprintf 'bmu-data' > \"$last\"\n")
|
|
||||||
if err := os.Chmod(ffmpegPath, 0o755); err != nil {
|
|
||||||
t.Fatalf("chmod ffmpeg: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Setenv("SOW_FFMPEG", ffmpegPath)
|
|
||||||
t.Setenv("SOW_FFPROBE", ffprobePath)
|
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
|
||||||
ctx := context{
|
|
||||||
stdout: &stdout,
|
|
||||||
stderr: &bytes.Buffer{},
|
|
||||||
cwd: root,
|
|
||||||
args: []string{"build-haks", "--verbose"},
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := runBuildHAKs(ctx); err != nil {
|
|
||||||
t.Fatalf("runBuildHAKs failed: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
output := stdout.String()
|
|
||||||
if !strings.Contains(output, "mappings:") {
|
|
||||||
t.Fatalf("expected verbose mappings header, got %q", output)
|
|
||||||
}
|
|
||||||
if !strings.Contains(output, "AleandAnecdotes.mp3 -> mus_wg_andnc.bmu") {
|
|
||||||
t.Fatalf("expected verbose mapping output, got %q", output)
|
|
||||||
}
|
|
||||||
if !strings.Contains(output, "wrote: envi (1 assets)") {
|
|
||||||
t.Fatalf("expected verbose archive action, got %q", output)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTopdataConsoleSuppressesProgressInNormalMode(t *testing.T) {
|
func TestTopdataConsoleSuppressesProgressInNormalMode(t *testing.T) {
|
||||||
var stdout bytes.Buffer
|
var stdout bytes.Buffer
|
||||||
console := &topdataConsole{
|
console := &topdataConsole{
|
||||||
@@ -250,16 +221,10 @@ func TestTopdataConsoleDebugProgressAndRelativePaths(t *testing.T) {
|
|||||||
console.emitWikiDeployResult(10, 1, 2, 3, 4, 5, 6, 0, "/workspace/project/build/wiki/deploy-manifest.json")
|
console.emitWikiDeployResult(10, 1, 2, 3, 4, 5, 6, 0, "/workspace/project/build/wiki/deploy-manifest.json")
|
||||||
|
|
||||||
output := stdout.String()
|
output := stdout.String()
|
||||||
if !strings.Contains(output, "[debug] NodeBB wiki plan: create 1, update 2, skip 3, stale 4, archive 5, purge 6, drift 0") {
|
if !strings.Contains(output, "NodeBB wiki plan") {
|
||||||
t.Fatalf("expected debug progress line, got %q", output)
|
t.Fatalf("expected debug progress line, got %q", output)
|
||||||
}
|
}
|
||||||
if !strings.Contains(output, "archived: 5") {
|
if !strings.Contains(output, "build/wiki/deploy-manifest.json") || strings.Contains(output, "/workspace/project/") {
|
||||||
t.Fatalf("expected archived deploy count, got %q", output)
|
|
||||||
}
|
|
||||||
if !strings.Contains(output, "purged: 6") {
|
|
||||||
t.Fatalf("expected purged deploy count, got %q", output)
|
|
||||||
}
|
|
||||||
if !strings.Contains(output, "manifest: build/wiki/deploy-manifest.json") {
|
|
||||||
t.Fatalf("expected relative deploy manifest path, got %q", output)
|
t.Fatalf("expected relative deploy manifest path, got %q", output)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -317,85 +282,11 @@ func TestProjectConsoleEmitsRelativePaths(t *testing.T) {
|
|||||||
})
|
})
|
||||||
|
|
||||||
output := stdout.String()
|
output := stdout.String()
|
||||||
if !strings.Contains(output, "module: build/test.mod") {
|
if !strings.Contains(output, "build/test.mod") || strings.Contains(output, "/workspace/project/") {
|
||||||
t.Fatalf("expected relative module path, got %q", output)
|
t.Fatalf("expected relative module path, got %q", output)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunMusicScanUsesConfiguredDatasetsWithoutWriting(t *testing.T) {
|
|
||||||
root := t.TempDir()
|
|
||||||
mkdirAll(t, filepath.Join(root, "assets", "audio", "westgate"))
|
|
||||||
mkdirAll(t, filepath.Join(root, "build"))
|
|
||||||
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
|
|
||||||
module:
|
|
||||||
name: Test Module
|
|
||||||
resref: testmod
|
|
||||||
paths:
|
|
||||||
assets: assets
|
|
||||||
build: build
|
|
||||||
music:
|
|
||||||
datasets:
|
|
||||||
westgate_audio:
|
|
||||||
source: audio/westgate
|
|
||||||
output: generated/music
|
|
||||||
prefix: wg_
|
|
||||||
`)
|
|
||||||
writeFile(t, filepath.Join(root, "assets", "audio", "westgate", "Theme Song.mp3"), "source-mp3")
|
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
|
||||||
ctx := context{
|
|
||||||
stdout: &stdout,
|
|
||||||
stderr: &bytes.Buffer{},
|
|
||||||
cwd: root,
|
|
||||||
args: []string{"music", "scan", "--dataset", "westgate_audio"},
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := runMusic(ctx); err != nil {
|
|
||||||
t.Fatalf("runMusic failed: %v", err)
|
|
||||||
}
|
|
||||||
output := stdout.String()
|
|
||||||
if !strings.Contains(output, "music scan") || !strings.Contains(output, "tracks: 1") {
|
|
||||||
t.Fatalf("unexpected music scan output: %q", output)
|
|
||||||
}
|
|
||||||
if _, err := os.Stat(filepath.Join(root, ".cache", "credits")); !os.IsNotExist(err) {
|
|
||||||
t.Fatalf("music scan should not write credits artifacts, stat err=%v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRunMusicListDatasets(t *testing.T) {
|
|
||||||
root := t.TempDir()
|
|
||||||
mkdirAll(t, filepath.Join(root, "assets"))
|
|
||||||
mkdirAll(t, filepath.Join(root, "build"))
|
|
||||||
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
|
|
||||||
module:
|
|
||||||
name: Test Module
|
|
||||||
resref: testmod
|
|
||||||
paths:
|
|
||||||
assets: assets
|
|
||||||
music:
|
|
||||||
datasets:
|
|
||||||
westgate_audio:
|
|
||||||
source: audio/westgate
|
|
||||||
output: generated/music
|
|
||||||
prefix: wg_
|
|
||||||
`)
|
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
|
||||||
ctx := context{
|
|
||||||
stdout: &stdout,
|
|
||||||
stderr: &bytes.Buffer{},
|
|
||||||
cwd: root,
|
|
||||||
args: []string{"music", "list-datasets"},
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := runMusic(ctx); err != nil {
|
|
||||||
t.Fatalf("runMusic failed: %v", err)
|
|
||||||
}
|
|
||||||
if !strings.Contains(stdout.String(), "westgate_audio: source=audio/westgate output=generated/music prefix=wg_") {
|
|
||||||
t.Fatalf("unexpected dataset list: %q", stdout.String())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRunConfigEffectiveJSONShowsDefaultsAndProvenance(t *testing.T) {
|
func TestRunConfigEffectiveJSONShowsDefaultsAndProvenance(t *testing.T) {
|
||||||
root := t.TempDir()
|
root := t.TempDir()
|
||||||
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
|
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
|
||||||
@@ -417,12 +308,16 @@ paths:
|
|||||||
if err := runConfig(ctx); err != nil {
|
if err := runConfig(ctx); err != nil {
|
||||||
t.Fatalf("runConfig failed: %v", err)
|
t.Fatalf("runConfig failed: %v", err)
|
||||||
}
|
}
|
||||||
output := stdout.String()
|
var effective map[string]any
|
||||||
if !strings.Contains(output, `"hak_manifest": "haks.json"`) {
|
if err := json.Unmarshal(stdout.Bytes(), &effective); err != nil {
|
||||||
t.Fatalf("expected HAK manifest default in effective config, got %q", output)
|
t.Fatalf("effective config is not JSON: %v\n%s", err, stdout.String())
|
||||||
}
|
}
|
||||||
if !strings.Contains(output, `"paths.build":`) || !strings.Contains(output, `"toolkit default"`) {
|
provenance, ok := effective["provenance"].(map[string]any)
|
||||||
t.Fatalf("expected default provenance in effective config, got %q", output)
|
if !ok || len(provenance) == 0 {
|
||||||
|
t.Fatalf("effective config missing provenance: %#v", effective["provenance"])
|
||||||
|
}
|
||||||
|
if _, ok := provenance["paths.build"]; !ok {
|
||||||
|
t.Fatalf("effective config missing provenance for omitted paths.build: %#v", provenance)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -448,10 +343,10 @@ paths:
|
|||||||
t.Fatalf("runConfig failed: %v", err)
|
t.Fatalf("runConfig failed: %v", err)
|
||||||
}
|
}
|
||||||
output := stdout.String()
|
output := stdout.String()
|
||||||
if !strings.Contains(output, "value: \"output\"") {
|
if !strings.Contains(output, "output") {
|
||||||
t.Fatalf("expected configured build value, got %q", output)
|
t.Fatalf("expected configured build value, got %q", output)
|
||||||
}
|
}
|
||||||
if !strings.Contains(output, "source: yaml") {
|
if !strings.Contains(strings.ToLower(output), "yaml") {
|
||||||
t.Fatalf("expected YAML source, got %q", output)
|
t.Fatalf("expected YAML source, got %q", output)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -468,9 +363,8 @@ paths:
|
|||||||
build: build
|
build: build
|
||||||
`)
|
`)
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
|
||||||
ctx := context{
|
ctx := context{
|
||||||
stdout: &stdout,
|
stdout: &bytes.Buffer{},
|
||||||
stderr: &bytes.Buffer{},
|
stderr: &bytes.Buffer{},
|
||||||
cwd: root,
|
cwd: root,
|
||||||
args: []string{"config", "validate"},
|
args: []string{"config", "validate"},
|
||||||
@@ -479,9 +373,6 @@ paths:
|
|||||||
if err := runConfig(ctx); err != nil {
|
if err := runConfig(ctx); err != nil {
|
||||||
t.Fatalf("runConfig failed: %v", err)
|
t.Fatalf("runConfig failed: %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(stdout.String(), "config: ok") {
|
|
||||||
t.Fatalf("expected config validation output, got %q", stdout.String())
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunConfigSourcesListsActiveOverrides(t *testing.T) {
|
func TestRunConfigSourcesListsActiveOverrides(t *testing.T) {
|
||||||
@@ -507,25 +398,19 @@ paths:
|
|||||||
t.Fatalf("runConfig failed: %v", err)
|
t.Fatalf("runConfig failed: %v", err)
|
||||||
}
|
}
|
||||||
output := stdout.String()
|
output := stdout.String()
|
||||||
if !strings.Contains(output, "active overrides:") {
|
if !strings.Contains(output, "build.keep_existing_haks") {
|
||||||
t.Fatalf("expected active overrides section, got %q", output)
|
|
||||||
}
|
|
||||||
if !strings.Contains(output, "build.keep_existing_haks=1") {
|
|
||||||
t.Fatalf("expected keep existing override, got %q", output)
|
t.Fatalf("expected keep existing override, got %q", output)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestInlineFlagParsersRejectEmptyValues(t *testing.T) {
|
func TestInlineFlagParsersRejectEmptyValues(t *testing.T) {
|
||||||
if _, err := parseBuildHAKArgs([]string{"--hak="}); err == nil || !strings.Contains(err.Error(), "--hak requires a value") {
|
if _, err := parseBuildHAKArgs([]string{"--hak="}); err == nil {
|
||||||
t.Fatalf("expected empty --hak inline value error, got %v", err)
|
t.Fatalf("expected empty --hak inline value error, got %v", err)
|
||||||
}
|
}
|
||||||
if _, err := parseMusicCommandArgs([]string{"--dataset="}); err == nil || !strings.Contains(err.Error(), "--dataset requires a value") {
|
if _, err := parseDeployWikiArgs("deploy-wiki", []string{"--endpoint="}); err == nil {
|
||||||
t.Fatalf("expected empty --dataset inline value error, got %v", err)
|
|
||||||
}
|
|
||||||
if _, err := parseDeployWikiArgs("deploy-wiki", []string{"--endpoint="}); err == nil || !strings.Contains(err.Error(), "--endpoint requires a value") {
|
|
||||||
t.Fatalf("expected empty --endpoint inline value error, got %v", err)
|
t.Fatalf("expected empty --endpoint inline value error, got %v", err)
|
||||||
}
|
}
|
||||||
if _, err := parseBuildChangelogArgs("build-changelog", []string{"--output="}); err == nil || !strings.Contains(err.Error(), "--output requires a value") {
|
if _, err := parseBuildChangelogArgs("build-changelog", []string{"--output="}); err == nil {
|
||||||
t.Fatalf("expected empty --output inline value error, got %v", err)
|
t.Fatalf("expected empty --output inline value error, got %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,67 @@
|
|||||||
|
package assets
|
||||||
|
|
||||||
|
import (
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
|
||||||
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/assets/mdl"
|
||||||
|
)
|
||||||
|
|
||||||
|
// mdlExt is the file set shared by the mdl commands.
|
||||||
|
var mdlExt = map[string]bool{".mdl": true}
|
||||||
|
|
||||||
|
// runCheckMDL reports uncompiled ASCII .mdl files and model-name mismatches.
|
||||||
|
// Read-only. Exit exitFail if any problem is found.
|
||||||
|
func runCheckMDL(args []string, stdout, stderr io.Writer) int {
|
||||||
|
fs := flag.NewFlagSet("check-mdl", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(stderr)
|
||||||
|
nonRecursive := fs.Bool("non-recursive", false, "do not descend into subdirectories")
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
paths := fs.Args()
|
||||||
|
if len(paths) == 0 {
|
||||||
|
fmt.Fprintln(stderr, "assets check-mdl: usage: check-mdl <path>...")
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
|
files, err := walk(paths, mdlExt, !*nonRecursive)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "assets check-mdl:", err)
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
|
fail := false
|
||||||
|
for _, f := range files {
|
||||||
|
ascii, err := mdl.IsASCII(f)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "assets check-mdl:", err)
|
||||||
|
return exitTool
|
||||||
|
}
|
||||||
|
if ascii {
|
||||||
|
fmt.Fprintf(stderr, "uncompiled ascii mdl: %s\n", f)
|
||||||
|
fail = true
|
||||||
|
}
|
||||||
|
mismatches, err := mdl.CheckNames(f)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "assets check-mdl:", err)
|
||||||
|
return exitTool
|
||||||
|
}
|
||||||
|
expected := mdl.ExpectedName(f)
|
||||||
|
for _, m := range mismatches {
|
||||||
|
if m.Line == 0 {
|
||||||
|
fmt.Fprintf(stderr, "%s: mdl model-name mismatch: root node is %s, expected %s\n", f, m.Got, expected)
|
||||||
|
} else {
|
||||||
|
fmt.Fprintf(stderr, "%s: mdl model-name mismatch: line %d: %s is %s, expected %s\n", f, m.Line, m.What, m.Got, expected)
|
||||||
|
}
|
||||||
|
fail = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if fail {
|
||||||
|
fmt.Fprintln(stderr, "check-mdl: found broken or uncompiled .mdl file(s)")
|
||||||
|
return exitFail
|
||||||
|
}
|
||||||
|
fmt.Fprintln(stdout, "check-mdl: OK")
|
||||||
|
return exitOK
|
||||||
|
}
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
package assets
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestCheckMDL(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
// Uncompiled ASCII model with a name mismatch.
|
||||||
|
bad := filepath.Join(dir, "foo.mdl")
|
||||||
|
if err := os.WriteFile(bad, []byte("newmodel wrong\nbeginmodelgeom wrong\nendmodelgeom wrong\ndonemodel wrong\n"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var out, errw bytes.Buffer
|
||||||
|
if code := runCheckMDL([]string{dir}, &out, &errw); code != exitFail {
|
||||||
|
t.Fatalf("bad mdl exit = %d, want %d\n%s", code, exitFail, errw.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
// A binary (compiled) model is fine.
|
||||||
|
good := t.TempDir()
|
||||||
|
if err := os.WriteFile(filepath.Join(good, "bar.mdl"), []byte("\x00\x00compiled binary blob"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
out.Reset()
|
||||||
|
errw.Reset()
|
||||||
|
if code := runCheckMDL([]string{good}, &out, &errw); code != exitOK {
|
||||||
|
t.Fatalf("binary mdl exit = %d, want %d\n%s", code, exitOK, errw.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,190 @@
|
|||||||
|
package assets
|
||||||
|
|
||||||
|
import (
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/assets/mdl"
|
||||||
|
)
|
||||||
|
|
||||||
|
// runCompile compiles ASCII .mdl models to binary in place, one at a time (the
|
||||||
|
// engine's development/ and modelcompiler/ folders are flat, single-slot).
|
||||||
|
func runCompile(args []string, stdout, stderr io.Writer, getenv func(string) string) int {
|
||||||
|
fs := flag.NewFlagSet("compile", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(stderr)
|
||||||
|
nwn := fs.String("nwn", "", "path to the NWN install root or nwmain-linux binary")
|
||||||
|
nonRecursive := fs.Bool("non-recursive", false, "do not descend into subdirectories")
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
dirs := fs.Args()
|
||||||
|
if len(dirs) == 0 {
|
||||||
|
fmt.Fprintln(stderr, "assets compile: usage: compile [--nwn INSTALL] <dir>...")
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
|
home := getenv("HOME")
|
||||||
|
userData := filepath.Join(home, ".local", "share", "Neverwinter Nights")
|
||||||
|
dev := filepath.Join(userData, "development")
|
||||||
|
mc := filepath.Join(userData, "modelcompiler")
|
||||||
|
|
||||||
|
nwmain := findNWMain(*nwn, home)
|
||||||
|
if nwmain == "" {
|
||||||
|
fmt.Fprintln(stderr, "assets compile: nwmain-linux not found — pass --nwn <install> "+
|
||||||
|
"(Steam/GOG/Beamdog install root or the nwmain-linux binary)")
|
||||||
|
return exitTool
|
||||||
|
}
|
||||||
|
binDir := filepath.Dir(nwmain)
|
||||||
|
|
||||||
|
// Always use a virtual X so compilation never opens the client UI.
|
||||||
|
xvfb := look("xvfb-run")
|
||||||
|
if xvfb == "" {
|
||||||
|
fmt.Fprintln(stderr, "assets compile: xvfb-run not found — install it to run the NWN model compiler headlessly")
|
||||||
|
return exitTool
|
||||||
|
}
|
||||||
|
wrap := []string{xvfb, "-a", "--server-args=-screen 0 1024x768x24"}
|
||||||
|
|
||||||
|
files, err := walk(dirs, mdlExt, !*nonRecursive)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "assets compile:", err)
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
|
failed := false
|
||||||
|
for _, src := range files {
|
||||||
|
if err := compileOne(src, binDir, nwmain, dev, mc, wrap, userData, stdout, stderr); err != nil {
|
||||||
|
fmt.Fprintf(stderr, "assets compile: %s: %v\n", src, err)
|
||||||
|
failed = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if failed {
|
||||||
|
return exitFail
|
||||||
|
}
|
||||||
|
return exitOK
|
||||||
|
}
|
||||||
|
|
||||||
|
// findNWMain resolves the nwmain-linux binary from --nwn or standard roots.
|
||||||
|
func findNWMain(override, home string) string {
|
||||||
|
if override != "" {
|
||||||
|
if fi, err := os.Stat(override); err == nil && !fi.IsDir() {
|
||||||
|
return override
|
||||||
|
}
|
||||||
|
cand := filepath.Join(override, "bin", "linux-x86", "nwmain-linux")
|
||||||
|
if fi, err := os.Stat(cand); err == nil && !fi.IsDir() {
|
||||||
|
return cand
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return look(
|
||||||
|
filepath.Join(home, ".local/share/Steam/steamapps/common/Neverwinter Nights/bin/linux-x86/nwmain-linux"),
|
||||||
|
filepath.Join(home, "GOG Games/Neverwinter Nights Enhanced Edition/game/bin/linux-x86/nwmain-linux"),
|
||||||
|
filepath.Join(home, ".steam/steam/steamapps/common/Neverwinter Nights/bin/linux-x86/nwmain-linux"),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// compileOne compiles a single model. Binary models are reported and skipped
|
||||||
|
// (not an error). A name-mismatched model is aborted with guidance.
|
||||||
|
func compileOne(src, binDir, nwmain, dev, mc string, wrap []string, userData string, stdout, stderr io.Writer) error {
|
||||||
|
ascii, err := mdl.IsASCII(src)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !ascii {
|
||||||
|
fmt.Fprintf(stdout, "skip (already compiled): %s\n", src)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
mismatches, err := mdl.CheckNames(src)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if len(mismatches) > 0 {
|
||||||
|
return fmt.Errorf("model names differ from the file stem; run `crucible assets fix-mdl` first")
|
||||||
|
}
|
||||||
|
|
||||||
|
stem := mdl.ExpectedNameStem(src)
|
||||||
|
name := strings.ToLower(filepath.Base(src))
|
||||||
|
devFile := filepath.Join(dev, name)
|
||||||
|
// Collision guard: a pre-existing slot aborts before any mutation.
|
||||||
|
if _, err := os.Stat(devFile); err == nil {
|
||||||
|
return fmt.Errorf("development slot already occupied: %s", devFile)
|
||||||
|
}
|
||||||
|
|
||||||
|
data, err := os.ReadFile(src)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(devFile, data, 0o644); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer os.Remove(devFile)
|
||||||
|
|
||||||
|
// Run the engine with cwd = the binary dir.
|
||||||
|
cmd := append(append([]string{}, wrap...), nwmain, "compilemodel", stem)
|
||||||
|
if out, err := runner(binDir, nil, cmd[0], cmd[1:]...); err != nil {
|
||||||
|
printEngineLog(userData, stderr)
|
||||||
|
return fmt.Errorf("engine: %v: %s", err, strings.TrimSpace(string(out)))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Collect the compiled artifact from modelcompiler/ (match by stem, any case).
|
||||||
|
compiled, err := findCompiled(mc, stem)
|
||||||
|
if err != nil {
|
||||||
|
printEngineLog(userData, stderr)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer os.Remove(compiled)
|
||||||
|
|
||||||
|
out, err := os.ReadFile(compiled)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
// Move it back over the source path, lowercased.
|
||||||
|
dst := filepath.Join(filepath.Dir(src), name)
|
||||||
|
if err := os.WriteFile(dst, out, 0o644); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if dst != src {
|
||||||
|
_ = os.Remove(src)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(stdout, "compiled: %s\n", dst)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// findCompiled returns the modelcompiler/ artifact whose stem matches (case-
|
||||||
|
// insensitively).
|
||||||
|
func findCompiled(mc, stem string) (string, error) {
|
||||||
|
entries, err := os.ReadDir(mc)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
want := strings.ToLower(stem) + ".mdl"
|
||||||
|
for _, e := range entries {
|
||||||
|
if e.IsDir() {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if strings.ToLower(e.Name()) == want {
|
||||||
|
return filepath.Join(mc, e.Name()), nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "", fmt.Errorf("engine produced no compiled model for %q", stem)
|
||||||
|
}
|
||||||
|
|
||||||
|
// printEngineLog appends a short tail of the engine log as advisory diagnostics.
|
||||||
|
func printEngineLog(userData string, stderr io.Writer) {
|
||||||
|
log := filepath.Join(userData, "logs", "nwengineLog.txt")
|
||||||
|
data, err := os.ReadFile(log)
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
lines := strings.Split(strings.TrimRight(string(data), "\n"), "\n")
|
||||||
|
if len(lines) > 8 {
|
||||||
|
lines = lines[len(lines)-8:]
|
||||||
|
}
|
||||||
|
fmt.Fprintf(stderr, " engine log tail (%s):\n", log)
|
||||||
|
for _, l := range lines {
|
||||||
|
fmt.Fprintf(stderr, " %s\n", l)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,171 @@
|
|||||||
|
package assets
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestCompileDrivesEngineAndReplacesInPlace(t *testing.T) {
|
||||||
|
home := t.TempDir()
|
||||||
|
userData := filepath.Join(home, ".local", "share", "Neverwinter Nights")
|
||||||
|
dev := filepath.Join(userData, "development")
|
||||||
|
mc := filepath.Join(userData, "modelcompiler")
|
||||||
|
for _, d := range []string{dev, mc} {
|
||||||
|
if err := os.MkdirAll(d, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Fake nwmain binary for discovery via --nwn.
|
||||||
|
binDir := t.TempDir()
|
||||||
|
nwmain := filepath.Join(binDir, "nwmain-linux")
|
||||||
|
if err := os.WriteFile(nwmain, []byte("#!/bin/sh\n"), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
xvfbDir := t.TempDir()
|
||||||
|
xvfb := filepath.Join(xvfbDir, "xvfb-run")
|
||||||
|
if err := os.WriteFile(xvfb, []byte("#!/bin/sh\n"), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Setenv("PATH", xvfbDir+string(os.PathListSeparator)+os.Getenv("PATH"))
|
||||||
|
|
||||||
|
getenv := func(k string) string {
|
||||||
|
switch k {
|
||||||
|
case "HOME":
|
||||||
|
return home
|
||||||
|
case "DISPLAY":
|
||||||
|
return ":0" // a desktop display must not make compilation interactive
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stub the engine: writes a binary compiled model into modelcompiler/.
|
||||||
|
orig := runner
|
||||||
|
defer func() { runner = orig }()
|
||||||
|
runner = func(dir string, env []string, name string, args ...string) ([]byte, error) {
|
||||||
|
if name != xvfb || len(args) != 5 || args[0] != "-a" ||
|
||||||
|
args[1] != "--server-args=-screen 0 1024x768x24" || args[2] != nwmain ||
|
||||||
|
args[3] != "compilemodel" {
|
||||||
|
t.Fatalf("engine command = %q %q, want xvfb-run wrapping nwmain", name, args)
|
||||||
|
}
|
||||||
|
stem := args[len(args)-1]
|
||||||
|
compiled := filepath.Join(mc, stem+".mdl")
|
||||||
|
return nil, os.WriteFile(compiled, []byte("\x00\x00compiled"), 0o644)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Source tree with one ASCII model whose names already match its stem.
|
||||||
|
srcDir := t.TempDir()
|
||||||
|
src := filepath.Join(srcDir, "foo.mdl")
|
||||||
|
body := "newmodel foo\nbeginmodelgeom foo\n node dummy foo\n parent null\n endnode\nendmodelgeom foo\ndonemodel foo\n"
|
||||||
|
if err := os.WriteFile(src, []byte(body), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
code := runCompile([]string{"--nwn", nwmain, srcDir}, &stdout, &stderr, getenv)
|
||||||
|
if code != exitOK {
|
||||||
|
t.Fatalf("compile exit = %d\n%s", code, stderr.String())
|
||||||
|
}
|
||||||
|
// The source is now binary (the compiled artifact moved back over it).
|
||||||
|
data, err := os.ReadFile(src)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("compiled model missing: %v", err)
|
||||||
|
}
|
||||||
|
if string(data) != "\x00\x00compiled" {
|
||||||
|
t.Fatalf("source not replaced by compiled binary: %q", data)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileAbortsOnNameMismatch(t *testing.T) {
|
||||||
|
home := t.TempDir()
|
||||||
|
if err := os.MkdirAll(filepath.Join(home, ".local", "share", "Neverwinter Nights", "development"), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
binDir := t.TempDir()
|
||||||
|
nwmain := filepath.Join(binDir, "nwmain-linux")
|
||||||
|
if err := os.WriteFile(nwmain, []byte("#!/bin/sh\n"), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
xvfbDir := t.TempDir()
|
||||||
|
if err := os.WriteFile(filepath.Join(xvfbDir, "xvfb-run"), []byte("#!/bin/sh\n"), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Setenv("PATH", xvfbDir+string(os.PathListSeparator)+os.Getenv("PATH"))
|
||||||
|
getenv := func(k string) string {
|
||||||
|
if k == "HOME" {
|
||||||
|
return home
|
||||||
|
}
|
||||||
|
if k == "DISPLAY" {
|
||||||
|
return ":0"
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
orig := runner
|
||||||
|
defer func() { runner = orig }()
|
||||||
|
engineCalled := false
|
||||||
|
runner = func(string, []string, string, ...string) ([]byte, error) {
|
||||||
|
engineCalled = true
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
srcDir := t.TempDir()
|
||||||
|
// Internal name "wrong" != stem "foo": must abort before touching the engine.
|
||||||
|
if err := os.WriteFile(filepath.Join(srcDir, "foo.mdl"),
|
||||||
|
[]byte("newmodel wrong\ndonemodel wrong\n"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
if code := runCompile([]string{"--nwn", nwmain, srcDir}, &stdout, &stderr, getenv); code != exitFail {
|
||||||
|
t.Fatalf("mismatch exit = %d, want %d", code, exitFail)
|
||||||
|
}
|
||||||
|
if engineCalled {
|
||||||
|
t.Fatal("engine should not run for a name-mismatched model")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileFailsClosedWithoutXvfb(t *testing.T) {
|
||||||
|
home := t.TempDir()
|
||||||
|
userData := filepath.Join(home, ".local", "share", "Neverwinter Nights")
|
||||||
|
for _, d := range []string{filepath.Join(userData, "development"), filepath.Join(userData, "modelcompiler")} {
|
||||||
|
if err := os.MkdirAll(d, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
nwmain := filepath.Join(t.TempDir(), "nwmain-linux")
|
||||||
|
if err := os.WriteFile(nwmain, []byte("#!/bin/sh\n"), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Setenv("PATH", t.TempDir())
|
||||||
|
|
||||||
|
getenv := func(k string) string {
|
||||||
|
if k == "HOME" {
|
||||||
|
return home
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
orig := runner
|
||||||
|
defer func() { runner = orig }()
|
||||||
|
engineCalled := false
|
||||||
|
runner = func(string, []string, string, ...string) ([]byte, error) {
|
||||||
|
engineCalled = true
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
srcDir := t.TempDir()
|
||||||
|
if err := os.WriteFile(filepath.Join(srcDir, "foo.mdl"),
|
||||||
|
[]byte("newmodel foo\nbeginmodelgeom foo\n node dummy foo\n parent null\n endnode\nendmodelgeom foo\ndonemodel foo\n"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
if code := runCompile([]string{"--nwn", nwmain, srcDir}, &stdout, &stderr, getenv); code != exitTool {
|
||||||
|
t.Fatalf("compile exit = %d, want %d\n%s", code, exitTool, stderr.String())
|
||||||
|
}
|
||||||
|
if engineCalled {
|
||||||
|
t.Fatal("engine must not run without xvfb-run")
|
||||||
|
}
|
||||||
|
if !strings.Contains(stderr.String(), "xvfb-run") {
|
||||||
|
t.Fatalf("missing actionable xvfb-run error: %s", stderr.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,116 @@
|
|||||||
|
package assets
|
||||||
|
|
||||||
|
import (
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
var textureExts = map[string]bool{".dds": true, ".png": true, ".tga": true}
|
||||||
|
|
||||||
|
// runConvert converts textures in place, flipping vertically exactly once per
|
||||||
|
// conversion. --to defaults to dds. Files already in the target format are
|
||||||
|
// skipped.
|
||||||
|
func runConvert(args []string, stdout, stderr io.Writer) int {
|
||||||
|
fs := flag.NewFlagSet("convert", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(stderr)
|
||||||
|
to := fs.String("to", "dds", "target format: dds|png|tga")
|
||||||
|
backend := fs.String("backend", "magick", "ImageMagick-compatible binary")
|
||||||
|
nonRecursive := fs.Bool("non-recursive", false, "do not descend into subdirectories")
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
target := strings.ToLower(*to)
|
||||||
|
if target != "dds" && target != "png" && target != "tga" {
|
||||||
|
fmt.Fprintln(stderr, "assets convert: --to must be dds, png, or tga")
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
dirs := fs.Args()
|
||||||
|
if len(dirs) == 0 {
|
||||||
|
fmt.Fprintln(stderr, "assets convert: usage: convert [--to dds|png|tga] <dir>...")
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
|
magick := look(*backend)
|
||||||
|
if magick == "" {
|
||||||
|
fmt.Fprintf(stderr, "assets convert: backend %q not found — install ImageMagick (magick)\n", *backend)
|
||||||
|
return exitTool
|
||||||
|
}
|
||||||
|
|
||||||
|
files, err := walk(dirs, textureExts, !*nonRecursive)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "assets convert:", err)
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
|
targetExt := "." + target
|
||||||
|
failed := false
|
||||||
|
for _, src := range files {
|
||||||
|
srcExt := strings.ToLower(filepath.Ext(src))
|
||||||
|
if srcExt == targetExt {
|
||||||
|
continue // already in the target format
|
||||||
|
}
|
||||||
|
dst := strings.TrimSuffix(src, filepath.Ext(src)) + targetExt
|
||||||
|
var convErr error
|
||||||
|
if target == "dds" {
|
||||||
|
convErr = pngToDDS(magick, src, dst)
|
||||||
|
} else {
|
||||||
|
convErr = ddsToPNG(magick, src, dst) // works for any decodable source
|
||||||
|
}
|
||||||
|
if convErr != nil {
|
||||||
|
fmt.Fprintf(stderr, "assets convert: %s: %v\n", src, convErr)
|
||||||
|
failed = true
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if dst != src {
|
||||||
|
if err := os.Remove(src); err != nil {
|
||||||
|
fmt.Fprintf(stderr, "assets convert: %s: %v\n", src, err)
|
||||||
|
failed = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if failed {
|
||||||
|
return exitFail
|
||||||
|
}
|
||||||
|
return exitOK
|
||||||
|
}
|
||||||
|
|
||||||
|
// ddsToPNG decodes src to dst (PNG/TGA by dst's extension), flipping vertically
|
||||||
|
// once. The single flip is the defining NWN behavior.
|
||||||
|
func ddsToPNG(magickBin, src, dst string) error {
|
||||||
|
if out, err := runner("", nil, magickBin, src, "-flip", dst); err != nil {
|
||||||
|
return fmt.Errorf("magick: %v: %s", err, strings.TrimSpace(string(out)))
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// pngToDDS encodes src to a DDS at dst, flipping vertically once and choosing
|
||||||
|
// DXT1 for opaque images, DXT5 for images with alpha.
|
||||||
|
func pngToDDS(magickBin, src, dst string) error {
|
||||||
|
compression := "dxt1"
|
||||||
|
if hasAlpha(magickBin, src) {
|
||||||
|
compression = "dxt5"
|
||||||
|
}
|
||||||
|
// ponytail: mipmaps rely on magick's default full chain (NWN wants a
|
||||||
|
// pyramid, which magick writes by default). Add `-define dds:mipmaps=N`
|
||||||
|
// here if a specific count is ever required.
|
||||||
|
out, err := runner("", nil, magickBin, src, "-flip",
|
||||||
|
"-define", "dds:compression="+compression, dst)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("magick: %v: %s", err, strings.TrimSpace(string(out)))
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// hasAlpha reports whether src has any non-opaque pixel, via `magick identify`.
|
||||||
|
// On any probe error it assumes alpha (DXT5), the safe/lossless-alpha default.
|
||||||
|
func hasAlpha(magickBin, src string) bool {
|
||||||
|
out, err := runner("", nil, magickBin, "identify", "-format", "%[opaque]", src)
|
||||||
|
if err != nil {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return !strings.EqualFold(strings.TrimSpace(string(out)), "true")
|
||||||
|
}
|
||||||
@@ -0,0 +1,108 @@
|
|||||||
|
package assets
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"image"
|
||||||
|
"image/color"
|
||||||
|
"image/png"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// writeTestPNG writes a 16x16 image: top half red, bottom half blue. The size
|
||||||
|
// and the half-way split keep every 4x4 DXT block a single flat color, so the
|
||||||
|
// DXT1 round trip stays lossless and only the flip is under test. (A 4x4 image
|
||||||
|
// is one mixed DXT block that magick's encoder collapses to a single color.)
|
||||||
|
func writeTestPNG(t *testing.T, path string) {
|
||||||
|
t.Helper()
|
||||||
|
const n = 16
|
||||||
|
img := image.NewRGBA(image.Rect(0, 0, n, n))
|
||||||
|
for y := 0; y < n; y++ {
|
||||||
|
c := color.RGBA{255, 0, 0, 255} // red
|
||||||
|
if y >= n/2 {
|
||||||
|
c = color.RGBA{0, 0, 255, 255} // blue
|
||||||
|
}
|
||||||
|
for x := 0; x < n; x++ {
|
||||||
|
img.Set(x, y, c)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
f, err := os.Create(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer f.Close()
|
||||||
|
if err := png.Encode(f, img); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func topRowIsBlue(t *testing.T, pngPath string) bool {
|
||||||
|
t.Helper()
|
||||||
|
f, err := os.Open(pngPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer f.Close()
|
||||||
|
img, err := png.Decode(f)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
r, _, b, _ := img.At(0, 0).RGBA()
|
||||||
|
return b > r // blue dominates the top-left after a flip
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConvertFlipsOnceAndRoundTrips(t *testing.T) {
|
||||||
|
magick := look("magick")
|
||||||
|
if magick == "" {
|
||||||
|
t.Skip("magick not on PATH")
|
||||||
|
}
|
||||||
|
dir := t.TempDir()
|
||||||
|
src := filepath.Join(dir, "tex.png")
|
||||||
|
writeTestPNG(t, src)
|
||||||
|
|
||||||
|
// Convert PNG -> DDS (in place: tex.png becomes tex.dds, original removed).
|
||||||
|
var out, errw bytes.Buffer
|
||||||
|
if code := runConvert([]string{"--to", "dds", dir}, &out, &errw); code != exitOK {
|
||||||
|
t.Fatalf("to-dds exit = %d\n%s", code, errw.String())
|
||||||
|
}
|
||||||
|
dds := filepath.Join(dir, "tex.dds")
|
||||||
|
if _, err := os.Stat(dds); err != nil {
|
||||||
|
t.Fatalf("dds not produced: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(src); !os.IsNotExist(err) {
|
||||||
|
t.Fatal("source png was not replaced")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Decode the DDS RAW (no extra flip) and confirm a single flip happened:
|
||||||
|
// the source top row was red, so the DDS top row must now be blue.
|
||||||
|
raw := filepath.Join(dir, "raw.png")
|
||||||
|
if err := exec.Command(magick, dds, raw).Run(); err != nil {
|
||||||
|
t.Fatalf("raw decode: %v", err)
|
||||||
|
}
|
||||||
|
if !topRowIsBlue(t, raw) {
|
||||||
|
t.Fatal("expected the DDS to be vertically flipped vs the source")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Convert DDS -> PNG (another flip). Result should match the original.
|
||||||
|
out.Reset()
|
||||||
|
errw.Reset()
|
||||||
|
if code := runConvert([]string{"--to", "png", dir}, &out, &errw); code != exitOK {
|
||||||
|
t.Fatalf("to-png exit = %d\n%s", code, errw.String())
|
||||||
|
}
|
||||||
|
restored := filepath.Join(dir, "tex.png")
|
||||||
|
if topRowIsBlue(t, restored) {
|
||||||
|
t.Fatal("round trip did not restore the original orientation")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConvertMissingBackendFailsClosed(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
writeTestPNG(t, filepath.Join(dir, "tex.png"))
|
||||||
|
var out, errw bytes.Buffer
|
||||||
|
code := runConvert([]string{"--backend", "/nonexistent/magick", dir}, &out, &errw)
|
||||||
|
if code != exitTool {
|
||||||
|
t.Fatalf("missing backend exit = %d, want %d", code, exitTool)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,151 @@
|
|||||||
|
package assets
|
||||||
|
|
||||||
|
import (
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// allFiles returns every regular file under root (recursive).
|
||||||
|
func allFiles(root string) ([]string, error) {
|
||||||
|
var out []string
|
||||||
|
err := filepath.WalkDir(root, func(p string, d os.DirEntry, err error) error {
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !d.IsDir() {
|
||||||
|
out = append(out, p)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
sort.Strings(out)
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// runCheckDupes reports files whose lower-cased basename collides across the
|
||||||
|
// given dirs. Read-only. Exit exitFail if any collision is found.
|
||||||
|
func runCheckDupes(args []string, stdout, stderr io.Writer) int {
|
||||||
|
fs := flag.NewFlagSet("check-dupes", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(stderr)
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
dirs := fs.Args()
|
||||||
|
if len(dirs) == 0 {
|
||||||
|
fmt.Fprintln(stderr, "assets check-dupes: usage: check-dupes <dir>...")
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
|
first := map[string]string{}
|
||||||
|
fail := false
|
||||||
|
for _, dir := range dirs {
|
||||||
|
if fi, err := os.Stat(dir); err != nil || !fi.IsDir() {
|
||||||
|
fmt.Fprintf(stderr, "assets check-dupes: no such dir: %s\n", dir)
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
files, err := allFiles(dir)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "assets check-dupes:", err)
|
||||||
|
return exitTool
|
||||||
|
}
|
||||||
|
for _, f := range files {
|
||||||
|
key := strings.ToLower(filepath.Base(f))
|
||||||
|
if prev, ok := first[key]; ok {
|
||||||
|
fmt.Fprintf(stderr, "runtime-name collision: %s collides with %s\n", f, prev)
|
||||||
|
fail = true
|
||||||
|
} else {
|
||||||
|
first[key] = f
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if fail {
|
||||||
|
fmt.Fprintln(stderr, "check-dupes: found runtime-name collision(s)")
|
||||||
|
return exitFail
|
||||||
|
}
|
||||||
|
fmt.Fprintln(stdout, "check-dupes: OK")
|
||||||
|
return exitOK
|
||||||
|
}
|
||||||
|
|
||||||
|
// runCleanDupes deletes files from <clean> whose lower-cased basename collides
|
||||||
|
// with any file in <primary>. Mutates only <clean>. Refuses to run if the two
|
||||||
|
// trees overlap.
|
||||||
|
func runCleanDupes(args []string, stdout, stderr io.Writer) int {
|
||||||
|
fs := flag.NewFlagSet("clean-dupes", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(stderr)
|
||||||
|
dryRun := fs.Bool("dry-run", false, "report deletions without removing")
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
pos := fs.Args()
|
||||||
|
if len(pos) != 2 {
|
||||||
|
fmt.Fprintln(stderr, "assets clean-dupes: usage: clean-dupes [--dry-run] <primary> <clean>")
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
primary, clean := pos[0], pos[1]
|
||||||
|
for _, d := range []string{primary, clean} {
|
||||||
|
if fi, err := os.Stat(d); err != nil || !fi.IsDir() {
|
||||||
|
fmt.Fprintf(stderr, "assets clean-dupes: no such dir: %s\n", d)
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
}
|
||||||
|
primaryReal, err1 := filepath.EvalSymlinks(primary)
|
||||||
|
cleanReal, err2 := filepath.EvalSymlinks(clean)
|
||||||
|
if err1 != nil || err2 != nil {
|
||||||
|
fmt.Fprintln(stderr, "assets clean-dupes: cannot resolve dirs")
|
||||||
|
return exitTool
|
||||||
|
}
|
||||||
|
if overlaps(primaryReal, cleanReal) {
|
||||||
|
fmt.Fprintln(stderr, "assets clean-dupes: primary and clean dirs must not overlap")
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
|
primaryFiles, err := allFiles(primary)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "assets clean-dupes:", err)
|
||||||
|
return exitTool
|
||||||
|
}
|
||||||
|
names := map[string]bool{}
|
||||||
|
for _, f := range primaryFiles {
|
||||||
|
names[strings.ToLower(filepath.Base(f))] = true
|
||||||
|
}
|
||||||
|
|
||||||
|
cleanFiles, err := allFiles(clean)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "assets clean-dupes:", err)
|
||||||
|
return exitTool
|
||||||
|
}
|
||||||
|
removed := 0
|
||||||
|
for _, f := range cleanFiles {
|
||||||
|
if !names[strings.ToLower(filepath.Base(f))] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if *dryRun {
|
||||||
|
fmt.Fprintf(stderr, "would delete clean-tree collision: %s\n", f)
|
||||||
|
} else {
|
||||||
|
if err := os.Remove(f); err != nil {
|
||||||
|
fmt.Fprintln(stderr, "assets clean-dupes:", err)
|
||||||
|
return exitTool
|
||||||
|
}
|
||||||
|
fmt.Fprintf(stderr, "deleted clean-tree collision: %s\n", f)
|
||||||
|
}
|
||||||
|
removed++
|
||||||
|
}
|
||||||
|
verb := "removed"
|
||||||
|
if *dryRun {
|
||||||
|
verb = "would remove"
|
||||||
|
}
|
||||||
|
fmt.Fprintf(stdout, "clean-dupes: %s %d file(s)\n", verb, removed)
|
||||||
|
return exitOK
|
||||||
|
}
|
||||||
|
|
||||||
|
// overlaps reports whether either directory contains the other (or they are
|
||||||
|
// equal), using cleaned absolute-ish paths with a trailing separator.
|
||||||
|
func overlaps(a, b string) bool {
|
||||||
|
as := a + string(filepath.Separator)
|
||||||
|
bs := b + string(filepath.Separator)
|
||||||
|
return strings.HasPrefix(as, bs) || strings.HasPrefix(bs, as)
|
||||||
|
}
|
||||||
@@ -0,0 +1,80 @@
|
|||||||
|
package assets
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func touch(t *testing.T, path string) {
|
||||||
|
t.Helper()
|
||||||
|
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(path, []byte("x"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCheckDupes(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
touch(t, filepath.Join(root, "tex", "Foo.tga"))
|
||||||
|
touch(t, filepath.Join(root, "plc", "foo.tga")) // basename collision (case-insensitive)
|
||||||
|
|
||||||
|
var out, errw bytes.Buffer
|
||||||
|
if code := runCheckDupes([]string{root}, &out, &errw); code != exitFail {
|
||||||
|
t.Fatalf("collision exit = %d, want %d\n%s", code, exitFail, errw.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
clean := t.TempDir()
|
||||||
|
touch(t, filepath.Join(clean, "a.tga"))
|
||||||
|
touch(t, filepath.Join(clean, "sub", "b.tga"))
|
||||||
|
out.Reset()
|
||||||
|
errw.Reset()
|
||||||
|
if code := runCheckDupes([]string{clean}, &out, &errw); code != exitOK {
|
||||||
|
t.Fatalf("no-collision exit = %d, want %d\n%s", code, exitOK, errw.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCleanDupes(t *testing.T) {
|
||||||
|
primary := t.TempDir()
|
||||||
|
clean := t.TempDir()
|
||||||
|
touch(t, filepath.Join(primary, "keep.tga"))
|
||||||
|
collide := filepath.Join(clean, "sub", "Keep.tga")
|
||||||
|
survive := filepath.Join(clean, "unique.tga")
|
||||||
|
touch(t, collide)
|
||||||
|
touch(t, survive)
|
||||||
|
|
||||||
|
// dry-run removes nothing.
|
||||||
|
var out, errw bytes.Buffer
|
||||||
|
if code := runCleanDupes([]string{"--dry-run", primary, clean}, &out, &errw); code != exitOK {
|
||||||
|
t.Fatalf("dry-run exit = %d\n%s", code, errw.String())
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(collide); err != nil {
|
||||||
|
t.Fatal("dry-run deleted a file")
|
||||||
|
}
|
||||||
|
|
||||||
|
// real run deletes the collision, keeps the unique file.
|
||||||
|
out.Reset()
|
||||||
|
errw.Reset()
|
||||||
|
if code := runCleanDupes([]string{primary, clean}, &out, &errw); code != exitOK {
|
||||||
|
t.Fatalf("clean exit = %d\n%s", code, errw.String())
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(collide); !os.IsNotExist(err) {
|
||||||
|
t.Fatal("collision file was not deleted")
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(survive); err != nil {
|
||||||
|
t.Fatal("unique file was wrongly deleted")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCleanDupesRejectsOverlap(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
sub := filepath.Join(root, "child")
|
||||||
|
touch(t, filepath.Join(sub, "x.tga"))
|
||||||
|
var out, errw bytes.Buffer
|
||||||
|
if code := runCleanDupes([]string{root, sub}, &out, &errw); code != exitUsage {
|
||||||
|
t.Fatalf("overlap exit = %d, want %d", code, exitUsage)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,103 @@
|
|||||||
|
package assets
|
||||||
|
|
||||||
|
import (
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/assets/mdl"
|
||||||
|
)
|
||||||
|
|
||||||
|
// runFixMDL lowercases .mdl filenames and rewrites ASCII model identity to
|
||||||
|
// match each file's stem. Binary models are skipped (the engine owns those).
|
||||||
|
func runFixMDL(args []string, stdout, stderr io.Writer) int {
|
||||||
|
fs := flag.NewFlagSet("fix-mdl", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(stderr)
|
||||||
|
dryRun := fs.Bool("dry-run", false, "report changes without touching disk")
|
||||||
|
nonRecursive := fs.Bool("non-recursive", false, "do not descend into subdirectories")
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
paths := fs.Args()
|
||||||
|
if len(paths) == 0 {
|
||||||
|
fmt.Fprintln(stderr, "assets fix-mdl: usage: fix-mdl [--dry-run] <path>...")
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
|
files, err := walk(paths, mdlExt, !*nonRecursive)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "assets fix-mdl:", err)
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
|
changed := 0
|
||||||
|
failed := false
|
||||||
|
for _, f := range files {
|
||||||
|
// 1. Lowercase the basename if needed.
|
||||||
|
//
|
||||||
|
// ponytail: 35k files are each read once here (CheckNames + the rewrite
|
||||||
|
// on candidates). The shell reference batched an awk pass to avoid
|
||||||
|
// per-file subprocess spawns; in Go a plain read is cheap, so the batch
|
||||||
|
// is not worth porting. If profiling ever shows this hot, parallelize
|
||||||
|
// the loop.
|
||||||
|
lower := f
|
||||||
|
if base := filepath.Base(f); base != strings.ToLower(base) {
|
||||||
|
lower = filepath.Join(filepath.Dir(f), strings.ToLower(base))
|
||||||
|
if _, err := os.Stat(lower); err == nil {
|
||||||
|
fmt.Fprintf(stderr, "assets fix-mdl: lowercase collision: %s -> %s\n", f, lower)
|
||||||
|
failed = true
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if *dryRun {
|
||||||
|
fmt.Fprintf(stdout, "would lowercase: %s -> %s\n", f, lower)
|
||||||
|
} else {
|
||||||
|
if err := os.Rename(f, lower); err != nil {
|
||||||
|
fmt.Fprintf(stderr, "assets fix-mdl: failed to lowercase %s: %v\n", f, err)
|
||||||
|
failed = true
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
fmt.Fprintf(stdout, "lowercased: %s -> %s\n", f, lower)
|
||||||
|
}
|
||||||
|
changed++
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. Rewrite model identity if the (lowercased) file has a mismatch.
|
||||||
|
// In dry-run the on-disk file is still the original path.
|
||||||
|
readPath := lower
|
||||||
|
if *dryRun && lower != f {
|
||||||
|
readPath = f
|
||||||
|
}
|
||||||
|
data, err := os.ReadFile(readPath)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "assets fix-mdl:", err)
|
||||||
|
failed = true
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out, wasChanged := mdl.FixNames(data, mdl.ExpectedName(lower))
|
||||||
|
if !wasChanged {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if *dryRun {
|
||||||
|
fmt.Fprintf(stdout, "would fix: %s\n", lower)
|
||||||
|
} else {
|
||||||
|
if err := os.WriteFile(lower, out, 0o644); err != nil {
|
||||||
|
fmt.Fprintln(stderr, "assets fix-mdl:", err)
|
||||||
|
failed = true
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
fmt.Fprintf(stdout, "fixed: %s\n", lower)
|
||||||
|
}
|
||||||
|
changed++
|
||||||
|
}
|
||||||
|
|
||||||
|
if changed == 0 {
|
||||||
|
fmt.Fprintln(stdout, "no broken mdl model names found")
|
||||||
|
}
|
||||||
|
if failed {
|
||||||
|
return exitFail
|
||||||
|
}
|
||||||
|
return exitOK
|
||||||
|
}
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
package assets
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/assets/mdl"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestFixMDL(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
// Uppercase name + internal mismatch.
|
||||||
|
upper := filepath.Join(dir, "Foo.MDL")
|
||||||
|
body := "newmodel wrong\nbeginmodelgeom wrong\nendmodelgeom wrong\ndonemodel wrong\n"
|
||||||
|
if err := os.WriteFile(upper, []byte(body), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// dry-run: nothing changes on disk.
|
||||||
|
var out, errw bytes.Buffer
|
||||||
|
if code := runFixMDL([]string{"--dry-run", dir}, &out, &errw); code != exitOK {
|
||||||
|
t.Fatalf("dry-run exit = %d\n%s", code, errw.String())
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(upper); err != nil {
|
||||||
|
t.Fatal("dry-run renamed a file")
|
||||||
|
}
|
||||||
|
|
||||||
|
// real run: file is lowercased and its identity rewritten to match.
|
||||||
|
out.Reset()
|
||||||
|
errw.Reset()
|
||||||
|
if code := runFixMDL([]string{dir}, &out, &errw); code != exitOK {
|
||||||
|
t.Fatalf("fix exit = %d\n%s", code, errw.String())
|
||||||
|
}
|
||||||
|
lowered := filepath.Join(dir, "foo.mdl")
|
||||||
|
if _, err := os.Stat(lowered); err != nil {
|
||||||
|
t.Fatalf("file was not lowercased: %v", err)
|
||||||
|
}
|
||||||
|
mismatches, err := mdl.CheckNames(lowered)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(mismatches) != 0 {
|
||||||
|
data, _ := os.ReadFile(lowered)
|
||||||
|
t.Fatalf("model still has mismatches after fix: %+v\n%s", mismatches, data)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,84 @@
|
|||||||
|
// Package assets is the crucible `assets` builder: NWN:EE model/texture tools
|
||||||
|
// that operate in place on a target directory. Mirrors internal/depot's
|
||||||
|
// Run(args, stdout, stderr, getenv) shape and bypasses the legacy internal/app
|
||||||
|
// surface entirely.
|
||||||
|
package assets
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// runner is the single external-command indirection so tests can observe and
|
||||||
|
// stub every engine / ImageMagick / upscaler invocation. dir is the working
|
||||||
|
// directory ("" = inherit); env replaces the child environment when non-nil.
|
||||||
|
var runner = func(dir string, env []string, name string, args ...string) ([]byte, error) {
|
||||||
|
cmd := exec.Command(name, args...)
|
||||||
|
cmd.Dir = dir
|
||||||
|
if env != nil {
|
||||||
|
cmd.Env = env
|
||||||
|
}
|
||||||
|
return cmd.CombinedOutput()
|
||||||
|
}
|
||||||
|
|
||||||
|
// walk collects files under each root whose lower-cased extension is in exts. A
|
||||||
|
// root that is itself a matching file is included. recursive controls descent
|
||||||
|
// into subdirectories. Results are sorted and deduplicated.
|
||||||
|
func walk(roots []string, exts map[string]bool, recursive bool) ([]string, error) {
|
||||||
|
seen := map[string]bool{}
|
||||||
|
var out []string
|
||||||
|
add := func(p string) {
|
||||||
|
if exts[strings.ToLower(filepath.Ext(p))] && !seen[p] {
|
||||||
|
seen[p] = true
|
||||||
|
out = append(out, p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, root := range roots {
|
||||||
|
fi, err := os.Stat(root)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if !fi.IsDir() {
|
||||||
|
add(root)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
err = filepath.WalkDir(root, func(p string, d os.DirEntry, err error) error {
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if d.IsDir() {
|
||||||
|
if !recursive && p != root {
|
||||||
|
return filepath.SkipDir
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
add(p)
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort.Strings(out)
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// look returns the first bare name found on PATH, or the first candidate that
|
||||||
|
// is an existing path. Returns "" if none resolve.
|
||||||
|
func look(candidates ...string) string {
|
||||||
|
for _, c := range candidates {
|
||||||
|
if strings.ContainsRune(c, filepath.Separator) {
|
||||||
|
if fi, err := os.Stat(c); err == nil && !fi.IsDir() {
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if p, err := exec.LookPath(c); err == nil {
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
package assets
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestWalk(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
must := func(rel string) {
|
||||||
|
p := filepath.Join(root, rel)
|
||||||
|
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(p, []byte("x"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
must("a.MDL")
|
||||||
|
must("sub/b.mdl")
|
||||||
|
must("sub/c.txt")
|
||||||
|
|
||||||
|
exts := map[string]bool{".mdl": true}
|
||||||
|
got, err := walk([]string{root}, exts, true)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(got) != 2 {
|
||||||
|
t.Fatalf("recursive walk = %v, want 2 mdl files", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
got, _ = walk([]string{root}, exts, false)
|
||||||
|
if len(got) != 1 || filepath.Base(got[0]) != "a.MDL" {
|
||||||
|
t.Fatalf("non-recursive walk = %v, want only a.MDL", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A file argument that matches is included directly.
|
||||||
|
got, _ = walk([]string{filepath.Join(root, "sub", "b.mdl")}, exts, true)
|
||||||
|
if len(got) != 1 {
|
||||||
|
t.Fatalf("file arg walk = %v, want the file itself", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLook(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
stub := filepath.Join(dir, "mytool")
|
||||||
|
if err := os.WriteFile(stub, []byte("#!/bin/sh\n"), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Setenv("PATH", dir)
|
||||||
|
|
||||||
|
if got := look("nope-not-here", "mytool"); got == "" {
|
||||||
|
t.Fatal("look should find mytool on PATH")
|
||||||
|
}
|
||||||
|
// Explicit existing path candidate.
|
||||||
|
if got := look(stub); got != stub {
|
||||||
|
t.Fatalf("look(%q) = %q, want the path itself", stub, got)
|
||||||
|
}
|
||||||
|
if got := look("definitely-absent-binary-xyz"); got != "" {
|
||||||
|
t.Fatalf("look = %q, want empty", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,218 @@
|
|||||||
|
// Package mdl ports the ASCII MDL model-name checks from sow-assets-manifest's
|
||||||
|
// mdl-name-lib.sh + mdl-scan.awk to pure Go. Binary (compiled) MDLs are opaque
|
||||||
|
// here; the NWN engine is the validator for those. Detection mirrors
|
||||||
|
// mdl-scan.awk exactly so behavior does not drift while that awk still runs in
|
||||||
|
// sow-assets-manifest.
|
||||||
|
package mdl
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Mismatch is one model-name problem. Line is 1-based; 0 means the geometry
|
||||||
|
// base-node mismatch, which the source has no single line for.
|
||||||
|
type Mismatch struct {
|
||||||
|
Line int
|
||||||
|
What string
|
||||||
|
Got string
|
||||||
|
}
|
||||||
|
|
||||||
|
// ExpectedName is the file stem with a trailing ".mdl" (any case) removed.
|
||||||
|
func ExpectedName(path string) string {
|
||||||
|
base := filepath.Base(path)
|
||||||
|
if len(base) >= 4 && strings.EqualFold(base[len(base)-4:], ".mdl") {
|
||||||
|
return base[:len(base)-4]
|
||||||
|
}
|
||||||
|
return base
|
||||||
|
}
|
||||||
|
|
||||||
|
// ExpectedNameStem is an alias of ExpectedName kept for call-site clarity where
|
||||||
|
// the value is used as the engine's model stem argument.
|
||||||
|
func ExpectedNameStem(path string) string { return ExpectedName(path) }
|
||||||
|
|
||||||
|
// IsASCII reports whether path is an uncompiled ASCII model: no NUL in the
|
||||||
|
// first 256 bytes, and a leading keyword of '#' / newmodel / node /
|
||||||
|
// setsupermodel. Mirrors mdl_is_ascii.
|
||||||
|
func IsASCII(path string) (bool, error) {
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return false, err
|
||||||
|
}
|
||||||
|
return isASCII(data), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func isASCII(data []byte) bool {
|
||||||
|
head := data
|
||||||
|
if len(head) > 256 {
|
||||||
|
head = head[:256]
|
||||||
|
}
|
||||||
|
if bytes.IndexByte(head, 0) >= 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
line := head
|
||||||
|
if i := bytes.IndexByte(line, '\n'); i >= 0 {
|
||||||
|
line = line[:i]
|
||||||
|
}
|
||||||
|
trimmed := strings.TrimLeft(string(line), " \t\r")
|
||||||
|
if strings.HasPrefix(trimmed, "#") {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
fields := strings.Fields(trimmed)
|
||||||
|
if len(fields) == 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
tok := strings.ToLower(fields[0])
|
||||||
|
return strings.HasPrefix(tok, "newmodel") ||
|
||||||
|
strings.HasPrefix(tok, "node") ||
|
||||||
|
strings.HasPrefix(tok, "setsupermodel")
|
||||||
|
}
|
||||||
|
|
||||||
|
// lineFields splits a line into whitespace-delimited tokens with the trailing
|
||||||
|
// CR removed, matching awk's default field split + \r strip.
|
||||||
|
func lineFields(line string) []string {
|
||||||
|
return strings.Fields(strings.TrimRight(line, "\r"))
|
||||||
|
}
|
||||||
|
|
||||||
|
// field returns the 1-based nth field or "" if absent.
|
||||||
|
func field(f []string, n int) string {
|
||||||
|
if n >= 1 && n <= len(f) {
|
||||||
|
return f[n-1]
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// CheckNames returns every model-name mismatch in an ASCII model. Returns nil
|
||||||
|
// for a binary model or one with no newmodel line (not a real model).
|
||||||
|
func CheckNames(path string) ([]Mismatch, error) {
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if !isASCII(data) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
return checkNames(data, ExpectedName(path)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func checkNames(data []byte, expected string) []Mismatch {
|
||||||
|
expLower := strings.ToLower(expected)
|
||||||
|
var out []Mismatch
|
||||||
|
seenModel := false
|
||||||
|
inGeom := false
|
||||||
|
curNode := ""
|
||||||
|
base := ""
|
||||||
|
baseFound := false
|
||||||
|
|
||||||
|
// header records one header-token mismatch (field n, 1-based) if the token
|
||||||
|
// is present and differs case-insensitively from the stem.
|
||||||
|
lines := strings.Split(string(data), "\n")
|
||||||
|
for i, raw := range lines {
|
||||||
|
f := lineFields(raw)
|
||||||
|
if len(f) == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
key := strings.ToLower(f[0])
|
||||||
|
mis := func(what string, n int) {
|
||||||
|
got := field(f, n)
|
||||||
|
if got != "" && !strings.EqualFold(got, expected) {
|
||||||
|
out = append(out, Mismatch{Line: i + 1, What: what, Got: got})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
switch key {
|
||||||
|
case "newmodel":
|
||||||
|
seenModel = true
|
||||||
|
mis("newmodel", 2)
|
||||||
|
case "setsupermodel":
|
||||||
|
mis("setsupermodel model", 2)
|
||||||
|
case "beginmodelgeom":
|
||||||
|
inGeom = true
|
||||||
|
mis("beginmodelgeom", 2)
|
||||||
|
case "endmodelgeom":
|
||||||
|
inGeom = false
|
||||||
|
mis("endmodelgeom", 2)
|
||||||
|
case "donemodel":
|
||||||
|
mis("donemodel", 2)
|
||||||
|
case "newanim":
|
||||||
|
mis("newanim model", 3)
|
||||||
|
case "doneanim":
|
||||||
|
mis("doneanim model", 3)
|
||||||
|
case "node":
|
||||||
|
if inGeom {
|
||||||
|
curNode = field(f, 3)
|
||||||
|
}
|
||||||
|
case "parent":
|
||||||
|
if inGeom && !baseFound && strings.EqualFold(field(f, 2), "null") {
|
||||||
|
base = curNode
|
||||||
|
baseFound = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !seenModel {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if baseFound && base != "" && !strings.EqualFold(base, expLower) {
|
||||||
|
out = append(out, Mismatch{Line: 0, What: "root node", Got: base})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// FixNames rewrites only the model identity to expected: the header tokens plus
|
||||||
|
// the geometry base node's declaration / any parent / animroot pointing at the
|
||||||
|
// old base name. Animation bone names and supermodel animroots are left alone
|
||||||
|
// so inheritance keeps working. Mirrors mdl_fix_model_name in intent.
|
||||||
|
//
|
||||||
|
// ponytail: a rewritten line is rebuilt joining fields with single spaces and
|
||||||
|
// drops a trailing CR, exactly like the awk it replaces; unchanged lines are
|
||||||
|
// byte-identical. The engine ignores identity-line whitespace, and FixNames'
|
||||||
|
// contract is only "produces a model that passes CheckNames", not byte parity.
|
||||||
|
func FixNames(in []byte, expected string) (out []byte, changed bool) {
|
||||||
|
if !isASCII(in) {
|
||||||
|
return in, false
|
||||||
|
}
|
||||||
|
// oldbase: the base-node name to rename, only if it differs from expected.
|
||||||
|
oldbase := ""
|
||||||
|
for _, m := range checkNames(in, expected) {
|
||||||
|
if m.What == "root node" {
|
||||||
|
oldbase = m.Got
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
lines := strings.Split(string(in), "\n")
|
||||||
|
for idx, raw := range lines {
|
||||||
|
f := lineFields(raw)
|
||||||
|
if len(f) == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
key := strings.ToLower(f[0])
|
||||||
|
set := func(n int) bool {
|
||||||
|
if field(f, n) == "" {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
f[n-1] = expected
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
modified := false
|
||||||
|
switch key {
|
||||||
|
case "newmodel", "setsupermodel", "beginmodelgeom", "endmodelgeom", "donemodel":
|
||||||
|
modified = set(2)
|
||||||
|
case "newanim", "doneanim":
|
||||||
|
modified = set(3)
|
||||||
|
case "node":
|
||||||
|
if oldbase != "" && strings.EqualFold(field(f, 3), oldbase) {
|
||||||
|
modified = set(3)
|
||||||
|
}
|
||||||
|
case "parent", "animroot":
|
||||||
|
if oldbase != "" && strings.EqualFold(field(f, 2), oldbase) {
|
||||||
|
modified = set(2)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if modified {
|
||||||
|
lines[idx] = strings.Join(f, " ")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out = []byte(strings.Join(lines, "\n"))
|
||||||
|
return out, !bytes.Equal(out, in)
|
||||||
|
}
|
||||||
@@ -0,0 +1,114 @@
|
|||||||
|
package mdl
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func writeMDL(t *testing.T, name, body string) string {
|
||||||
|
t.Helper()
|
||||||
|
p := filepath.Join(t.TempDir(), name)
|
||||||
|
if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIsASCII(t *testing.T) {
|
||||||
|
ascii := writeMDL(t, "foo.mdl", "newmodel foo\nbeginmodelgeom foo\n")
|
||||||
|
if ok, err := IsASCII(ascii); err != nil || !ok {
|
||||||
|
t.Fatalf("ascii: ok=%v err=%v, want true nil", ok, err)
|
||||||
|
}
|
||||||
|
bin := writeMDL(t, "bar.mdl", "\x00\x01binary\x00garbage")
|
||||||
|
if ok, err := IsASCII(bin); err != nil || ok {
|
||||||
|
t.Fatalf("binary: ok=%v err=%v, want false nil", ok, err)
|
||||||
|
}
|
||||||
|
hash := writeMDL(t, "baz.mdl", "# a comment\nnewmodel baz\n")
|
||||||
|
if ok, _ := IsASCII(hash); !ok {
|
||||||
|
t.Fatal("leading # should be ascii")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExpectedName(t *testing.T) {
|
||||||
|
for in, want := range map[string]string{
|
||||||
|
"a/b/Foo.MDL": "Foo",
|
||||||
|
"waxbt_b_091.mdl": "waxbt_b_091",
|
||||||
|
"x.mdl": "x",
|
||||||
|
} {
|
||||||
|
if got := ExpectedName(in); got != want {
|
||||||
|
t.Errorf("ExpectedName(%q)=%q want %q", in, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCheckNames(t *testing.T) {
|
||||||
|
// Header token mismatch + base-node mismatch in one file.
|
||||||
|
body := "newmodel wrong\n" +
|
||||||
|
"setsupermodel wrong a_base\n" +
|
||||||
|
"beginmodelgeom foo\n" +
|
||||||
|
" node dummy Wmgst_m_081\n" +
|
||||||
|
" parent null\n" +
|
||||||
|
" endnode\n" +
|
||||||
|
"endmodelgeom foo\n" +
|
||||||
|
"donemodel foo\n"
|
||||||
|
p := writeMDL(t, "foo.mdl", body)
|
||||||
|
got, err := CheckNames(p)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
// Expected stem "foo": newmodel(wrong), setsupermodel(wrong), base node(Wmgst_m_081).
|
||||||
|
if len(got) != 3 {
|
||||||
|
t.Fatalf("got %d mismatches, want 3: %+v", len(got), got)
|
||||||
|
}
|
||||||
|
if got[0].What != "newmodel" || got[0].Got != "wrong" || got[0].Line != 1 {
|
||||||
|
t.Errorf("first mismatch = %+v", got[0])
|
||||||
|
}
|
||||||
|
base := got[len(got)-1]
|
||||||
|
if base.What != "root node" || base.Got != "Wmgst_m_081" || base.Line != 0 {
|
||||||
|
t.Errorf("base mismatch = %+v", base)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Clean model -> no mismatches.
|
||||||
|
clean := writeMDL(t, "bar.mdl",
|
||||||
|
"newmodel bar\nbeginmodelgeom bar\n node dummy bar\n parent null\n endnode\nendmodelgeom bar\ndonemodel bar\n")
|
||||||
|
if got, _ := CheckNames(clean); len(got) != 0 {
|
||||||
|
t.Fatalf("clean model reported mismatches: %+v", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Binary model -> nil, no error.
|
||||||
|
bin := writeMDL(t, "b.mdl", "\x00\x00binary")
|
||||||
|
if got, _ := CheckNames(bin); got != nil {
|
||||||
|
t.Fatalf("binary reported mismatches: %+v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFixNamesRoundTrips(t *testing.T) {
|
||||||
|
body := "newmodel wrong\n" +
|
||||||
|
"setsupermodel wrong\n" +
|
||||||
|
"beginmodelgeom wrong\n" +
|
||||||
|
" node dummy Wmgst_m_081\n" +
|
||||||
|
" parent null\n" +
|
||||||
|
" endnode\n" +
|
||||||
|
"endmodelgeom wrong\n" +
|
||||||
|
"donemodel wrong\n"
|
||||||
|
out, changed := FixNames([]byte(body), "foo")
|
||||||
|
if !changed {
|
||||||
|
t.Fatal("expected changed=true")
|
||||||
|
}
|
||||||
|
p := filepath.Join(t.TempDir(), "foo.mdl")
|
||||||
|
if err := os.WriteFile(p, out, 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
got, err := CheckNames(p)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(got) != 0 {
|
||||||
|
t.Fatalf("fixed model still has mismatches: %+v\n%s", got, out)
|
||||||
|
}
|
||||||
|
// Idempotent: fixing a clean model changes nothing.
|
||||||
|
if _, changed := FixNames(out, "foo"); changed {
|
||||||
|
t.Fatal("second fix reported a change")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
package assets
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
exitOK = 0
|
||||||
|
exitFail = 1 // problems found / per-file failures
|
||||||
|
exitUsage = 64 // bad invocation / unknown subcommand / bad flags
|
||||||
|
exitTool = 70 // missing external tool / internal error
|
||||||
|
)
|
||||||
|
|
||||||
|
// Run executes an assets subcommand. args[0] is the subcommand; returns the
|
||||||
|
// process exit code.
|
||||||
|
func Run(args []string, stdout, stderr io.Writer, getenv func(string) string) int {
|
||||||
|
if len(args) == 0 {
|
||||||
|
printUsage(stderr)
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
rest := args[1:]
|
||||||
|
switch args[0] {
|
||||||
|
case "check-dupes":
|
||||||
|
return runCheckDupes(rest, stdout, stderr)
|
||||||
|
case "clean-dupes":
|
||||||
|
return runCleanDupes(rest, stdout, stderr)
|
||||||
|
case "check-mdl":
|
||||||
|
return runCheckMDL(rest, stdout, stderr)
|
||||||
|
case "fix-mdl":
|
||||||
|
return runFixMDL(rest, stdout, stderr)
|
||||||
|
case "convert":
|
||||||
|
return runConvert(rest, stdout, stderr)
|
||||||
|
case "upscale":
|
||||||
|
return runUpscale(rest, stdout, stderr)
|
||||||
|
case "compile":
|
||||||
|
return runCompile(rest, stdout, stderr, getenv)
|
||||||
|
default:
|
||||||
|
fmt.Fprintf(stderr, "assets: unknown subcommand %q\n\n", args[0])
|
||||||
|
printUsage(stderr)
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func printUsage(w io.Writer) {
|
||||||
|
fmt.Fprint(w, `usage:
|
||||||
|
assets compile [--nwn INSTALL] [--non-recursive] <dir>...
|
||||||
|
assets convert [--to dds|png|tga] [--backend PATH] [--non-recursive] <dir>...
|
||||||
|
assets upscale [--scale N] [--backend PATH] [--non-recursive] <dir>...
|
||||||
|
assets check-mdl <path>...
|
||||||
|
assets fix-mdl [--dry-run] <path>...
|
||||||
|
assets check-dupes <dir>...
|
||||||
|
assets clean-dupes [--dry-run] <primary> <clean>
|
||||||
|
`)
|
||||||
|
}
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
package assets
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func env(string) string { return "" }
|
||||||
|
|
||||||
|
func TestRunNoArgsIsUsage(t *testing.T) {
|
||||||
|
var out, errw bytes.Buffer
|
||||||
|
if code := Run(nil, &out, &errw, env); code != exitUsage {
|
||||||
|
t.Fatalf("no args exit = %d, want %d", code, exitUsage)
|
||||||
|
}
|
||||||
|
if !strings.Contains(errw.String(), "usage") {
|
||||||
|
t.Fatalf("no args should print usage, got: %q", errw.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunUnknownSubcommandIsUsage(t *testing.T) {
|
||||||
|
var out, errw bytes.Buffer
|
||||||
|
if code := Run([]string{"frobnicate"}, &out, &errw, env); code != exitUsage {
|
||||||
|
t.Fatalf("unknown subcommand exit = %d, want %d", code, exitUsage)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,98 @@
|
|||||||
|
package assets
|
||||||
|
|
||||||
|
import (
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
var upscaleBackends = []string{"upscayl-bin", "upscayl", "realesrgan-ncnn-vulkan", "waifu2x-ncnn-vulkan"}
|
||||||
|
|
||||||
|
// runUpscale upscales textures in place through an installed ncnn-vulkan
|
||||||
|
// backend. DDS inputs are bridged through PNG so the NWN flip stays correct.
|
||||||
|
func runUpscale(args []string, stdout, stderr io.Writer) int {
|
||||||
|
fs := flag.NewFlagSet("upscale", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(stderr)
|
||||||
|
scale := fs.Int("scale", 4, "upscale factor")
|
||||||
|
backend := fs.String("backend", "", "override the upscaler binary")
|
||||||
|
nonRecursive := fs.Bool("non-recursive", false, "do not descend into subdirectories")
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
dirs := fs.Args()
|
||||||
|
if len(dirs) == 0 {
|
||||||
|
fmt.Fprintln(stderr, "assets upscale: usage: upscale [--scale N] <dir>...")
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
|
candidates := upscaleBackends
|
||||||
|
if *backend != "" {
|
||||||
|
candidates = []string{*backend}
|
||||||
|
}
|
||||||
|
tool := look(candidates...)
|
||||||
|
if tool == "" {
|
||||||
|
fmt.Fprintf(stderr, "assets upscale: no upscaler found — install one of: %s\n", strings.Join(upscaleBackends, ", "))
|
||||||
|
return exitTool
|
||||||
|
}
|
||||||
|
|
||||||
|
files, err := walk(dirs, textureExts, !*nonRecursive)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "assets upscale:", err)
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
|
// magick is only needed if a .dds input is present; resolve lazily.
|
||||||
|
magick := ""
|
||||||
|
failed := false
|
||||||
|
for _, src := range files {
|
||||||
|
var upErr error
|
||||||
|
if strings.EqualFold(filepath.Ext(src), ".dds") {
|
||||||
|
if magick == "" {
|
||||||
|
if magick = look("magick"); magick == "" {
|
||||||
|
fmt.Fprintln(stderr, "assets upscale: .dds input needs ImageMagick (magick) for the png bridge")
|
||||||
|
return exitTool
|
||||||
|
}
|
||||||
|
}
|
||||||
|
upErr = upscaleDDS(tool, magick, src, *scale)
|
||||||
|
} else {
|
||||||
|
upErr = upscaleImage(tool, src, src, *scale)
|
||||||
|
}
|
||||||
|
if upErr != nil {
|
||||||
|
fmt.Fprintf(stderr, "assets upscale: %s: %v\n", src, upErr)
|
||||||
|
failed = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if failed {
|
||||||
|
return exitFail
|
||||||
|
}
|
||||||
|
return exitOK
|
||||||
|
}
|
||||||
|
|
||||||
|
// upscaleImage runs the ncnn-vulkan backend to upscale src into dst (may be the
|
||||||
|
// same path).
|
||||||
|
func upscaleImage(tool, src, dst string, scale int) error {
|
||||||
|
tmp := dst + ".upscaled.png"
|
||||||
|
out, err := runner("", nil, tool, "-i", src, "-o", tmp, "-s", strconv.Itoa(scale))
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("%s: %v: %s", filepath.Base(tool), err, strings.TrimSpace(string(out)))
|
||||||
|
}
|
||||||
|
return os.Rename(tmp, dst)
|
||||||
|
}
|
||||||
|
|
||||||
|
// upscaleDDS bridges a DDS through PNG: dds->png (flip), upscale, png->dds
|
||||||
|
// (flip back), yielding a correctly-flipped upscaled DDS.
|
||||||
|
func upscaleDDS(tool, magick, src string, scale int) error {
|
||||||
|
tmpPNG := src + ".bridge.png"
|
||||||
|
if err := ddsToPNG(magick, src, tmpPNG); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer os.Remove(tmpPNG)
|
||||||
|
if err := upscaleImage(tool, tmpPNG, tmpPNG, scale); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return pngToDDS(magick, tmpPNG, src)
|
||||||
|
}
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
package assets
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestUpscalePNGUsesBackend(t *testing.T) {
|
||||||
|
// A fake backend binary on PATH so look() resolves it.
|
||||||
|
binDir := t.TempDir()
|
||||||
|
fake := filepath.Join(binDir, "upscayl-bin")
|
||||||
|
if err := os.WriteFile(fake, []byte("#!/bin/sh\n"), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Setenv("PATH", binDir)
|
||||||
|
|
||||||
|
// Stub runner: emulate `-i in -o out -s scale` by copying in->out.
|
||||||
|
orig := runner
|
||||||
|
defer func() { runner = orig }()
|
||||||
|
var gotScale string
|
||||||
|
runner = func(dir string, env []string, name string, args ...string) ([]byte, error) {
|
||||||
|
var in, out string
|
||||||
|
for i := 0; i < len(args)-1; i++ {
|
||||||
|
switch args[i] {
|
||||||
|
case "-i":
|
||||||
|
in = args[i+1]
|
||||||
|
case "-o":
|
||||||
|
out = args[i+1]
|
||||||
|
case "-s":
|
||||||
|
gotScale = args[i+1]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
data, err := os.ReadFile(in)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return nil, os.WriteFile(out, data, 0o644)
|
||||||
|
}
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
src := filepath.Join(dir, "tex.png")
|
||||||
|
if err := os.WriteFile(src, []byte("pngdata"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
if code := runUpscale([]string{"--scale", "2", dir}, &stdout, &stderr); code != exitOK {
|
||||||
|
t.Fatalf("upscale exit = %d\n%s", code, stderr.String())
|
||||||
|
}
|
||||||
|
if gotScale != "2" {
|
||||||
|
t.Fatalf("scale passed to backend = %q, want 2", gotScale)
|
||||||
|
}
|
||||||
|
if data, _ := os.ReadFile(src); string(data) != "pngdata" {
|
||||||
|
t.Fatalf("upscaled file content = %q, want the backend output", data)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUpscaleNoBackendFailsClosed(t *testing.T) {
|
||||||
|
t.Setenv("PATH", t.TempDir()) // empty PATH: no backend resolvable
|
||||||
|
dir := t.TempDir()
|
||||||
|
if err := os.WriteFile(filepath.Join(dir, "tex.png"), []byte("x"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
if code := runUpscale([]string{dir}, &stdout, &stderr); code != exitTool {
|
||||||
|
t.Fatalf("no-backend exit = %d, want %d", code, exitTool)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -178,11 +178,17 @@ func TestGenerateIncludesPullRequestsAndDirectPushes(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
rendered := stdout.String()
|
rendered := stdout.String()
|
||||||
if !strings.Contains(rendered, "- Add release summary ([#12]("+repoURL+"/pulls/12)) - From Patch Author") {
|
for _, want := range []string{
|
||||||
t.Fatalf("rendered changelog missing pull entry:\n%s", rendered)
|
"Add release summary",
|
||||||
|
repoURL + "/pulls/12",
|
||||||
|
"Patch Author",
|
||||||
|
"Fix direct push handling",
|
||||||
|
repoURL + "/commit/" + directHash,
|
||||||
|
"Test User",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(rendered, want) {
|
||||||
|
t.Errorf("rendered changelog missing %q:\n%s", want, rendered)
|
||||||
}
|
}
|
||||||
if !strings.Contains(rendered, "- Fix direct push handling (["+shortCommitHash(directHash)+"]("+repoURL+"/commit/"+directHash+")) - From Test User") {
|
|
||||||
t.Fatalf("rendered changelog missing direct-push entry:\n%s", rendered)
|
|
||||||
}
|
}
|
||||||
if pullRequests != 1 {
|
if pullRequests != 1 {
|
||||||
t.Fatalf("expected 1 pull lookup, got %d", pullRequests)
|
t.Fatalf("expected 1 pull lookup, got %d", pullRequests)
|
||||||
@@ -192,7 +198,7 @@ func TestGenerateIncludesPullRequestsAndDirectPushes(t *testing.T) {
|
|||||||
func runGit(t *testing.T, repoRoot string, args ...string) string {
|
func runGit(t *testing.T, repoRoot string, args ...string) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
cmd := exec.Command("git", args...)
|
cmd := exec.Command("git", append([]string{"-c", "commit.gpgsign=false", "-c", "tag.gpgsign=false"}, args...)...)
|
||||||
cmd.Dir = repoRoot
|
cmd.Dir = repoRoot
|
||||||
output, err := cmd.CombinedOutput()
|
output, err := cmd.CombinedOutput()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -0,0 +1,24 @@
|
|||||||
|
package depot
|
||||||
|
|
||||||
|
import "context"
|
||||||
|
|
||||||
|
type ProbeState int
|
||||||
|
|
||||||
|
const (
|
||||||
|
Present ProbeState = iota
|
||||||
|
Absent
|
||||||
|
Unconfirmed
|
||||||
|
)
|
||||||
|
|
||||||
|
// Backend defines the interface for blob storage backends.
|
||||||
|
type Backend interface {
|
||||||
|
// Name returns the backend name.
|
||||||
|
Name() string
|
||||||
|
// Probe returns the existence state of one sha. transient=true means a
|
||||||
|
// retry might change the answer (feeds the serial confirm loop).
|
||||||
|
Probe(ctx context.Context, sha string) (state ProbeState, transient bool, err error)
|
||||||
|
// Get fetches sha into dest (temp file + rename), re-hashes, deletes on mismatch.
|
||||||
|
Get(ctx context.Context, sha, dest string) error
|
||||||
|
// Put uploads bytes from src for sha. Read-only backends return an error.
|
||||||
|
Put(ctx context.Context, sha, src string) error
|
||||||
|
}
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
package depot
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strconv"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Config struct {
|
||||||
|
CDNBase string
|
||||||
|
StorageHost string
|
||||||
|
StorageZone string
|
||||||
|
ReadKey string
|
||||||
|
WriteKey string
|
||||||
|
ProbeJobs int
|
||||||
|
Jobs int
|
||||||
|
ConnectTimeout time.Duration
|
||||||
|
ProbeMaxTime time.Duration
|
||||||
|
ConfirmRetries int
|
||||||
|
ConfirmMax int
|
||||||
|
}
|
||||||
|
|
||||||
|
func LoadConfig(getenv func(string) string) Config {
|
||||||
|
cfg := Config{
|
||||||
|
CDNBase: "https://cdn-a7f3k9.westgate.pw",
|
||||||
|
StorageZone: "sow-assets-depot",
|
||||||
|
ProbeJobs: 16,
|
||||||
|
Jobs: 16,
|
||||||
|
ConnectTimeout: 10 * time.Second,
|
||||||
|
ProbeMaxTime: 30 * time.Second,
|
||||||
|
ConfirmRetries: 3,
|
||||||
|
ConfirmMax: 200,
|
||||||
|
}
|
||||||
|
|
||||||
|
// DEPOT_CDN_BASE || BUNNY_CDN_BASE
|
||||||
|
if v := getenv("DEPOT_CDN_BASE"); v != "" {
|
||||||
|
cfg.CDNBase = v
|
||||||
|
} else if v := getenv("BUNNY_CDN_BASE"); v != "" {
|
||||||
|
cfg.CDNBase = v
|
||||||
|
}
|
||||||
|
|
||||||
|
// BUNNY_STORAGE_HOST (no default)
|
||||||
|
cfg.StorageHost = getenv("BUNNY_STORAGE_HOST")
|
||||||
|
|
||||||
|
// BUNNY_STORAGE_ZONE (default "sow-assets-depot")
|
||||||
|
if v := getenv("BUNNY_STORAGE_ZONE"); v != "" {
|
||||||
|
cfg.StorageZone = v
|
||||||
|
}
|
||||||
|
|
||||||
|
// BUNNY_STORAGE_PASSWORD
|
||||||
|
cfg.WriteKey = getenv("BUNNY_STORAGE_PASSWORD")
|
||||||
|
|
||||||
|
// BUNNY_STORAGE_READ_PASSWORD || BUNNY_STORAGE_PASSWORD
|
||||||
|
if v := getenv("BUNNY_STORAGE_READ_PASSWORD"); v != "" {
|
||||||
|
cfg.ReadKey = v
|
||||||
|
} else {
|
||||||
|
cfg.ReadKey = cfg.WriteKey
|
||||||
|
}
|
||||||
|
|
||||||
|
// DEPOT_PROBE_JOBS (default 16)
|
||||||
|
if v := getenv("DEPOT_PROBE_JOBS"); v != "" {
|
||||||
|
if i, err := strconv.Atoi(v); err == nil {
|
||||||
|
cfg.ProbeJobs = i
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// DEPOT_JOBS (default 16)
|
||||||
|
if v := getenv("DEPOT_JOBS"); v != "" {
|
||||||
|
if i, err := strconv.Atoi(v); err == nil {
|
||||||
|
cfg.Jobs = i
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// DEPOT_CONNECT_TIMEOUT (default 10s)
|
||||||
|
if v := getenv("DEPOT_CONNECT_TIMEOUT"); v != "" {
|
||||||
|
if d, err := time.ParseDuration(v + "s"); err == nil {
|
||||||
|
cfg.ConnectTimeout = d
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// DEPOT_PROBE_MAX_TIME (default 30s)
|
||||||
|
if v := getenv("DEPOT_PROBE_MAX_TIME"); v != "" {
|
||||||
|
if d, err := time.ParseDuration(v + "s"); err == nil {
|
||||||
|
cfg.ProbeMaxTime = d
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// DEPOT_CONFIRM_RETRIES (default 3)
|
||||||
|
if v := getenv("DEPOT_CONFIRM_RETRIES"); v != "" {
|
||||||
|
if i, err := strconv.Atoi(v); err == nil {
|
||||||
|
cfg.ConfirmRetries = i
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// DEPOT_CONFIRM_MAX (default 200, 0 = unlimited)
|
||||||
|
if v := getenv("DEPOT_CONFIRM_MAX"); v != "" {
|
||||||
|
if i, err := strconv.Atoi(v); err == nil {
|
||||||
|
cfg.ConfirmMax = i
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return cfg
|
||||||
|
}
|
||||||
@@ -0,0 +1,115 @@
|
|||||||
|
package depot
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestLoadConfigDefaults(t *testing.T) {
|
||||||
|
getenv := func(key string) string {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
cfg := LoadConfig(getenv)
|
||||||
|
if cfg.CDNBase != "https://cdn-a7f3k9.westgate.pw" {
|
||||||
|
t.Errorf("CDNBase: got %q, want %q", cfg.CDNBase, "https://cdn-a7f3k9.westgate.pw")
|
||||||
|
}
|
||||||
|
if cfg.StorageZone != "sow-assets-depot" {
|
||||||
|
t.Errorf("StorageZone: got %q, want %q", cfg.StorageZone, "sow-assets-depot")
|
||||||
|
}
|
||||||
|
if cfg.ProbeJobs != 16 {
|
||||||
|
t.Errorf("ProbeJobs: got %d, want 16", cfg.ProbeJobs)
|
||||||
|
}
|
||||||
|
if cfg.Jobs != 16 {
|
||||||
|
t.Errorf("Jobs: got %d, want 16", cfg.Jobs)
|
||||||
|
}
|
||||||
|
if cfg.ConnectTimeout != 10*time.Second {
|
||||||
|
t.Errorf("ConnectTimeout: got %v, want 10s", cfg.ConnectTimeout)
|
||||||
|
}
|
||||||
|
if cfg.ProbeMaxTime != 30*time.Second {
|
||||||
|
t.Errorf("ProbeMaxTime: got %v, want 30s", cfg.ProbeMaxTime)
|
||||||
|
}
|
||||||
|
if cfg.ConfirmRetries != 3 {
|
||||||
|
t.Errorf("ConfirmRetries: got %d, want 3", cfg.ConfirmRetries)
|
||||||
|
}
|
||||||
|
if cfg.ConfirmMax != 200 {
|
||||||
|
t.Errorf("ConfirmMax: got %d, want 200", cfg.ConfirmMax)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadConfigReadKeyFallback(t *testing.T) {
|
||||||
|
// When READ unset, falls back to WriteKey
|
||||||
|
getenv := func(key string) string {
|
||||||
|
if key == "BUNNY_STORAGE_PASSWORD" {
|
||||||
|
return "write-key"
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
cfg := LoadConfig(getenv)
|
||||||
|
if cfg.ReadKey != "write-key" {
|
||||||
|
t.Errorf("ReadKey fallback: got %q, want %q", cfg.ReadKey, "write-key")
|
||||||
|
}
|
||||||
|
if cfg.WriteKey != "write-key" {
|
||||||
|
t.Errorf("WriteKey: got %q, want %q", cfg.WriteKey, "write-key")
|
||||||
|
}
|
||||||
|
|
||||||
|
// When READ is set, uses its own value
|
||||||
|
getenv = func(key string) string {
|
||||||
|
if key == "BUNNY_STORAGE_READ_PASSWORD" {
|
||||||
|
return "read-key"
|
||||||
|
}
|
||||||
|
if key == "BUNNY_STORAGE_PASSWORD" {
|
||||||
|
return "write-key"
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
cfg = LoadConfig(getenv)
|
||||||
|
if cfg.ReadKey != "read-key" {
|
||||||
|
t.Errorf("ReadKey explicit: got %q, want %q", cfg.ReadKey, "read-key")
|
||||||
|
}
|
||||||
|
if cfg.WriteKey != "write-key" {
|
||||||
|
t.Errorf("WriteKey: got %q, want %q", cfg.WriteKey, "write-key")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadConfigBadIntFallsBack(t *testing.T) {
|
||||||
|
getenv := func(key string) string {
|
||||||
|
if key == "DEPOT_PROBE_JOBS" {
|
||||||
|
return "not-a-number"
|
||||||
|
}
|
||||||
|
if key == "DEPOT_JOBS" {
|
||||||
|
return "invalid"
|
||||||
|
}
|
||||||
|
if key == "DEPOT_CONNECT_TIMEOUT" {
|
||||||
|
return "bad"
|
||||||
|
}
|
||||||
|
if key == "DEPOT_PROBE_MAX_TIME" {
|
||||||
|
return "wrong"
|
||||||
|
}
|
||||||
|
if key == "DEPOT_CONFIRM_RETRIES" {
|
||||||
|
return "nope"
|
||||||
|
}
|
||||||
|
if key == "DEPOT_CONFIRM_MAX" {
|
||||||
|
return "nah"
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
cfg := LoadConfig(getenv)
|
||||||
|
if cfg.ProbeJobs != 16 {
|
||||||
|
t.Errorf("ProbeJobs fallback: got %d, want 16", cfg.ProbeJobs)
|
||||||
|
}
|
||||||
|
if cfg.Jobs != 16 {
|
||||||
|
t.Errorf("Jobs fallback: got %d, want 16", cfg.Jobs)
|
||||||
|
}
|
||||||
|
if cfg.ConnectTimeout != 10*time.Second {
|
||||||
|
t.Errorf("ConnectTimeout fallback: got %v, want 10s", cfg.ConnectTimeout)
|
||||||
|
}
|
||||||
|
if cfg.ProbeMaxTime != 30*time.Second {
|
||||||
|
t.Errorf("ProbeMaxTime fallback: got %v, want 30s", cfg.ProbeMaxTime)
|
||||||
|
}
|
||||||
|
if cfg.ConfirmRetries != 3 {
|
||||||
|
t.Errorf("ConfirmRetries fallback: got %d, want 3", cfg.ConfirmRetries)
|
||||||
|
}
|
||||||
|
if cfg.ConfirmMax != 200 {
|
||||||
|
t.Errorf("ConfirmMax fallback: got %d, want 200", cfg.ConfirmMax)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,183 @@
|
|||||||
|
package depot
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"fmt"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestIntegrationPushStatusPull drives status/push/pull end-to-end against a
|
||||||
|
// fake Bunny backend (real bunny HTTP path, in-memory blob store) since the
|
||||||
|
// command surface has no local->local push (push --target must be bunny).
|
||||||
|
func TestIntegrationPushStatusPull(t *testing.T) {
|
||||||
|
manifestsDir := t.TempDir()
|
||||||
|
sourceDir := t.TempDir()
|
||||||
|
|
||||||
|
blobs := map[string]string{
|
||||||
|
shaOf("blob-one"): "blob-one",
|
||||||
|
shaOf("blob-two"): "blob-two",
|
||||||
|
shaOf("blob-three"): "blob-three",
|
||||||
|
}
|
||||||
|
var manifest strings.Builder
|
||||||
|
manifest.WriteString("assets:\n")
|
||||||
|
for sha, content := range blobs {
|
||||||
|
manifest.WriteString(fmt.Sprintf(" - path: %s\n sha256: %s\n size: %d\n", sha, sha, len(content)))
|
||||||
|
blobPath := filepath.Join(sourceDir, BlobKey(sha))
|
||||||
|
if err := os.MkdirAll(filepath.Dir(blobPath), 0755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(blobPath, []byte(content), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(manifestsDir, "manifest.yml"), []byte(manifest.String()), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
f := newFakeBunny()
|
||||||
|
// Pre-seed one of the three blobs on the target so status starts with drift=2.
|
||||||
|
var oneSHA string
|
||||||
|
for sha := range blobs {
|
||||||
|
oneSHA = sha
|
||||||
|
break
|
||||||
|
}
|
||||||
|
f.blobs[oneSHA] = []byte(blobs[oneSHA])
|
||||||
|
srv := httptest.NewServer(f.handler())
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
env := map[string]string{
|
||||||
|
"BUNNY_STORAGE_HOST": hostPort(srv),
|
||||||
|
"BUNNY_STORAGE_READ_PASSWORD": "readkey",
|
||||||
|
"BUNNY_STORAGE_PASSWORD": "writekey",
|
||||||
|
}
|
||||||
|
getenv := testGetenv(env)
|
||||||
|
|
||||||
|
// status: expect exit 1, missing=2
|
||||||
|
var out, errb bytes.Buffer
|
||||||
|
code := Run([]string{"status", "--manifests", manifestsDir, "--target", "bunny"}, &out, &errb, getenv)
|
||||||
|
if code != 1 {
|
||||||
|
t.Fatalf("status: expected 1, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
|
||||||
|
}
|
||||||
|
if !bytesContains(out.String(), "missing=2") {
|
||||||
|
t.Fatalf("status: expected missing=2, got %s", out.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
// push: uploads the 2 missing blobs
|
||||||
|
out.Reset()
|
||||||
|
errb.Reset()
|
||||||
|
code = Run([]string{"push", "--manifests", manifestsDir, "--source", sourceDir, "--target", "bunny"}, &out, &errb, getenv)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("push: expected 0, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
|
||||||
|
}
|
||||||
|
if !bytesContains(out.String(), "uploaded=2 failed=0") {
|
||||||
|
t.Fatalf("push: expected uploaded=2 failed=0, got %s", out.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
putCountAfterFirstPush := 0
|
||||||
|
for _, r := range f.requestsSnapshot() {
|
||||||
|
if r.Method == "PUT" {
|
||||||
|
putCountAfterFirstPush++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if putCountAfterFirstPush != 2 {
|
||||||
|
t.Fatalf("expected 2 PUTs after first push, got %d", putCountAfterFirstPush)
|
||||||
|
}
|
||||||
|
|
||||||
|
// status: now clean
|
||||||
|
out.Reset()
|
||||||
|
errb.Reset()
|
||||||
|
code = Run([]string{"status", "--manifests", manifestsDir, "--target", "bunny"}, &out, &errb, getenv)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("status after push: expected 0, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
// push again: idempotent, no new PUTs, uploaded=0
|
||||||
|
out.Reset()
|
||||||
|
errb.Reset()
|
||||||
|
code = Run([]string{"push", "--manifests", manifestsDir, "--source", sourceDir, "--target", "bunny"}, &out, &errb, getenv)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("second push: expected 0, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
|
||||||
|
}
|
||||||
|
if !bytesContains(out.String(), "uploaded=0 failed=0") {
|
||||||
|
t.Fatalf("second push: expected uploaded=0 failed=0, got %s", out.String())
|
||||||
|
}
|
||||||
|
putCountAfterSecondPush := 0
|
||||||
|
for _, r := range f.requestsSnapshot() {
|
||||||
|
if r.Method == "PUT" {
|
||||||
|
putCountAfterSecondPush++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if putCountAfterSecondPush != putCountAfterFirstPush {
|
||||||
|
t.Fatalf("expected no additional PUTs on idempotent push, before=%d after=%d", putCountAfterFirstPush, putCountAfterSecondPush)
|
||||||
|
}
|
||||||
|
|
||||||
|
// pull into a dest dir: one pre-populated correct file (skipped, no GET),
|
||||||
|
// one pre-populated corrupt file (re-downloaded), one absent (downloaded).
|
||||||
|
destDir := t.TempDir()
|
||||||
|
shas := make([]string, 0, len(blobs))
|
||||||
|
for sha := range blobs {
|
||||||
|
shas = append(shas, sha)
|
||||||
|
}
|
||||||
|
correctSHA := shas[0]
|
||||||
|
corruptSHA := shas[1]
|
||||||
|
// absentSHA := shas[2] // left absent on purpose
|
||||||
|
|
||||||
|
correctPath := filepath.Join(destDir, BlobKey(correctSHA))
|
||||||
|
if err := os.MkdirAll(filepath.Dir(correctPath), 0755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(correctPath, []byte(blobs[correctSHA]), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
corruptPath := filepath.Join(destDir, BlobKey(corruptSHA))
|
||||||
|
if err := os.MkdirAll(filepath.Dir(corruptPath), 0755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(corruptPath, []byte("corrupted-content"), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
getCountBeforePull := 0
|
||||||
|
for _, r := range f.requestsSnapshot() {
|
||||||
|
if r.Method == "GET" && !strings.Contains(r.Range, "0-0") {
|
||||||
|
getCountBeforePull++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
out.Reset()
|
||||||
|
errb.Reset()
|
||||||
|
code = Run([]string{"pull", "--manifests", manifestsDir, "--dest", destDir, "--target", "bunny"}, &out, &errb, getenv)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("pull: expected 0, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
|
||||||
|
}
|
||||||
|
if !bytesContains(out.String(), "pulled=2 present=1") {
|
||||||
|
t.Fatalf("pull: expected pulled=2 present=1, got %s", out.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
// verify the correct pre-existing file was never re-fetched with a full GET.
|
||||||
|
fullGETsForCorrect := 0
|
||||||
|
for _, r := range f.requestsSnapshot() {
|
||||||
|
if r.Method == "GET" && strings.HasSuffix(r.Path, correctSHA) && r.Range != "bytes=0-0" {
|
||||||
|
fullGETsForCorrect++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if fullGETsForCorrect != 0 {
|
||||||
|
t.Fatalf("expected no full GET for already-correct blob, got %d", fullGETsForCorrect)
|
||||||
|
}
|
||||||
|
|
||||||
|
// all three blobs should now be present and correct in destDir.
|
||||||
|
for sha, content := range blobs {
|
||||||
|
got, err := os.ReadFile(filepath.Join(destDir, BlobKey(sha)))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("dest blob %s: %v", sha, err)
|
||||||
|
}
|
||||||
|
if string(got) != content {
|
||||||
|
t.Fatalf("dest blob %s: expected %q, got %q", sha, content, string(got))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,121 @@
|
|||||||
|
package depot
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// KeyStore is the zone addressed by object key rather than by depot sha. The
|
||||||
|
// depot names every object after the sha256 of its contents; NWSync does not —
|
||||||
|
// a blob is named after the sha1 of its *uncompressed* bytes while the body
|
||||||
|
// uploaded is the compressed form, and a per-artifact index is named after its
|
||||||
|
// artifact. Both addressing modes want the same transport, retry and probe
|
||||||
|
// discipline, so the sha-addressed Backend rides on this rather than the other
|
||||||
|
// way round.
|
||||||
|
type KeyStore interface {
|
||||||
|
// ProbeKey returns the existence state of one key. transient=true means a
|
||||||
|
// retry might change the answer — never read it as "missing, re-upload".
|
||||||
|
ProbeKey(ctx context.Context, key string) (state ProbeState, transient bool, err error)
|
||||||
|
// PutReader uploads size bytes read from r to key. checksum is the
|
||||||
|
// uppercase hex sha256 of those bytes, which Bunny verifies server-side.
|
||||||
|
PutReader(ctx context.Context, key string, r io.Reader, size int64, checksum string) error
|
||||||
|
// GetKey fetches the whole object at key. Small objects only — it holds
|
||||||
|
// the body in memory and does no hash check, because a key is not always
|
||||||
|
// a content hash.
|
||||||
|
GetKey(ctx context.Context, key string) ([]byte, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewKeyStore returns a KeyStore for cfg's storage zone. Fails closed on a
|
||||||
|
// missing host or read key, matching NewBackend.
|
||||||
|
func NewKeyStore(cfg Config) (KeyStore, error) {
|
||||||
|
if cfg.StorageHost == "" {
|
||||||
|
return nil, errors.New("storage backend requires a storage host")
|
||||||
|
}
|
||||||
|
if cfg.StorageZone == "" {
|
||||||
|
return nil, errors.New("storage backend requires a storage zone")
|
||||||
|
}
|
||||||
|
if cfg.ReadKey == "" {
|
||||||
|
return nil, errors.New("storage backend requires a read key")
|
||||||
|
}
|
||||||
|
return &httpBackend{name: "bunny", client: newHTTPClient(cfg), cfg: cfg}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// keyURL is the storage URL of one object key.
|
||||||
|
func (b *httpBackend) keyURL(key string) string {
|
||||||
|
host := b.cfg.StorageHost
|
||||||
|
// StorageHost is normally a bare host ("storage.bunnycdn.com"); allow a
|
||||||
|
// full scheme (used by tests against httptest.NewServer) to pass through
|
||||||
|
// unchanged.
|
||||||
|
if !strings.Contains(host, "://") {
|
||||||
|
host = "https://" + host
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("%s/%s/%s", strings.TrimSuffix(host, "/"), b.cfg.StorageZone, key)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (b *httpBackend) ProbeKey(ctx context.Context, key string) (ProbeState, bool, error) {
|
||||||
|
return b.rangeProbe(ctx, b.keyURL(key), map[string]string{"AccessKey": b.cfg.ReadKey})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (b *httpBackend) PutReader(ctx context.Context, key string, r io.Reader, size int64, checksum string) error {
|
||||||
|
if b.name == "cdn" {
|
||||||
|
return errors.New("cdn backend is read-only")
|
||||||
|
}
|
||||||
|
if b.cfg.WriteKey == "" {
|
||||||
|
return errors.New("storage backend requires a write key to write")
|
||||||
|
}
|
||||||
|
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodPut, b.keyURL(key), r)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
req.ContentLength = size
|
||||||
|
req.Header.Set("AccessKey", b.cfg.WriteKey)
|
||||||
|
// Bunny defines Checksum as sha256 of the body and rejects a mismatch, so
|
||||||
|
// this is server-side integrity checking, not decoration.
|
||||||
|
req.Header.Set("Checksum", strings.ToUpper(checksum))
|
||||||
|
|
||||||
|
resp, err := b.client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
_, _ = io.Copy(io.Discard, resp.Body)
|
||||||
|
|
||||||
|
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||||
|
return fmt.Errorf("put %s: unexpected status %d", key, resp.StatusCode)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (b *httpBackend) GetKey(ctx context.Context, key string) ([]byte, error) {
|
||||||
|
resp, err := b.get(ctx, b.keyURL(key), map[string]string{"AccessKey": b.cfg.ReadKey})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||||
|
_, _ = io.Copy(io.Discard, resp.Body)
|
||||||
|
return nil, fmt.Errorf("get %s: unexpected status %d", key, resp.StatusCode)
|
||||||
|
}
|
||||||
|
return io.ReadAll(resp.Body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// putFile uploads the file at src to key, streaming it. checksum is the
|
||||||
|
// uppercase hex sha256 of the file's bytes.
|
||||||
|
func (b *httpBackend) putFile(ctx context.Context, key, src, checksum string) error {
|
||||||
|
f, err := os.Open(src)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer f.Close()
|
||||||
|
info, err := f.Stat()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return b.PutReader(ctx, key, f, info.Size(), checksum)
|
||||||
|
}
|
||||||
@@ -0,0 +1,113 @@
|
|||||||
|
package depot
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
)
|
||||||
|
|
||||||
|
// LocalBackend implements Backend for local filesystem storage.
|
||||||
|
type LocalBackend struct {
|
||||||
|
Root string
|
||||||
|
}
|
||||||
|
|
||||||
|
// Name returns the backend name.
|
||||||
|
func (b *LocalBackend) Name() string {
|
||||||
|
return "local"
|
||||||
|
}
|
||||||
|
|
||||||
|
// Probe checks if a blob exists.
|
||||||
|
func (b *LocalBackend) Probe(ctx context.Context, sha string) (ProbeState, bool, error) {
|
||||||
|
blobPath := filepath.Join(b.Root, BlobKey(sha))
|
||||||
|
_, err := os.Stat(blobPath)
|
||||||
|
if err == nil {
|
||||||
|
return Present, false, nil
|
||||||
|
}
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return Absent, false, nil
|
||||||
|
}
|
||||||
|
// Real I/O error (permission, etc.)
|
||||||
|
return Absent, false, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Put copies the file from src to the blob storage.
|
||||||
|
func (b *LocalBackend) Put(ctx context.Context, sha, src string) error {
|
||||||
|
blobPath := filepath.Join(b.Root, BlobKey(sha))
|
||||||
|
blobDir := filepath.Dir(blobPath)
|
||||||
|
|
||||||
|
// Create parent directories
|
||||||
|
if err := os.MkdirAll(blobDir, 0755); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create temp file in the same directory for atomic rename
|
||||||
|
tmpFile, err := os.CreateTemp(blobDir, ".tmp-")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer os.Remove(tmpFile.Name())
|
||||||
|
|
||||||
|
// Copy source to temp file
|
||||||
|
srcFile, err := os.Open(src)
|
||||||
|
if err != nil {
|
||||||
|
tmpFile.Close()
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer srcFile.Close()
|
||||||
|
|
||||||
|
_, err = io.Copy(tmpFile, srcFile)
|
||||||
|
tmpFile.Close()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Atomic rename
|
||||||
|
return os.Rename(tmpFile.Name(), blobPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get fetches the blob, re-hashes it, and deletes it if the hash doesn't match.
|
||||||
|
func (b *LocalBackend) Get(ctx context.Context, sha, dest string) error {
|
||||||
|
blobPath := filepath.Join(b.Root, BlobKey(sha))
|
||||||
|
|
||||||
|
// Create temp file in dest directory for atomic rename
|
||||||
|
destDir := filepath.Dir(dest)
|
||||||
|
if err := os.MkdirAll(destDir, 0755); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
tmpFile, err := os.CreateTemp(destDir, ".tmp-")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer os.Remove(tmpFile.Name())
|
||||||
|
|
||||||
|
// Copy and hash simultaneously
|
||||||
|
srcFile, err := os.Open(blobPath)
|
||||||
|
if err != nil {
|
||||||
|
tmpFile.Close()
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer srcFile.Close()
|
||||||
|
|
||||||
|
hasher := sha256.New()
|
||||||
|
teeReader := io.TeeReader(srcFile, hasher)
|
||||||
|
|
||||||
|
_, err = io.Copy(tmpFile, teeReader)
|
||||||
|
tmpFile.Close()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check hash
|
||||||
|
gotHash := fmt.Sprintf("%x", hasher.Sum(nil))
|
||||||
|
if gotHash != sha {
|
||||||
|
os.Remove(tmpFile.Name())
|
||||||
|
return fmt.Errorf("hash mismatch: expected %s, got %s", sha, gotHash)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Atomic rename
|
||||||
|
return os.Rename(tmpFile.Name(), dest)
|
||||||
|
}
|
||||||
@@ -0,0 +1,139 @@
|
|||||||
|
package depot
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestLocalRoundTrip(t *testing.T) {
|
||||||
|
tmpDir := t.TempDir()
|
||||||
|
backend := &LocalBackend{Root: tmpDir}
|
||||||
|
|
||||||
|
// Generate test data
|
||||||
|
testData := []byte("hello world")
|
||||||
|
testSHA := fmt.Sprintf("%x", sha256.Sum256(testData))
|
||||||
|
|
||||||
|
// Create source file
|
||||||
|
srcFile := filepath.Join(tmpDir, "source.txt")
|
||||||
|
if err := os.WriteFile(srcFile, testData, 0644); err != nil {
|
||||||
|
t.Fatalf("failed to create source file: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Put
|
||||||
|
ctx := context.Background()
|
||||||
|
if err := backend.Put(ctx, testSHA, srcFile); err != nil {
|
||||||
|
t.Fatalf("Put failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Probe should be Present
|
||||||
|
state, transient, err := backend.Probe(ctx, testSHA)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Probe failed: %v", err)
|
||||||
|
}
|
||||||
|
if state != Present {
|
||||||
|
t.Fatalf("expected Present, got %v", state)
|
||||||
|
}
|
||||||
|
if transient {
|
||||||
|
t.Fatalf("local backend should never be transient")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get
|
||||||
|
destFile := filepath.Join(tmpDir, "dest.txt")
|
||||||
|
if err := backend.Get(ctx, testSHA, destFile); err != nil {
|
||||||
|
t.Fatalf("Get failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify bytes match
|
||||||
|
gotData, err := os.ReadFile(destFile)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to read dest file: %v", err)
|
||||||
|
}
|
||||||
|
if !bytes.Equal(gotData, testData) {
|
||||||
|
t.Fatalf("data mismatch: want %s, got %s", testData, gotData)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLocalProbeAbsent(t *testing.T) {
|
||||||
|
tmpDir := t.TempDir()
|
||||||
|
backend := &LocalBackend{Root: tmpDir}
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
state, transient, err := backend.Probe(ctx, "0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Probe should not error on missing blob: %v", err)
|
||||||
|
}
|
||||||
|
if state != Absent {
|
||||||
|
t.Fatalf("expected Absent, got %v", state)
|
||||||
|
}
|
||||||
|
if transient {
|
||||||
|
t.Fatalf("local backend should never be transient")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLocalGetHashMismatch(t *testing.T) {
|
||||||
|
tmpDir := t.TempDir()
|
||||||
|
backend := &LocalBackend{Root: tmpDir}
|
||||||
|
|
||||||
|
// Create a corrupt blob at the expected path
|
||||||
|
sha := "0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d0b1d"
|
||||||
|
blobKey := BlobKey(sha)
|
||||||
|
blobPath := filepath.Join(tmpDir, blobKey)
|
||||||
|
|
||||||
|
if err := os.MkdirAll(filepath.Dir(blobPath), 0755); err != nil {
|
||||||
|
t.Fatalf("failed to create blob dir: %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(blobPath, []byte("wrong data"), 0644); err != nil {
|
||||||
|
t.Fatalf("failed to create blob file: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get should error
|
||||||
|
ctx := context.Background()
|
||||||
|
destFile := filepath.Join(tmpDir, "dest.txt")
|
||||||
|
err := backend.Get(ctx, sha, destFile)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("Get should error on hash mismatch")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Destination file should not exist
|
||||||
|
_, err = os.Stat(destFile)
|
||||||
|
if !os.IsNotExist(err) {
|
||||||
|
t.Fatalf("dest file should not exist after Get error: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLocalPutIdempotent(t *testing.T) {
|
||||||
|
tmpDir := t.TempDir()
|
||||||
|
backend := &LocalBackend{Root: tmpDir}
|
||||||
|
|
||||||
|
testData := []byte("hello world")
|
||||||
|
testSHA := fmt.Sprintf("%x", sha256.Sum256(testData))
|
||||||
|
|
||||||
|
srcFile := filepath.Join(tmpDir, "source.txt")
|
||||||
|
if err := os.WriteFile(srcFile, testData, 0644); err != nil {
|
||||||
|
t.Fatalf("failed to create source file: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
// Put twice
|
||||||
|
if err := backend.Put(ctx, testSHA, srcFile); err != nil {
|
||||||
|
t.Fatalf("first Put failed: %v", err)
|
||||||
|
}
|
||||||
|
if err := backend.Put(ctx, testSHA, srcFile); err != nil {
|
||||||
|
t.Fatalf("second Put failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify blob exists
|
||||||
|
state, _, err := backend.Probe(ctx, testSHA)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Probe failed: %v", err)
|
||||||
|
}
|
||||||
|
if state != Present {
|
||||||
|
t.Fatalf("expected Present after Put, got %v", state)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
package depot
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
|
||||||
|
"gopkg.in/yaml.v3"
|
||||||
|
)
|
||||||
|
|
||||||
|
type manifestFile struct {
|
||||||
|
Assets []struct {
|
||||||
|
Path string `yaml:"path"`
|
||||||
|
SHA256 string `yaml:"sha256"`
|
||||||
|
Size int64 `yaml:"size"`
|
||||||
|
} `yaml:"assets"`
|
||||||
|
}
|
||||||
|
|
||||||
|
var sha256Pattern = regexp.MustCompile(`^[0-9a-f]{64}$`)
|
||||||
|
|
||||||
|
func ValidSHA(s string) bool {
|
||||||
|
return sha256Pattern.MatchString(s)
|
||||||
|
}
|
||||||
|
|
||||||
|
func BlobKey(sha string) string {
|
||||||
|
return fmt.Sprintf("sha256/%s/%s/%s", sha[0:2], sha[2:4], sha)
|
||||||
|
}
|
||||||
|
|
||||||
|
func ReferencedSHAs(dir string) (map[string]int64, error) {
|
||||||
|
entries, err := os.ReadDir(dir)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
result := make(map[string]int64)
|
||||||
|
foundAny := false
|
||||||
|
|
||||||
|
for _, entry := range entries {
|
||||||
|
if entry.IsDir() {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if filepath.Ext(entry.Name()) != ".yml" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
foundAny = true
|
||||||
|
path := filepath.Join(dir, entry.Name())
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
var manifest manifestFile
|
||||||
|
if err := yaml.Unmarshal(data, &manifest); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, asset := range manifest.Assets {
|
||||||
|
if !ValidSHA(asset.SHA256) {
|
||||||
|
return nil, fmt.Errorf("%s: asset %q: invalid sha256 %q", entry.Name(), asset.Path, asset.SHA256)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Keep the largest size for each sha
|
||||||
|
if asset.Size > result[asset.SHA256] {
|
||||||
|
result[asset.SHA256] = asset.Size
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if !foundAny {
|
||||||
|
return nil, fmt.Errorf("no *.yml files found in %s", dir)
|
||||||
|
}
|
||||||
|
|
||||||
|
return result, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,129 @@
|
|||||||
|
package depot
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestValidSHA(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
input string
|
||||||
|
want bool
|
||||||
|
}{
|
||||||
|
{"0b1d1234567890abcdef1234567890abcdef1234567890abcdef1234567890ab", true},
|
||||||
|
{"0000000000000000000000000000000000000000000000000000000000000000", true},
|
||||||
|
{"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", true},
|
||||||
|
{"0b1d1234567890ABCDEF1234567890abcdef1234567890abcdef1234567890ab", false}, // uppercase
|
||||||
|
{"0b1d1234567890abcde", false}, // too short
|
||||||
|
{"0b1d1234567890abcdef1234567890abcdef1234567890abcdef1234567890abff", false}, // too long
|
||||||
|
{"0b1d1234567890abcdef1234567890abcdef1234567890abcdef1234567890ag", false}, // invalid hex
|
||||||
|
{"", false},
|
||||||
|
}
|
||||||
|
for _, tc := range tests {
|
||||||
|
got := ValidSHA(tc.input)
|
||||||
|
if got != tc.want {
|
||||||
|
t.Errorf("ValidSHA(%q): got %v, want %v", tc.input, got, tc.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBlobKey(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
sha string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{"0b1d1234567890abcdef1234567890abcdef1234567890abcdef1234567890ab", "sha256/0b/1d/0b1d1234567890abcdef1234567890abcdef1234567890abcdef1234567890ab"},
|
||||||
|
{"abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789", "sha256/ab/cd/abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789"},
|
||||||
|
}
|
||||||
|
for _, tc := range tests {
|
||||||
|
got := BlobKey(tc.sha)
|
||||||
|
if got != tc.want {
|
||||||
|
t.Errorf("BlobKey(%q): got %q, want %q", tc.sha, got, tc.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReferencedSHAs(t *testing.T) {
|
||||||
|
// Test: temp dir with two yml files sharing one sha → dedup
|
||||||
|
tmpdir := t.TempDir()
|
||||||
|
|
||||||
|
// First manifest with sha1 and sha2
|
||||||
|
manifest1 := `assets:
|
||||||
|
- path: file1.txt
|
||||||
|
sha256: 0b1d1234567890abcdef1234567890abcdef1234567890abcdef1234567890ab
|
||||||
|
size: 100
|
||||||
|
- path: file2.txt
|
||||||
|
sha256: abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789
|
||||||
|
size: 200
|
||||||
|
`
|
||||||
|
if err := os.WriteFile(filepath.Join(tmpdir, "manifest1.yml"), []byte(manifest1), 0644); err != nil {
|
||||||
|
t.Fatalf("write manifest1: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Second manifest with sha2 and sha3
|
||||||
|
manifest2 := `assets:
|
||||||
|
- path: file3.txt
|
||||||
|
sha256: abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789
|
||||||
|
size: 300
|
||||||
|
- path: file4.txt
|
||||||
|
sha256: fedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210
|
||||||
|
size: 400
|
||||||
|
`
|
||||||
|
if err := os.WriteFile(filepath.Join(tmpdir, "manifest2.yml"), []byte(manifest2), 0644); err != nil {
|
||||||
|
t.Fatalf("write manifest2: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
result, err := ReferencedSHAs(tmpdir)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReferencedSHAs: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Should have 3 unique SHAs
|
||||||
|
if len(result) != 3 {
|
||||||
|
t.Errorf("ReferencedSHAs: got %d unique SHAs, want 3", len(result))
|
||||||
|
}
|
||||||
|
|
||||||
|
expectedSizes := map[string]int64{
|
||||||
|
"0b1d1234567890abcdef1234567890abcdef1234567890abcdef1234567890ab": 100,
|
||||||
|
"abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789": 300, // Largest size for duplicated sha
|
||||||
|
"fedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210": 400,
|
||||||
|
}
|
||||||
|
for sha, expectedSize := range expectedSizes {
|
||||||
|
size, exists := result[sha]
|
||||||
|
if !exists {
|
||||||
|
t.Errorf("ReferencedSHAs: sha %q missing", sha)
|
||||||
|
}
|
||||||
|
if size != expectedSize {
|
||||||
|
t.Errorf("ReferencedSHAs: sha %q size: got %d, want %d", sha, size, expectedSize)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test: bad sha → error naming the file
|
||||||
|
badManifest := `assets:
|
||||||
|
- path: file5.txt
|
||||||
|
sha256: not_a_valid_sha
|
||||||
|
size: 500
|
||||||
|
`
|
||||||
|
if err := os.WriteFile(filepath.Join(tmpdir, "badsha.yml"), []byte(badManifest), 0644); err != nil {
|
||||||
|
t.Fatalf("write badsha: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = ReferencedSHAs(tmpdir)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("ReferencedSHAs with bad sha: got nil error, want error naming file and asset path")
|
||||||
|
}
|
||||||
|
want := `badsha.yml: asset "file5.txt": invalid sha256 "not_a_valid_sha"`
|
||||||
|
if err.Error() != want {
|
||||||
|
t.Errorf("ReferencedSHAs error message: got %q, want %q", err.Error(), want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReferencedSHAsEmpty(t *testing.T) {
|
||||||
|
// Test: empty dir → error
|
||||||
|
tmpdir := t.TempDir()
|
||||||
|
_, err := ReferencedSHAs(tmpdir)
|
||||||
|
if err == nil {
|
||||||
|
t.Errorf("ReferencedSHAs on empty dir: got nil error, want error")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,237 @@
|
|||||||
|
package depot
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// NewBackend returns the backend for name ("local"|"cdn"|"bunny").
|
||||||
|
// localRoot is used only for "local". Fails closed: "bunny" with empty
|
||||||
|
// StorageHost or empty ReadKey returns an error (never prompts);
|
||||||
|
// unknown name returns an error.
|
||||||
|
func NewBackend(name, localRoot string, cfg Config) (Backend, error) {
|
||||||
|
switch name {
|
||||||
|
case "local":
|
||||||
|
if localRoot == "" {
|
||||||
|
return nil, errors.New("local backend requires a non-empty root")
|
||||||
|
}
|
||||||
|
return &LocalBackend{Root: localRoot}, nil
|
||||||
|
case "cdn":
|
||||||
|
return &httpBackend{
|
||||||
|
name: "cdn",
|
||||||
|
client: newHTTPClient(cfg),
|
||||||
|
cfg: cfg,
|
||||||
|
}, nil
|
||||||
|
case "bunny":
|
||||||
|
if cfg.StorageHost == "" {
|
||||||
|
return nil, errors.New("bunny backend requires BUNNY_STORAGE_HOST")
|
||||||
|
}
|
||||||
|
if cfg.ReadKey == "" {
|
||||||
|
return nil, errors.New("bunny backend requires BUNNY_STORAGE_READ_PASSWORD or BUNNY_STORAGE_PASSWORD")
|
||||||
|
}
|
||||||
|
return &httpBackend{
|
||||||
|
name: "bunny",
|
||||||
|
client: newHTTPClient(cfg),
|
||||||
|
cfg: cfg,
|
||||||
|
}, nil
|
||||||
|
default:
|
||||||
|
return nil, fmt.Errorf("unknown backend %q", name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// newHTTPClient builds an IPv4-only client per spec (HEAD probes are banned;
|
||||||
|
// IPv6 dial hazards are out of scope for this depot).
|
||||||
|
func newHTTPClient(cfg Config) *http.Client {
|
||||||
|
transport := &http.Transport{
|
||||||
|
DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
|
||||||
|
d := net.Dialer{Timeout: cfg.ConnectTimeout}
|
||||||
|
return d.DialContext(ctx, "tcp4", addr)
|
||||||
|
},
|
||||||
|
MaxIdleConnsPerHost: cfg.ProbeJobs,
|
||||||
|
}
|
||||||
|
return &http.Client{Transport: transport, Timeout: cfg.ProbeMaxTime}
|
||||||
|
}
|
||||||
|
|
||||||
|
// httpBackend implements Backend for both cdn (read-only) and bunny
|
||||||
|
// (read/write) over HTTP, sharing probe/get/put logic.
|
||||||
|
type httpBackend struct {
|
||||||
|
name string
|
||||||
|
client *http.Client
|
||||||
|
cfg Config
|
||||||
|
}
|
||||||
|
|
||||||
|
func (b *httpBackend) Name() string { return b.name }
|
||||||
|
|
||||||
|
func (b *httpBackend) storageURL(sha string) string { return b.keyURL(BlobKey(sha)) }
|
||||||
|
|
||||||
|
func (b *httpBackend) cdnURL(sha string) string {
|
||||||
|
return fmt.Sprintf("%s/%s", b.cfg.CDNBase, BlobKey(sha))
|
||||||
|
}
|
||||||
|
|
||||||
|
// probeRetrySleep is the backoff sleeper for transient probe retries;
|
||||||
|
// tests stub it (same pattern as sweep.go's confirmSleep).
|
||||||
|
var probeRetrySleep = time.Sleep
|
||||||
|
|
||||||
|
// rangeProbe issues GET <url> with Range: bytes=0-0 and classifies the
|
||||||
|
// response. Never HEAD (banned by spec). Transient outcomes (transport
|
||||||
|
// error, or a status that is neither 2xx nor 404/410) retry up to 2 more
|
||||||
|
// times with linear backoff (1s, 2s) — the CDN edge throttles sustained
|
||||||
|
// sweeps; the bash this ports absorbed that with curl --retry 2.
|
||||||
|
func (b *httpBackend) rangeProbe(ctx context.Context, url string, headers map[string]string) (ProbeState, bool, error) {
|
||||||
|
for attempt := 0; ; attempt++ {
|
||||||
|
state, transient, err := b.rangeProbeOnce(ctx, url, headers)
|
||||||
|
if !transient || attempt >= 2 {
|
||||||
|
return state, transient, err
|
||||||
|
}
|
||||||
|
probeRetrySleep(time.Duration(attempt+1) * time.Second)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (b *httpBackend) rangeProbeOnce(ctx context.Context, url string, headers map[string]string) (ProbeState, bool, error) {
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||||
|
if err != nil {
|
||||||
|
return Unconfirmed, true, err
|
||||||
|
}
|
||||||
|
req.Header.Set("Range", "bytes=0-0")
|
||||||
|
for k, v := range headers {
|
||||||
|
req.Header.Set(k, v)
|
||||||
|
}
|
||||||
|
|
||||||
|
resp, err := b.client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return Unconfirmed, true, err
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
_, _ = io.Copy(io.Discard, resp.Body)
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case resp.StatusCode >= 200 && resp.StatusCode < 300:
|
||||||
|
return Present, false, nil
|
||||||
|
case resp.StatusCode == 404 || resp.StatusCode == 410:
|
||||||
|
return Absent, false, nil
|
||||||
|
default:
|
||||||
|
return Unconfirmed, true, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Probe returns the existence state of sha at this backend.
|
||||||
|
func (b *httpBackend) Probe(ctx context.Context, sha string) (ProbeState, bool, error) {
|
||||||
|
if b.name == "cdn" {
|
||||||
|
return b.rangeProbe(ctx, b.cdnURL(sha), nil)
|
||||||
|
}
|
||||||
|
return b.rangeProbe(ctx, b.storageURL(sha), map[string]string{"AccessKey": b.cfg.ReadKey})
|
||||||
|
}
|
||||||
|
|
||||||
|
// Put uploads src for sha. cdn is read-only. A depot object is named after the
|
||||||
|
// sha256 of its own bytes, so the key's sha doubles as the Checksum header.
|
||||||
|
func (b *httpBackend) Put(ctx context.Context, sha, src string) error {
|
||||||
|
if b.name == "cdn" {
|
||||||
|
return errors.New("cdn backend is read-only")
|
||||||
|
}
|
||||||
|
if b.cfg.WriteKey == "" {
|
||||||
|
return errors.New("bunny backend requires BUNNY_STORAGE_PASSWORD to write")
|
||||||
|
}
|
||||||
|
|
||||||
|
state, _, err := b.Probe(ctx, sha)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if state == Present {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return b.putFile(ctx, BlobKey(sha), src, sha)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get fetches sha into dest via temp file + rename, re-hashing and deleting
|
||||||
|
// on mismatch. Mirrors bash _depot_bunny_get: try CDN first, fall back to
|
||||||
|
// storage with ReadKey.
|
||||||
|
func (b *httpBackend) Get(ctx context.Context, sha, dest string) error {
|
||||||
|
destDir := filepath.Dir(dest)
|
||||||
|
if err := os.MkdirAll(destDir, 0755); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
tmpFile, err := os.CreateTemp(destDir, ".tmp-")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
tmpName := tmpFile.Name()
|
||||||
|
defer os.Remove(tmpName)
|
||||||
|
|
||||||
|
resp, err := b.fetch(ctx, sha)
|
||||||
|
if err != nil {
|
||||||
|
tmpFile.Close()
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
hasher := sha256.New()
|
||||||
|
tee := io.TeeReader(resp.Body, hasher)
|
||||||
|
_, err = io.Copy(tmpFile, tee)
|
||||||
|
tmpFile.Close()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
gotHash := fmt.Sprintf("%x", hasher.Sum(nil))
|
||||||
|
if gotHash != sha {
|
||||||
|
os.Remove(tmpName)
|
||||||
|
return fmt.Errorf("hash mismatch: expected %s, got %s", sha, gotHash)
|
||||||
|
}
|
||||||
|
|
||||||
|
return os.Rename(tmpName, dest)
|
||||||
|
}
|
||||||
|
|
||||||
|
// fetch tries the CDN URL first (no auth), falling back to the storage URL
|
||||||
|
// with ReadKey on any non-2xx response or transport error.
|
||||||
|
func (b *httpBackend) fetch(ctx context.Context, sha string) (*http.Response, error) {
|
||||||
|
if b.name != "cdn" {
|
||||||
|
if resp, err := b.get(ctx, b.cdnURL(sha), nil); err == nil && resp.StatusCode >= 200 && resp.StatusCode < 300 {
|
||||||
|
return resp, nil
|
||||||
|
} else if err == nil {
|
||||||
|
_, _ = io.Copy(io.Discard, resp.Body)
|
||||||
|
resp.Body.Close()
|
||||||
|
}
|
||||||
|
resp, err := b.get(ctx, b.storageURL(sha), map[string]string{"AccessKey": b.cfg.ReadKey})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||||
|
_, _ = io.Copy(io.Discard, resp.Body)
|
||||||
|
resp.Body.Close()
|
||||||
|
return nil, fmt.Errorf("bunny get %s: unexpected status %d", sha, resp.StatusCode)
|
||||||
|
}
|
||||||
|
return resp, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
resp, err := b.get(ctx, b.cdnURL(sha), nil)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||||
|
_, _ = io.Copy(io.Discard, resp.Body)
|
||||||
|
resp.Body.Close()
|
||||||
|
return nil, fmt.Errorf("cdn get %s: unexpected status %d", sha, resp.StatusCode)
|
||||||
|
}
|
||||||
|
return resp, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (b *httpBackend) get(ctx context.Context, url string, headers map[string]string) (*http.Response, error) {
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
for k, v := range headers {
|
||||||
|
req.Header.Set(k, v)
|
||||||
|
}
|
||||||
|
return b.client.Do(req)
|
||||||
|
}
|
||||||
@@ -0,0 +1,549 @@
|
|||||||
|
package depot
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func shaOf(s string) string {
|
||||||
|
return fmt.Sprintf("%x", sha256.Sum256([]byte(s)))
|
||||||
|
}
|
||||||
|
|
||||||
|
type recordedReq struct {
|
||||||
|
Method string
|
||||||
|
Path string
|
||||||
|
Range string
|
||||||
|
Headers http.Header
|
||||||
|
}
|
||||||
|
|
||||||
|
// fakeBunny is a minimal recording fake for Bunny storage/CDN endpoints.
|
||||||
|
type fakeBunny struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
requests []recordedReq
|
||||||
|
headCount int
|
||||||
|
blobs map[string][]byte
|
||||||
|
statusFor map[string]int // sha -> status code override for probe/get
|
||||||
|
}
|
||||||
|
|
||||||
|
func newFakeBunny() *fakeBunny {
|
||||||
|
return &fakeBunny{blobs: map[string][]byte{}, statusFor: map[string]int{}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeBunny) handler() http.HandlerFunc {
|
||||||
|
return func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
f.mu.Lock()
|
||||||
|
f.requests = append(f.requests, recordedReq{
|
||||||
|
Method: r.Method,
|
||||||
|
Path: r.URL.Path,
|
||||||
|
Range: r.Header.Get("Range"),
|
||||||
|
Headers: r.Header.Clone(),
|
||||||
|
})
|
||||||
|
if r.Method == http.MethodHead {
|
||||||
|
f.headCount++
|
||||||
|
f.mu.Unlock()
|
||||||
|
w.WriteHeader(500)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
f.mu.Unlock()
|
||||||
|
|
||||||
|
sha := strings.TrimPrefix(r.URL.Path, "/")
|
||||||
|
// last path element is the sha
|
||||||
|
parts := strings.Split(sha, "/")
|
||||||
|
sha = parts[len(parts)-1]
|
||||||
|
|
||||||
|
if r.Method == http.MethodPut {
|
||||||
|
buf, _ := io.ReadAll(r.Body)
|
||||||
|
f.mu.Lock()
|
||||||
|
f.blobs[sha] = buf
|
||||||
|
f.mu.Unlock()
|
||||||
|
w.WriteHeader(201)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// GET (range probe / actual get)
|
||||||
|
f.mu.Lock()
|
||||||
|
status, hasStatus := f.statusFor[sha]
|
||||||
|
data, ok := f.blobs[sha]
|
||||||
|
f.mu.Unlock()
|
||||||
|
|
||||||
|
if hasStatus {
|
||||||
|
w.WriteHeader(status)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !ok {
|
||||||
|
w.WriteHeader(404)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.WriteHeader(206)
|
||||||
|
_, _ = w.Write(data)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeBunny) requestsSnapshot() []recordedReq {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
out := make([]recordedReq, len(f.requests))
|
||||||
|
copy(out, f.requests)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func testCfg(storageHost, readKey, writeKey string) Config {
|
||||||
|
return Config{
|
||||||
|
CDNBase: "http://127.0.0.1:1", // unreachable by default; overridden per test
|
||||||
|
StorageHost: storageHost,
|
||||||
|
StorageZone: "sow-assets-depot",
|
||||||
|
ReadKey: readKey,
|
||||||
|
WriteKey: writeKey,
|
||||||
|
ProbeJobs: 4,
|
||||||
|
Jobs: 4,
|
||||||
|
ConnectTimeout: 2 * time.Second,
|
||||||
|
ProbeMaxTime: 2 * time.Second,
|
||||||
|
ConfirmRetries: 3,
|
||||||
|
ConfirmMax: 200,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// hostPort returns a StorageHost value for tests: a full "http://host:port"
|
||||||
|
// base URL, which storageURL() passes through unchanged (bypassing the
|
||||||
|
// production https:// default so httptest.NewServer works without TLS).
|
||||||
|
func hostPort(srv *httptest.Server) string {
|
||||||
|
return srv.URL
|
||||||
|
}
|
||||||
|
|
||||||
|
// stubProbeSleep replaces probeRetrySleep for the test, recording durations.
|
||||||
|
func stubProbeSleep(t *testing.T) *[]time.Duration {
|
||||||
|
t.Helper()
|
||||||
|
var slept []time.Duration
|
||||||
|
old := probeRetrySleep
|
||||||
|
probeRetrySleep = func(d time.Duration) { slept = append(slept, d) }
|
||||||
|
t.Cleanup(func() { probeRetrySleep = old })
|
||||||
|
return &slept
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProbeRetriesTransientThenPresent(t *testing.T) {
|
||||||
|
slept := stubProbeSleep(t)
|
||||||
|
sha := shaOf("throttled-blob")
|
||||||
|
|
||||||
|
var calls int
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
calls++
|
||||||
|
if calls == 1 {
|
||||||
|
w.WriteHeader(428)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.WriteHeader(206)
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
cfg := testCfg(hostPort(srv), "readkey", "writekey")
|
||||||
|
b, err := NewBackend("bunny", "", cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewBackend: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
state, transient, err := b.Probe(context.Background(), sha)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Probe: %v", err)
|
||||||
|
}
|
||||||
|
if state != Present || transient {
|
||||||
|
t.Fatalf("expected Present/non-transient after retry, got %v/%v", state, transient)
|
||||||
|
}
|
||||||
|
if calls != 2 {
|
||||||
|
t.Fatalf("expected 2 requests (428 then 206), got %d", calls)
|
||||||
|
}
|
||||||
|
if len(*slept) != 1 || (*slept)[0] != time.Second {
|
||||||
|
t.Fatalf("expected one 1s backoff, got %v", *slept)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProbe404NoRetry(t *testing.T) {
|
||||||
|
slept := stubProbeSleep(t)
|
||||||
|
sha := shaOf("gone-blob")
|
||||||
|
|
||||||
|
f := newFakeBunny()
|
||||||
|
srv := httptest.NewServer(f.handler())
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
cfg := testCfg(hostPort(srv), "readkey", "writekey")
|
||||||
|
b, err := NewBackend("bunny", "", cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewBackend: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
state, transient, err := b.Probe(context.Background(), sha)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Probe: %v", err)
|
||||||
|
}
|
||||||
|
if state != Absent || transient {
|
||||||
|
t.Fatalf("expected Absent/non-transient, got %v/%v", state, transient)
|
||||||
|
}
|
||||||
|
if got := len(f.requestsSnapshot()); got != 1 {
|
||||||
|
t.Fatalf("expected exactly 1 request for 404, got %d", got)
|
||||||
|
}
|
||||||
|
if len(*slept) != 0 {
|
||||||
|
t.Fatalf("expected zero backoffs for 404, got %v", *slept)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProbeUsesRangeGetNotHead(t *testing.T) {
|
||||||
|
sha := shaOf("present-blob")
|
||||||
|
f := newFakeBunny()
|
||||||
|
f.blobs[sha] = []byte("hello")
|
||||||
|
srv := httptest.NewServer(f.handler())
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
cfg := testCfg(hostPort(srv), "readkey", "writekey")
|
||||||
|
b, err := NewBackend("bunny", "", cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewBackend: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
state, transient, err := b.Probe(context.Background(), sha)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Probe: %v", err)
|
||||||
|
}
|
||||||
|
if state != Present || transient {
|
||||||
|
t.Fatalf("expected Present/non-transient, got %v/%v", state, transient)
|
||||||
|
}
|
||||||
|
|
||||||
|
reqs := f.requestsSnapshot()
|
||||||
|
if len(reqs) != 1 {
|
||||||
|
t.Fatalf("expected exactly 1 request, got %d", len(reqs))
|
||||||
|
}
|
||||||
|
if reqs[0].Method != "GET" {
|
||||||
|
t.Fatalf("expected GET, got %s", reqs[0].Method)
|
||||||
|
}
|
||||||
|
if reqs[0].Range != "bytes=0-0" {
|
||||||
|
t.Fatalf("expected Range bytes=0-0, got %q", reqs[0].Range)
|
||||||
|
}
|
||||||
|
if f.headCount != 0 {
|
||||||
|
t.Fatalf("expected zero HEAD requests, got %d", f.headCount)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProbeStates(t *testing.T) {
|
||||||
|
stubProbeSleep(t)
|
||||||
|
cases := []struct {
|
||||||
|
status int
|
||||||
|
wantState ProbeState
|
||||||
|
wantTransient bool
|
||||||
|
}{
|
||||||
|
{200, Present, false},
|
||||||
|
{206, Present, false},
|
||||||
|
{404, Absent, false},
|
||||||
|
{428, Unconfirmed, true},
|
||||||
|
{503, Unconfirmed, true},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, c := range cases {
|
||||||
|
t.Run(fmt.Sprintf("status_%d", c.status), func(t *testing.T) {
|
||||||
|
sha := shaOf(fmt.Sprintf("blob-%d", c.status))
|
||||||
|
f := newFakeBunny()
|
||||||
|
f.statusFor[sha] = c.status
|
||||||
|
srv := httptest.NewServer(f.handler())
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
cfg := testCfg(hostPort(srv), "readkey", "writekey")
|
||||||
|
b, err := NewBackend("bunny", "", cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewBackend: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
state, transient, err := b.Probe(context.Background(), sha)
|
||||||
|
if err != nil && c.status < 500 {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
if state != c.wantState {
|
||||||
|
t.Fatalf("status %d: expected state %v, got %v", c.status, c.wantState, state)
|
||||||
|
}
|
||||||
|
if transient != c.wantTransient {
|
||||||
|
t.Fatalf("status %d: expected transient=%v, got %v", c.status, c.wantTransient, transient)
|
||||||
|
}
|
||||||
|
if f.headCount != 0 {
|
||||||
|
t.Fatalf("expected zero HEAD requests, got %d", f.headCount)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBunnyPutSequence(t *testing.T) {
|
||||||
|
t.Run("absent sha", func(t *testing.T) {
|
||||||
|
sha := shaOf("new-blob")
|
||||||
|
f := newFakeBunny()
|
||||||
|
srv := httptest.NewServer(f.handler())
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
cfg := testCfg(hostPort(srv), "readkey", "writekey")
|
||||||
|
b, err := NewBackend("bunny", "", cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewBackend: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
srcPath := filepath.Join(dir, "src")
|
||||||
|
if err := os.WriteFile(srcPath, []byte("new-blob-content"), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := b.Put(context.Background(), sha, srcPath); err != nil {
|
||||||
|
t.Fatalf("Put: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
reqs := f.requestsSnapshot()
|
||||||
|
if len(reqs) != 2 {
|
||||||
|
t.Fatalf("expected 2 requests (probe, put), got %d: %+v", len(reqs), reqs)
|
||||||
|
}
|
||||||
|
if reqs[0].Method != "GET" || reqs[0].Range != "bytes=0-0" {
|
||||||
|
t.Fatalf("expected first request to be range probe GET, got %+v", reqs[0])
|
||||||
|
}
|
||||||
|
if reqs[1].Method != "PUT" {
|
||||||
|
t.Fatalf("expected second request to be PUT, got %+v", reqs[1])
|
||||||
|
}
|
||||||
|
if got := reqs[1].Headers.Get("Checksum"); got != strings.ToUpper(sha) {
|
||||||
|
t.Fatalf("expected Checksum %s, got %s", strings.ToUpper(sha), got)
|
||||||
|
}
|
||||||
|
if got := reqs[1].Headers.Get("AccessKey"); got != "writekey" {
|
||||||
|
t.Fatalf("expected AccessKey writekey, got %s", got)
|
||||||
|
}
|
||||||
|
if f.headCount != 0 {
|
||||||
|
t.Fatalf("expected zero HEAD requests, got %d", f.headCount)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("present sha", func(t *testing.T) {
|
||||||
|
sha := shaOf("existing-blob")
|
||||||
|
f := newFakeBunny()
|
||||||
|
f.blobs[sha] = []byte("existing-blob-content")
|
||||||
|
srv := httptest.NewServer(f.handler())
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
cfg := testCfg(hostPort(srv), "readkey", "writekey")
|
||||||
|
b, err := NewBackend("bunny", "", cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewBackend: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
srcPath := filepath.Join(dir, "src")
|
||||||
|
if err := os.WriteFile(srcPath, []byte("existing-blob-content"), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := b.Put(context.Background(), sha, srcPath); err != nil {
|
||||||
|
t.Fatalf("Put: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
reqs := f.requestsSnapshot()
|
||||||
|
if len(reqs) != 1 {
|
||||||
|
t.Fatalf("expected probe-only (1 request) since present, got %d: %+v", len(reqs), reqs)
|
||||||
|
}
|
||||||
|
if reqs[0].Method != "GET" {
|
||||||
|
t.Fatalf("expected GET probe, got %+v", reqs[0])
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReadWriteKeySplit(t *testing.T) {
|
||||||
|
sha := shaOf("split-blob")
|
||||||
|
f := newFakeBunny()
|
||||||
|
srv := httptest.NewServer(f.handler())
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
cfg := testCfg(hostPort(srv), "readkeyXXX", "writekeyYYY")
|
||||||
|
b, err := NewBackend("bunny", "", cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewBackend: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
srcPath := filepath.Join(dir, "src")
|
||||||
|
if err := os.WriteFile(srcPath, []byte("split-blob-content"), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := b.Put(context.Background(), sha, srcPath); err != nil {
|
||||||
|
t.Fatalf("Put: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
reqs := f.requestsSnapshot()
|
||||||
|
if len(reqs) != 2 {
|
||||||
|
t.Fatalf("expected 2 requests, got %d", len(reqs))
|
||||||
|
}
|
||||||
|
if got := reqs[0].Headers.Get("AccessKey"); got != "readkeyXXX" {
|
||||||
|
t.Fatalf("probe expected AccessKey readkeyXXX, got %s", got)
|
||||||
|
}
|
||||||
|
if got := reqs[1].Headers.Get("AccessKey"); got != "writekeyYYY" {
|
||||||
|
t.Fatalf("put expected AccessKey writekeyYYY, got %s", got)
|
||||||
|
}
|
||||||
|
if reqs[0].Headers.Get("AccessKey") == reqs[1].Headers.Get("AccessKey") {
|
||||||
|
t.Fatalf("expected distinct read/write keys")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBunnyNoReadKeyFailsClosed(t *testing.T) {
|
||||||
|
// Grep-based tripwire: no non-test depot source may reference stdin/tty.
|
||||||
|
dir := "."
|
||||||
|
entries, err := os.ReadDir(dir)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadDir: %v", err)
|
||||||
|
}
|
||||||
|
for _, e := range entries {
|
||||||
|
name := e.Name()
|
||||||
|
if e.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
data, err := os.ReadFile(filepath.Join(dir, name))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile %s: %v", name, err)
|
||||||
|
}
|
||||||
|
s := string(data)
|
||||||
|
if strings.Contains(s, "os.Stdin") || strings.Contains(s, "bufio.NewReader(os.Stdin)") || strings.Contains(s, "/dev/tty") {
|
||||||
|
t.Fatalf("%s references stdin/tty; credential prompts are banned", name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Confirm undrained pipe: swap in an os.Pipe as stdin-equivalent to prove
|
||||||
|
// no read occurs.
|
||||||
|
r, w, err := os.Pipe()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Pipe: %v", err)
|
||||||
|
}
|
||||||
|
oldStdin := os.Stdin
|
||||||
|
os.Stdin = r
|
||||||
|
defer func() { os.Stdin = oldStdin; r.Close() }()
|
||||||
|
|
||||||
|
cfg := testCfg("storage.example.com", "", "writekey")
|
||||||
|
_, err = NewBackend("bunny", "", cfg)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for empty ReadKey")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "ReadKey") && !strings.Contains(err.Error(), "BUNNY_STORAGE_READ_PASSWORD") && !strings.Contains(err.Error(), "BUNNY_STORAGE_PASSWORD") {
|
||||||
|
t.Fatalf("expected error to mention missing env var, got: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
w.Close()
|
||||||
|
// If code read from stdin it would have blocked already (pipe with no
|
||||||
|
// writer-side data yet); assert the pipe is still open/undrained by
|
||||||
|
// writing now and reading it back ourselves.
|
||||||
|
if _, err := w.Write([]byte("x")); err == nil {
|
||||||
|
t.Fatal("expected write to closed pipe writer to fail")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBunnyNoStorageHostFailsClosed(t *testing.T) {
|
||||||
|
cfg := testCfg("", "readkey", "writekey")
|
||||||
|
_, err := NewBackend("bunny", "", cfg)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for empty StorageHost")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "StorageHost") && !strings.Contains(err.Error(), "BUNNY_STORAGE_HOST") {
|
||||||
|
t.Fatalf("expected error to mention missing StorageHost/BUNNY_STORAGE_HOST, got: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGetHashMismatchDeletes(t *testing.T) {
|
||||||
|
sha := shaOf("get-mismatch-blob")
|
||||||
|
f := newFakeBunny()
|
||||||
|
f.blobs[sha] = []byte("wrong content")
|
||||||
|
srv := httptest.NewServer(f.handler())
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
cfg := testCfg(hostPort(srv), "readkey", "writekey")
|
||||||
|
// force CDN unreachable so Get falls back to storage
|
||||||
|
cfg.CDNBase = "http://127.0.0.1:1"
|
||||||
|
b, err := NewBackend("bunny", "", cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewBackend: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
dest := filepath.Join(dir, "dest")
|
||||||
|
|
||||||
|
err = b.Get(context.Background(), sha, dest)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected hash mismatch error")
|
||||||
|
}
|
||||||
|
if _, statErr := os.Stat(dest); !os.IsNotExist(statErr) {
|
||||||
|
t.Fatalf("expected dest to be removed on mismatch, stat err: %v", statErr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCDNPutReadOnly(t *testing.T) {
|
||||||
|
cfg := testCfg("storage.example.com", "readkey", "writekey")
|
||||||
|
b, err := NewBackend("cdn", "", cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewBackend: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
srcPath := filepath.Join(dir, "src")
|
||||||
|
if err := os.WriteFile(srcPath, []byte("data"), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = b.Put(context.Background(), shaOf("anything"), srcPath)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for cdn Put (read-only)")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "read-only") {
|
||||||
|
t.Fatalf("expected read-only error, got: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestNoHeadInSources is the anti-HEAD regression tripwire from field note 3:
|
||||||
|
// grep-assert no http.MethodHead/.Head( usage in non-test depot sources.
|
||||||
|
func TestNoHeadInSources(t *testing.T) {
|
||||||
|
dir := "."
|
||||||
|
entries, err := os.ReadDir(dir)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadDir: %v", err)
|
||||||
|
}
|
||||||
|
for _, e := range entries {
|
||||||
|
name := e.Name()
|
||||||
|
if e.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
data, err := os.ReadFile(filepath.Join(dir, name))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile %s: %v", name, err)
|
||||||
|
}
|
||||||
|
s := string(data)
|
||||||
|
if strings.Contains(s, "http.MethodHead") || strings.Contains(s, ".Head(") {
|
||||||
|
t.Fatalf("%s references HEAD probe (banned)", name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLocalBackendViaFactory sanity-checks NewBackend("local", ...).
|
||||||
|
func TestLocalBackendViaFactory(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
b, err := NewBackend("local", root, Config{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewBackend: %v", err)
|
||||||
|
}
|
||||||
|
lb, ok := b.(*LocalBackend)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("expected *LocalBackend, got %T", b)
|
||||||
|
}
|
||||||
|
if lb.Root != root {
|
||||||
|
t.Fatalf("expected Root=%s, got %s", root, lb.Root)
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := NewBackend("local", "", Config{}); err == nil {
|
||||||
|
t.Fatal("expected error for empty local root")
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := NewBackend("nonsense", "", Config{}); err == nil {
|
||||||
|
t.Fatal("expected error for unknown backend name")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,497 @@
|
|||||||
|
package depot
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"errors"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"math/rand"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
exitOK = 0
|
||||||
|
exitDrift = 1
|
||||||
|
exitUnconfirmed = 2
|
||||||
|
exitUsage = 64
|
||||||
|
exitInternal = 70
|
||||||
|
)
|
||||||
|
|
||||||
|
// Run executes a depot subcommand. args[0] is the subcommand
|
||||||
|
// (status|push|verify|get|pull); returns the process exit code.
|
||||||
|
func Run(args []string, stdout, stderr io.Writer, getenv func(string) string) int {
|
||||||
|
if len(args) == 0 {
|
||||||
|
printRunUsage(stderr)
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
rest := args[1:]
|
||||||
|
switch args[0] {
|
||||||
|
case "status":
|
||||||
|
return runStatus(rest, stdout, stderr, getenv)
|
||||||
|
case "push":
|
||||||
|
return runPush(rest, stdout, stderr, getenv)
|
||||||
|
case "verify":
|
||||||
|
return runVerify(rest, stdout, stderr, getenv)
|
||||||
|
case "get":
|
||||||
|
return runGet(rest, stdout, stderr, getenv)
|
||||||
|
case "pull":
|
||||||
|
return runPull(rest, stdout, stderr, getenv)
|
||||||
|
default:
|
||||||
|
fmt.Fprintf(stderr, "depot: unknown subcommand %q\n\n", args[0])
|
||||||
|
printRunUsage(stderr)
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func printRunUsage(w io.Writer) {
|
||||||
|
fmt.Fprint(w, `usage:
|
||||||
|
depot status [--manifests DIR] --target bunny|cdn | --out DIR
|
||||||
|
depot push [--manifests DIR] --source LOCAL_DEPOT --target bunny
|
||||||
|
depot verify [--manifests DIR] --target bunny|cdn [--sample N]
|
||||||
|
depot get <sha> <dest> --target cdn|bunny | --out DIR
|
||||||
|
depot pull [--manifests DIR] --dest DIR --target cdn|bunny
|
||||||
|
|
||||||
|
--out DIR reads and writes a depot tree on disk at DIR instead of a remote
|
||||||
|
backend. --target names a remote backend only; the two are mutually exclusive.
|
||||||
|
--target local is a deprecated alias for --out $DEPOT_DIR.
|
||||||
|
`)
|
||||||
|
}
|
||||||
|
|
||||||
|
// errUsage marks errors that are the caller's fault (usage, exit 64) rather
|
||||||
|
// than internal/backend failures (exit 70).
|
||||||
|
var errUsage = errors.New("usage error")
|
||||||
|
|
||||||
|
// resolveBackend picks the backend for a command that can work either against a
|
||||||
|
// depot tree on disk (--out DIR) or a remote backend (--target bunny|cdn).
|
||||||
|
// "--target local" stays as a deprecated alias for --out $DEPOT_DIR so older
|
||||||
|
// callers keep working.
|
||||||
|
func resolveBackend(out, target string, getenv func(string) string, cfg Config) (Backend, error) {
|
||||||
|
switch {
|
||||||
|
case out != "" && target != "":
|
||||||
|
return nil, fmt.Errorf("--out and --target are mutually exclusive: %w", errUsage)
|
||||||
|
case out != "":
|
||||||
|
return NewBackend("local", out, cfg)
|
||||||
|
case target == "local":
|
||||||
|
root := getenv("DEPOT_DIR")
|
||||||
|
if root == "" {
|
||||||
|
return nil, fmt.Errorf("--target local requires DEPOT_DIR to be set (prefer --out DIR): %w", errUsage)
|
||||||
|
}
|
||||||
|
return NewBackend("local", root, cfg)
|
||||||
|
case target == "":
|
||||||
|
return nil, fmt.Errorf("--out DIR or --target bunny|cdn is required: %w", errUsage)
|
||||||
|
default:
|
||||||
|
return NewBackend(target, "", cfg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// backendErrExit maps a resolveBackend error onto the exit contract.
|
||||||
|
func backendErrExit(err error) int {
|
||||||
|
if errors.Is(err, errUsage) {
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
return exitInternal
|
||||||
|
}
|
||||||
|
|
||||||
|
func printSweepStatus(stdout io.Writer, referenced int, res SweepResult) {
|
||||||
|
fmt.Fprintf(stdout, "referenced=%d present=%d missing=%d unconfirmed=%d\n",
|
||||||
|
referenced, len(res.Present), len(res.Missing), len(res.Unconfirmed))
|
||||||
|
for _, sha := range res.Missing {
|
||||||
|
fmt.Fprintf(stdout, "missing %s\n", sha)
|
||||||
|
}
|
||||||
|
for _, sha := range res.Unconfirmed {
|
||||||
|
fmt.Fprintf(stdout, "unconfirmed %s\n", sha)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func sweepExitCode(res SweepResult) int {
|
||||||
|
if len(res.Missing) > 0 {
|
||||||
|
return exitDrift
|
||||||
|
}
|
||||||
|
if len(res.Unconfirmed) > 0 {
|
||||||
|
return exitUnconfirmed
|
||||||
|
}
|
||||||
|
return exitOK
|
||||||
|
}
|
||||||
|
|
||||||
|
func shaKeys(m map[string]int64) []string {
|
||||||
|
out := make([]string, 0, len(m))
|
||||||
|
for sha := range m {
|
||||||
|
out = append(out, sha)
|
||||||
|
}
|
||||||
|
sort.Strings(out)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func runStatus(args []string, stdout, stderr io.Writer, getenv func(string) string) int {
|
||||||
|
fs := flag.NewFlagSet("status", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(stderr)
|
||||||
|
manifests := fs.String("manifests", "assets", "directory of *.yml manifests")
|
||||||
|
source := fs.String("source", "", "deprecated and ignored (use --out DIR for a depot tree on disk)")
|
||||||
|
out := fs.String("out", "", "depot tree on disk to read instead of a remote backend")
|
||||||
|
target := fs.String("target", "", "bunny|cdn")
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
if *source != "" {
|
||||||
|
fmt.Fprintln(stderr, "depot status: --source is ignored; use --out DIR")
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg := LoadConfig(getenv)
|
||||||
|
backend, err := resolveBackend(*out, *target, getenv, cfg)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "depot status:", err)
|
||||||
|
return backendErrExit(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
shaSizes, err := ReferencedSHAs(*manifests)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "depot status:", err)
|
||||||
|
return exitInternal
|
||||||
|
}
|
||||||
|
shas := shaKeys(shaSizes)
|
||||||
|
|
||||||
|
res, err := Sweep(context.Background(), backend, shas, cfg, stderr)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "depot status:", err)
|
||||||
|
return exitInternal
|
||||||
|
}
|
||||||
|
|
||||||
|
printSweepStatus(stdout, len(shas), res)
|
||||||
|
return sweepExitCode(res)
|
||||||
|
}
|
||||||
|
|
||||||
|
func runPush(args []string, stdout, stderr io.Writer, getenv func(string) string) int {
|
||||||
|
fs := flag.NewFlagSet("push", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(stderr)
|
||||||
|
manifests := fs.String("manifests", "assets", "directory of *.yml manifests")
|
||||||
|
source := fs.String("source", "", "local depot root to read blobs from")
|
||||||
|
target := fs.String("target", "", "must be bunny")
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
if *target != "bunny" {
|
||||||
|
fmt.Fprintln(stderr, "depot push: --target must be bunny")
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
if *source == "" {
|
||||||
|
fmt.Fprintln(stderr, "depot push: --source is required")
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg := LoadConfig(getenv)
|
||||||
|
backend, err := NewBackend("bunny", "", cfg)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "depot push:", err)
|
||||||
|
return exitInternal
|
||||||
|
}
|
||||||
|
|
||||||
|
shaSizes, err := ReferencedSHAs(*manifests)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "depot push:", err)
|
||||||
|
return exitInternal
|
||||||
|
}
|
||||||
|
shas := shaKeys(shaSizes)
|
||||||
|
|
||||||
|
res, err := Sweep(context.Background(), backend, shas, cfg, stderr)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "depot push:", err)
|
||||||
|
return exitInternal
|
||||||
|
}
|
||||||
|
|
||||||
|
toUpload := append(append([]string(nil), res.Missing...), res.Unconfirmed...)
|
||||||
|
sort.Strings(toUpload)
|
||||||
|
|
||||||
|
var (
|
||||||
|
mu sync.Mutex
|
||||||
|
uploaded int
|
||||||
|
failed int
|
||||||
|
transportErr error
|
||||||
|
)
|
||||||
|
|
||||||
|
work := func(sha string) {
|
||||||
|
srcPath := filepath.Join(*source, BlobKey(sha))
|
||||||
|
if _, statErr := os.Stat(srcPath); statErr != nil {
|
||||||
|
mu.Lock()
|
||||||
|
failed++
|
||||||
|
mu.Unlock()
|
||||||
|
fmt.Fprintf(stderr, "push: missing source blob %s\n", sha)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := backend.Put(context.Background(), sha, srcPath); err != nil {
|
||||||
|
mu.Lock()
|
||||||
|
failed++
|
||||||
|
if transportErr == nil {
|
||||||
|
transportErr = err
|
||||||
|
}
|
||||||
|
mu.Unlock()
|
||||||
|
fmt.Fprintf(stderr, "push: upload %s: %v\n", sha, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
mu.Lock()
|
||||||
|
uploaded++
|
||||||
|
mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
runWorkers(cfg.Jobs, toUpload, work)
|
||||||
|
|
||||||
|
fmt.Fprintf(stdout, "uploaded=%d failed=%d\n", uploaded, failed)
|
||||||
|
|
||||||
|
if transportErr != nil {
|
||||||
|
return exitInternal
|
||||||
|
}
|
||||||
|
if failed > 0 {
|
||||||
|
return exitDrift
|
||||||
|
}
|
||||||
|
return exitOK
|
||||||
|
}
|
||||||
|
|
||||||
|
func runVerify(args []string, stdout, stderr io.Writer, getenv func(string) string) int {
|
||||||
|
fs := flag.NewFlagSet("verify", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(stderr)
|
||||||
|
manifests := fs.String("manifests", "assets", "directory of *.yml manifests")
|
||||||
|
target := fs.String("target", "", "bunny|cdn")
|
||||||
|
sample := fs.Int("sample", 3, "number of present blobs to spot-check by download (0=skip)")
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
if *target != "bunny" && *target != "cdn" {
|
||||||
|
fmt.Fprintln(stderr, "depot verify: --target must be bunny or cdn")
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg := LoadConfig(getenv)
|
||||||
|
backend, err := NewBackend(*target, "", cfg)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "depot verify:", err)
|
||||||
|
return exitInternal
|
||||||
|
}
|
||||||
|
|
||||||
|
shaSizes, err := ReferencedSHAs(*manifests)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "depot verify:", err)
|
||||||
|
return exitInternal
|
||||||
|
}
|
||||||
|
shas := shaKeys(shaSizes)
|
||||||
|
|
||||||
|
res, err := Sweep(context.Background(), backend, shas, cfg, stderr)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "depot verify:", err)
|
||||||
|
return exitInternal
|
||||||
|
}
|
||||||
|
|
||||||
|
printSweepStatus(stdout, len(shas), res)
|
||||||
|
if code := sweepExitCode(res); code != exitOK {
|
||||||
|
return code
|
||||||
|
}
|
||||||
|
|
||||||
|
if *sample <= 0 || len(res.Present) == 0 {
|
||||||
|
return exitOK
|
||||||
|
}
|
||||||
|
|
||||||
|
n := *sample
|
||||||
|
if n > len(res.Present) {
|
||||||
|
n = len(res.Present)
|
||||||
|
}
|
||||||
|
rng := rand.New(rand.NewSource(time.Now().UnixNano()))
|
||||||
|
picks := rng.Perm(len(res.Present))[:n]
|
||||||
|
|
||||||
|
tmpDir, err := os.MkdirTemp("", "depot-verify-")
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "depot verify:", err)
|
||||||
|
return exitInternal
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tmpDir)
|
||||||
|
|
||||||
|
for _, idx := range picks {
|
||||||
|
sha := res.Present[idx]
|
||||||
|
dest := filepath.Join(tmpDir, sha)
|
||||||
|
if err := backend.Get(context.Background(), sha, dest); err != nil {
|
||||||
|
fmt.Fprintf(stderr, "depot verify: sample %s: %v\n", sha, err)
|
||||||
|
return exitDrift
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return exitOK
|
||||||
|
}
|
||||||
|
|
||||||
|
func runGet(args []string, stdout, stderr io.Writer, getenv func(string) string) int {
|
||||||
|
fs := flag.NewFlagSet("get", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(stderr)
|
||||||
|
out := fs.String("out", "", "depot tree on disk to read instead of a remote backend")
|
||||||
|
target := fs.String("target", "", "cdn|bunny")
|
||||||
|
// Positionals come first in the documented usage, and Go's flag package
|
||||||
|
// stops parsing at the first one, so split them off by hand.
|
||||||
|
split := 0
|
||||||
|
for split < len(args) && !strings.HasPrefix(args[split], "-") {
|
||||||
|
split++
|
||||||
|
}
|
||||||
|
positional := args[:split]
|
||||||
|
if err := fs.Parse(args[split:]); err != nil {
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
positional = append(positional, fs.Args()...)
|
||||||
|
if len(positional) != 2 {
|
||||||
|
fmt.Fprintln(stderr, "depot get: usage: depot get <sha> <dest> --target cdn|bunny | --out DIR")
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
sha, dest := positional[0], positional[1]
|
||||||
|
if !ValidSHA(sha) {
|
||||||
|
fmt.Fprintf(stderr, "depot get: invalid sha %q\n", sha)
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg := LoadConfig(getenv)
|
||||||
|
backend, err := resolveBackend(*out, *target, getenv, cfg)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "depot get:", err)
|
||||||
|
return backendErrExit(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := backend.Get(context.Background(), sha, dest); err != nil {
|
||||||
|
fmt.Fprintln(stderr, "depot get:", err)
|
||||||
|
return exitInternal
|
||||||
|
}
|
||||||
|
return exitOK
|
||||||
|
}
|
||||||
|
|
||||||
|
func runPull(args []string, stdout, stderr io.Writer, getenv func(string) string) int {
|
||||||
|
fs := flag.NewFlagSet("pull", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(stderr)
|
||||||
|
manifests := fs.String("manifests", "assets", "directory of *.yml manifests")
|
||||||
|
dest := fs.String("dest", "", "local directory to pull blobs into")
|
||||||
|
target := fs.String("target", "", "cdn|bunny")
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
if *target != "bunny" && *target != "cdn" {
|
||||||
|
fmt.Fprintln(stderr, "depot pull: --target must be bunny or cdn")
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
if *dest == "" {
|
||||||
|
fmt.Fprintln(stderr, "depot pull: --dest is required")
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg := LoadConfig(getenv)
|
||||||
|
backend, err := NewBackend(*target, "", cfg)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "depot pull:", err)
|
||||||
|
return exitInternal
|
||||||
|
}
|
||||||
|
|
||||||
|
shaSizes, err := ReferencedSHAs(*manifests)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "depot pull:", err)
|
||||||
|
return exitInternal
|
||||||
|
}
|
||||||
|
shas := shaKeys(shaSizes)
|
||||||
|
|
||||||
|
// Split into "already present and correct locally" (skip) vs "needs a
|
||||||
|
// probe/download decision".
|
||||||
|
var present int
|
||||||
|
var toCheck []string
|
||||||
|
for _, sha := range shas {
|
||||||
|
destPath := filepath.Join(*dest, BlobKey(sha))
|
||||||
|
if localFileMatchesSHA(destPath, sha) {
|
||||||
|
present++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
toCheck = append(toCheck, sha)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sweep the source to distinguish "not there" (exit 1, listed) from
|
||||||
|
// "there, download it".
|
||||||
|
res, err := Sweep(context.Background(), backend, toCheck, cfg, stderr)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "depot pull:", err)
|
||||||
|
return exitInternal
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
mu sync.Mutex
|
||||||
|
pulled int
|
||||||
|
anyFail bool
|
||||||
|
)
|
||||||
|
work := func(sha string) {
|
||||||
|
destPath := filepath.Join(*dest, BlobKey(sha))
|
||||||
|
if err := backend.Get(context.Background(), sha, destPath); err != nil {
|
||||||
|
mu.Lock()
|
||||||
|
anyFail = true
|
||||||
|
mu.Unlock()
|
||||||
|
fmt.Fprintf(stderr, "pull: %s: %v\n", sha, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
mu.Lock()
|
||||||
|
pulled++
|
||||||
|
mu.Unlock()
|
||||||
|
}
|
||||||
|
runWorkers(cfg.Jobs, res.Present, work)
|
||||||
|
|
||||||
|
// ponytail: unconfirmed source state (probe budget exhausted / flaky
|
||||||
|
// responses) is treated as a download failure rather than a third exit
|
||||||
|
// path; if that proves too coarse in practice, give pull its own
|
||||||
|
// unconfirmed accounting like status.
|
||||||
|
if len(res.Unconfirmed) > 0 {
|
||||||
|
anyFail = true
|
||||||
|
for _, sha := range res.Unconfirmed {
|
||||||
|
fmt.Fprintf(stderr, "pull: %s: unconfirmed at source\n", sha)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, sha := range res.Missing {
|
||||||
|
fmt.Fprintf(stdout, "missing %s\n", sha)
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Fprintf(stdout, "pulled=%d present=%d\n", pulled, present)
|
||||||
|
|
||||||
|
if anyFail {
|
||||||
|
return exitInternal
|
||||||
|
}
|
||||||
|
if len(res.Missing) > 0 {
|
||||||
|
return exitDrift
|
||||||
|
}
|
||||||
|
return exitOK
|
||||||
|
}
|
||||||
|
|
||||||
|
// localFileMatchesSHA reports whether path exists and hashes to sha.
|
||||||
|
func localFileMatchesSHA(path, sha string) bool {
|
||||||
|
f, err := os.Open(path)
|
||||||
|
if err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
defer f.Close()
|
||||||
|
h := sha256.New()
|
||||||
|
if _, err := io.Copy(h, f); err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("%x", h.Sum(nil)) == sha
|
||||||
|
}
|
||||||
|
|
||||||
|
// runWorkers runs work(sha) for each sha in items using up to jobs goroutines.
|
||||||
|
func runWorkers(jobs int, items []string, work func(sha string)) {
|
||||||
|
if jobs < 1 {
|
||||||
|
jobs = 1
|
||||||
|
}
|
||||||
|
ch := make(chan string)
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := 0; i < jobs; i++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
for sha := range ch {
|
||||||
|
work(sha)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
for _, sha := range items {
|
||||||
|
ch <- sha
|
||||||
|
}
|
||||||
|
close(ch)
|
||||||
|
wg.Wait()
|
||||||
|
}
|
||||||
@@ -0,0 +1,277 @@
|
|||||||
|
package depot
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func testGetenv(vals map[string]string) func(string) string {
|
||||||
|
return func(k string) string { return vals[k] }
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunUsage(t *testing.T) {
|
||||||
|
t.Run("no args", func(t *testing.T) {
|
||||||
|
var out, errb bytes.Buffer
|
||||||
|
code := Run(nil, &out, &errb, testGetenv(nil))
|
||||||
|
if code != 64 {
|
||||||
|
t.Fatalf("expected 64, got %d (stderr=%s)", code, errb.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("unknown subcommand", func(t *testing.T) {
|
||||||
|
var out, errb bytes.Buffer
|
||||||
|
code := Run([]string{"bogus"}, &out, &errb, testGetenv(nil))
|
||||||
|
if code != 64 {
|
||||||
|
t.Fatalf("expected 64, got %d (stderr=%s)", code, errb.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("status --target local without DEPOT_DIR", func(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
writeManifest(t, dir, shaOf("blob-a"), 5)
|
||||||
|
var out, errb bytes.Buffer
|
||||||
|
code := Run([]string{"status", "--manifests", dir, "--target", "local"}, &out, &errb, testGetenv(nil))
|
||||||
|
if code != 64 {
|
||||||
|
t.Fatalf("expected 64, got %d (stderr=%s)", code, errb.String())
|
||||||
|
}
|
||||||
|
if !bytesContains(errb.String(), "DEPOT_DIR") {
|
||||||
|
t.Fatalf("expected stderr to mention DEPOT_DIR, got %s", errb.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("push --target cdn rejected", func(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
var out, errb bytes.Buffer
|
||||||
|
code := Run([]string{"push", "--manifests", dir, "--source", dir, "--target", "cdn"}, &out, &errb, testGetenv(nil))
|
||||||
|
if code != 64 {
|
||||||
|
t.Fatalf("expected 64, got %d (stderr=%s)", code, errb.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOutFlag(t *testing.T) {
|
||||||
|
t.Run("status --out reads the given tree", func(t *testing.T) {
|
||||||
|
sha := shaOf("out-blob")
|
||||||
|
manifests := t.TempDir()
|
||||||
|
writeManifest(t, manifests, sha, 8)
|
||||||
|
depotDir := t.TempDir()
|
||||||
|
writeBlob(t, depotDir, sha, "out-blob")
|
||||||
|
|
||||||
|
var out, errb bytes.Buffer
|
||||||
|
code := Run([]string{"status", "--manifests", manifests, "--out", depotDir}, &out, &errb, testGetenv(nil))
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("expected 0, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
|
||||||
|
}
|
||||||
|
if !bytesContains(out.String(), "present=1") {
|
||||||
|
t.Fatalf("expected present=1, got %s", out.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("get --out fetches from the given tree", func(t *testing.T) {
|
||||||
|
sha := shaOf("get-blob")
|
||||||
|
depotDir := t.TempDir()
|
||||||
|
writeBlob(t, depotDir, sha, "get-blob")
|
||||||
|
dest := filepath.Join(t.TempDir(), "fetched")
|
||||||
|
|
||||||
|
var out, errb bytes.Buffer
|
||||||
|
code := Run([]string{"get", sha, dest, "--out", depotDir}, &out, &errb, testGetenv(nil))
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("expected 0, got %d (stderr=%s)", code, errb.String())
|
||||||
|
}
|
||||||
|
got, err := os.ReadFile(dest)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if string(got) != "get-blob" {
|
||||||
|
t.Fatalf("got %q", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("--out with any --target is rejected", func(t *testing.T) {
|
||||||
|
for _, target := range []string{"bunny", "cdn", "local"} {
|
||||||
|
var out, errb bytes.Buffer
|
||||||
|
code := Run([]string{"status", "--manifests", t.TempDir(), "--out", t.TempDir(), "--target", target}, &out, &errb, testGetenv(nil))
|
||||||
|
if code != 64 {
|
||||||
|
t.Fatalf("--target %s: expected 64, got %d (stderr=%s)", target, code, errb.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("get accepts flags before the positionals", func(t *testing.T) {
|
||||||
|
sha := shaOf("flags-first-blob")
|
||||||
|
depotDir := t.TempDir()
|
||||||
|
writeBlob(t, depotDir, sha, "flags-first-blob")
|
||||||
|
dest := filepath.Join(t.TempDir(), "fetched")
|
||||||
|
|
||||||
|
var out, errb bytes.Buffer
|
||||||
|
code := Run([]string{"get", "--out", depotDir, sha, dest}, &out, &errb, testGetenv(nil))
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("expected 0, got %d (stderr=%s)", code, errb.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("neither --out nor --target is rejected", func(t *testing.T) {
|
||||||
|
var out, errb bytes.Buffer
|
||||||
|
code := Run([]string{"status", "--manifests", t.TempDir()}, &out, &errb, testGetenv(nil))
|
||||||
|
if code != 64 {
|
||||||
|
t.Fatalf("expected 64, got %d (stderr=%s)", code, errb.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("--target local still works as a hidden alias", func(t *testing.T) {
|
||||||
|
sha := shaOf("alias-blob")
|
||||||
|
manifests := t.TempDir()
|
||||||
|
writeManifest(t, manifests, sha, 11)
|
||||||
|
depotDir := t.TempDir()
|
||||||
|
writeBlob(t, depotDir, sha, "alias-blob")
|
||||||
|
|
||||||
|
var out, errb bytes.Buffer
|
||||||
|
code := Run([]string{"status", "--manifests", manifests, "--target", "local"}, &out, &errb,
|
||||||
|
testGetenv(map[string]string{"DEPOT_DIR": depotDir}))
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("expected 0, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGetInvalidSHA(t *testing.T) {
|
||||||
|
var out, errb bytes.Buffer
|
||||||
|
dir := t.TempDir()
|
||||||
|
code := Run([]string{"get", "not-a-sha", dir + "/dest", "--target", "local"}, &out, &errb, testGetenv(map[string]string{"DEPOT_DIR": dir}))
|
||||||
|
if code != 64 {
|
||||||
|
t.Fatalf("expected 64, got %d (stderr=%s)", code, errb.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNoKeyStatusBunnyFailsClosedNoStdin(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
writeManifest(t, dir, shaOf("blob-a"), 5)
|
||||||
|
|
||||||
|
var out, errb bytes.Buffer
|
||||||
|
code := Run([]string{"status", "--manifests", dir, "--target", "bunny"}, &out, &errb, testGetenv(nil))
|
||||||
|
if code != 70 {
|
||||||
|
t.Fatalf("expected 70, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
|
||||||
|
}
|
||||||
|
if !bytesContains(errb.String(), "BUNNY_STORAGE_HOST") {
|
||||||
|
t.Fatalf("expected stderr to mention BUNNY_STORAGE_HOST, got %s", errb.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStatusExitCodes(t *testing.T) {
|
||||||
|
t.Run("drift", func(t *testing.T) {
|
||||||
|
f := newFakeBunny()
|
||||||
|
srv := httptest.NewServer(f.handler())
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
sha := shaOf("missing-blob")
|
||||||
|
dir := t.TempDir()
|
||||||
|
writeManifest(t, dir, sha, 5)
|
||||||
|
|
||||||
|
var out, errb bytes.Buffer
|
||||||
|
code := Run([]string{"status", "--manifests", dir, "--target", "bunny"}, &out, &errb,
|
||||||
|
testGetenv(map[string]string{
|
||||||
|
"BUNNY_STORAGE_HOST": hostPort(srv),
|
||||||
|
"BUNNY_STORAGE_READ_PASSWORD": "readkey",
|
||||||
|
}))
|
||||||
|
if code != 1 {
|
||||||
|
t.Fatalf("expected 1, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
|
||||||
|
}
|
||||||
|
if !bytesContains(out.String(), "missing="+"1") {
|
||||||
|
t.Fatalf("expected missing=1 in output, got %s", out.String())
|
||||||
|
}
|
||||||
|
if !bytesContains(out.String(), "missing "+sha) {
|
||||||
|
t.Fatalf("expected missing line for sha, got %s", out.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("unconfirmed-only", func(t *testing.T) {
|
||||||
|
stubProbeSleep(t)
|
||||||
|
f := newFakeBunny()
|
||||||
|
sha := shaOf("flaky-blob")
|
||||||
|
f.statusFor[sha] = 428
|
||||||
|
srv := httptest.NewServer(f.handler())
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
writeManifest(t, dir, sha, 5)
|
||||||
|
|
||||||
|
var out, errb bytes.Buffer
|
||||||
|
code := Run([]string{"status", "--manifests", dir, "--target", "bunny"}, &out, &errb,
|
||||||
|
testGetenv(map[string]string{
|
||||||
|
"BUNNY_STORAGE_HOST": hostPort(srv),
|
||||||
|
"BUNNY_STORAGE_READ_PASSWORD": "readkey",
|
||||||
|
"DEPOT_CONFIRM_RETRIES": "1",
|
||||||
|
}))
|
||||||
|
if code != 2 {
|
||||||
|
t.Fatalf("expected 2, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
|
||||||
|
}
|
||||||
|
if !bytesContains(out.String(), "unconfirmed="+"1") {
|
||||||
|
t.Fatalf("expected unconfirmed=1 in output, got %s", out.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPushTransportErrorCountsFailed(t *testing.T) {
|
||||||
|
// Probes 404 (blob absent), PUTs 500 (transport-level upload failure).
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.Method == http.MethodPut {
|
||||||
|
w.WriteHeader(500)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.WriteHeader(404)
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
sha := shaOf("push-fail-blob")
|
||||||
|
manifestsDir := t.TempDir()
|
||||||
|
writeManifest(t, manifestsDir, sha, 14)
|
||||||
|
sourceDir := t.TempDir()
|
||||||
|
blobPath := filepath.Join(sourceDir, BlobKey(sha))
|
||||||
|
if err := os.MkdirAll(filepath.Dir(blobPath), 0755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(blobPath, []byte("push-fail-blob"), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var out, errb bytes.Buffer
|
||||||
|
code := Run([]string{"push", "--manifests", manifestsDir, "--source", sourceDir, "--target", "bunny"}, &out, &errb,
|
||||||
|
testGetenv(map[string]string{
|
||||||
|
"BUNNY_STORAGE_HOST": hostPort(srv),
|
||||||
|
"BUNNY_STORAGE_PASSWORD": "writekey",
|
||||||
|
}))
|
||||||
|
if code != 70 {
|
||||||
|
t.Fatalf("expected 70, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
|
||||||
|
}
|
||||||
|
if !bytesContains(out.String(), "uploaded=0 failed=1") {
|
||||||
|
t.Fatalf("expected uploaded=0 failed=1, got %s", out.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func bytesContains(s, substr string) bool {
|
||||||
|
return bytes.Contains([]byte(s), []byte(substr))
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeBlob(t *testing.T, root, sha, content string) {
|
||||||
|
t.Helper()
|
||||||
|
path := filepath.Join(root, BlobKey(sha))
|
||||||
|
if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(path, []byte(content), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeManifest(t *testing.T, dir, sha string, size int64) {
|
||||||
|
t.Helper()
|
||||||
|
content := fmt.Sprintf("assets:\n - path: foo\n sha256: %s\n size: %d\n", sha, size)
|
||||||
|
if err := os.WriteFile(filepath.Join(dir, "manifest.yml"), []byte(content), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,156 @@
|
|||||||
|
package depot
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"sort"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// confirmSleep is the backoff sleeper for the serial confirm phase;
|
||||||
|
// overridden in tests.
|
||||||
|
var confirmSleep = time.Sleep
|
||||||
|
|
||||||
|
// SweepResult partitions the swept shas by confirmed state. Unconfirmed is
|
||||||
|
// distinct from Missing and must never be collapsed into it: it means the
|
||||||
|
// probe budget was exhausted, not that the blob is known absent.
|
||||||
|
type SweepResult struct {
|
||||||
|
Present []string
|
||||||
|
Missing []string
|
||||||
|
Unconfirmed []string
|
||||||
|
}
|
||||||
|
|
||||||
|
type probeOutcome struct {
|
||||||
|
sha string
|
||||||
|
state ProbeState
|
||||||
|
transient bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sweep probes every sha against b: one parallel pass (cfg.ProbeJobs workers)
|
||||||
|
// followed by a serial confirm phase over every non-Present result (unless
|
||||||
|
// the non-Present count exceeds cfg.ConfirmMax, in which case all of them
|
||||||
|
// are reported Unconfirmed with zero re-probes). Progress is reported to
|
||||||
|
// progress roughly every 1000 blobs.
|
||||||
|
func Sweep(ctx context.Context, b Backend, shas []string, cfg Config, progress io.Writer) (SweepResult, error) {
|
||||||
|
sorted := append([]string(nil), shas...)
|
||||||
|
sort.Strings(sorted)
|
||||||
|
|
||||||
|
outcomes := make([]probeOutcome, len(sorted))
|
||||||
|
jobs := cfg.ProbeJobs
|
||||||
|
if jobs < 1 {
|
||||||
|
jobs = 1
|
||||||
|
}
|
||||||
|
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
work := make(chan int)
|
||||||
|
var probed int64
|
||||||
|
var mu sync.Mutex
|
||||||
|
var firstErr error
|
||||||
|
|
||||||
|
worker := func() {
|
||||||
|
defer wg.Done()
|
||||||
|
for i := range work {
|
||||||
|
state, transient, err := b.Probe(ctx, sorted[i])
|
||||||
|
mu.Lock()
|
||||||
|
if err != nil && !transient {
|
||||||
|
if firstErr == nil {
|
||||||
|
firstErr = fmt.Errorf("probe %s: %w", sorted[i], err)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
outcomes[i] = probeOutcome{sha: sorted[i], state: state, transient: transient}
|
||||||
|
}
|
||||||
|
probed++
|
||||||
|
n := probed
|
||||||
|
mu.Unlock()
|
||||||
|
if n%1000 == 0 {
|
||||||
|
fmt.Fprintf(progress, "probed %d/%d\n", n, len(sorted))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for w := 0; w < jobs; w++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go worker()
|
||||||
|
}
|
||||||
|
for i := range sorted {
|
||||||
|
work <- i
|
||||||
|
}
|
||||||
|
close(work)
|
||||||
|
wg.Wait()
|
||||||
|
|
||||||
|
if firstErr != nil {
|
||||||
|
return SweepResult{}, firstErr
|
||||||
|
}
|
||||||
|
|
||||||
|
var res SweepResult
|
||||||
|
var pending []probeOutcome
|
||||||
|
for _, o := range outcomes {
|
||||||
|
switch o.state {
|
||||||
|
case Present:
|
||||||
|
res.Present = append(res.Present, o.sha)
|
||||||
|
case Absent:
|
||||||
|
res.Missing = append(res.Missing, o.sha)
|
||||||
|
default:
|
||||||
|
pending = append(pending, o)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if cfg.ConfirmMax > 0 && len(pending) > cfg.ConfirmMax {
|
||||||
|
for _, o := range pending {
|
||||||
|
res.Unconfirmed = append(res.Unconfirmed, o.sha)
|
||||||
|
}
|
||||||
|
return res, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, o := range pending {
|
||||||
|
state, err := confirm(ctx, b, o.sha, cfg.ConfirmRetries)
|
||||||
|
if err != nil {
|
||||||
|
return SweepResult{}, err
|
||||||
|
}
|
||||||
|
switch state {
|
||||||
|
case Present:
|
||||||
|
res.Present = append(res.Present, o.sha)
|
||||||
|
case Absent:
|
||||||
|
res.Missing = append(res.Missing, o.sha)
|
||||||
|
default:
|
||||||
|
res.Unconfirmed = append(res.Unconfirmed, o.sha)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
sort.Strings(res.Present)
|
||||||
|
sort.Strings(res.Missing)
|
||||||
|
sort.Strings(res.Unconfirmed)
|
||||||
|
|
||||||
|
if len(sorted)%1000 != 0 {
|
||||||
|
fmt.Fprintf(progress, "probed %d/%d\n", len(sorted), len(sorted))
|
||||||
|
}
|
||||||
|
|
||||||
|
return res, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// confirm re-probes sha serially up to retries attempts with linear backoff
|
||||||
|
// (1s, 2s, ... between attempts). A 2xx result confirms Present, a 404/410
|
||||||
|
// confirms Missing (Absent) immediately. If it is still transient after all
|
||||||
|
// attempts, the state is left Unconfirmed rather than guessed as Missing.
|
||||||
|
func confirm(ctx context.Context, b Backend, sha string, retries int) (ProbeState, error) {
|
||||||
|
if retries < 1 {
|
||||||
|
retries = 1
|
||||||
|
}
|
||||||
|
var last ProbeState = Unconfirmed
|
||||||
|
for attempt := 1; attempt <= retries; attempt++ {
|
||||||
|
state, transient, err := b.Probe(ctx, sha)
|
||||||
|
if err != nil && !transient {
|
||||||
|
return Unconfirmed, fmt.Errorf("confirm %s: %w", sha, err)
|
||||||
|
}
|
||||||
|
if state == Present || state == Absent {
|
||||||
|
return state, nil
|
||||||
|
}
|
||||||
|
last = Unconfirmed
|
||||||
|
if attempt < retries {
|
||||||
|
confirmSleep(time.Duration(attempt) * time.Second)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return last, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,169 @@
|
|||||||
|
package depot
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"io"
|
||||||
|
"sort"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// stubBackend lets tests script Probe behavior per call.
|
||||||
|
type stubBackend struct {
|
||||||
|
probe func(ctx context.Context, sha string) (ProbeState, bool, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *stubBackend) Name() string { return "stub" }
|
||||||
|
func (s *stubBackend) Probe(ctx context.Context, sha string) (ProbeState, bool, error) {
|
||||||
|
return s.probe(ctx, sha)
|
||||||
|
}
|
||||||
|
func (s *stubBackend) Get(ctx context.Context, sha, dest string) error { return nil }
|
||||||
|
func (s *stubBackend) Put(ctx context.Context, sha, src string) error { return nil }
|
||||||
|
|
||||||
|
func sweepTestCfg() Config {
|
||||||
|
return Config{ProbeJobs: 4, ConfirmRetries: 3, ConfirmMax: 200}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSweepAllPresent(t *testing.T) {
|
||||||
|
b := &stubBackend{probe: func(ctx context.Context, sha string) (ProbeState, bool, error) {
|
||||||
|
return Present, false, nil
|
||||||
|
}}
|
||||||
|
shas := []string{"bbb", "aaa", "ccc"}
|
||||||
|
res, err := Sweep(context.Background(), b, shas, sweepTestCfg(), io.Discard)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
want := []string{"aaa", "bbb", "ccc"}
|
||||||
|
sort.Strings(want)
|
||||||
|
if len(res.Present) != 3 || len(res.Missing) != 0 || len(res.Unconfirmed) != 0 {
|
||||||
|
t.Fatalf("got %+v", res)
|
||||||
|
}
|
||||||
|
for i, sha := range res.Present {
|
||||||
|
if sha != want[i] {
|
||||||
|
t.Fatalf("expected sorted output, got %v", res.Present)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSweepMissing(t *testing.T) {
|
||||||
|
b := &stubBackend{probe: func(ctx context.Context, sha string) (ProbeState, bool, error) {
|
||||||
|
return Absent, false, nil
|
||||||
|
}}
|
||||||
|
shas := []string{"aaa"}
|
||||||
|
res, err := Sweep(context.Background(), b, shas, sweepTestCfg(), io.Discard)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
if len(res.Missing) != 1 || res.Missing[0] != "aaa" {
|
||||||
|
t.Fatalf("got %+v", res)
|
||||||
|
}
|
||||||
|
if len(res.Present) != 0 || len(res.Unconfirmed) != 0 {
|
||||||
|
t.Fatalf("got %+v", res)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSweepThrottledUnconfirmed(t *testing.T) {
|
||||||
|
var sleeps []time.Duration
|
||||||
|
orig := confirmSleep
|
||||||
|
confirmSleep = func(d time.Duration) { sleeps = append(sleeps, d) }
|
||||||
|
defer func() { confirmSleep = orig }()
|
||||||
|
|
||||||
|
b := &stubBackend{probe: func(ctx context.Context, sha string) (ProbeState, bool, error) {
|
||||||
|
return Unconfirmed, true, nil
|
||||||
|
}}
|
||||||
|
shas := []string{"aaa"}
|
||||||
|
cfg := sweepTestCfg()
|
||||||
|
res, err := Sweep(context.Background(), b, shas, cfg, io.Discard)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
if len(res.Unconfirmed) != 1 || res.Unconfirmed[0] != "aaa" {
|
||||||
|
t.Fatalf("got %+v", res)
|
||||||
|
}
|
||||||
|
if len(res.Missing) != 0 {
|
||||||
|
t.Fatalf("must never collapse into missing: %+v", res)
|
||||||
|
}
|
||||||
|
if len(sleeps) != 2 || sleeps[0] != 1*time.Second || sleeps[1] != 2*time.Second {
|
||||||
|
t.Fatalf("expected sleeps [1s 2s], got %v", sleeps)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSweepTransientRecovers(t *testing.T) {
|
||||||
|
orig := confirmSleep
|
||||||
|
confirmSleep = func(d time.Duration) {}
|
||||||
|
defer func() { confirmSleep = orig }()
|
||||||
|
|
||||||
|
var calls int32
|
||||||
|
b := &stubBackend{probe: func(ctx context.Context, sha string) (ProbeState, bool, error) {
|
||||||
|
n := atomic.AddInt32(&calls, 1)
|
||||||
|
if n == 1 {
|
||||||
|
return Unconfirmed, true, nil // initial parallel probe: 428
|
||||||
|
}
|
||||||
|
return Present, false, nil // serial re-probe: 206
|
||||||
|
}}
|
||||||
|
res, err := Sweep(context.Background(), b, []string{"aaa"}, sweepTestCfg(), io.Discard)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
if len(res.Present) != 1 || res.Present[0] != "aaa" {
|
||||||
|
t.Fatalf("got %+v", res)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSweepConfirmMaxCap(t *testing.T) {
|
||||||
|
orig := confirmSleep
|
||||||
|
var sleepCalls int32
|
||||||
|
confirmSleep = func(d time.Duration) { atomic.AddInt32(&sleepCalls, 1) }
|
||||||
|
defer func() { confirmSleep = orig }()
|
||||||
|
|
||||||
|
b := &stubBackend{probe: func(ctx context.Context, sha string) (ProbeState, bool, error) {
|
||||||
|
return Unconfirmed, true, nil
|
||||||
|
}}
|
||||||
|
shas := []string{"a", "b", "c", "d", "e"}
|
||||||
|
cfg := sweepTestCfg()
|
||||||
|
cfg.ConfirmMax = 2
|
||||||
|
res, err := Sweep(context.Background(), b, shas, cfg, io.Discard)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
if len(res.Unconfirmed) != 5 {
|
||||||
|
t.Fatalf("expected all 5 unconfirmed, got %+v", res)
|
||||||
|
}
|
||||||
|
if len(res.Missing) != 0 || len(res.Present) != 0 {
|
||||||
|
t.Fatalf("got %+v", res)
|
||||||
|
}
|
||||||
|
if sleepCalls != 0 {
|
||||||
|
t.Fatalf("expected zero serial re-probes/sleeps when over ConfirmMax, got %d calls", sleepCalls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSweepParallelBounded(t *testing.T) {
|
||||||
|
var inFlight int32
|
||||||
|
var maxInFlight int32
|
||||||
|
shas := make([]string, 100)
|
||||||
|
for i := range shas {
|
||||||
|
shas[i] = string(rune('a'+i%26)) + string(rune('A'+i/26))
|
||||||
|
}
|
||||||
|
b := &stubBackend{probe: func(ctx context.Context, sha string) (ProbeState, bool, error) {
|
||||||
|
n := atomic.AddInt32(&inFlight, 1)
|
||||||
|
for {
|
||||||
|
cur := atomic.LoadInt32(&maxInFlight)
|
||||||
|
if n <= cur || atomic.CompareAndSwapInt32(&maxInFlight, cur, n) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
time.Sleep(time.Millisecond)
|
||||||
|
atomic.AddInt32(&inFlight, -1)
|
||||||
|
return Present, false, nil
|
||||||
|
}}
|
||||||
|
cfg := sweepTestCfg()
|
||||||
|
cfg.ProbeJobs = 4
|
||||||
|
_, err := Sweep(context.Background(), b, shas, cfg, io.Discard)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
if maxInFlight > 4 {
|
||||||
|
t.Fatalf("expected max concurrency <= 4, got %d", maxInFlight)
|
||||||
|
}
|
||||||
|
}
|
||||||
+282
-38
@@ -2,22 +2,26 @@
|
|||||||
// builders shared by the `crucible` dispatcher and the standalone
|
// builders shared by the `crucible` dispatcher and the standalone
|
||||||
// `crucible-<name>` shims.
|
// `crucible-<name>` shims.
|
||||||
//
|
//
|
||||||
// Cutover state (Phase 5 → 6): the internal pipeline/topdata/erf/wiki/music
|
// Cutover state (Phase 5 → 6): the internal pipeline/topdata/erf/wiki
|
||||||
// packages migrated from gitea/sow-tools now live in this tree, so wired
|
// packages migrated from gitea/sow-tools now live in this tree, so wired
|
||||||
// builders delegate to internal/app's legacy command surface (mapped per
|
// builders delegate to internal/app's legacy command surface (mapped per
|
||||||
// docs/command-surface.md). A builder with no migrated logic yet (depot) keeps
|
// docs/command-surface.md). depot is wired but bypasses that legacy surface
|
||||||
// the Wired=false fail-closed path: exit 70, never a faked artifact.
|
// entirely, delegating straight to internal/depot.Run. A builder with no
|
||||||
|
// migrated logic yet keeps the Wired=false fail-closed path: exit 70, never a
|
||||||
|
// faked artifact.
|
||||||
package dispatch
|
package dispatch
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
"text/tabwriter"
|
|
||||||
|
|
||||||
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/app"
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/app"
|
||||||
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/assets"
|
||||||
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo"
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo"
|
||||||
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/depot"
|
||||||
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/menu"
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/menu"
|
||||||
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/nwsync"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Exit codes follow the sysexits(3) convention so CI can distinguish
|
// Exit codes follow the sysexits(3) convention so CI can distinguish
|
||||||
@@ -33,22 +37,56 @@ type Builder struct {
|
|||||||
Name string // canonical dispatcher subcommand, e.g. "module"
|
Name string // canonical dispatcher subcommand, e.g. "module"
|
||||||
Bin string // standalone binary name, e.g. "crucible-module"
|
Bin string // standalone binary name, e.g. "crucible-module"
|
||||||
Summary string // one-line description
|
Summary string // one-line description
|
||||||
Legacy []string // nwn-tool commands this subsumes (migration aid; see docs/command-surface.md)
|
Commands []Command // visible commands plus hidden compatibility aliases
|
||||||
Extra []string // additional callable subcommands beyond Legacy (e.g. cross-listed "music")
|
|
||||||
Wired bool // true once the builder delegates to migrated internal/app logic
|
Wired bool // true once the builder delegates to migrated internal/app logic
|
||||||
}
|
}
|
||||||
|
|
||||||
// subcommands returns every legacy command a wired builder accepts: the
|
// Command is one public builder action. AppCommand names the unchanged
|
||||||
// canonical Legacy set plus any cross-listed Extra commands.
|
// internal/app implementation command. Aliases remain callable for compatibility
|
||||||
func (b Builder) subcommands() []string { return append(append([]string{}, b.Legacy...), b.Extra...) }
|
// but are omitted from routine help and the interactive menu.
|
||||||
|
type Command struct {
|
||||||
|
Name string
|
||||||
|
Summary string
|
||||||
|
AppCommand string
|
||||||
|
Usage string
|
||||||
|
Options []string
|
||||||
|
Aliases []CommandAlias
|
||||||
|
}
|
||||||
|
|
||||||
|
type CommandAlias struct {
|
||||||
|
Name string
|
||||||
|
AppCommand string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (b Builder) subcommands() []string {
|
||||||
|
commands := make([]string, 0, len(b.Commands))
|
||||||
|
for _, command := range b.Commands {
|
||||||
|
commands = append(commands, command.Name)
|
||||||
|
}
|
||||||
|
return commands
|
||||||
|
}
|
||||||
|
|
||||||
|
func (b Builder) command(name string) (Command, string, bool) {
|
||||||
|
for _, command := range b.Commands {
|
||||||
|
if command.Name == name {
|
||||||
|
return command, command.AppCommand, true
|
||||||
|
}
|
||||||
|
for _, alias := range command.Aliases {
|
||||||
|
if alias.Name == name {
|
||||||
|
target := alias.AppCommand
|
||||||
|
if target == "" {
|
||||||
|
target = command.AppCommand
|
||||||
|
}
|
||||||
|
return command, target, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return Command{}, "", false
|
||||||
|
}
|
||||||
|
|
||||||
func (b Builder) accepts(sub string) bool {
|
func (b Builder) accepts(sub string) bool {
|
||||||
for _, s := range b.subcommands() {
|
_, _, ok := b.command(sub)
|
||||||
if s == sub {
|
return ok
|
||||||
return true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return false
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Registry is the single source of truth for the Crucible command surface.
|
// Registry is the single source of truth for the Crucible command surface.
|
||||||
@@ -57,40 +95,205 @@ var Registry = []Builder{
|
|||||||
{
|
{
|
||||||
Name: "depot",
|
Name: "depot",
|
||||||
Bin: "crucible-depot",
|
Bin: "crucible-depot",
|
||||||
Summary: "verify/move content-addressed depot blobs (SeaweedFS)",
|
Summary: "content-addressed asset depot (disk/cdn/bunny)",
|
||||||
Legacy: nil,
|
Commands: []Command{
|
||||||
Wired: false, // no migrated logic yet; fails closed (exit 70)
|
{Name: "status", Summary: "report referenced-vs-present drift against a backend", Usage: "crucible depot status [--manifests DIR] --target bunny|cdn | --out DIR"},
|
||||||
|
{Name: "push", Summary: "upload referenced-but-absent blobs from a local depot", Usage: "crucible depot push [--manifests DIR] --source DIR --target bunny"},
|
||||||
|
{Name: "verify", Summary: "existence sweep plus sampled download re-hash", Usage: "crucible depot verify [--manifests DIR] --target bunny|cdn [--sample N]"},
|
||||||
|
{Name: "get", Summary: "fetch one blob with sha re-verify", Usage: "crucible depot get <sha> <dest> --target cdn|bunny | --out DIR"},
|
||||||
|
{Name: "pull", Summary: "incremental verified pull of every referenced blob", Usage: "crucible depot pull [--manifests DIR] --dest DIR --target cdn|bunny"},
|
||||||
|
},
|
||||||
|
Wired: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "nwsync",
|
||||||
|
Bin: "crucible-nwsync",
|
||||||
|
Summary: "publish NWSync blobs and manifests (emit/assemble/verify)",
|
||||||
|
Commands: []Command{
|
||||||
|
{Name: "emit", Summary: "explode one artifact into blobs plus its own NSYM manifest", Usage: "crucible nwsync emit <artifact> --out DIR"},
|
||||||
|
{Name: "assemble", Summary: "merge per-artifact NSYM manifests into one", Usage: "crucible nwsync assemble --order NAMES --entries DIR --out DIR [--group-id N]"},
|
||||||
|
{Name: "verify", Summary: "read a published manifest's blobs back through the pull zone and hash them", Usage: "crucible nwsync verify <manifest-sha1> [--sample N]"},
|
||||||
|
},
|
||||||
|
Wired: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "assets",
|
||||||
|
Bin: "crucible-assets",
|
||||||
|
Summary: "compile/convert/upscale NWN assets + mdl/dupe integrity",
|
||||||
|
Commands: []Command{
|
||||||
|
{Name: "compile", Summary: "compile ASCII .mdl models to binary in place", Usage: "crucible assets compile [--nwn INSTALL] [--non-recursive] <dir>..."},
|
||||||
|
{Name: "convert", Summary: "convert textures to/from NWN DDS (flips vertically)", Usage: "crucible assets convert [--to dds|png|tga] [--backend PATH] [--non-recursive] <dir>..."},
|
||||||
|
{Name: "upscale", Summary: "upscale textures through an installed backend", Usage: "crucible assets upscale [--scale N] [--backend PATH] [--non-recursive] <dir>..."},
|
||||||
|
{Name: "check-mdl", Summary: "report uncompiled ASCII .mdl and model-name mismatches", Usage: "crucible assets check-mdl <path>..."},
|
||||||
|
{Name: "fix-mdl", Summary: "lowercase .mdl names and rewrite model identity to match", Usage: "crucible assets fix-mdl [--dry-run] <path>..."},
|
||||||
|
{Name: "check-dupes", Summary: "report runtime-name (basename) collisions across dirs", Usage: "crucible assets check-dupes <dir>..."},
|
||||||
|
{Name: "clean-dupes", Summary: "delete clean-tree files whose basename collides with primary", Usage: "crucible assets clean-dupes [--dry-run] <primary> <clean>"},
|
||||||
|
},
|
||||||
|
Wired: true,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
Name: "hak",
|
Name: "hak",
|
||||||
Bin: "crucible-hak",
|
Bin: "crucible-hak",
|
||||||
Summary: "pack/unpack ERF/HAK archives + hak manifests",
|
Summary: "pack/unpack ERF/HAK archives + hak manifests",
|
||||||
Legacy: []string{"build-haks", "apply-hak-manifest"},
|
Commands: []Command{
|
||||||
Extra: []string{"music"}, // music BMUs are packed into HAKs (command-surface.md)
|
{
|
||||||
|
Name: "build",
|
||||||
|
Summary: "build configured HAK archives and their manifest",
|
||||||
|
AppCommand: "build-haks",
|
||||||
|
Usage: "crucible hak build [options]",
|
||||||
|
Options: []string{
|
||||||
|
"--hak <name> build selected configured HAKs",
|
||||||
|
"--archive <name> build selected archive members",
|
||||||
|
"--source-manifest <path> read a generated source manifest",
|
||||||
|
"--content-addressed-root <path> resolve source-manifest blobs here",
|
||||||
|
"--plan-only plan outputs without writing archives",
|
||||||
|
"--quiet | --verbose | --debug select output detail",
|
||||||
|
},
|
||||||
|
Aliases: []CommandAlias{{Name: "build-haks"}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "manifest",
|
||||||
|
Summary: "apply a generated HAK list to the module source",
|
||||||
|
AppCommand: "apply-hak-manifest",
|
||||||
|
Usage: "crucible hak manifest [manifest-path]",
|
||||||
|
Aliases: []CommandAlias{{Name: "apply-hak-manifest"}},
|
||||||
|
},
|
||||||
|
},
|
||||||
Wired: true,
|
Wired: true,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
Name: "module",
|
Name: "module",
|
||||||
Bin: "crucible-module",
|
Bin: "crucible-module",
|
||||||
Summary: "build/extract/validate/compare the .mod",
|
Summary: "build/extract/validate/compare the .mod",
|
||||||
Legacy: []string{"build", "build-module", "extract", "validate", "compare"},
|
Commands: []Command{
|
||||||
// apply-hak-manifest is canonically a hak command but reachable here too;
|
{
|
||||||
// music is folded into the module build pipeline (command-surface.md).
|
Name: "build",
|
||||||
Extra: []string{"apply-hak-manifest", "music"},
|
Summary: "build the module and configured project outputs",
|
||||||
|
AppCommand: "build",
|
||||||
|
Usage: "crucible module build [--quiet|--verbose|--debug]",
|
||||||
|
Aliases: []CommandAlias{{Name: "build-module", AppCommand: "build-module"}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "extract",
|
||||||
|
Summary: "extract built archives into configured source trees",
|
||||||
|
AppCommand: "extract",
|
||||||
|
Usage: "crucible module extract [resource ...]",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "validate",
|
||||||
|
Summary: "validate project layout, config, and source inventory",
|
||||||
|
AppCommand: "validate",
|
||||||
|
Usage: "crucible module validate",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "compare",
|
||||||
|
Summary: "compare source content with built module and HAK archives",
|
||||||
|
AppCommand: "compare",
|
||||||
|
Usage: "crucible module compare",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "manifest",
|
||||||
|
Summary: "apply a generated HAK list to the module source",
|
||||||
|
AppCommand: "apply-hak-manifest",
|
||||||
|
Usage: "crucible module manifest [manifest-path]",
|
||||||
|
Aliases: []CommandAlias{{Name: "apply-hak-manifest"}},
|
||||||
|
},
|
||||||
|
},
|
||||||
Wired: true,
|
Wired: true,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
Name: "topdata",
|
Name: "topdata",
|
||||||
Bin: "crucible-topdata",
|
Bin: "crucible-topdata",
|
||||||
Summary: "compile 2da/tlk topdata + packages",
|
Summary: "compile 2da/tlk topdata + packages",
|
||||||
Legacy: []string{"validate-topdata", "build-topdata", "build-top-package", "compare-topdata", "convert-topdata"},
|
Commands: []Command{
|
||||||
|
{
|
||||||
|
Name: "validate",
|
||||||
|
Summary: "validate topdata layout and canonical JSON compatibility",
|
||||||
|
AppCommand: "validate-topdata",
|
||||||
|
Usage: "crucible topdata validate",
|
||||||
|
Aliases: []CommandAlias{{Name: "validate-topdata"}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "build",
|
||||||
|
Summary: "compile topdata and refresh the configured package",
|
||||||
|
AppCommand: "build-topdata",
|
||||||
|
Usage: "crucible topdata build [--force] [--wiki] [--skip-lfs]",
|
||||||
|
Options: []string{
|
||||||
|
"--force rebuild outputs even when current",
|
||||||
|
"--wiki build wiki drafts after compilation",
|
||||||
|
"--skip-lfs skip Git LFS materialization",
|
||||||
|
},
|
||||||
|
Aliases: []CommandAlias{{Name: "build-topdata"}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "package",
|
||||||
|
Summary: "package cached topdata outputs into HAK and TLK files",
|
||||||
|
AppCommand: "build-top-package",
|
||||||
|
Usage: "crucible topdata package [--force] [--skip-lfs]",
|
||||||
|
Options: []string{
|
||||||
|
"--force rebuild outputs even when current",
|
||||||
|
"--skip-lfs skip Git LFS materialization",
|
||||||
|
},
|
||||||
|
Aliases: []CommandAlias{{Name: "build-top-package"}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "compare",
|
||||||
|
Summary: "compare current output with a fresh native build",
|
||||||
|
AppCommand: "compare-topdata",
|
||||||
|
Usage: "crucible topdata compare",
|
||||||
|
Aliases: []CommandAlias{{Name: "compare-topdata"}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "convert",
|
||||||
|
Summary: "convert between 2DA and native JSON/module formats",
|
||||||
|
AppCommand: "convert-topdata",
|
||||||
|
Usage: "crucible topdata convert <2da-to-json|2da-to-module|json-to-2da> ...",
|
||||||
|
Options: []string{
|
||||||
|
"2da-to-json <input.2da> <output.json>",
|
||||||
|
"2da-to-module [flags] <input.2da> [output.json]",
|
||||||
|
"json-to-2da <input.json> <output.2da>",
|
||||||
|
},
|
||||||
|
Aliases: []CommandAlias{{Name: "convert-topdata"}},
|
||||||
|
},
|
||||||
|
},
|
||||||
Wired: true,
|
Wired: true,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
Name: "wiki",
|
Name: "wiki",
|
||||||
Bin: "crucible-wiki",
|
Bin: "crucible-wiki",
|
||||||
Summary: "render + deploy mechanical wiki pages",
|
Summary: "render + deploy mechanical wiki pages",
|
||||||
Legacy: []string{"build-wiki", "deploy-wiki"},
|
Commands: []Command{
|
||||||
|
{
|
||||||
|
Name: "build",
|
||||||
|
Summary: "render wiki page drafts from compiled topdata",
|
||||||
|
AppCommand: "build-wiki",
|
||||||
|
Usage: "crucible wiki build [--force]",
|
||||||
|
Options: []string{"--force rebuild drafts even when current"},
|
||||||
|
Aliases: []CommandAlias{{Name: "build-wiki"}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "deploy",
|
||||||
|
Summary: "deploy generated wiki pages to NodeBB",
|
||||||
|
AppCommand: "deploy-wiki",
|
||||||
|
Usage: "crucible wiki deploy [options]",
|
||||||
|
Options: []string{
|
||||||
|
"--source-dir <path> read generated pages here",
|
||||||
|
"--endpoint <url> override the NodeBB endpoint",
|
||||||
|
"--token <token> authenticate with an API token",
|
||||||
|
"--username/--password <value> authenticate with credentials",
|
||||||
|
"--version <version> label the deployed revision",
|
||||||
|
"--namespace <name> deploy selected namespaces",
|
||||||
|
"--category <namespace=id> override a namespace category",
|
||||||
|
"--manifest <path> write deployment state here",
|
||||||
|
"--stale-policy <policy> report, archive, or purge",
|
||||||
|
"--dry-run report changes without writing",
|
||||||
|
"--create allow missing pages to be created",
|
||||||
|
"--force update unchanged pages",
|
||||||
|
"--reset-managed-namespaces reset managed namespace state",
|
||||||
|
},
|
||||||
|
Aliases: []CommandAlias{{Name: "deploy-wiki"}},
|
||||||
|
},
|
||||||
|
},
|
||||||
Wired: true,
|
Wired: true,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -129,10 +332,14 @@ func menuItems() []menu.Item {
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
for _, sub := range b.subcommands() {
|
for _, sub := range b.subcommands() {
|
||||||
|
command, _, _ := b.command(sub)
|
||||||
items = append(items, menu.Item{
|
items = append(items, menu.Item{
|
||||||
Label: b.Name + " " + sub,
|
Label: b.Name + " " + sub,
|
||||||
Desc: b.Summary,
|
Desc: command.Summary,
|
||||||
Args: []string{b.Name, sub},
|
Args: []string{b.Name, sub},
|
||||||
|
Usage: command.Usage,
|
||||||
|
Options: append([]string(nil),
|
||||||
|
command.Options...),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -196,6 +403,19 @@ func runBuilder(name string, args []string, out, errw io.Writer) int {
|
|||||||
return exitOK
|
return exitOK
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if b.Wired {
|
||||||
|
// depot, assets and nwsync are self-contained builders: they parse their own
|
||||||
|
// subcommands and own their exit contract, bypassing the
|
||||||
|
// b.Commands/delegateLegacy legacy routing entirely.
|
||||||
|
switch b.Name {
|
||||||
|
case "depot":
|
||||||
|
return depot.Run(args, out, errw, os.Getenv)
|
||||||
|
case "assets":
|
||||||
|
return assets.Run(args, out, errw, os.Getenv)
|
||||||
|
case "nwsync":
|
||||||
|
return nwsync.Run(args, out, errw)
|
||||||
|
}
|
||||||
|
}
|
||||||
if !b.Wired {
|
if !b.Wired {
|
||||||
// No migrated logic yet (depot): fail closed, never fake an artifact.
|
// No migrated logic yet (depot): fail closed, never fake an artifact.
|
||||||
fmt.Fprintf(errw, unwiredMsg, b.Name, b.Bin)
|
fmt.Fprintf(errw, unwiredMsg, b.Name, b.Bin)
|
||||||
@@ -207,14 +427,21 @@ func runBuilder(name string, args []string, out, errw io.Writer) int {
|
|||||||
return exitUsage
|
return exitUsage
|
||||||
}
|
}
|
||||||
sub := args[0]
|
sub := args[0]
|
||||||
if !b.accepts(sub) {
|
command, appCommand, ok := b.command(sub)
|
||||||
|
if !ok {
|
||||||
fmt.Fprintf(errw, "crucible %s: unknown subcommand %q\n\n", b.Name, sub)
|
fmt.Fprintf(errw, "crucible %s: unknown subcommand %q\n\n", b.Name, sub)
|
||||||
builderHelp(errw, b)
|
builderHelp(errw, b)
|
||||||
return exitUsage
|
return exitUsage
|
||||||
}
|
}
|
||||||
// Delegate to the migrated legacy command surface. args[0] is already the
|
if len(args) > 1 {
|
||||||
// legacy command name, so it is forwarded unchanged.
|
switch args[1] {
|
||||||
return delegateLegacy(args, errw)
|
case "-h", "--help", "help":
|
||||||
|
commandHelp(out, command)
|
||||||
|
return exitOK
|
||||||
|
}
|
||||||
|
}
|
||||||
|
legacyArgs := append([]string{appCommand}, args[1:]...)
|
||||||
|
return delegateLegacy(legacyArgs, errw)
|
||||||
}
|
}
|
||||||
|
|
||||||
// delegateLegacy runs a migrated nwn-tool command via internal/app and maps its
|
// delegateLegacy runs a migrated nwn-tool command via internal/app and maps its
|
||||||
@@ -250,17 +477,15 @@ func usage(w io.Writer) {
|
|||||||
fmt.Fprintf(w, " config [args] inspect/validate effective configuration\n")
|
fmt.Fprintf(w, " config [args] inspect/validate effective configuration\n")
|
||||||
fmt.Fprintf(w, " changelog [args] generate the release changelog\n")
|
fmt.Fprintf(w, " changelog [args] generate the release changelog\n")
|
||||||
fmt.Fprintf(w, "\nBuilders read all inputs from the project tree (nwn-tool.yaml + data/); no\n")
|
fmt.Fprintf(w, "\nBuilders read all inputs from the project tree (nwn-tool.yaml + data/); no\n")
|
||||||
fmt.Fprintf(w, "NWN install is required. If a builder ever needs an NWN root, it must be\n")
|
fmt.Fprintf(w, "NWN install is required. A builder that one day needs NWN game data\n")
|
||||||
fmt.Fprintf(w, "passed explicitly via NWN_ROOT; crucible never guesses from $HOME. See\n")
|
fmt.Fprintf(w, "auto-detects the install; it never requires a hand-set path. See\n")
|
||||||
fmt.Fprintf(w, "docs/consumer-contract.md.\n")
|
fmt.Fprintf(w, "docs/consumer-contract.md.\n")
|
||||||
}
|
}
|
||||||
|
|
||||||
func list(w io.Writer) {
|
func list(w io.Writer) {
|
||||||
tw := tabwriter.NewWriter(w, 0, 2, 2, ' ', 0)
|
|
||||||
for _, b := range Registry {
|
for _, b := range Registry {
|
||||||
fmt.Fprintf(tw, " %s\t%s\t%s\n", b.Name, b.Bin, b.Summary)
|
fmt.Fprintf(w, "%s\t%s\t%s\n", b.Name, b.Bin, b.Summary)
|
||||||
}
|
}
|
||||||
tw.Flush()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func builderHelp(w io.Writer, b Builder) {
|
func builderHelp(w io.Writer, b Builder) {
|
||||||
@@ -272,8 +497,27 @@ func builderHelp(w io.Writer, b Builder) {
|
|||||||
}
|
}
|
||||||
fmt.Fprintf(w, "usage: crucible %s <subcommand> [args] (or: %s <subcommand> [args])\n\n", b.Name, b.Bin)
|
fmt.Fprintf(w, "usage: crucible %s <subcommand> [args] (or: %s <subcommand> [args])\n\n", b.Name, b.Bin)
|
||||||
fmt.Fprintf(w, "subcommands:\n")
|
fmt.Fprintf(w, "subcommands:\n")
|
||||||
for _, c := range b.subcommands() {
|
width := 0
|
||||||
fmt.Fprintf(w, " %s\n", c)
|
for _, command := range b.Commands {
|
||||||
|
if len(command.Name) > width {
|
||||||
|
width = len(command.Name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, command := range b.Commands {
|
||||||
|
fmt.Fprintf(w, " %-*s %s\n", width, command.Name, command.Summary)
|
||||||
}
|
}
|
||||||
fmt.Fprintf(w, "\nstatus: wired to migrated nwn-tool logic. See docs/command-surface.md.\n")
|
fmt.Fprintf(w, "\nstatus: wired to migrated nwn-tool logic. See docs/command-surface.md.\n")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func commandHelp(w io.Writer, command Command) {
|
||||||
|
fmt.Fprintf(w, "%s\n", command.Summary)
|
||||||
|
if command.Usage != "" {
|
||||||
|
fmt.Fprintf(w, "\nusage: %s\n", command.Usage)
|
||||||
|
}
|
||||||
|
if len(command.Options) > 0 {
|
||||||
|
fmt.Fprintln(w, "\noptions:")
|
||||||
|
for _, option := range command.Options {
|
||||||
|
fmt.Fprintf(w, " %s\n", option)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -25,22 +25,54 @@ func TestHelpAndNoArgs(t *testing.T) {
|
|||||||
if code := run([]string{"help"}, &out, &errw); code != exitOK {
|
if code := run([]string{"help"}, &out, &errw); code != exitOK {
|
||||||
t.Fatalf("help exit=%d want %d", code, exitOK)
|
t.Fatalf("help exit=%d want %d", code, exitOK)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "builders:") {
|
for _, builder := range Registry {
|
||||||
t.Fatalf("help output missing builders section:\n%s", out.String())
|
if !strings.Contains(out.String(), builder.Name) {
|
||||||
|
t.Errorf("help output missing builder %q:\n%s", builder.Name, out.String())
|
||||||
|
}
|
||||||
}
|
}
|
||||||
// No args is a usage error (exit 64) but still prints help.
|
// No args is a usage error (exit 64) but still prints help.
|
||||||
out.Reset()
|
out.Reset()
|
||||||
if code := run(nil, &out, &errw); code != exitUsage {
|
if code := run(nil, &out, &errw); code != exitUsage {
|
||||||
t.Fatalf("no-args exit=%d want %d", code, exitUsage)
|
t.Fatalf("no-args exit=%d want %d", code, exitUsage)
|
||||||
}
|
}
|
||||||
|
if out.Len() == 0 {
|
||||||
|
t.Fatal("no-args usage error should include help")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestListCoversRegistry(t *testing.T) {
|
func TestListCoversRegistry(t *testing.T) {
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
list(&out)
|
list(&out)
|
||||||
for _, b := range Registry {
|
lines := strings.Split(strings.TrimSpace(out.String()), "\n")
|
||||||
if !strings.Contains(out.String(), b.Name) || !strings.Contains(out.String(), b.Bin) {
|
if len(lines) != len(Registry) {
|
||||||
t.Fatalf("list missing %s/%s:\n%s", b.Name, b.Bin, out.String())
|
t.Fatalf("list returned %d rows for %d builders:\n%s", len(lines), len(Registry), out.String())
|
||||||
|
}
|
||||||
|
builders := make(map[string]Builder, len(Registry))
|
||||||
|
for _, builder := range Registry {
|
||||||
|
builders[builder.Name] = builder
|
||||||
|
}
|
||||||
|
seen := make(map[string]bool, len(lines))
|
||||||
|
for _, line := range lines {
|
||||||
|
fields := strings.Split(strings.TrimSpace(line), "\t")
|
||||||
|
if len(fields) != 3 {
|
||||||
|
t.Fatalf("list row must be name<TAB>bin<TAB>summary, got %q", line)
|
||||||
|
}
|
||||||
|
builder, ok := builders[fields[0]]
|
||||||
|
if !ok {
|
||||||
|
t.Errorf("list returned unregistered builder %q", fields[0])
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if fields[1] != builder.Bin {
|
||||||
|
t.Errorf("builder %q listed binary %q, want %q", builder.Name, fields[1], builder.Bin)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(fields[2]) == "" {
|
||||||
|
t.Errorf("builder %q listed an empty summary", builder.Name)
|
||||||
|
}
|
||||||
|
seen[fields[0]] = true
|
||||||
|
}
|
||||||
|
for _, builder := range Registry {
|
||||||
|
if !seen[builder.Name] {
|
||||||
|
t.Errorf("list missing builder %q", builder.Name)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -50,8 +82,8 @@ func TestUnknownBuilderFailsUsage(t *testing.T) {
|
|||||||
if code := run([]string{"frobnicate"}, &out, &errw); code != exitUsage {
|
if code := run([]string{"frobnicate"}, &out, &errw); code != exitUsage {
|
||||||
t.Fatalf("unknown builder exit=%d want %d", code, exitUsage)
|
t.Fatalf("unknown builder exit=%d want %d", code, exitUsage)
|
||||||
}
|
}
|
||||||
if !strings.Contains(errw.String(), "unknown builder") {
|
if errw.Len() == 0 {
|
||||||
t.Fatalf("unknown builder stderr=%q", errw.String())
|
t.Fatal("unknown builder should explain the usage error")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -65,8 +97,8 @@ func TestUnwiredBuilderFailsClosed(t *testing.T) {
|
|||||||
if code := run([]string{b.Name}, &out, &errw); code != exitUnwired {
|
if code := run([]string{b.Name}, &out, &errw); code != exitUnwired {
|
||||||
t.Errorf("crucible %s: exit=%d want %d (must fail closed)", b.Name, code, exitUnwired)
|
t.Errorf("crucible %s: exit=%d want %d (must fail closed)", b.Name, code, exitUnwired)
|
||||||
}
|
}
|
||||||
if !strings.Contains(errw.String(), "not wired") {
|
if errw.Len() == 0 {
|
||||||
t.Errorf("crucible %s: stderr missing fail-closed message: %q", b.Name, errw.String())
|
t.Errorf("crucible %s: missing fail-closed explanation", b.Name)
|
||||||
}
|
}
|
||||||
// Via standalone shim path.
|
// Via standalone shim path.
|
||||||
out.Reset()
|
out.Reset()
|
||||||
@@ -93,8 +125,8 @@ func TestWiredBuilderRejectsBadInvocation(t *testing.T) {
|
|||||||
if code := runBuilder(b.Name, []string{"frobnicate"}, &out, &errw); code != exitUsage {
|
if code := runBuilder(b.Name, []string{"frobnicate"}, &out, &errw); code != exitUsage {
|
||||||
t.Errorf("crucible %s frobnicate: exit=%d want %d", b.Name, code, exitUsage)
|
t.Errorf("crucible %s frobnicate: exit=%d want %d", b.Name, code, exitUsage)
|
||||||
}
|
}
|
||||||
if !strings.Contains(errw.String(), "unknown subcommand") {
|
if errw.Len() == 0 {
|
||||||
t.Errorf("crucible %s frobnicate: stderr=%q", b.Name, errw.String())
|
t.Errorf("crucible %s frobnicate: missing usage explanation", b.Name)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -105,34 +137,159 @@ func TestBuilderHelpIsOK(t *testing.T) {
|
|||||||
if code := runBuilder(b.Name, []string{"--help"}, &out, &errw); code != exitOK {
|
if code := runBuilder(b.Name, []string{"--help"}, &out, &errw); code != exitOK {
|
||||||
t.Errorf("%s --help: exit=%d want %d", b.Name, code, exitOK)
|
t.Errorf("%s --help: exit=%d want %d", b.Name, code, exitOK)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), b.Summary) {
|
if !strings.Contains(out.String(), b.Name) || !strings.Contains(out.String(), b.Bin) {
|
||||||
t.Errorf("%s --help: missing summary", b.Name)
|
t.Errorf("%s --help: missing builder identity", b.Name)
|
||||||
|
}
|
||||||
|
for _, subcommand := range b.subcommands() {
|
||||||
|
if !strings.Contains(out.String(), subcommand) {
|
||||||
|
t.Errorf("%s --help: missing subcommand %q", b.Name, subcommand)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Every legacy command named in command-surface.md must have exactly one
|
// selfContained builders parse their own subcommands instead of delegating to
|
||||||
// canonical home (Builder.Legacy), so the migrated surface has no gaps or
|
// the legacy internal/app surface.
|
||||||
// ambiguous homes.
|
var selfContained = map[string]bool{"depot": true, "assets": true, "nwsync": true}
|
||||||
func TestLegacyCommandsHaveExactlyOneHome(t *testing.T) {
|
|
||||||
legacy := []string{
|
func TestCanonicalCommandSurface(t *testing.T) {
|
||||||
"build", "build-module", "extract", "validate", "compare",
|
want := map[string][]string{
|
||||||
"build-haks", "apply-hak-manifest",
|
"depot": {"status", "push", "verify", "get", "pull"},
|
||||||
"validate-topdata", "build-topdata", "build-top-package", "compare-topdata", "convert-topdata",
|
"assets": {"compile", "convert", "upscale", "check-mdl", "fix-mdl", "check-dupes", "clean-dupes"},
|
||||||
"build-wiki", "deploy-wiki",
|
"hak": {"build", "manifest"},
|
||||||
|
"module": {"build", "extract", "validate", "compare", "manifest"},
|
||||||
|
"topdata": {"validate", "build", "package", "compare", "convert"},
|
||||||
|
"wiki": {"build", "deploy"},
|
||||||
|
"nwsync": {"emit", "assemble", "verify"},
|
||||||
}
|
}
|
||||||
for _, cmd := range legacy {
|
for _, builder := range Registry {
|
||||||
homes := 0
|
got := builder.subcommands()
|
||||||
for _, b := range Registry {
|
expected, ok := want[builder.Name]
|
||||||
for _, c := range b.Legacy {
|
if !ok {
|
||||||
if c == cmd {
|
t.Fatalf("unexpected builder %q", builder.Name)
|
||||||
homes++
|
}
|
||||||
|
if strings.Join(got, ",") != strings.Join(expected, ",") {
|
||||||
|
t.Errorf("%s commands = %v, want %v", builder.Name, got, expected)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if homes != 1 {
|
|
||||||
t.Errorf("legacy command %q has %d canonical homes, want 1", cmd, homes)
|
func TestRegistryCommandNamesAndAliasesAreUnambiguous(t *testing.T) {
|
||||||
|
for _, builder := range Registry {
|
||||||
|
seen := map[string]bool{}
|
||||||
|
for _, command := range builder.Commands {
|
||||||
|
// depot, assets and nwsync parse their own subcommands and bypass
|
||||||
|
// AppCommand routing entirely (see the self-contained-builder branch
|
||||||
|
// in runBuilder), so their Commands carry no AppCommand.
|
||||||
|
requireAppCommand := !selfContained[builder.Name]
|
||||||
|
if command.Name == "" || command.Summary == "" || command.Usage == "" || (requireAppCommand && command.AppCommand == "") {
|
||||||
|
t.Errorf("%s has incomplete command metadata: %#v", builder.Name, command)
|
||||||
}
|
}
|
||||||
|
if seen[command.Name] {
|
||||||
|
t.Errorf("%s repeats command name %q", builder.Name, command.Name)
|
||||||
|
}
|
||||||
|
seen[command.Name] = true
|
||||||
|
for _, alias := range command.Aliases {
|
||||||
|
if alias.Name == "" {
|
||||||
|
t.Errorf("%s %s has an empty alias", builder.Name, command.Name)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if seen[alias.Name] {
|
||||||
|
t.Errorf("%s repeats command or alias %q", builder.Name, alias.Name)
|
||||||
|
}
|
||||||
|
seen[alias.Name] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMusicCommandsAreNotAccepted(t *testing.T) {
|
||||||
|
for _, builderName := range []string{"hak", "module"} {
|
||||||
|
builder, ok := find(builderName)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("missing builder %q", builderName)
|
||||||
|
}
|
||||||
|
if builder.accepts("music") {
|
||||||
|
t.Errorf("%s must not accept the removed music command", builderName)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHiddenAliasesPreserveImplementationTargets(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
builder string
|
||||||
|
alias string
|
||||||
|
target string
|
||||||
|
}{
|
||||||
|
{"hak", "build-haks", "build-haks"},
|
||||||
|
{"hak", "apply-hak-manifest", "apply-hak-manifest"},
|
||||||
|
{"module", "build-module", "build-module"},
|
||||||
|
{"module", "apply-hak-manifest", "apply-hak-manifest"},
|
||||||
|
{"topdata", "validate-topdata", "validate-topdata"},
|
||||||
|
{"topdata", "build-topdata", "build-topdata"},
|
||||||
|
{"topdata", "build-top-package", "build-top-package"},
|
||||||
|
{"topdata", "compare-topdata", "compare-topdata"},
|
||||||
|
{"topdata", "convert-topdata", "convert-topdata"},
|
||||||
|
{"wiki", "build-wiki", "build-wiki"},
|
||||||
|
{"wiki", "deploy-wiki", "deploy-wiki"},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.builder+"/"+tt.alias, func(t *testing.T) {
|
||||||
|
builder, ok := find(tt.builder)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("missing builder %q", tt.builder)
|
||||||
|
}
|
||||||
|
command, target, ok := builder.command(tt.alias)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("hidden alias %q is not accepted", tt.alias)
|
||||||
|
}
|
||||||
|
if target != tt.target {
|
||||||
|
t.Fatalf("alias %q target = %q, want %q", tt.alias, target, tt.target)
|
||||||
|
}
|
||||||
|
if command.Name == tt.alias {
|
||||||
|
t.Fatalf("alias %q must not be a visible command", tt.alias)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuilderHelpHidesCompatibilityAliases(t *testing.T) {
|
||||||
|
aliases := map[string][]string{
|
||||||
|
"hak": {"build-haks", "apply-hak-manifest"},
|
||||||
|
"module": {"build-module", "apply-hak-manifest"},
|
||||||
|
"topdata": {"validate-topdata", "build-topdata", "build-top-package", "compare-topdata", "convert-topdata"},
|
||||||
|
"wiki": {"build-wiki", "deploy-wiki"},
|
||||||
|
}
|
||||||
|
for builderName, hidden := range aliases {
|
||||||
|
var out, errw bytes.Buffer
|
||||||
|
if code := runBuilder(builderName, []string{"--help"}, &out, &errw); code != exitOK {
|
||||||
|
t.Fatalf("%s help exit = %d", builderName, code)
|
||||||
|
}
|
||||||
|
for _, alias := range hidden {
|
||||||
|
if strings.Contains(out.String(), alias) {
|
||||||
|
t.Errorf("%s help exposed hidden alias %q:\n%s", builderName, alias, out.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCommandHelpUsesCanonicalGuidanceWithoutLoadingProject(t *testing.T) {
|
||||||
|
var out, errw bytes.Buffer
|
||||||
|
if code := runBuilder("topdata", []string{"build", "--help"}, &out, &errw); code != exitOK {
|
||||||
|
t.Fatalf("command help exit = %d, stderr:\n%s", code, errw.String())
|
||||||
|
}
|
||||||
|
for _, want := range []string{
|
||||||
|
"compile topdata",
|
||||||
|
"usage: crucible topdata build",
|
||||||
|
"--force",
|
||||||
|
"--skip-lfs",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(out.String(), want) {
|
||||||
|
t.Errorf("command help missing %q:\n%s", want, out.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if errw.Len() != 0 {
|
||||||
|
t.Fatalf("command help must not load a project or emit errors:\n%s", errw.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -142,15 +299,35 @@ func TestMenuItemsSkipsUnwiredIncludesWired(t *testing.T) {
|
|||||||
t.Fatal("expected menu items")
|
t.Fatal("expected menu items")
|
||||||
}
|
}
|
||||||
sawModuleBuild := false
|
sawModuleBuild := false
|
||||||
|
sawDepotStatus := false
|
||||||
for _, it := range items {
|
for _, it := range items {
|
||||||
if it.Args[0] == "depot" {
|
|
||||||
t.Errorf("unwired builder %q must not appear in the menu", it.Args[0])
|
|
||||||
}
|
|
||||||
if len(it.Args) == 2 && it.Args[0] == "module" && it.Args[1] == "build" {
|
if len(it.Args) == 2 && it.Args[0] == "module" && it.Args[1] == "build" {
|
||||||
sawModuleBuild = true
|
sawModuleBuild = true
|
||||||
}
|
}
|
||||||
|
if len(it.Args) == 2 && it.Args[0] == "depot" && it.Args[1] == "status" {
|
||||||
|
sawDepotStatus = true
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if !sawModuleBuild {
|
if !sawModuleBuild {
|
||||||
t.Error("expected 'module build' in the menu")
|
t.Error("expected 'module build' in the menu")
|
||||||
}
|
}
|
||||||
|
if !sawDepotStatus {
|
||||||
|
t.Error("expected wired builder 'depot status' in the menu")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestDepotRoutesToDepotRun asserts the depot special-case in runBuilder
|
||||||
|
// reaches depot.Run rather than the generic Commands/delegateLegacy path or
|
||||||
|
// the unwired fail-closed path. With no manifests dir present, depot.Run's
|
||||||
|
// status command fails resolving the backend/manifest (exit 70), but the
|
||||||
|
// stderr text must come from depot, never the dispatcher's unwiredMsg.
|
||||||
|
func TestDepotRoutesToDepotRun(t *testing.T) {
|
||||||
|
var out, errw bytes.Buffer
|
||||||
|
code := runBuilder("depot", []string{"status", "--target", "local"}, &out, &errw)
|
||||||
|
if errw.Len() == 0 {
|
||||||
|
t.Fatalf("depot status with no manifests dir: expected an error from depot.Run, got no stderr (exit=%d)", code)
|
||||||
|
}
|
||||||
|
if strings.Contains(errw.String(), "not wired yet") {
|
||||||
|
t.Fatalf("depot status: stderr shows the dispatcher's unwired message, want depot.Run's own error:\n%s", errw.String())
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+147
-64
@@ -2,14 +2,19 @@ package erf
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
|
"crypto/sha256"
|
||||||
"encoding/binary"
|
"encoding/binary"
|
||||||
|
"encoding/hex"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
|
"regexp"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
var expectedSHA256Pattern = regexp.MustCompile(`^[0-9a-f]{64}$`)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
headerSize = 160
|
headerSize = 160
|
||||||
versionV10 = "V1.0"
|
versionV10 = "V1.0"
|
||||||
@@ -30,6 +35,9 @@ type Resource struct {
|
|||||||
Data []byte
|
Data []byte
|
||||||
SourcePath string
|
SourcePath string
|
||||||
Size int64
|
Size int64
|
||||||
|
// ExpectedSHA256, when set, is the lowercase hex SHA-256 the streamed
|
||||||
|
// SourcePath payload must hash to. A mismatch fails the write.
|
||||||
|
ExpectedSHA256 string
|
||||||
}
|
}
|
||||||
|
|
||||||
type header struct {
|
type header struct {
|
||||||
@@ -59,6 +67,19 @@ type resourceEntry struct {
|
|||||||
Size uint32
|
Size uint32
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// extensionTypes and typeExtensions must stay exact inverses of each other;
|
||||||
|
// init() panics if they drift apart.
|
||||||
|
//
|
||||||
|
// Numbers below 0x0BB8 follow upstream neverwinter.nim
|
||||||
|
// (neverwinter/restype.nim). The 0x0BB8 and up entries are lyt/vis/mdx: real
|
||||||
|
// Aurora archive types that NWN1 ships in its own data/*.bif but upstream
|
||||||
|
// happens not to register. xoreos corroborates those three numbers
|
||||||
|
// (src/aurora/types.h).
|
||||||
|
//
|
||||||
|
// This table is NWN:EE only. Do not add NWN2 formats (mdb, gr2, wlk, xml):
|
||||||
|
// NWN:EE either gives that number to something else (2070 is xbc here and mdb
|
||||||
|
// in NWN2) or has no number for it at all, so packing one into a HAK writes a
|
||||||
|
// resource the game misreads. See the reserved list in erf_test.go.
|
||||||
var extensionTypes = map[string]uint16{
|
var extensionTypes = map[string]uint16{
|
||||||
"res": 0x0000,
|
"res": 0x0000,
|
||||||
"bmp": 0x0001,
|
"bmp": 0x0001,
|
||||||
@@ -88,12 +109,13 @@ var extensionTypes = map[string]uint16{
|
|||||||
"utt": 0x07F0,
|
"utt": 0x07F0,
|
||||||
"dds": 0x07F1,
|
"dds": 0x07F1,
|
||||||
"uts": 0x07F3,
|
"uts": 0x07F3,
|
||||||
|
"ltr": 0x07F4,
|
||||||
|
"gff": 0x07F5,
|
||||||
"fac": 0x07F6,
|
"fac": 0x07F6,
|
||||||
"gff": 0x07F7,
|
|
||||||
"ute": 0x07F8,
|
"ute": 0x07F8,
|
||||||
"utd": 0x07FA,
|
"utd": 0x07FA,
|
||||||
"utp": 0x07FC,
|
"utp": 0x07FC,
|
||||||
"dfa": 0x07FD,
|
"dft": 0x07FD,
|
||||||
"gic": 0x07FE,
|
"gic": 0x07FE,
|
||||||
"gui": 0x07FF,
|
"gui": 0x07FF,
|
||||||
"utm": 0x0803,
|
"utm": 0x0803,
|
||||||
@@ -109,19 +131,15 @@ var extensionTypes = map[string]uint16{
|
|||||||
"ndb": 0x0810,
|
"ndb": 0x0810,
|
||||||
"ptm": 0x0811,
|
"ptm": 0x0811,
|
||||||
"ptt": 0x0812,
|
"ptt": 0x0812,
|
||||||
"ltr": 0x0813,
|
|
||||||
"shd": 0x0815,
|
"shd": 0x0815,
|
||||||
"mdb": 0x0816,
|
|
||||||
"mtr": 0x0818,
|
"mtr": 0x0818,
|
||||||
"jpg": 0x081C,
|
|
||||||
"lod": 0x081E,
|
"lod": 0x081E,
|
||||||
|
"gif": 0x081F,
|
||||||
"png": 0x0820,
|
"png": 0x0820,
|
||||||
|
"jpg": 0x0821,
|
||||||
"lyt": 0x0BB8,
|
"lyt": 0x0BB8,
|
||||||
"vis": 0x0BB9,
|
"vis": 0x0BB9,
|
||||||
"mdx": 0x0BC0,
|
"mdx": 0x0BC0,
|
||||||
"wlk": 0x0BCC,
|
|
||||||
"xml": 0x0BCD,
|
|
||||||
"gr2": 0x0FA3,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
var typeExtensions = map[uint16]string{
|
var typeExtensions = map[uint16]string{
|
||||||
@@ -153,12 +171,13 @@ var typeExtensions = map[uint16]string{
|
|||||||
0x07F0: "utt",
|
0x07F0: "utt",
|
||||||
0x07F1: "dds",
|
0x07F1: "dds",
|
||||||
0x07F3: "uts",
|
0x07F3: "uts",
|
||||||
|
0x07F4: "ltr",
|
||||||
|
0x07F5: "gff",
|
||||||
0x07F6: "fac",
|
0x07F6: "fac",
|
||||||
0x07F7: "gff",
|
|
||||||
0x07F8: "ute",
|
0x07F8: "ute",
|
||||||
0x07FA: "utd",
|
0x07FA: "utd",
|
||||||
0x07FC: "utp",
|
0x07FC: "utp",
|
||||||
0x07FD: "dfa",
|
0x07FD: "dft",
|
||||||
0x07FE: "gic",
|
0x07FE: "gic",
|
||||||
0x07FF: "gui",
|
0x07FF: "gui",
|
||||||
0x0803: "utm",
|
0x0803: "utm",
|
||||||
@@ -174,19 +193,15 @@ var typeExtensions = map[uint16]string{
|
|||||||
0x0810: "ndb",
|
0x0810: "ndb",
|
||||||
0x0811: "ptm",
|
0x0811: "ptm",
|
||||||
0x0812: "ptt",
|
0x0812: "ptt",
|
||||||
0x0813: "ltr",
|
|
||||||
0x0815: "shd",
|
0x0815: "shd",
|
||||||
0x0816: "mdb",
|
|
||||||
0x0818: "mtr",
|
0x0818: "mtr",
|
||||||
0x081C: "jpg",
|
|
||||||
0x081E: "lod",
|
0x081E: "lod",
|
||||||
|
0x081F: "gif",
|
||||||
0x0820: "png",
|
0x0820: "png",
|
||||||
|
0x0821: "jpg",
|
||||||
0x0BB8: "lyt",
|
0x0BB8: "lyt",
|
||||||
0x0BB9: "vis",
|
0x0BB9: "vis",
|
||||||
0x0BC0: "mdx",
|
0x0BC0: "mdx",
|
||||||
0x0BCC: "wlk",
|
|
||||||
0x0BCD: "xml",
|
|
||||||
0x0FA3: "gr2",
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func init() {
|
func init() {
|
||||||
@@ -195,8 +210,13 @@ func init() {
|
|||||||
if !ok {
|
if !ok {
|
||||||
panic(fmt.Sprintf("missing canonical extension for resource type 0x%04X", resourceType))
|
panic(fmt.Sprintf("missing canonical extension for resource type 0x%04X", resourceType))
|
||||||
}
|
}
|
||||||
if canonicalExt == ext {
|
if canonicalExt != ext {
|
||||||
continue
|
panic(fmt.Sprintf("resource type 0x%04X is %q in extensionTypes but %q in typeExtensions", resourceType, ext, canonicalExt))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for resourceType, ext := range typeExtensions {
|
||||||
|
if registered, ok := extensionTypes[ext]; !ok || registered != resourceType {
|
||||||
|
panic(fmt.Sprintf("extension %q is 0x%04X in typeExtensions but 0x%04X in extensionTypes (registered=%v)", ext, resourceType, registered, ok))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -291,63 +311,110 @@ func Write(w io.Writer, archive Archive) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// IndexEntry locates one resource inside an archive without holding its
|
||||||
|
// payload. Streaming callers read one payload at a time from these, so peak
|
||||||
|
// memory tracks the largest resource instead of the whole archive.
|
||||||
|
type IndexEntry struct {
|
||||||
|
Name string
|
||||||
|
Type uint16
|
||||||
|
Offset int64
|
||||||
|
Size int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// Index is the header plus the resource table of an ERF: everything except the
|
||||||
|
// payloads.
|
||||||
|
type Index struct {
|
||||||
|
FileType string
|
||||||
|
Version string
|
||||||
|
Entries []IndexEntry
|
||||||
|
}
|
||||||
|
|
||||||
|
// ReadIndex parses the tables of an ERF of the given size, reading only the
|
||||||
|
// header, the key list and the resource list.
|
||||||
|
func ReadIndex(r io.ReaderAt, size int64) (Index, error) {
|
||||||
|
if size < headerSize {
|
||||||
|
return Index{}, fmt.Errorf("erf file too small: %d bytes", size)
|
||||||
|
}
|
||||||
|
|
||||||
|
var hdr header
|
||||||
|
if err := binary.Read(io.NewSectionReader(r, 0, headerSize), binary.LittleEndian, &hdr); err != nil {
|
||||||
|
return Index{}, fmt.Errorf("decode erf header: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if int64(hdr.KeyListOffset)+int64(hdr.EntryCount)*24 > size {
|
||||||
|
return Index{}, fmt.Errorf("erf key list exceeds file bounds")
|
||||||
|
}
|
||||||
|
keys := make([]keyEntry, hdr.EntryCount)
|
||||||
|
keyReader := io.NewSectionReader(r, int64(hdr.KeyListOffset), int64(hdr.EntryCount)*24)
|
||||||
|
if err := binary.Read(keyReader, binary.LittleEndian, &keys); err != nil {
|
||||||
|
return Index{}, fmt.Errorf("decode key list: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if int64(hdr.ResourceListOffset)+int64(hdr.EntryCount)*8 > size {
|
||||||
|
return Index{}, fmt.Errorf("erf resource list exceeds file bounds")
|
||||||
|
}
|
||||||
|
entries := make([]resourceEntry, hdr.EntryCount)
|
||||||
|
entryReader := io.NewSectionReader(r, int64(hdr.ResourceListOffset), int64(hdr.EntryCount)*8)
|
||||||
|
if err := binary.Read(entryReader, binary.LittleEndian, &entries); err != nil {
|
||||||
|
return Index{}, fmt.Errorf("decode resource list: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
index := Index{
|
||||||
|
FileType: string(hdr.FileType[:]),
|
||||||
|
Version: string(hdr.Version[:]),
|
||||||
|
Entries: make([]IndexEntry, 0, hdr.EntryCount),
|
||||||
|
}
|
||||||
|
for position, key := range keys {
|
||||||
|
entry := entries[position]
|
||||||
|
if int64(entry.Offset)+int64(entry.Size) > size {
|
||||||
|
return Index{}, fmt.Errorf("resource %d exceeds file bounds", position)
|
||||||
|
}
|
||||||
|
index.Entries = append(index.Entries, IndexEntry{
|
||||||
|
Name: string(bytes.TrimRight(key.ResRef[:], "\x00")),
|
||||||
|
Type: key.ResourceType,
|
||||||
|
Offset: int64(entry.Offset),
|
||||||
|
Size: int64(entry.Size),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return index, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ReadPayload returns one resource's bytes.
|
||||||
|
func ReadPayload(r io.ReaderAt, entry IndexEntry) ([]byte, error) {
|
||||||
|
payload := make([]byte, entry.Size)
|
||||||
|
if _, err := r.ReadAt(payload, entry.Offset); err != nil {
|
||||||
|
return nil, fmt.Errorf("read resource %q: %w", entry.Name, err)
|
||||||
|
}
|
||||||
|
return payload, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read materialises a whole archive. Payloads are subslices of the buffer the
|
||||||
|
// archive was read into, so nothing is copied twice: a caller must not mutate
|
||||||
|
// Data. Callers that only need one resource at a time should use ReadIndex
|
||||||
|
// instead, which never holds the archive at all.
|
||||||
func Read(r io.Reader) (Archive, error) {
|
func Read(r io.Reader) (Archive, error) {
|
||||||
data, err := io.ReadAll(r)
|
data, err := io.ReadAll(r)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return Archive{}, fmt.Errorf("read erf: %w", err)
|
return Archive{}, fmt.Errorf("read erf: %w", err)
|
||||||
}
|
}
|
||||||
if len(data) < headerSize {
|
index, err := ReadIndex(bytes.NewReader(data), int64(len(data)))
|
||||||
return Archive{}, fmt.Errorf("erf file too small: %d bytes", len(data))
|
if err != nil {
|
||||||
|
return Archive{}, err
|
||||||
}
|
}
|
||||||
|
|
||||||
var hdr header
|
resources := make([]Resource, 0, len(index.Entries))
|
||||||
if err := binary.Read(bytes.NewReader(data[:headerSize]), binary.LittleEndian, &hdr); err != nil {
|
for _, entry := range index.Entries {
|
||||||
return Archive{}, fmt.Errorf("decode erf header: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
keyStart := int(hdr.KeyListOffset)
|
|
||||||
keyEnd := keyStart + int(hdr.EntryCount)*24
|
|
||||||
if keyEnd > len(data) {
|
|
||||||
return Archive{}, fmt.Errorf("erf key list exceeds file bounds")
|
|
||||||
}
|
|
||||||
keys := make([]keyEntry, hdr.EntryCount)
|
|
||||||
if err := binary.Read(bytes.NewReader(data[keyStart:keyEnd]), binary.LittleEndian, &keys); err != nil {
|
|
||||||
return Archive{}, fmt.Errorf("decode key list: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
resourceStart := int(hdr.ResourceListOffset)
|
|
||||||
resourceEnd := resourceStart + int(hdr.EntryCount)*8
|
|
||||||
if resourceEnd > len(data) {
|
|
||||||
return Archive{}, fmt.Errorf("erf resource list exceeds file bounds")
|
|
||||||
}
|
|
||||||
entries := make([]resourceEntry, hdr.EntryCount)
|
|
||||||
if err := binary.Read(bytes.NewReader(data[resourceStart:resourceEnd]), binary.LittleEndian, &entries); err != nil {
|
|
||||||
return Archive{}, fmt.Errorf("decode resource list: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
resources := make([]Resource, 0, hdr.EntryCount)
|
|
||||||
for index, key := range keys {
|
|
||||||
entry := entries[index]
|
|
||||||
start := int(entry.Offset)
|
|
||||||
end := start + int(entry.Size)
|
|
||||||
if end > len(data) {
|
|
||||||
return Archive{}, fmt.Errorf("resource %d exceeds file bounds", index)
|
|
||||||
}
|
|
||||||
|
|
||||||
resref := string(bytes.TrimRight(key.ResRef[:], "\x00"))
|
|
||||||
payload := make([]byte, entry.Size)
|
|
||||||
copy(payload, data[start:end])
|
|
||||||
resources = append(resources, Resource{
|
resources = append(resources, Resource{
|
||||||
Name: resref,
|
Name: entry.Name,
|
||||||
Type: key.ResourceType,
|
Type: entry.Type,
|
||||||
Data: payload,
|
Data: data[entry.Offset : entry.Offset+entry.Size],
|
||||||
Size: int64(entry.Size),
|
Size: entry.Size,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
return Archive{
|
return Archive{
|
||||||
FileType: string(hdr.FileType[:]),
|
FileType: index.FileType,
|
||||||
Version: string(hdr.Version[:]),
|
Version: index.Version,
|
||||||
Resources: resources,
|
Resources: resources,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
@@ -407,18 +474,34 @@ func writeResourceData(w io.Writer, resource Resource) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if resource.ExpectedSHA256 != "" && !expectedSHA256Pattern.MatchString(resource.ExpectedSHA256) {
|
||||||
|
return fmt.Errorf("resource %q has invalid expected sha256", resource.SourcePath)
|
||||||
|
}
|
||||||
|
|
||||||
file, err := os.Open(resource.SourcePath)
|
file, err := os.Open(resource.SourcePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("open resource %q: %w", resource.SourcePath, err)
|
return fmt.Errorf("open resource %q: %w", resource.SourcePath, err)
|
||||||
}
|
}
|
||||||
defer file.Close()
|
defer file.Close()
|
||||||
|
|
||||||
written, err := io.Copy(w, file)
|
// The hash is computed while streaming into w, so size/SHA mismatches are
|
||||||
|
// only detected AFTER the (bad) bytes have already been written. A non-nil
|
||||||
|
// return therefore means w holds partially-written, unverified output; the
|
||||||
|
// caller must discard it. writeHAKArchive does this by writing to a temp file
|
||||||
|
// and removing it on any Write error rather than renaming it into place.
|
||||||
|
hash := sha256.New()
|
||||||
|
written, err := io.Copy(io.MultiWriter(w, hash), file)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("copy resource %q: %w", resource.SourcePath, err)
|
return fmt.Errorf("copy resource %q: %w", resource.SourcePath, err)
|
||||||
}
|
}
|
||||||
if resource.Size > 0 && written != resource.Size {
|
if resource.Size > 0 && written != resource.Size {
|
||||||
return fmt.Errorf("copy resource %q: expected %d bytes, wrote %d", resource.SourcePath, resource.Size, written)
|
return fmt.Errorf("copy resource %q: expected %d bytes, wrote %d", resource.SourcePath, resource.Size, written)
|
||||||
}
|
}
|
||||||
|
if resource.ExpectedSHA256 != "" {
|
||||||
|
got := hex.EncodeToString(hash.Sum(nil))
|
||||||
|
if got != resource.ExpectedSHA256 {
|
||||||
|
return fmt.Errorf("resource %q sha256 mismatch: expected %s, got %s", resource.SourcePath, resource.ExpectedSHA256, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
+132
-5
@@ -2,9 +2,72 @@ package erf
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
func writeTempPayload(t *testing.T, payload []byte) string {
|
||||||
|
t.Helper()
|
||||||
|
path := filepath.Join(t.TempDir(), "payload.bin")
|
||||||
|
if err := os.WriteFile(path, payload, 0o644); err != nil {
|
||||||
|
t.Fatalf("write payload: %v", err)
|
||||||
|
}
|
||||||
|
return path
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWriteRejectsSourcePathSHA256Mismatch(t *testing.T) {
|
||||||
|
payload := []byte("hello world")
|
||||||
|
path := writeTempPayload(t, payload)
|
||||||
|
|
||||||
|
archive := New("HAK ", []Resource{{
|
||||||
|
Name: "asset",
|
||||||
|
Type: 0x0003,
|
||||||
|
SourcePath: path,
|
||||||
|
Size: int64(len(payload)),
|
||||||
|
ExpectedSHA256: strings.Repeat("a", 64),
|
||||||
|
}})
|
||||||
|
|
||||||
|
var buf bytes.Buffer
|
||||||
|
err := Write(&buf, archive)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected sha256 mismatch error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "sha256 mismatch") {
|
||||||
|
t.Fatalf("error = %v, want sha256 mismatch", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWriteAcceptsMatchingSourcePathSHA256(t *testing.T) {
|
||||||
|
payload := []byte("hello world")
|
||||||
|
path := writeTempPayload(t, payload)
|
||||||
|
sum := sha256.Sum256(payload)
|
||||||
|
|
||||||
|
archive := New("HAK ", []Resource{{
|
||||||
|
Name: "asset",
|
||||||
|
Type: 0x0003,
|
||||||
|
SourcePath: path,
|
||||||
|
Size: int64(len(payload)),
|
||||||
|
ExpectedSHA256: hex.EncodeToString(sum[:]),
|
||||||
|
}})
|
||||||
|
|
||||||
|
var buf bytes.Buffer
|
||||||
|
if err := Write(&buf, archive); err != nil {
|
||||||
|
t.Fatalf("write: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
decoded, err := Read(bytes.NewReader(buf.Bytes()))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read: %v", err)
|
||||||
|
}
|
||||||
|
if len(decoded.Resources) != 1 || string(decoded.Resources[0].Data) != string(payload) {
|
||||||
|
t.Fatalf("unexpected payload: %#v", decoded.Resources)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestArchiveRoundTrip(t *testing.T) {
|
func TestArchiveRoundTrip(t *testing.T) {
|
||||||
archive := New("MOD ", []Resource{
|
archive := New("MOD ", []Resource{
|
||||||
{Name: "module", Type: 0x07DE, Data: []byte("ifo")},
|
{Name: "module", Type: 0x07DE, Data: []byte("ifo")},
|
||||||
@@ -71,14 +134,10 @@ func TestExtensionMappingsSupportModernAssetTypes(t *testing.T) {
|
|||||||
"mtr": 0x0818,
|
"mtr": 0x0818,
|
||||||
"shd": 0x0815,
|
"shd": 0x0815,
|
||||||
"txi": 0x07E6,
|
"txi": 0x07E6,
|
||||||
"jpg": 0x081C,
|
"jpg": 0x0821,
|
||||||
"mdb": 0x0816,
|
|
||||||
"lyt": 0x0BB8,
|
"lyt": 0x0BB8,
|
||||||
"vis": 0x0BB9,
|
"vis": 0x0BB9,
|
||||||
"mdx": 0x0BC0,
|
"mdx": 0x0BC0,
|
||||||
"xml": 0x0BCD,
|
|
||||||
"wlk": 0x0BCC,
|
|
||||||
"gr2": 0x0FA3,
|
|
||||||
}
|
}
|
||||||
for ext, wantType := range cases {
|
for ext, wantType := range cases {
|
||||||
gotType, ok := ResourceTypeForExtension(ext)
|
gotType, ok := ResourceTypeForExtension(ext)
|
||||||
@@ -128,3 +187,71 @@ func TestExtensionMappingsSupportAllUTBlueprintTypes(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestRestypeTableMatchesUpstream pins the restype numbers Crucible shares with
|
||||||
|
// neverwinter.nim's restype.nim. The values below are upstream's; a mismatch
|
||||||
|
// means a HAK we write is mislabelled for the game.
|
||||||
|
func TestRestypeTableMatchesUpstream(t *testing.T) {
|
||||||
|
upstream := map[string]uint16{
|
||||||
|
"res": 0, "bmp": 1, "mve": 2, "tga": 3, "wav": 4, "plt": 6, "ini": 7,
|
||||||
|
"bmu": 8, "txt": 10, "mdl": 2002, "nss": 2009, "ncs": 2010, "are": 2012,
|
||||||
|
"set": 2013, "ifo": 2014, "bic": 2015, "wok": 2016, "2da": 2017,
|
||||||
|
"tlk": 2018, "txi": 2022, "git": 2023, "uti": 2025, "utc": 2027,
|
||||||
|
"dlg": 2029, "itp": 2030, "utt": 2032, "dds": 2033, "uts": 2035,
|
||||||
|
"ltr": 2036, "gff": 2037, "fac": 2038, "ute": 2040, "utd": 2042,
|
||||||
|
"utp": 2044, "dft": 2045, "gic": 2046, "gui": 2047, "utm": 2051,
|
||||||
|
"dwk": 2052, "pwk": 2053, "utg": 2055, "jrl": 2056, "utw": 2058,
|
||||||
|
"ssf": 2060, "hak": 2061, "nwm": 2062, "bik": 2063, "ndb": 2064,
|
||||||
|
"ptm": 2065, "ptt": 2066, "shd": 2069, "mtr": 2072, "lod": 2078,
|
||||||
|
"gif": 2079, "png": 2080, "jpg": 2081,
|
||||||
|
}
|
||||||
|
for ext, want := range upstream {
|
||||||
|
got, ok := ResourceTypeForExtension(ext)
|
||||||
|
if !ok {
|
||||||
|
t.Errorf("%s: not registered, upstream has %d", ext, want)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if got != want {
|
||||||
|
t.Errorf("%s: registered as %d, upstream has %d", ext, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Extensions upstream registers that Crucible must not reuse for anything else.
|
||||||
|
reserved := map[uint16]string{2039: "bte", 2067: "bak", 2070: "xbc", 2076: "tml"}
|
||||||
|
for number, upstreamExt := range reserved {
|
||||||
|
if ext, ok := ExtensionForResourceType(number); ok {
|
||||||
|
t.Errorf("%d: registered as %s, upstream reserves it for %s", number, ext, upstreamExt)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestNWN2FormatsAreNotRegistered keeps NWN2 file formats out of an NWN:EE
|
||||||
|
// table. mdb and gr2 have Aurora numbers that mean something else (or nothing)
|
||||||
|
// in NWN:EE; wlk and xml have no archive number in any Aurora game, so the
|
||||||
|
// values Crucible used for them were invented. Packing any of these into a HAK
|
||||||
|
// writes a resource the game misreads.
|
||||||
|
func TestNWN2FormatsAreNotRegistered(t *testing.T) {
|
||||||
|
for _, ext := range []string{"mdb", "gr2", "wlk", "xml"} {
|
||||||
|
if number, ok := ResourceTypeForExtension(ext); ok {
|
||||||
|
t.Errorf("%s is an NWN2 format but is registered as 0x%04X", ext, number)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRestypeTablesAreMutualInverses is the check init() is meant to enforce.
|
||||||
|
func TestRestypeTablesAreMutualInverses(t *testing.T) {
|
||||||
|
for ext, number := range extensionTypes {
|
||||||
|
canonical, ok := typeExtensions[number]
|
||||||
|
if !ok {
|
||||||
|
t.Errorf("%s: number 0x%04X missing from typeExtensions", ext, number)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if canonical != ext {
|
||||||
|
t.Errorf("%s: maps to 0x%04X, which maps back to %s", ext, number, canonical)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for number, ext := range typeExtensions {
|
||||||
|
if got, ok := extensionTypes[ext]; !ok || got != number {
|
||||||
|
t.Errorf("0x%04X: maps to %s, which maps back to 0x%04X (ok=%v)", number, ext, got, ok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user