docs: scope check-dupes to paths-only; add depot pipeline migration charter
check-dupes ports directory mode only; the --manifests hak-scoped check is manifest-integrity and moves to the depot pipeline migration. Add that follow-up as a charter spec (import/sync/export + mdl gate + manifest dupes into `crucible depot`), marked draft pending its own brainstorming. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -30,8 +30,8 @@ Seven commands:
|
|||||||
model-name mismatches (read-only).
|
model-name mismatches (read-only).
|
||||||
- `crucible assets fix-mdl` — lowercase `.mdl` filenames and rewrite ASCII
|
- `crucible assets fix-mdl` — lowercase `.mdl` filenames and rewrite ASCII
|
||||||
model/root names to match each file's stem.
|
model/root names to match each file's stem.
|
||||||
- `crucible assets check-dupes` — report runtime-name (basename) collisions in
|
- `crucible assets check-dupes` — report runtime-name (basename) collisions
|
||||||
directories, or hak-scoped collisions across `assets/*.yml` manifests.
|
across directories.
|
||||||
- `crucible assets clean-dupes` — delete files from a "clean" tree whose
|
- `crucible assets clean-dupes` — delete files from a "clean" tree whose
|
||||||
basename collides with anything in a "primary" tree.
|
basename collides with anything in a "primary" tree.
|
||||||
|
|
||||||
@@ -235,21 +235,21 @@ candidate only if it needs a content fix (name mismatch) or a filename
|
|||||||
lowercase, so the expensive per-file read/rewrite runs only on the few that
|
lowercase, so the expensive per-file read/rewrite runs only on the few that
|
||||||
matter, not the whole tree.
|
matter, not the whole tree.
|
||||||
|
|
||||||
## `crucible assets check-dupes [--manifests] <dir>…`
|
## `crucible assets check-dupes <dir>…`
|
||||||
|
|
||||||
Port of `check-duplicate-names.sh`. Read-only. NWN packs hak entries by
|
Port of `check-duplicate-names.sh`'s **directory mode only**. Read-only. NWN
|
||||||
basename, not path, so files at different paths with the same basename
|
packs hak entries by basename, not path, so files at different paths with the
|
||||||
(case-insensitively) silently clobber on pack. This flags them.
|
same basename (case-insensitively) silently clobber on pack. Across all `<dir>`
|
||||||
|
args, report any two files whose lowercased basename matches.
|
||||||
- Directory mode (default): across all `<dir>` args, report any two files whose
|
|
||||||
lowercased basename matches.
|
|
||||||
- `--manifests`: additionally read every `assets/*.yml` in the current repo and
|
|
||||||
report basename collisions **within the same hak** (`.assets[].path` grouped
|
|
||||||
by `.assets[].hak`), reusing Crucible's existing `gopkg.in/yaml.v3` parsing.
|
|
||||||
Requires no external `yq`.
|
|
||||||
|
|
||||||
Exit non-zero if any collision is found.
|
Exit non-zero if any collision is found.
|
||||||
|
|
||||||
|
The script's `--manifests` mode (hak-scoped collisions read from `assets/*.yml`)
|
||||||
|
is **not** ported here — it inspects manifest structure, not files on disk, so
|
||||||
|
it belongs with the manifest/depot tooling. It moves in the follow-up spec (see
|
||||||
|
Out of scope); until then `sow-assets-manifest` keeps
|
||||||
|
`check-duplicate-names.sh` solely for that mode.
|
||||||
|
|
||||||
## `crucible assets clean-dupes [--dry-run] <primary> <clean>`
|
## `crucible assets clean-dupes [--dry-run] <primary> <clean>`
|
||||||
|
|
||||||
Port of `clean-duplicate-names.sh`. Deletes files from the `<clean>` tree whose
|
Port of `clean-duplicate-names.sh`. Deletes files from the `<clean>` tree whose
|
||||||
@@ -269,22 +269,24 @@ Exit zero on success with a count of removed (or would-remove) files.
|
|||||||
(`crucible.sh`) and uses `crucible depot` from its Makefile, so it resolves a
|
(`crucible.sh`) and uses `crucible depot` from its Makefile, so it resolves a
|
||||||
released `crucible` with zero extra setup.
|
released `crucible` with zero extra setup.
|
||||||
|
|
||||||
- **Delete** the four folded driver scripts: `scripts/check-ascii-mdl.sh`,
|
- **Delete** the three fully-folded driver scripts: `scripts/check-ascii-mdl.sh`,
|
||||||
`scripts/fix-mdl-model-names.sh`, `scripts/check-duplicate-names.sh`,
|
`scripts/fix-mdl-model-names.sh`, `scripts/clean-duplicate-names.sh`.
|
||||||
`scripts/clean-duplicate-names.sh`.
|
|
||||||
- **Repoint** the Makefile targets to the wrapper:
|
- **Repoint** the Makefile targets to the wrapper:
|
||||||
- `check-mdl` → `./crucible.sh assets check-mdl …`
|
- `check-mdl` → `./crucible.sh assets check-mdl …`
|
||||||
- `fix-mdl` → `./crucible.sh assets fix-mdl [--dry-run] …`
|
- `fix-mdl` → `./crucible.sh assets fix-mdl [--dry-run] …`
|
||||||
- `check-dupes` → `./crucible.sh assets check-dupes …` and
|
- `check-dupes` — its directory scan → `./crucible.sh assets check-dupes …`;
|
||||||
`./crucible.sh assets check-dupes --manifests`
|
the `check-duplicate-names.sh --manifests` line stays as-is.
|
||||||
- `clean-dupes` → `./crucible.sh assets clean-dupes [--dry-run] <PRIMARY> <CLEAN>`
|
- `clean-dupes` → `./crucible.sh assets clean-dupes [--dry-run] <PRIMARY> <CLEAN>`
|
||||||
- the `haks` target's `check-duplicate-names.sh --manifests` guard →
|
- **Keep for now:**
|
||||||
`./crucible.sh assets check-dupes --manifests`
|
- `scripts/check-duplicate-names.sh` — kept **only** for its `--manifests`
|
||||||
- **Keep for now:** `scripts/mdl-scan.awk` and `scripts/mdl-name-lib.sh`. They
|
mode, still called by the `check` and `haks` targets. Its directory mode is
|
||||||
are still sourced by `import.sh` for its batched pre-import gate (~28× faster
|
superseded by `crucible assets check-dupes`. Ported and deleted in the
|
||||||
than per-file, on a hot path). They retire in the follow-up below. Because
|
follow-up.
|
||||||
both the awk classifier and the Go `mdl` module must agree, the port keeps
|
- `scripts/mdl-scan.awk` and `scripts/mdl-name-lib.sh` — still sourced by
|
||||||
detection byte-for-byte identical (a shared fixture set checks this).
|
`import.sh` for its batched pre-import gate (~28× faster than per-file, on a
|
||||||
|
hot path). They retire in the follow-up too. Because both the awk classifier
|
||||||
|
and the Go `mdl` module must agree, the port keeps detection byte-for-byte
|
||||||
|
identical (a shared fixture set checks this).
|
||||||
|
|
||||||
Nothing in `sow-assets-manifest` is committed by this project's plan except the
|
Nothing in `sow-assets-manifest` is committed by this project's plan except the
|
||||||
script deletions and Makefile edits; that repo's change lands as its own commit
|
script deletions and Makefile edits; that repo's change lands as its own commit
|
||||||
@@ -292,12 +294,15 @@ once the new `crucible` release with `assets` is available.
|
|||||||
|
|
||||||
## Out of scope — follow-up spec
|
## Out of scope — follow-up spec
|
||||||
|
|
||||||
Moving the **depot-interacting pipeline** (`import.sh`, `sync-assets.sh`,
|
Moving the **depot-interacting pipeline** into `crucible depot` is a separate
|
||||||
`export.sh`, and the inline mdl gate) into `crucible depot` is a separate
|
project with its own spec
|
||||||
project with its own spec. When `import.sh` becomes `crucible depot import`, its
|
(`2026-07-12-crucible-depot-pipeline-migration-design.md`). It covers
|
||||||
gate uses the in-process `internal/assets/mdl` module and
|
`import.sh`, `sync-assets.sh`, `export.sh`, the inline mdl gate, and the
|
||||||
`scripts/mdl-scan.awk` + `scripts/mdl-name-lib.sh` are deleted. This spec
|
`check-duplicate-names.sh --manifests` manifest-collision check. When that lands:
|
||||||
deliberately stops at the standalone integrity tools to keep one focused plan.
|
its gate uses the in-process `internal/assets/mdl` module, and
|
||||||
|
`scripts/mdl-scan.awk`, `scripts/mdl-name-lib.sh`, and
|
||||||
|
`scripts/check-duplicate-names.sh` are all deleted. This spec deliberately stops
|
||||||
|
at the standalone, file-on-disk integrity tools to keep one focused plan.
|
||||||
|
|
||||||
## Error handling
|
## Error handling
|
||||||
|
|
||||||
@@ -327,8 +332,7 @@ Go tests in `internal/assets`:
|
|||||||
a model that then passes `CheckNames`. A parity fixture set shared in intent
|
a model that then passes `CheckNames`. A parity fixture set shared in intent
|
||||||
with `mdl-scan.awk` so the Go port and the still-present awk agree.
|
with `mdl-scan.awk` so the Go port and the still-present awk agree.
|
||||||
- `check-dupes`/`clean-dupes`: temp trees asserting basename-collision
|
- `check-dupes`/`clean-dupes`: temp trees asserting basename-collision
|
||||||
detection (case-insensitive), the `--manifests` hak-scoped path against a
|
detection (case-insensitive), the overlap guard, and `--dry-run` mutating
|
||||||
small `assets/*.yml` fixture, the overlap guard, and `--dry-run` mutating
|
|
||||||
nothing.
|
nothing.
|
||||||
|
|
||||||
External engine/upscaler calls are exercised through the `run` indirection with
|
External engine/upscaler calls are exercised through the `run` indirection with
|
||||||
|
|||||||
@@ -0,0 +1,90 @@
|
|||||||
|
# Crucible `depot` pipeline migration — design (charter / draft)
|
||||||
|
|
||||||
|
Date: 2026-07-12
|
||||||
|
Status: **draft — needs its own brainstorming pass before implementation.**
|
||||||
|
Depends on: `2026-07-12-crucible-assets-builder-design.md` (the `assets` builder
|
||||||
|
and `internal/assets/mdl` module) landing first.
|
||||||
|
|
||||||
|
This is a scoping charter, not an implementation-ready design. The scripts it
|
||||||
|
covers are nontrivial and lean heavily on `sow-assets-manifest/scripts/lib.sh`
|
||||||
|
(~35 KB). A full design requires studying `lib.sh` and the manifest format in
|
||||||
|
depth; that work happens in this spec's own brainstorming before any code.
|
||||||
|
|
||||||
|
## Why this exists
|
||||||
|
|
||||||
|
`sow-assets-manifest` is the last consumer still carrying real builder logic in
|
||||||
|
shell. The `assets` spec folded its standalone integrity tools into Crucible but
|
||||||
|
deliberately left the **depot-interacting pipeline** behind, because those
|
||||||
|
scripts read/write the content-addressed depot and the asset manifests — depot
|
||||||
|
domain, and a bigger lift. This charter completes the migration so
|
||||||
|
`sow-assets-manifest` becomes what the other consumers already are: a thin
|
||||||
|
caller of released `crucible` binaries with no vendored toolkit logic.
|
||||||
|
|
||||||
|
## What moves into `crucible depot`
|
||||||
|
|
||||||
|
The depot builder already owns `status/push/verify/get/pull`. This adds the
|
||||||
|
authoring side:
|
||||||
|
|
||||||
|
| Script | Proposed command | What it does today |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `import.sh` (318 lines) | `crucible depot import <edit-dir> [prefix…]` | Import an edit tree into the depot + manifests: sticky per-path member assignment, new paths to the tail member, per-category `max_bytes` rebalance (spill overflow forward, no full repack), a local stat-cache to skip unchanged files, batched hashing + parallel `depot_put_many`. |
|
||||||
|
| `sync-assets.sh` (61 lines) | `crucible depot sync <tree> [prefix…]` | Full reconcile = import + prune manifest entries whose path no longer exists (scoped by prefix when given). Delegates to `import.sh`. |
|
||||||
|
| `export.sh` (96 lines) | `crucible depot export [glob] [--hak m] [--restype ext] --to <dir>` | Materialize a manifest-selected subset out of the depot into a clean edit tree (category paths preserved, no metadata). |
|
||||||
|
| `check-duplicate-names.sh --manifests` | `crucible depot check-dupes` (name TBD) | Hak-scoped basename collisions read from `assets/*.yml` (`.assets[].path` grouped by `.assets[].hak`). Manifest-structure check, not files on disk. |
|
||||||
|
|
||||||
|
The inline **pre-import mdl gate** in `import.sh` (batched `mdl-scan.awk` pass
|
||||||
|
that aborts on an H/B model-name mismatch before any blob is hashed/uploaded)
|
||||||
|
is reimplemented with the in-process `internal/assets/mdl` module from the
|
||||||
|
`assets` spec — same H/B contract, no subprocess.
|
||||||
|
|
||||||
|
## What retires when this lands
|
||||||
|
|
||||||
|
Deleted from `sow-assets-manifest/scripts/` once the commands above are wired
|
||||||
|
and the Makefile is repointed:
|
||||||
|
|
||||||
|
- `import.sh`, `sync-assets.sh`, `export.sh`
|
||||||
|
- `mdl-scan.awk`, `mdl-name-lib.sh` (only remaining consumer was `import.sh`)
|
||||||
|
- `check-duplicate-names.sh` (its directory mode already superseded by
|
||||||
|
`crucible assets check-dupes`; the `--manifests` mode moves here)
|
||||||
|
- whatever else in `lib.sh` becomes dead once the above are gone (assess during
|
||||||
|
design — `lib.sh` also serves `build-haks.sh`, `pack-haks.sh`, `promote.sh`,
|
||||||
|
etc., which are **not** in scope here, so `lib.sh` likely shrinks, not dies)
|
||||||
|
|
||||||
|
Makefile targets `import`, `sync`, `haks` (its sync/import/dupe steps), `export`
|
||||||
|
repoint to `./crucible.sh depot …`.
|
||||||
|
|
||||||
|
## Known hard parts (resolve in brainstorming)
|
||||||
|
|
||||||
|
- **Manifest read/write parity.** Byte-stable output, member `max_bytes`
|
||||||
|
rebalance semantics, sticky path→member assignment. Must match the current
|
||||||
|
awk/`lib.sh` behavior or diff-review of manifests becomes noise. Crucible
|
||||||
|
already parses these manifests (`internal/pipeline`, hak build) — reuse, don't
|
||||||
|
reinvent.
|
||||||
|
- **Depot backends.** `import`/`export` drive bunny/cdn/local through
|
||||||
|
`lib.sh`'s `depot_put_many`/`depot_require_write` (incl. the
|
||||||
|
`BUNNY_STORAGE_PASSWORD` prompt and cdn→bunny write normalization). Crucible's
|
||||||
|
`internal/depot` already speaks these backends for pull/push — the authoring
|
||||||
|
side should share that client, not a second implementation.
|
||||||
|
- **Stat-cache.** `import.sh`'s `.cache/import/<target>/<cat>.tsv` skip-unchanged
|
||||||
|
optimization is the dominant re-import speedup. Decide whether Crucible
|
||||||
|
reproduces it, replaces it (content-addressing already dedupes uploads), or
|
||||||
|
drops it with a measured justification.
|
||||||
|
- **Throughput.** The shell versions are already batched to avoid per-file
|
||||||
|
forks; a Go port should be at least as fast (in-process, no forks) — verify,
|
||||||
|
don't assume.
|
||||||
|
- **Scoping semantics.** Prefix-scoped import/prune (segment-boundary match) and
|
||||||
|
export glob matching (`**`/`*`/literal dots, no shell expansion) must port
|
||||||
|
exactly.
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
The HAK build/pack/promote/publish scripts (`build-haks.sh`, `pack-haks.sh`,
|
||||||
|
`promote.sh`, `publish-release*.sh`, `hak-artifact-record.sh`) and their
|
||||||
|
`lib.sh` support. `crucible hak build` already exists; whether these thin
|
||||||
|
release wrappers also migrate is a separate question, not this charter.
|
||||||
|
|
||||||
|
## Next step
|
||||||
|
|
||||||
|
Brainstorm this charter into a full design: read `lib.sh` and the manifest
|
||||||
|
format, settle the hard parts above, then write the implementation-ready spec
|
||||||
|
and plan. Do not implement from this document.
|
||||||
Reference in New Issue
Block a user