Files

T

archvillainetteandClaude Opus 4.8 5fff8353da docs: scope check-dupes to paths-only; add depot pipeline migration charter

check-dupes ports directory mode only; the --manifests hak-scoped check is
manifest-integrity and moves to the depot pipeline migration. Add that
follow-up as a charter spec (import/sync/export + mdl gate + manifest dupes
into `crucible depot`), marked draft pending its own brainstorming.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-07-12 09:26:48 +02:00

5.2 KiB

Raw Blame History

Crucible `depot` pipeline migration — design (charter / draft)

Date: 2026-07-12 Status: draft — needs its own brainstorming pass before implementation. Depends on: 2026-07-12-crucible-assets-builder-design.md (the assets builder and internal/assets/mdl module) landing first.

This is a scoping charter, not an implementation-ready design. The scripts it covers are nontrivial and lean heavily on sow-assets-manifest/scripts/lib.sh (~35 KB). A full design requires studying lib.sh and the manifest format in depth; that work happens in this spec's own brainstorming before any code.

Why this exists

sow-assets-manifest is the last consumer still carrying real builder logic in shell. The assets spec folded its standalone integrity tools into Crucible but deliberately left the depot-interacting pipeline behind, because those scripts read/write the content-addressed depot and the asset manifests — depot domain, and a bigger lift. This charter completes the migration so sow-assets-manifest becomes what the other consumers already are: a thin caller of released crucible binaries with no vendored toolkit logic.

What moves into `crucible depot`

The depot builder already owns status/push/verify/get/pull. This adds the authoring side:

Script	Proposed command	What it does today
`import.sh` (318 lines)	`crucible depot import <edit-dir> [prefix…]`	Import an edit tree into the depot + manifests: sticky per-path member assignment, new paths to the tail member, per-category `max_bytes` rebalance (spill overflow forward, no full repack), a local stat-cache to skip unchanged files, batched hashing + parallel `depot_put_many`.
`sync-assets.sh` (61 lines)	`crucible depot sync <tree> [prefix…]`	Full reconcile = import + prune manifest entries whose path no longer exists (scoped by prefix when given). Delegates to `import.sh`.
`export.sh` (96 lines)	`crucible depot export [glob] [--hak m] [--restype ext] --to <dir>`	Materialize a manifest-selected subset out of the depot into a clean edit tree (category paths preserved, no metadata).
`check-duplicate-names.sh --manifests`	`crucible depot check-dupes` (name TBD)	Hak-scoped basename collisions read from `assets/*.yml` (`.assets[].path` grouped by `.assets[].hak`). Manifest-structure check, not files on disk.

The inline pre-import mdl gate in import.sh (batched mdl-scan.awk pass that aborts on an H/B model-name mismatch before any blob is hashed/uploaded) is reimplemented with the in-process internal/assets/mdl module from the assets spec — same H/B contract, no subprocess.

What retires when this lands

Deleted from sow-assets-manifest/scripts/ once the commands above are wired and the Makefile is repointed:

import.sh, sync-assets.sh, export.sh
mdl-scan.awk, mdl-name-lib.sh (only remaining consumer was import.sh)
check-duplicate-names.sh (its directory mode already superseded by crucible assets check-dupes; the --manifests mode moves here)
whatever else in lib.sh becomes dead once the above are gone (assess during design — lib.sh also serves build-haks.sh, pack-haks.sh, promote.sh, etc., which are not in scope here, so lib.sh likely shrinks, not dies)

Makefile targets import, sync, haks (its sync/import/dupe steps), export repoint to ./crucible.sh depot ….

Known hard parts (resolve in brainstorming)

Manifest read/write parity. Byte-stable output, member max_bytes rebalance semantics, sticky path→member assignment. Must match the current awk/lib.sh behavior or diff-review of manifests becomes noise. Crucible already parses these manifests (internal/pipeline, hak build) — reuse, don't reinvent.
Depot backends. import/export drive bunny/cdn/local through lib.sh's depot_put_many/depot_require_write (incl. the BUNNY_STORAGE_PASSWORD prompt and cdn→bunny write normalization). Crucible's internal/depot already speaks these backends for pull/push — the authoring side should share that client, not a second implementation.
Stat-cache. import.sh's .cache/import/<target>/<cat>.tsv skip-unchanged optimization is the dominant re-import speedup. Decide whether Crucible reproduces it, replaces it (content-addressing already dedupes uploads), or drops it with a measured justification.
Throughput. The shell versions are already batched to avoid per-file forks; a Go port should be at least as fast (in-process, no forks) — verify, don't assume.
Scoping semantics. Prefix-scoped import/prune (segment-boundary match) and export glob matching (**/*/literal dots, no shell expansion) must port exactly.

Out of scope

The HAK build/pack/promote/publish scripts (build-haks.sh, pack-haks.sh, promote.sh, publish-release*.sh, hak-artifact-record.sh) and their lib.sh support. crucible hak build already exists; whether these thin release wrappers also migrate is a separate question, not this charter.

Next step

Brainstorm this charter into a full design: read lib.sh and the manifest format, settle the hard parts above, then write the implementation-ready spec and plan. Do not implement from this document.

5.2 KiB Raw Blame History

Crucible depot pipeline migration — design (charter / draft)