25 KiB
Crucible Depot Core (Increment 1) Implementation Plan
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: Implement crucible depot {status,push,verify,get,pull} with local/cdn/bunny backends in sow-tools, then cut sow-assets-manifest over to it and delete the bash it replaces.
Architecture: New stdlib-only internal/depot package with its own Run(args) int entrypoint (exit contract 0/1/2/70 is richer than internal/app.Run's 0/1, so dispatch special-cases depot instead of routing through app.Run). Backends implement a small interface; a sweep engine does parallel range-GET probes with serial re-confirm. Cutover repo work happens in sow-assets-manifest on its own branch.
Tech Stack: Go 1.26, stdlib net/http/crypto/sha256/sync, gopkg.in/yaml.v3 (already a dep — add nothing to go.mod, vendorHash must not change).
Spec: docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md — read it before any task; it is authoritative.
Global Constraints
- No new Go dependencies. stdlib + existing
gopkg.in/yaml.v3only. - HEAD is banned for existence checks. All probes: IPv4,
GETwithRange: bytes=0-0, expect206(accept any 2xx). A test must fail if HEAD is introduced. - Never prompt, never read stdin for credentials — read path included. Missing key → clear error on stderr, exit
70. - No cache of "what I uploaded" — presence is always probed against the real target backend.
unconfirmedis a distinct state frommissing; exit2when unconfirmed-only.- Exit codes:
0clean,1drift (referenced-but-absent),2unconfirmed-only,64usage,70internal/backend error. - Blob key:
sha256/<sha[0:2]>/<sha[2:4]>/<sha>; sha must match^[0-9a-f]{64}$. - Env names (unchanged from bash):
DEPOT_CDN_BASE/BUNNY_CDN_BASE(cdn read base, defaulthttps://cdn-a7f3k9.westgate.pw),BUNNY_STORAGE_HOST(required for bunny, no default — spec overrides bash),BUNNY_STORAGE_ZONE(defaultsow-assets-depot),BUNNY_STORAGE_READ_PASSWORD(falls back toBUNNY_STORAGE_PASSWORD) for reads/probes,BUNNY_STORAGE_PASSWORDfor PUT,DEPOT_PROBE_JOBS(default 16),DEPOT_JOBS(default 16),DEPOT_CONNECT_TIMEOUT(default 10s),DEPOT_PROBE_MAX_TIME(default 30s),DEPOT_CONFIRM_RETRIES(default 3),DEPOT_CONFIRM_MAX(default 200, 0 = unlimited). - Upload:
PUTwithAccessKey: <write key>andChecksum: <UPPERCASE sha>headers; each sha uploaded at most once per run; probe-before-put. - Every downloaded blob is re-hashed; deleted on mismatch.
cdnbackend writes fail closed (error, never fake success).- Verify with
make vet testin sow-tools (run insidenix developif go missing from PATH). - Commit after each task. Never commit to main. sow-tools branch:
depot-core-increment-1. sow-assets-manifest branch:crucible-depot-cutover. sow-tools has an unrelated modified spec file on main — leave it out of commits unless the task says otherwise.
Task 1: Package skeleton — config + manifest parsing
Files:
- Create:
internal/depot/config.go,internal/depot/config_test.go - Create:
internal/depot/manifest.go,internal/depot/manifest_test.go
Interfaces (Produces):
package depot
// config.go
type Config struct {
CDNBase string // DEPOT_CDN_BASE || BUNNY_CDN_BASE || default
StorageHost string // BUNNY_STORAGE_HOST, no default
StorageZone string // BUNNY_STORAGE_ZONE, default "sow-assets-depot"
ReadKey string // BUNNY_STORAGE_READ_PASSWORD || BUNNY_STORAGE_PASSWORD
WriteKey string // BUNNY_STORAGE_PASSWORD
ProbeJobs int // DEPOT_PROBE_JOBS, default 16
Jobs int // DEPOT_JOBS, default 16
ConnectTimeout time.Duration // DEPOT_CONNECT_TIMEOUT seconds, default 10s
ProbeMaxTime time.Duration // DEPOT_PROBE_MAX_TIME seconds, default 30s
ConfirmRetries int // DEPOT_CONFIRM_RETRIES, default 3
ConfirmMax int // DEPOT_CONFIRM_MAX, default 200 (0 = unlimited)
}
func LoadConfig(getenv func(string) string) Config
// manifest.go
// ReferencedSHAs parses every *.yml in dir (schema: assets[].{path,sha256,size,restype,hak})
// and returns the deduplicated sha set plus per-sha size (for pull).
func ReferencedSHAs(dir string) (map[string]int64, error)
func ValidSHA(s string) bool // ^[0-9a-f]{64}$
func BlobKey(sha string) string // "sha256/ab/cd/<sha>"
LoadConfig takes getenv so tests inject env without t.Setenv races. Invalid ints/durations in env → fall back to default (do not error). ReferencedSHAs errors on: unreadable dir, no *.yml files, YAML parse failure, or an entry whose sha256 fails ValidSHA (report file + path). Manifest struct:
type manifestFile struct {
Assets []struct {
Path string `yaml:"path"`
SHA256 string `yaml:"sha256"`
Size int64 `yaml:"size"`
} `yaml:"assets"`
}
- Step 1:
git -C sow-tools switch -c depot-core-increment-1(from main). - Step 2: Write failing tests:
TestLoadConfigDefaults,TestLoadConfigReadKeyFallback(READ unset → WriteKey value; READ set → its own),TestLoadConfigBadIntFallsBack,TestBlobKey(BlobKey("0b1d…")=="sha256/0b/1d/0b1d…"),TestValidSHA(rejects uppercase, short, non-hex),TestReferencedSHAs(temp dir with two yml files sharing one sha → dedup; bad sha → error naming the file; empty dir → error). - Step 3:
go test ./internal/depot/— verify FAIL (compile error is fine). - Step 4: Implement
config.go+manifest.gominimally. - Step 5:
go test ./internal/depot/PASS;gofmt -l -w internal/depot. - Step 6: Commit:
feat(depot): config resolution and manifest sha parsing.
Task 2: Local backend
Files:
- Create:
internal/depot/backend.go,internal/depot/local.go,internal/depot/local_test.go
Interfaces (Produces):
// backend.go
type ProbeState int
const (
Present ProbeState = iota
Absent
Unconfirmed
)
type Backend interface {
Name() string
// Probe returns the existence state of one sha. transient=true means a
// retry might change the answer (feeds the serial confirm loop).
Probe(ctx context.Context, sha string) (state ProbeState, transient bool, err error)
// Get fetches sha into dest (temp file + rename), re-hashes, deletes on mismatch.
Get(ctx context.Context, sha, dest string) error
// Put uploads bytes from src for sha. Read-only backends return an error.
Put(ctx context.Context, sha, src string) error
}
// local.go
type LocalBackend struct{ Root string } // implements Backend; Name()=="local"
Local semantics: Probe = os.Stat(Root/BlobKey(sha)), never transient. Put = copy to temp file in dest dir then os.Rename (MkdirAll parents). Get = copy out + re-hash (hash the source bytes while copying via io.TeeReader into sha256.New()), delete dest and error on mismatch. err from Probe only for real I/O errors (permission), not absence.
- Step 1: Failing tests:
TestLocalRoundTrip(Put→Probe Present→Get→bytes equal),TestLocalProbeAbsent,TestLocalGetHashMismatch(hand-write a corrupt blob file at the keyed path, Get returns error and dest does not exist),TestLocalPutIdempotent. - Step 2: Verify FAIL. Step 3: Implement. Step 4: PASS + gofmt.
- Step 5: Commit:
feat(depot): backend interface and local backend.
Task 3: Remote backends (cdn, bunny) with fake-Bunny tests
Files:
- Create:
internal/depot/remote.go,internal/depot/remote_test.go
Interfaces (Consumes): Backend, ProbeState, Config, BlobKey.
Produces:
// NewBackend returns the backend for name ("local"|"cdn"|"bunny").
// localRoot is used only for "local". Fails closed: "bunny" with empty
// StorageHost or empty ReadKey returns an error (never prompts);
// unknown name returns an error.
func NewBackend(name, localRoot string, cfg Config) (Backend, error)
type httpBackend struct { ... } // shared by cdn and bunny
Implementation requirements:
// IPv4-only transport — REQUIRED, spec bans IPv6/HEAD probe hazards.
transport := &http.Transport{
DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
d := net.Dialer{Timeout: cfg.ConnectTimeout}
return d.DialContext(ctx, "tcp4", addr)
},
MaxIdleConnsPerHost: cfg.ProbeJobs,
}
client := &http.Client{Transport: transport, Timeout: cfg.ProbeMaxTime}
- Probe (both remotes):
GET <base>/<BlobKey(sha)>with headerRange: bytes=0-0; drain/close body. 2xx →Present; 404/410 →Absent; anything else (incl. transport error) →Unconfirmed, transient=true. Never http.Head / MethodHead. - cdn: base =
cfg.CDNBase, no auth headers,Putreturnserrors.New("cdn backend is read-only"). - bunny: probe/read against storage
https://<StorageHost>/<StorageZone>/<BlobKey(sha)>withAccessKey: <ReadKey>.Get: try CDN base first, on non-2xx fall back to storage URL with ReadKey (mirrors bash_depot_bunny_get); re-hash, delete on mismatch.Put: probe first, skip if Present; elsePUTstorage URL, headersAccessKey: <WriteKey>,Checksum: <strings.ToUpper(sha)>, body streamed from src file; non-2xx → error.Putwith empty WriteKey → error before any I/O. - Both
Gets write via temp file + rename and re-hash exactly like local.
httptest note: httptest.NewServer listens on 127.0.0.1, which the tcp4 dialer reaches — no test shim needed.
- Step 1: Failing tests against an
httptestfake Bunny (a handler recording method+path+headers, serving a blob map):TestProbeUsesRangeGetNotHead— probe a present sha; assert recorded request hasMethod=="GET"andRange=="bytes=0-0"; also assert the depot source contains no HEAD probe: the handler returns 500 for any HEAD request, and the test asserts zero HEAD requests were received across the whole test run (this is the anti-HEAD regression tripwire from field note 3).TestProbeStates— 200/206→Present, 404→Absent, 428→Unconfirmed+transient, 503→Unconfirmed+transient.TestBunnyPutSequence— Put of an absent sha: recorded sequence is [GET range probe, PUT]; PUT hasChecksum= uppercase sha andAccessKey= write key; Put of a present sha: probe only, no PUT.TestReadWriteKeySplit— probe uses ReadKey, PUT uses WriteKey (distinct values asserted).TestBunnyNoReadKeyFailsClosed—NewBackend("bunny",…)with empty read key returns error without reading stdin: replaceos.Stdinwith the read end of anos.Pipe()for the test; after the call, write+close and confirm the pipe is undrained (or simply assert error occurs with no stdin interaction because the code path has no stdin reference — grep-based assertion acceptable: test fails ifinternal/depotsources mentionos.Stdinor/dev/tty).TestBunnyNoStorageHostFailsClosed.TestGetHashMismatchDeletes— server returns wrong bytes; Get errors, dest removed.TestCDNPutReadOnly.
- Step 2: Verify FAIL. Step 3: Implement
remote.go. Step 4: PASS + gofmt +go vet ./.... - Step 5: Commit:
feat(depot): cdn and bunny HTTP backends with fake-Bunny tests.
Task 4: Sweep engine — parallel probe + serial confirm
Files:
- Create:
internal/depot/sweep.go,internal/depot/sweep_test.go
Interfaces (Consumes): Backend, Config, ProbeState.
Produces:
type SweepResult struct {
Present []string
Missing []string // confirmed absent (404/410 after retries)
Unconfirmed []string // probe budget exhausted, state unknown
}
// Sweep probes every sha against b: one parallel pass (cfg.ProbeJobs workers,
// sync.WaitGroup + buffered-channel work queue), then any non-Present result
// is re-probed serially with linear backoff (attempt t sleeps t seconds,
// cfg.ConfirmRetries attempts) — unless the non-Present count exceeds
// cfg.ConfirmMax (>0), in which case the remainder is reported Unconfirmed
// without serial re-probe (never collapsed into Missing).
// A transient result that stays non-2xx after retries is Unconfirmed;
// a 404/410 is Missing. Backend errors (non-transient) abort with error.
func Sweep(ctx context.Context, b Backend, shas []string, cfg Config, progress io.Writer) (SweepResult, error)
Sort shas before sweeping (deterministic output). Progress: print probed X/N roughly every 1000 blobs to progress (may be io.Discard). For tests, make the backoff sleeper injectable: package-level var confirmSleep = time.Sleep, overridden in tests.
- Step 1: Failing tests using a stub Backend (function fields):
TestSweepAllPresent,TestSweepMissing(404 → Missing, no retries beyond first serial confirm).TestSweepThrottledUnconfirmed— stub always returns Unconfirmed/transient → after retries lands inUnconfirmed, neverMissing; assertconfirmSleepcalled with 1s then 2s.TestSweepTransientRecovers— first probe 428, serial re-probe 206 → Present.TestSweepConfirmMaxCap— 5 non-present, ConfirmMax=2 → all 5 Unconfirmed, zero serial re-probes.TestSweepParallelBounded— 100 shas, ProbeJobs=4; stub counts concurrent in-flight probes (atomic), assert max ≤ 4.
- Step 2: FAIL. Step 3: Implement. Step 4: PASS + gofmt.
- Step 5: Commit:
feat(depot): parallel probe sweep with serial confirm and unconfirmed state.
Task 5: Commands + Run entrypoint + integration test
Files:
- Create:
internal/depot/run.go,internal/depot/run_test.go,internal/depot/integration_test.go
Interfaces (Consumes): everything above. Produces:
// Run executes a depot subcommand. args[0] is the subcommand
// (status|push|verify|get|pull); returns the process exit code.
// Exit: 0 clean, 1 drift, 2 unconfirmed-only, 64 usage, 70 internal.
func Run(args []string, stdout, stderr io.Writer, getenv func(string) string) int
Flag parsing: one flag.FlagSet per subcommand (stdlib; internal/depot is a fresh package, the hand-rolled loop in internal/app is not a constraint here). Flags per spec:
status [--manifests DIR] [--source LOCAL_DEPOT] --target bunny|cdn|local
push [--manifests DIR] --source LOCAL_DEPOT --target bunny
verify [--manifests DIR] --target bunny|cdn [--sample N]
get <sha> <dest> --target cdn|bunny|local
pull [--manifests DIR] --dest DIR --target cdn|bunny
--manifests default assets. --target local uses --source as root for status, and --target-side root comes from DEPOT_DIR env when target is local (matches bash DEPOT_DIR); document in usage string. Missing required flag / unknown subcommand / no args → usage on stderr, exit 64.
Command semantics (all build shas via ReferencedSHAs, backend via NewBackend, sweep via Sweep):
- status: sweep; print
referenced=<n> present=<n> missing=<n> unconfirmed=<n>to stdout, then each missing sha on its own line prefixedmissingand each unconfirmed prefixedunconfirmed. Exit 1 if missing>0; else 2 if unconfirmed>0; else 0. - push: target must be
bunny(enforce). Sweep; treat missing∪unconfirmed as to-upload (unconfirmed→re-upload is safe/idempotent per spec). For each, read bytes from--sourcelocal root; a sha absent from source → report and count as failed. Upload withcfg.Jobsparallel workers, each sha at most once. Exit 70 on upload error, 1 if any sha was missing from source, else 0. Printuploaded=<n> failed=<n>. - verify: sweep (exit 1/2 as status) plus, when clean, download
--sampleN (default 3, 0=skip) random blobs (math/randseeded — pick via deterministic-enoughrand.New(rand.NewSource(time.Now().UnixNano()))) to temp files viaGet(which re-hashes); any failure → exit 1. - get:
Get(sha, dest); validate sha; exit 0/70. - pull: for every referenced sha, dest path
--dest/BlobKey(sha). Incremental: if dest file exists, re-hash it; matching → skip, mismatch → re-download. Download absent/mismatched withcfg.Jobsworkers viaGet. Printpulled=<n> present=<n>; any failed download → exit 70 after finishing the batch (report each failure). Non-zero missing at the source (404) → exit 1 listing them.
Wire the binary: Modify internal/dispatch/dispatch.go — in runBuilder, before the Wired check/delegateLegacy routing, special-case:
if b.Name == "depot" && b.Wired {
return depot.Run(args[1:], os.Stdout, errw, os.Getenv)
}
(Exact insertion point: the implementer must read runBuilder and place this after subcommand resolution is bypassed — depot does its own subcommand parsing, so pass the raw args through, i.e. args[1:] where args[0]=="depot" at the dispatcher level, or the full arg slice in RunBuilder("depot", …) context. Verify both crucible depot status … and crucible-depot status … paths hit depot.Run.)
- Step 1: Failing unit tests (
run_test.go):TestRunUsage(no args→64, unknown cmd→64, push --target cdn→64),TestStatusExitCodes(drift→1, unconfirmed-only→2 using a bunny backend pointed at an httptest server that 428s),TestGetInvalidSHA(→64 or 70, pick 64),TestNoKeyStatusBunnyFailsClosedNoStdin(getenv returns empty keys → exit 70, stderr mentions the env var name). - Step 2: Failing integration test (
integration_test.go), pure local→local in temp dirs: build manifests yml referencing 3 blobs; source depot has all 3; target depot has 1. Then:status --target local→ exit 1, missing=2 →push(local→local: for the test, allowpush --target localonly via an unexported hook? No — keep the spec surface: instead run the integration through bunny backend backed by httptest fake-Bunny with an in-memory blob map, which exercises the real push path) →status→ 0 →pushagain → 0 uploads (idempotent, assert fake recorded no second PUT). Alsopullinto a pre-populated dest: present-and-hash-ok skipped (fake records no GET for it), corrupt pre-existing file re-downloaded. - Step 3: FAIL. Step 4: Implement
run.go. Step 5:make vet testPASS + gofmt. - Step 6: Commit:
feat(depot): status/push/verify/get/pull commands with exit-code contract.
Task 6: Registry wiring + docs
Files:
- Modify:
internal/dispatch/dispatch.go(depot Registry entry) - Modify:
internal/dispatch/dispatch_test.go(if it asserts wired/unwired sets)
Flip the depot entry:
{
Name: "depot",
Bin: "crucible-depot",
Summary: "content-addressed asset depot (local/cdn/bunny)",
Commands: []Command{
{Name: "status", Summary: "report referenced-vs-present drift against a backend", Usage: "crucible depot status [--manifests DIR] [--source DIR] --target bunny|cdn|local"},
{Name: "push", Summary: "upload referenced-but-absent blobs from a local depot", Usage: "crucible depot push [--manifests DIR] --source DIR --target bunny"},
{Name: "verify", Summary: "existence sweep plus sampled download re-hash", Usage: "crucible depot verify [--manifests DIR] --target bunny|cdn [--sample N]"},
{Name: "get", Summary: "fetch one blob with sha re-verify", Usage: "crucible depot get <sha> <dest> --target cdn|bunny|local"},
{Name: "pull", Summary: "incremental verified pull of every referenced blob", Usage: "crucible depot pull [--manifests DIR] --dest DIR --target cdn|bunny"},
},
Wired: true,
},
The stale "(SeaweedFS)" wording must not survive anywhere: grep -ri seaweedfs across the repo and fix hits.
- Step 1: Update registry; run
go test ./internal/dispatch/— fix any test that pinned depot as unwired; add/extend a test assertingcrucible depotroutes todepot.Run(e.g.RunBuilder("depot", []string{"status"})with no manifests dir returns 70/64, not the unwired 70 message — assert on stderr text). - Step 2:
make check(vet+test+shellcheck+yamllint) PASS. Alsomake build && ./bin/crucible depotprints depot usage;./bin/crucibleinteractive menu shows depot entries (manual eyeball viaprintf 'q\n' | ./bin/crucibleor equivalent). - Step 3: Commit:
feat(depot): wire depot into dispatch registry and menu.
Task 7: sow-assets-manifest cutover
Repo: /home/vicky/Projects/westgate/repositories/migration/sow-assets-manifest, new branch crucible-depot-cutover from main.
Files:
- Delete:
scripts/mirror.sh - Modify:
scripts/build-haks.sh(the--fulldepot-verify block, ~L138-192),Makefile(mirrortarget,hakstarget),.gitea/workflows/release.yml,.gitea/workflows/publish-haks.yml,.gitea/workflows/build-haks.yml,scripts/lib.sh(header comments only),AGENTS.md/README.md(author-facing docs)
Consumes: the crucible depot CLI from Tasks 1–6. NOTE: CI resolves crucible via the flake input tracking sow-tools' default branch — the sow-tools branch must be merged before this repo's CI can pass. Local verification uses a locally built binary (sow-tools/bin/crucible on PATH). Do not run nix flake update here; note in the PR description that a nix flake update sow-tools (or lock bump) must land with/after the sow-tools merge.
Changes:
release.yml"Warm depot cache (verified full pull)" step body (keep name, job env unchanged; drop the ticker/compgen bash):
- name: Warm depot cache (verified full pull)
env:
BUNNY_STORAGE_READ_PASSWORD: ${{ secrets.BUNNY_STORAGE_READ_PASSWORD }}
BUNNY_STORAGE_HOST: storage.bunnycdn.com
BUNNY_STORAGE_ZONE: sow-assets-depot
run: |
nix develop --command bash -c '
set -euo pipefail
crucible depot status --manifests assets --target bunny
crucible depot pull --manifests assets --dest "$DEPOT_CACHE_DIR" --target cdn
'
(If the BUNNY_STORAGE_READ_PASSWORD secret does not exist in the repo, use secrets.BUNNY_STORAGE_PASSWORD and say so in the PR body. Check what other steps use.)
2. publish-haks.yml — same swap of the identical step.
3. build-haks.yml — the PR-check step keeps build-haks.sh --dry-run --no-verify --base "$base" (structural check) and adds crucible depot verify --manifests assets --target cdn --sample 3 as the cdn-backend verification.
4. scripts/build-haks.sh — delete the depot existence-sweep + sample-download block inside --dry-run (the depot_has_many/depot_get verify at ~L138-185); replace with a comment pointing at crucible depot verify. Remove now-unused locals. shellcheck scripts/build-haks.sh must pass.
5. Makefile — mirror: target now runs crucible depot pull --manifests assets --dest "$(DEPOT_DIR)" --target cdn. haks: gains a trailing non-fatal drift report: -crucible depot status --manifests assets --target cdn (leading - so drift never fails a local build) and an echo pointing at crucible depot push --source workspace/depot --target bunny as the standard post-edit step.
6. scripts/lib.sh — no function deletions (caller analysis: every depot function used by mirror.sh/build-haks.sh retains Increment 2–3 callers). Add a header comment block at the top of the depot section: # NOTE: this bash depot layer is being retired. Increment 1 moved mirror/verify to 'crucible depot' (sow-tools). import/sync go in Increment 2; export/pack/publish/artifact/cdn_purge in Increment 3. Do not add new callers. Do not touch the backend-keyed stat-cache in import.sh (spec forbids it).
7. Docs — in README.md (or AGENTS.md where depot workflow is documented): document crucible depot status --manifests assets --target cdn as the canonical credential-free "is my content live?" author check, and crucible depot push --source workspace/depot --target bunny as the standard post-edit publish step.
- Step 1: Branch; make all edits above.
- Step 2: Verify:
shellcheck scripts/*.sh;yamllint .gitea;grep -rn "mirror.sh" . --exclude-dir=.gitreturns nothing; with sow-tools'binon PATH runcrucible depot status --manifests assets --target cdnfor real (credential-free) and confirm it produces a sane report (exit 0 or 2 acceptable; exit 1 means real drift — report it, don't hide it). - Step 3: Commit:
feat: cut depot mirror/verify over to crucible depot, retire mirror.sh.
Self-Review Notes
- Spec coverage: command surface (T5), drift model + unconfirmed/exit 2 (T4/T5), backends + env + no-prompt (T1/T3), IPv4 range probe + HEAD ban test (T3), Checksum PUT (T3), pull incremental (T5), cutover of all three workflows + Makefile + lib.sh headers + author docs (T7), registry/SeaweedFS fix + menu (T6), testing section requirements all mapped (T1–T5), interim contract
make haksdrift report (T7.5). - Deliberate deviations from spec, justified:
status --target localroot viaDEPOT_DIR; push treats source-missing blobs as exit 1; pull download failure exits 70 (backend error) but source-404 exits 1 (drift). - Out of scope, per spec non-goals: import/sync, prune/gc/export, mdl checks, rich TUI, cdn_purge.