crucible depot: Increment 1 core (status/push/verify/get/pull + local/cdn/bunny backends) (#30)
Implements Increment 1 of `docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md`: a new stdlib-only `internal/depot` package wired into the dispatcher. - `crucible depot status|push|verify|get|pull` with backends `local`/`cdn`/`bunny`; exit contract `0` clean / `1` drift / `2` unconfirmed-only / `64` usage / `70` internal. - Presence is always probed against the real target (IPv4 1-byte range GET; HEAD is banned with a regression-tripwire test). `unconfirmed` is a distinct state, never collapsed into `missing`. - No prompting anywhere: missing `BUNNY_STORAGE_*` env fails closed (read path included), enforced by a no-stdin test. - Uploads: probe-then-PUT with `Checksum: <UPPER-sha>`; read/write key split (`BUNNY_STORAGE_READ_PASSWORD` falls back to `BUNNY_STORAGE_PASSWORD`). - Field-driven fix included: per-probe transient retry (curl `--retry 2` equivalent) — without it a real 1490-blob CDN sweep reported 1222 false-unconfirmed; with it, 1490/1490 present in 74s, exit 0. - Registry: depot `Wired: true`, joins the interactive menu; stale "(SeaweedFS)" wording removed. Tests: unit + httptest fake-Bunny (probe sequence, Checksum header, key split, 428 throttling → exit 2) + local→bunny integration (drift → push → clean → idempotent no-second-PUT; incremental pull). `make check` green. **Merge ordering:** this merges FIRST; the companion `sow-assets-manifest#crucible-depot-cutover` PR needs its flake input bumped to include this. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: #30 Reviewed-by: xtul <mpiasecki720@protonmail.com> Co-authored-by: vickydotbat <vickydotbat@tutamail.com> Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
This commit was merged in pull request #30.
This commit is contained in:
@@ -0,0 +1,330 @@
|
||||
# Crucible Depot Core (Increment 1) Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Implement `crucible depot {status,push,verify,get,pull}` with local/cdn/bunny backends in sow-tools, then cut sow-assets-manifest over to it and delete the bash it replaces.
|
||||
|
||||
**Architecture:** New stdlib-only `internal/depot` package with its own `Run(args) int` entrypoint (exit contract 0/1/2/70 is richer than `internal/app.Run`'s 0/1, so dispatch special-cases depot instead of routing through `app.Run`). Backends implement a small interface; a sweep engine does parallel range-GET probes with serial re-confirm. Cutover repo work happens in sow-assets-manifest on its own branch.
|
||||
|
||||
**Tech Stack:** Go 1.26, stdlib `net/http`/`crypto/sha256`/`sync`, `gopkg.in/yaml.v3` (already a dep — add nothing to go.mod, vendorHash must not change).
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md` — read it before any task; it is authoritative.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- **No new Go dependencies.** stdlib + existing `gopkg.in/yaml.v3` only.
|
||||
- **HEAD is banned** for existence checks. All probes: IPv4, `GET` with `Range: bytes=0-0`, expect `206` (accept any 2xx). A test must fail if HEAD is introduced.
|
||||
- **Never prompt, never read stdin** for credentials — read path included. Missing key → clear error on stderr, exit `70`.
|
||||
- **No cache of "what I uploaded"** — presence is always probed against the real target backend.
|
||||
- `unconfirmed` is a distinct state from `missing`; exit `2` when unconfirmed-only.
|
||||
- Exit codes: `0` clean, `1` drift (referenced-but-absent), `2` unconfirmed-only, `64` usage, `70` internal/backend error.
|
||||
- Blob key: `sha256/<sha[0:2]>/<sha[2:4]>/<sha>`; sha must match `^[0-9a-f]{64}$`.
|
||||
- Env names (unchanged from bash): `DEPOT_CDN_BASE`/`BUNNY_CDN_BASE` (cdn read base, default `https://cdn-a7f3k9.westgate.pw`), `BUNNY_STORAGE_HOST` (**required for bunny, no default** — spec overrides bash), `BUNNY_STORAGE_ZONE` (default `sow-assets-depot`), `BUNNY_STORAGE_READ_PASSWORD` (falls back to `BUNNY_STORAGE_PASSWORD`) for reads/probes, `BUNNY_STORAGE_PASSWORD` for PUT, `DEPOT_PROBE_JOBS` (default 16), `DEPOT_JOBS` (default 16), `DEPOT_CONNECT_TIMEOUT` (default 10s), `DEPOT_PROBE_MAX_TIME` (default 30s), `DEPOT_CONFIRM_RETRIES` (default 3), `DEPOT_CONFIRM_MAX` (default 200, 0 = unlimited).
|
||||
- Upload: `PUT` with `AccessKey: <write key>` and `Checksum: <UPPERCASE sha>` headers; each sha uploaded at most once per run; probe-before-put.
|
||||
- Every downloaded blob is re-hashed; deleted on mismatch.
|
||||
- `cdn` backend writes fail closed (error, never fake success).
|
||||
- Verify with `make vet test` in sow-tools (run inside `nix develop` if go missing from PATH).
|
||||
- Commit after each task. **Never commit to main.** sow-tools branch: `depot-core-increment-1`. sow-assets-manifest branch: `crucible-depot-cutover`. sow-tools has an unrelated modified spec file on main — leave it out of commits unless the task says otherwise.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Package skeleton — config + manifest parsing
|
||||
|
||||
**Files:**
|
||||
- Create: `internal/depot/config.go`, `internal/depot/config_test.go`
|
||||
- Create: `internal/depot/manifest.go`, `internal/depot/manifest_test.go`
|
||||
|
||||
**Interfaces (Produces):**
|
||||
|
||||
```go
|
||||
package depot
|
||||
|
||||
// config.go
|
||||
type Config struct {
|
||||
CDNBase string // DEPOT_CDN_BASE || BUNNY_CDN_BASE || default
|
||||
StorageHost string // BUNNY_STORAGE_HOST, no default
|
||||
StorageZone string // BUNNY_STORAGE_ZONE, default "sow-assets-depot"
|
||||
ReadKey string // BUNNY_STORAGE_READ_PASSWORD || BUNNY_STORAGE_PASSWORD
|
||||
WriteKey string // BUNNY_STORAGE_PASSWORD
|
||||
ProbeJobs int // DEPOT_PROBE_JOBS, default 16
|
||||
Jobs int // DEPOT_JOBS, default 16
|
||||
ConnectTimeout time.Duration // DEPOT_CONNECT_TIMEOUT seconds, default 10s
|
||||
ProbeMaxTime time.Duration // DEPOT_PROBE_MAX_TIME seconds, default 30s
|
||||
ConfirmRetries int // DEPOT_CONFIRM_RETRIES, default 3
|
||||
ConfirmMax int // DEPOT_CONFIRM_MAX, default 200 (0 = unlimited)
|
||||
}
|
||||
func LoadConfig(getenv func(string) string) Config
|
||||
|
||||
// manifest.go
|
||||
// ReferencedSHAs parses every *.yml in dir (schema: assets[].{path,sha256,size,restype,hak})
|
||||
// and returns the deduplicated sha set plus per-sha size (for pull).
|
||||
func ReferencedSHAs(dir string) (map[string]int64, error)
|
||||
func ValidSHA(s string) bool // ^[0-9a-f]{64}$
|
||||
func BlobKey(sha string) string // "sha256/ab/cd/<sha>"
|
||||
```
|
||||
|
||||
`LoadConfig` takes `getenv` so tests inject env without `t.Setenv` races. Invalid ints/durations in env → fall back to default (do not error). `ReferencedSHAs` errors on: unreadable dir, no `*.yml` files, YAML parse failure, or an entry whose `sha256` fails `ValidSHA` (report file + path). Manifest struct:
|
||||
|
||||
```go
|
||||
type manifestFile struct {
|
||||
Assets []struct {
|
||||
Path string `yaml:"path"`
|
||||
SHA256 string `yaml:"sha256"`
|
||||
Size int64 `yaml:"size"`
|
||||
} `yaml:"assets"`
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 1:** `git -C sow-tools switch -c depot-core-increment-1` (from main).
|
||||
- [ ] **Step 2:** Write failing tests: `TestLoadConfigDefaults`, `TestLoadConfigReadKeyFallback` (READ unset → WriteKey value; READ set → its own), `TestLoadConfigBadIntFallsBack`, `TestBlobKey` (`BlobKey("0b1d…")=="sha256/0b/1d/0b1d…"`), `TestValidSHA` (rejects uppercase, short, non-hex), `TestReferencedSHAs` (temp dir with two yml files sharing one sha → dedup; bad sha → error naming the file; empty dir → error).
|
||||
- [ ] **Step 3:** `go test ./internal/depot/` — verify FAIL (compile error is fine).
|
||||
- [ ] **Step 4:** Implement `config.go` + `manifest.go` minimally.
|
||||
- [ ] **Step 5:** `go test ./internal/depot/` PASS; `gofmt -l -w internal/depot`.
|
||||
- [ ] **Step 6:** Commit: `feat(depot): config resolution and manifest sha parsing`.
|
||||
|
||||
### Task 2: Local backend
|
||||
|
||||
**Files:**
|
||||
- Create: `internal/depot/backend.go`, `internal/depot/local.go`, `internal/depot/local_test.go`
|
||||
|
||||
**Interfaces (Produces):**
|
||||
|
||||
```go
|
||||
// backend.go
|
||||
type ProbeState int
|
||||
const (
|
||||
Present ProbeState = iota
|
||||
Absent
|
||||
Unconfirmed
|
||||
)
|
||||
type Backend interface {
|
||||
Name() string
|
||||
// Probe returns the existence state of one sha. transient=true means a
|
||||
// retry might change the answer (feeds the serial confirm loop).
|
||||
Probe(ctx context.Context, sha string) (state ProbeState, transient bool, err error)
|
||||
// Get fetches sha into dest (temp file + rename), re-hashes, deletes on mismatch.
|
||||
Get(ctx context.Context, sha, dest string) error
|
||||
// Put uploads bytes from src for sha. Read-only backends return an error.
|
||||
Put(ctx context.Context, sha, src string) error
|
||||
}
|
||||
|
||||
// local.go
|
||||
type LocalBackend struct{ Root string } // implements Backend; Name()=="local"
|
||||
```
|
||||
|
||||
Local semantics: `Probe` = `os.Stat(Root/BlobKey(sha))`, never transient. `Put` = copy to temp file in dest dir then `os.Rename` (MkdirAll parents). `Get` = copy out + re-hash (hash the source bytes while copying via `io.TeeReader` into `sha256.New()`), delete dest and error on mismatch. `err` from Probe only for real I/O errors (permission), not absence.
|
||||
|
||||
- [ ] **Step 1:** Failing tests: `TestLocalRoundTrip` (Put→Probe Present→Get→bytes equal), `TestLocalProbeAbsent`, `TestLocalGetHashMismatch` (hand-write a corrupt blob file at the keyed path, Get returns error and dest does not exist), `TestLocalPutIdempotent`.
|
||||
- [ ] **Step 2:** Verify FAIL. **Step 3:** Implement. **Step 4:** PASS + gofmt.
|
||||
- [ ] **Step 5:** Commit: `feat(depot): backend interface and local backend`.
|
||||
|
||||
### Task 3: Remote backends (cdn, bunny) with fake-Bunny tests
|
||||
|
||||
**Files:**
|
||||
- Create: `internal/depot/remote.go`, `internal/depot/remote_test.go`
|
||||
|
||||
**Interfaces (Consumes):** `Backend`, `ProbeState`, `Config`, `BlobKey`.
|
||||
**Produces:**
|
||||
|
||||
```go
|
||||
// NewBackend returns the backend for name ("local"|"cdn"|"bunny").
|
||||
// localRoot is used only for "local". Fails closed: "bunny" with empty
|
||||
// StorageHost or empty ReadKey returns an error (never prompts);
|
||||
// unknown name returns an error.
|
||||
func NewBackend(name, localRoot string, cfg Config) (Backend, error)
|
||||
|
||||
type httpBackend struct { ... } // shared by cdn and bunny
|
||||
```
|
||||
|
||||
Implementation requirements:
|
||||
|
||||
```go
|
||||
// IPv4-only transport — REQUIRED, spec bans IPv6/HEAD probe hazards.
|
||||
transport := &http.Transport{
|
||||
DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
|
||||
d := net.Dialer{Timeout: cfg.ConnectTimeout}
|
||||
return d.DialContext(ctx, "tcp4", addr)
|
||||
},
|
||||
MaxIdleConnsPerHost: cfg.ProbeJobs,
|
||||
}
|
||||
client := &http.Client{Transport: transport, Timeout: cfg.ProbeMaxTime}
|
||||
```
|
||||
|
||||
- **Probe** (both remotes): `GET <base>/<BlobKey(sha)>` with header `Range: bytes=0-0`; drain/close body. 2xx → `Present`; 404/410 → `Absent`; anything else (incl. transport error) → `Unconfirmed, transient=true`. **Never http.Head / MethodHead.**
|
||||
- cdn: base = `cfg.CDNBase`, no auth headers, `Put` returns `errors.New("cdn backend is read-only")`.
|
||||
- bunny: probe/read against storage `https://<StorageHost>/<StorageZone>/<BlobKey(sha)>` with `AccessKey: <ReadKey>`. `Get`: try CDN base first, on non-2xx fall back to storage URL with ReadKey (mirrors bash `_depot_bunny_get`); re-hash, delete on mismatch. `Put`: probe first, skip if Present; else `PUT` storage URL, headers `AccessKey: <WriteKey>`, `Checksum: <strings.ToUpper(sha)>`, body streamed from src file; non-2xx → error. `Put` with empty WriteKey → error before any I/O.
|
||||
- Both `Get`s write via temp file + rename and re-hash exactly like local.
|
||||
|
||||
httptest note: `httptest.NewServer` listens on `127.0.0.1`, which the tcp4 dialer reaches — no test shim needed.
|
||||
|
||||
- [ ] **Step 1:** Failing tests against an `httptest` fake Bunny (a handler recording method+path+headers, serving a blob map):
|
||||
- `TestProbeUsesRangeGetNotHead` — probe a present sha; assert recorded request has `Method=="GET"` and `Range=="bytes=0-0"`; **also** assert the depot source contains no HEAD probe: the handler returns 500 for any HEAD request, and the test asserts zero HEAD requests were received across the whole test run (this is the anti-HEAD regression tripwire from field note 3).
|
||||
- `TestProbeStates` — 200/206→Present, 404→Absent, 428→Unconfirmed+transient, 503→Unconfirmed+transient.
|
||||
- `TestBunnyPutSequence` — Put of an absent sha: recorded sequence is [GET range probe, PUT]; PUT has `Checksum` = uppercase sha and `AccessKey` = write key; Put of a present sha: probe only, no PUT.
|
||||
- `TestReadWriteKeySplit` — probe uses ReadKey, PUT uses WriteKey (distinct values asserted).
|
||||
- `TestBunnyNoReadKeyFailsClosed` — `NewBackend("bunny",…)` with empty read key returns error **without reading stdin**: replace `os.Stdin` with the read end of an `os.Pipe()` for the test; after the call, write+close and confirm the pipe is undrained (or simply assert error occurs with no stdin interaction because the code path has no stdin reference — grep-based assertion acceptable: test fails if `internal/depot` sources mention `os.Stdin` or `/dev/tty`).
|
||||
- `TestBunnyNoStorageHostFailsClosed`.
|
||||
- `TestGetHashMismatchDeletes` — server returns wrong bytes; Get errors, dest removed.
|
||||
- `TestCDNPutReadOnly`.
|
||||
- [ ] **Step 2:** Verify FAIL. **Step 3:** Implement `remote.go`. **Step 4:** PASS + gofmt + `go vet ./...`.
|
||||
- [ ] **Step 5:** Commit: `feat(depot): cdn and bunny HTTP backends with fake-Bunny tests`.
|
||||
|
||||
### Task 4: Sweep engine — parallel probe + serial confirm
|
||||
|
||||
**Files:**
|
||||
- Create: `internal/depot/sweep.go`, `internal/depot/sweep_test.go`
|
||||
|
||||
**Interfaces (Consumes):** `Backend`, `Config`, `ProbeState`.
|
||||
**Produces:**
|
||||
|
||||
```go
|
||||
type SweepResult struct {
|
||||
Present []string
|
||||
Missing []string // confirmed absent (404/410 after retries)
|
||||
Unconfirmed []string // probe budget exhausted, state unknown
|
||||
}
|
||||
// Sweep probes every sha against b: one parallel pass (cfg.ProbeJobs workers,
|
||||
// sync.WaitGroup + buffered-channel work queue), then any non-Present result
|
||||
// is re-probed serially with linear backoff (attempt t sleeps t seconds,
|
||||
// cfg.ConfirmRetries attempts) — unless the non-Present count exceeds
|
||||
// cfg.ConfirmMax (>0), in which case the remainder is reported Unconfirmed
|
||||
// without serial re-probe (never collapsed into Missing).
|
||||
// A transient result that stays non-2xx after retries is Unconfirmed;
|
||||
// a 404/410 is Missing. Backend errors (non-transient) abort with error.
|
||||
func Sweep(ctx context.Context, b Backend, shas []string, cfg Config, progress io.Writer) (SweepResult, error)
|
||||
```
|
||||
|
||||
Sort `shas` before sweeping (deterministic output). Progress: print `probed X/N` roughly every 1000 blobs to `progress` (may be `io.Discard`). For tests, make the backoff sleeper injectable: package-level `var confirmSleep = time.Sleep`, overridden in tests.
|
||||
|
||||
- [ ] **Step 1:** Failing tests using a stub Backend (function fields):
|
||||
- `TestSweepAllPresent`, `TestSweepMissing` (404 → Missing, no retries beyond first serial confirm).
|
||||
- `TestSweepThrottledUnconfirmed` — stub always returns Unconfirmed/transient → after retries lands in `Unconfirmed`, never `Missing`; assert `confirmSleep` called with 1s then 2s.
|
||||
- `TestSweepTransientRecovers` — first probe 428, serial re-probe 206 → Present.
|
||||
- `TestSweepConfirmMaxCap` — 5 non-present, ConfirmMax=2 → all 5 Unconfirmed, zero serial re-probes.
|
||||
- `TestSweepParallelBounded` — 100 shas, ProbeJobs=4; stub counts concurrent in-flight probes (atomic), assert max ≤ 4.
|
||||
- [ ] **Step 2:** FAIL. **Step 3:** Implement. **Step 4:** PASS + gofmt.
|
||||
- [ ] **Step 5:** Commit: `feat(depot): parallel probe sweep with serial confirm and unconfirmed state`.
|
||||
|
||||
### Task 5: Commands + Run entrypoint + integration test
|
||||
|
||||
**Files:**
|
||||
- Create: `internal/depot/run.go`, `internal/depot/run_test.go`, `internal/depot/integration_test.go`
|
||||
|
||||
**Interfaces (Consumes):** everything above.
|
||||
**Produces:**
|
||||
|
||||
```go
|
||||
// Run executes a depot subcommand. args[0] is the subcommand
|
||||
// (status|push|verify|get|pull); returns the process exit code.
|
||||
// Exit: 0 clean, 1 drift, 2 unconfirmed-only, 64 usage, 70 internal.
|
||||
func Run(args []string, stdout, stderr io.Writer, getenv func(string) string) int
|
||||
```
|
||||
|
||||
Flag parsing: one `flag.FlagSet` per subcommand (stdlib; `internal/depot` is a fresh package, the hand-rolled loop in `internal/app` is not a constraint here). Flags per spec:
|
||||
|
||||
```
|
||||
status [--manifests DIR] [--source LOCAL_DEPOT] --target bunny|cdn|local
|
||||
push [--manifests DIR] --source LOCAL_DEPOT --target bunny
|
||||
verify [--manifests DIR] --target bunny|cdn [--sample N]
|
||||
get <sha> <dest> --target cdn|bunny|local
|
||||
pull [--manifests DIR] --dest DIR --target cdn|bunny
|
||||
```
|
||||
|
||||
`--manifests` default `assets`. `--target local` uses `--source` as root for status, and `--target`-side root comes from `DEPOT_DIR` env when target is local (matches bash `DEPOT_DIR`); document in usage string. Missing required flag / unknown subcommand / no args → usage on stderr, exit 64.
|
||||
|
||||
Command semantics (all build shas via `ReferencedSHAs`, backend via `NewBackend`, sweep via `Sweep`):
|
||||
|
||||
- **status**: sweep; print `referenced=<n> present=<n> missing=<n> unconfirmed=<n>` to stdout, then each missing sha on its own line prefixed `missing ` and each unconfirmed prefixed `unconfirmed `. Exit 1 if missing>0; else 2 if unconfirmed>0; else 0.
|
||||
- **push**: target must be `bunny` (enforce). Sweep; treat missing∪unconfirmed as to-upload (unconfirmed→re-upload is safe/idempotent per spec). For each, read bytes from `--source` local root; a sha absent from source → report and count as failed. Upload with `cfg.Jobs` parallel workers, each sha at most once. Exit 70 on upload error, 1 if any sha was missing from source, else 0. Print `uploaded=<n> failed=<n>`.
|
||||
- **verify**: sweep (exit 1/2 as status) **plus**, when clean, download `--sample` N (default 3, 0=skip) random blobs (`math/rand` seeded — pick via deterministic-enough `rand.New(rand.NewSource(time.Now().UnixNano()))`) to temp files via `Get` (which re-hashes); any failure → exit 1.
|
||||
- **get**: `Get(sha, dest)`; validate sha; exit 0/70.
|
||||
- **pull**: for every referenced sha, dest path `--dest/BlobKey(sha)`. Incremental: if dest file exists, re-hash it; matching → skip, mismatch → re-download. Download absent/mismatched with `cfg.Jobs` workers via `Get`. Print `pulled=<n> present=<n>`; any failed download → exit 70 after finishing the batch (report each failure). Non-zero missing at the source (404) → exit 1 listing them.
|
||||
|
||||
Wire the binary: **Modify** `internal/dispatch/dispatch.go` — in `runBuilder`, before the `Wired` check/`delegateLegacy` routing, special-case:
|
||||
|
||||
```go
|
||||
if b.Name == "depot" && b.Wired {
|
||||
return depot.Run(args[1:], os.Stdout, errw, os.Getenv)
|
||||
}
|
||||
```
|
||||
|
||||
(Exact insertion point: the implementer must read `runBuilder` and place this after subcommand resolution is bypassed — depot does its own subcommand parsing, so pass the raw args through, i.e. `args[1:]` where `args[0]=="depot"` at the dispatcher level, or the full arg slice in `RunBuilder("depot", …)` context. Verify both `crucible depot status …` and `crucible-depot status …` paths hit `depot.Run`.)
|
||||
|
||||
- [ ] **Step 1:** Failing unit tests (`run_test.go`): `TestRunUsage` (no args→64, unknown cmd→64, push --target cdn→64), `TestStatusExitCodes` (drift→1, unconfirmed-only→2 using a bunny backend pointed at an httptest server that 428s), `TestGetInvalidSHA` (→64 or 70, pick 64), `TestNoKeyStatusBunnyFailsClosedNoStdin` (getenv returns empty keys → exit 70, stderr mentions the env var name).
|
||||
- [ ] **Step 2:** Failing integration test (`integration_test.go`), pure local→local in temp dirs: build manifests yml referencing 3 blobs; source depot has all 3; target depot has 1. Then: `status --target local` → exit 1, missing=2 → `push` (local→local: for the test, allow `push --target local` **only** via an unexported hook? No — keep the spec surface: instead run the integration through bunny backend backed by httptest fake-Bunny with an in-memory blob map, which exercises the real push path) → `status` → 0 → `push` again → 0 uploads (idempotent, assert fake recorded no second PUT). Also `pull` into a pre-populated dest: present-and-hash-ok skipped (fake records no GET for it), corrupt pre-existing file re-downloaded.
|
||||
- [ ] **Step 3:** FAIL. **Step 4:** Implement `run.go`. **Step 5:** `make vet test` PASS + gofmt.
|
||||
- [ ] **Step 6:** Commit: `feat(depot): status/push/verify/get/pull commands with exit-code contract`.
|
||||
|
||||
### Task 6: Registry wiring + docs
|
||||
|
||||
**Files:**
|
||||
- Modify: `internal/dispatch/dispatch.go` (depot Registry entry)
|
||||
- Modify: `internal/dispatch/dispatch_test.go` (if it asserts wired/unwired sets)
|
||||
|
||||
Flip the depot entry:
|
||||
|
||||
```go
|
||||
{
|
||||
Name: "depot",
|
||||
Bin: "crucible-depot",
|
||||
Summary: "content-addressed asset depot (local/cdn/bunny)",
|
||||
Commands: []Command{
|
||||
{Name: "status", Summary: "report referenced-vs-present drift against a backend", Usage: "crucible depot status [--manifests DIR] [--source DIR] --target bunny|cdn|local"},
|
||||
{Name: "push", Summary: "upload referenced-but-absent blobs from a local depot", Usage: "crucible depot push [--manifests DIR] --source DIR --target bunny"},
|
||||
{Name: "verify", Summary: "existence sweep plus sampled download re-hash", Usage: "crucible depot verify [--manifests DIR] --target bunny|cdn [--sample N]"},
|
||||
{Name: "get", Summary: "fetch one blob with sha re-verify", Usage: "crucible depot get <sha> <dest> --target cdn|bunny|local"},
|
||||
{Name: "pull", Summary: "incremental verified pull of every referenced blob", Usage: "crucible depot pull [--manifests DIR] --dest DIR --target cdn|bunny"},
|
||||
},
|
||||
Wired: true,
|
||||
},
|
||||
```
|
||||
|
||||
The stale "(SeaweedFS)" wording must not survive anywhere: `grep -ri seaweedfs` across the repo and fix hits.
|
||||
|
||||
- [ ] **Step 1:** Update registry; run `go test ./internal/dispatch/` — fix any test that pinned depot as unwired; add/extend a test asserting `crucible depot` routes to `depot.Run` (e.g. `RunBuilder("depot", []string{"status"})` with no manifests dir returns 70/64, not the unwired 70 message — assert on stderr text).
|
||||
- [ ] **Step 2:** `make check` (vet+test+shellcheck+yamllint) PASS. Also `make build && ./bin/crucible depot` prints depot usage; `./bin/crucible` interactive menu shows depot entries (manual eyeball via `printf 'q\n' | ./bin/crucible` or equivalent).
|
||||
- [ ] **Step 3:** Commit: `feat(depot): wire depot into dispatch registry and menu`.
|
||||
|
||||
### Task 7: sow-assets-manifest cutover
|
||||
|
||||
**Repo:** `/home/vicky/Projects/westgate/repositories/migration/sow-assets-manifest`, new branch `crucible-depot-cutover` from main.
|
||||
|
||||
**Files:**
|
||||
- Delete: `scripts/mirror.sh`
|
||||
- Modify: `scripts/build-haks.sh` (the `--full` depot-verify block, ~L138-192), `Makefile` (`mirror` target, `haks` target), `.gitea/workflows/release.yml`, `.gitea/workflows/publish-haks.yml`, `.gitea/workflows/build-haks.yml`, `scripts/lib.sh` (header comments only), `AGENTS.md`/`README.md` (author-facing docs)
|
||||
|
||||
**Consumes:** the `crucible depot` CLI from Tasks 1–6. NOTE: CI resolves crucible via the flake input tracking sow-tools' default branch — the sow-tools branch must be merged before this repo's CI can pass. Local verification uses a locally built binary (`sow-tools/bin/crucible` on PATH). Do not run `nix flake update` here; note in the PR description that a `nix flake update sow-tools` (or lock bump) must land with/after the sow-tools merge.
|
||||
|
||||
Changes:
|
||||
|
||||
1. **`release.yml`** "Warm depot cache (verified full pull)" step body (keep name, job env unchanged; drop the ticker/compgen bash):
|
||||
```yaml
|
||||
- name: Warm depot cache (verified full pull)
|
||||
env:
|
||||
BUNNY_STORAGE_READ_PASSWORD: ${{ secrets.BUNNY_STORAGE_READ_PASSWORD }}
|
||||
BUNNY_STORAGE_HOST: storage.bunnycdn.com
|
||||
BUNNY_STORAGE_ZONE: sow-assets-depot
|
||||
run: |
|
||||
nix develop --command bash -c '
|
||||
set -euo pipefail
|
||||
crucible depot status --manifests assets --target bunny
|
||||
crucible depot pull --manifests assets --dest "$DEPOT_CACHE_DIR" --target cdn
|
||||
'
|
||||
```
|
||||
(If the `BUNNY_STORAGE_READ_PASSWORD` secret does not exist in the repo, use `secrets.BUNNY_STORAGE_PASSWORD` and say so in the PR body. Check what other steps use.)
|
||||
2. **`publish-haks.yml`** — same swap of the identical step.
|
||||
3. **`build-haks.yml`** — the PR-check step keeps `build-haks.sh --dry-run --no-verify --base "$base"` (structural check) and **adds** `crucible depot verify --manifests assets --target cdn --sample 3` as the cdn-backend verification.
|
||||
4. **`scripts/build-haks.sh`** — delete the depot existence-sweep + sample-download block inside `--dry-run` (the `depot_has_many`/`depot_get` verify at ~L138-185); replace with a comment pointing at `crucible depot verify`. Remove now-unused locals. `shellcheck scripts/build-haks.sh` must pass.
|
||||
5. **`Makefile`** — `mirror:` target now runs `crucible depot pull --manifests assets --dest "$(DEPOT_DIR)" --target cdn`. `haks:` gains a trailing non-fatal drift report: `-crucible depot status --manifests assets --target cdn` (leading `-` so drift never fails a local build) and an echo pointing at `crucible depot push --source workspace/depot --target bunny` as the standard post-edit step.
|
||||
6. **`scripts/lib.sh`** — no function deletions (caller analysis: every depot function used by mirror.sh/build-haks.sh retains Increment 2–3 callers). Add a header comment block at the top of the depot section: `# NOTE: this bash depot layer is being retired. Increment 1 moved mirror/verify to 'crucible depot' (sow-tools). import/sync go in Increment 2; export/pack/publish/artifact/cdn_purge in Increment 3. Do not add new callers.` **Do not touch the backend-keyed stat-cache in import.sh** (spec forbids it).
|
||||
7. **Docs** — in README.md (or AGENTS.md where depot workflow is documented): document `crucible depot status --manifests assets --target cdn` as the canonical credential-free "is my content live?" author check, and `crucible depot push --source workspace/depot --target bunny` as the standard post-edit publish step.
|
||||
|
||||
- [ ] **Step 1:** Branch; make all edits above.
|
||||
- [ ] **Step 2:** Verify: `shellcheck scripts/*.sh`; `yamllint .gitea`; `grep -rn "mirror.sh" . --exclude-dir=.git` returns nothing; with sow-tools' `bin` on PATH run `crucible depot status --manifests assets --target cdn` for real (credential-free) and confirm it produces a sane report (exit 0 or 2 acceptable; exit 1 means real drift — report it, don't hide it).
|
||||
- [ ] **Step 3:** Commit: `feat: cut depot mirror/verify over to crucible depot, retire mirror.sh`.
|
||||
|
||||
## Self-Review Notes
|
||||
|
||||
- Spec coverage: command surface (T5), drift model + unconfirmed/exit 2 (T4/T5), backends + env + no-prompt (T1/T3), IPv4 range probe + HEAD ban test (T3), Checksum PUT (T3), pull incremental (T5), cutover of all three workflows + Makefile + lib.sh headers + author docs (T7), registry/SeaweedFS fix + menu (T6), testing section requirements all mapped (T1–T5), interim contract `make haks` drift report (T7.5).
|
||||
- Deliberate deviations from spec, justified: `status --target local` root via `DEPOT_DIR`; push treats source-missing blobs as exit 1; pull download failure exits 70 (backend error) but source-404 exits 1 (drift).
|
||||
- Out of scope, per spec non-goals: import/sync, prune/gc/export, mdl checks, rich TUI, cdn_purge.
|
||||
Reference in New Issue
Block a user