Bunny zone: how a blob is uploaded and how existence is checked without mounting #55

New Issue

2026-07-27T18:51:19Z

archvillainette commented

2026-07-27 18:51:19 +00:00

Question

How does emit put a blob into the Bunny zone, and how does it know a blob is already there — without mounting the zone and without a local mirror?

Part of #54. Blob paths are data/sha1/<h0h1>/<h2h3>/<sha1>, names are content hashes, so name equality is content equality and the check is a name diff rather than a byte comparison.

Find out, from primary sources and from what the repos already do:

The Bunny storage HTTP API for upload, existence check and listing: exact endpoints, auth header, what a missing object returns, whether a bulk or prefixed listing exists or whether existence must be probed one object at a time.
Listing cost at our scale — 71,436 objects. If the only way to know what is present is a full listing, how long it takes and how large the response is.
How sow-assets-manifest publishes to the depot today: the credential name, where it comes from in CI, and which client it uses. This is the path a new writer copies rather than invents.
Whether internal/depot's remote.go backend already speaks this API, and if so whether emit reuses it or needs something different.
Idempotency and concurrency: what happens on re-PUT of an existing object, and on two producers writing the same blob at once.

## Question How does `emit` put a blob into the Bunny zone, and how does it know a blob is already there — without mounting the zone and without a local mirror? Part of #54. Blob paths are `data/sha1/<h0h1>/<h2h3>/<sha1>`, names are content hashes, so name equality is content equality and the check is a name diff rather than a byte comparison. Find out, from primary sources and from what the repos already do: - The Bunny storage HTTP API for upload, existence check and listing: exact endpoints, auth header, what a missing object returns, whether a bulk or prefixed listing exists or whether existence must be probed one object at a time. - Listing cost at our scale — 71,436 objects. If the only way to know what is present is a full listing, how long it takes and how large the response is. - How `sow-assets-manifest` publishes to the depot today: the credential name, where it comes from in CI, and which client it uses. This is the path a new writer copies rather than invents. - Whether `internal/depot`'s `remote.go` backend already speaks this API, and if so whether `emit` reuses it or needs something different. - Idempotency and concurrency: what happens on re-PUT of an existing object, and on two producers writing the same blob at once.

archvillainette added the wayfinder:research label 2026-07-27 18:51:19 +00:00

archvillainette added a new dependency 2026-07-27 18:51:30 +00:00

#60 The emit upload path and NWCompressedBuffer round-trip

archvillainette self-assigned this 2026-07-27 18:53:41 +00:00

archvillainette commented

2026-07-27 18:59:40 +00:00

Resolved: probe per object, and extend `internal/depot`'s backend rather than writing a new client

The API

Flat REST filesystem: PUT|GET|DELETE https://{host}/{zone}/{path}, auth via an AccessKey: <storage-zone-password> header on every call — the zone password from the FTP & API Access tab, not the account API key (that one is CDN purge only). Host for us is storage.bunnycdn.com (release.yml:41).

Op	Request	Success	Missing
Upload	`PUT /{zone}/{path}` raw body	`201`	n/a
Download	`GET /{zone}/{path}`	`200` + body	`404` + JSON
Delete	`DELETE /{zone}/{path}`	`200`	not distinguished
List	`GET /{zone}/{path}/` trailing slash	`200` + JSON array	undocumented

Refs: storage API, put, get, list.

HEAD is banned, and not by preference — docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md:98-103,237 records the edge throttling concurrent HEADs with 428/000, producing both false 404s and false 200s. The probe is a 1-byte range GET (Range: bytes=0-0) over forced IPv4, expecting 206.

Listing is worse than probing, so probe

There is no bulk, recursive or prefix listing. Enumerating a depth-2 tree costs 1 + 256 + 65,536 = 65,793 requests, and Bunny only creates directories that hold objects — but with 71,436 objects over 65,536 buckets, ~1.09 per bucket, almost every bucket exists. That is ~92% of the cost of just probing all 71,436 objects, and it returns a whole-zone snapshot we would have to hold and trust as fresh.

So: probe per object, which is what Sweep() already does — ProbeJobs workers (default 16, config.go:26), then every non-Present result re-confirmed serially before being called missing.

Bunny's S3-compatible API would give ListObjectsV2 with prefixes and 1000-key pages (~72 calls), but it is still in phased closed preview as of March 2026 (roadmap post). Worth revisiting, not worth building on.

What sow-assets-manifest does today

curl, driven from scripts/lib.sh: URL builder :368, upload :465-475 (existence check, then PUT with AccessKey and Checksum), existence :411-425 (curl -r 0-0, accepting 200/206), delete :552-562 plus a scoped CDN purge. Newer steps already call the Go tool — crucible depot status --manifests assets --target bunny (release.yml:51, publish-haks.yml:53).

Credential names, values not read: BUNNY_STORAGE_PASSWORD (zone write), BUNNY_STORAGE_READ_PASSWORD (read, fed from the same secret), BUNNY_STORAGE_ZONE, and BUNNY_API_KEY (account key, purge only). BUNNY_STORAGE_HOST and BUNNY_CDN_BASE are plain env values.

One behaviour emit must not inherit: depot_require_write (lib.sh:390-411) prompts on a TTY for the write password when unset. The Go design doc calls this out as a bug not to repeat (:82-90), and NewBackend already does the right thing — env only, error rather than prompt (remote.go:35-40).

Reuse `httpBackend`, with a small extension

internal/depot already speaks this exact API and already gets the hard parts right: IPv4-only transport with per-host connection limits (remote.go:53-62, there because of a documented IPv6/HTTP-2 reset problem), range-GET probe with retry and a tri-state result where Unconfirmed is distinct from Absent (remote.go:98-141), and upload-that-skips-what-exists (remote.go:144-184) — which is precisely the flow emit needs, already written.

Three sha256 assumptions must become per-instance:

Key layout. BlobKey() (manifest.go:26-28) hardcodes sha256/<a>/<b>/<sha>; NWSync needs data/sha1/<h0h1>/<h2h3>/<sha1>. Same shape, different prefix — so a keyFn func(string) string field on httpBackend, not a second backend type.
Verify hash. Get() (remote.go:209-221) tees through sha256.New() and deletes on mismatch, which would reject every NWSync blob. Same fix, a newHash func() hash.Hash field.
The Checksum header, which is the real trap. remote.go:171 sends Checksum: strings.ToUpper(sha). Bunny documents that header as SHA-256 and validates the body against it — handing it a 40-char SHA-1 will make Bunny reject the upload. Either drop the header for NWSync blobs, or compute a SHA-256 of the same bytes in the same pass with io.MultiWriter and send that. The latter costs almost nothing and keeps server-side integrity checking on a 71k-object upload, which is worth having.

Roughly 20 lines across remote.go and backend.go, and emit inherits the retry logic, IPv4 pinning, tri-state probe, skip-if-present upload and Sweep(). A separate NWSync-only client would duplicate every one of those hard-won behaviours and drift from them.

The one genuinely new need: Put(ctx, sha, src string) takes a filesystem path, but NWSync blobs come from inside a hak and must never be staged to disk. So emit needs PutReader(ctx, key string, r io.Reader, size int64), with Put reimplemented on top of it. Folded into #60.

Idempotency and concurrency

Re-PUT is a plain overwrite returning 201; Bunny documents no conflict or precondition response, and both the bash and Go paths treat re-upload as safe. Two producers writing the same key is harmless because the key is the content hash — both bodies are byte-identical, so whichever wins is correct. A reader seeing a partially-written object mid-overwrite is unverified (Bunny documents nothing either way), but the Checksum header rejects a truncated upload and Get() re-hashes on download and deletes on mismatch (remote.go:217-221).

The practical risk is throttling, not correctness. The design doc's field notes (:53-56) record 428/000 under concurrent probing — which is exactly why ProbeJobs is 16 and why Unconfirmed exists. emit keeps the same ceiling and the same discipline: a throttled probe must never be read as "missing, re-upload" or as "present, skip".

## Resolved: probe per object, and extend `internal/depot`'s backend rather than writing a new client ### The API Flat REST filesystem: `PUT|GET|DELETE https://{host}/{zone}/{path}`, auth via an **`AccessKey: <storage-zone-password>`** header on every call — the zone password from the FTP & API Access tab, not the account API key (that one is CDN purge only). Host for us is `storage.bunnycdn.com` (`release.yml:41`). | Op | Request | Success | Missing | |---|---|---|---| | Upload | `PUT /{zone}/{path}` raw body | `201` | n/a | | Download | `GET /{zone}/{path}` | `200` + body | **`404`** + JSON | | Delete | `DELETE /{zone}/{path}` | `200` | not distinguished | | List | `GET /{zone}/{path}/` trailing slash | `200` + JSON array | undocumented | Refs: [storage API](https://docs.bunny.net/reference/storage-api), [put](https://docs.bunny.net/reference/put_-storagezonename-path-filename), [get](https://docs.bunny.net/reference/get_-storagezonename-path-filename), [list](https://docs.bunny.net/reference/get_-storagezonename-path-). **HEAD is banned**, and not by preference — `docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md:98-103,237` records the edge throttling concurrent HEADs with `428`/`000`, producing both false 404s and false 200s. The probe is a 1-byte range GET (`Range: bytes=0-0`) over forced IPv4, expecting `206`. ### Listing is worse than probing, so probe There is no bulk, recursive or prefix listing. Enumerating a depth-2 tree costs `1 + 256 + 65,536 = 65,793` requests, and Bunny only creates directories that hold objects — but with 71,436 objects over 65,536 buckets, ~1.09 per bucket, almost every bucket exists. That is ~92% of the cost of just probing all 71,436 objects, and it returns a whole-zone snapshot we would have to hold and trust as fresh. So: **probe per object**, which is what `Sweep()` already does — `ProbeJobs` workers (default 16, `config.go:26`), then every non-Present result re-confirmed serially before being called missing. Bunny's S3-compatible API would give `ListObjectsV2` with prefixes and 1000-key pages (~72 calls), but it is still in phased closed preview as of March 2026 ([roadmap post](https://bunny.net/blog/whats-happening-with-s3-compatibility/)). Worth revisiting, not worth building on. ### What sow-assets-manifest does today `curl`, driven from `scripts/lib.sh`: URL builder `:368`, upload `:465-475` (existence check, then PUT with `AccessKey` and `Checksum`), existence `:411-425` (`curl -r 0-0`, accepting 200/206), delete `:552-562` plus a scoped CDN purge. Newer steps already call the Go tool — `crucible depot status --manifests assets --target bunny` (`release.yml:51`, `publish-haks.yml:53`). Credential names, values not read: `BUNNY_STORAGE_PASSWORD` (zone write), `BUNNY_STORAGE_READ_PASSWORD` (read, fed from the same secret), `BUNNY_STORAGE_ZONE`, and `BUNNY_API_KEY` (account key, purge only). `BUNNY_STORAGE_HOST` and `BUNNY_CDN_BASE` are plain env values. One behaviour `emit` must **not** inherit: `depot_require_write` (`lib.sh:390-411`) prompts on a TTY for the write password when unset. The Go design doc calls this out as a bug not to repeat (`:82-90`), and `NewBackend` already does the right thing — env only, error rather than prompt (`remote.go:35-40`). ### Reuse `httpBackend`, with a small extension `internal/depot` already speaks this exact API and already gets the hard parts right: IPv4-only transport with per-host connection limits (`remote.go:53-62`, there because of a documented IPv6/HTTP-2 reset problem), range-GET probe with retry and a **tri-state** result where `Unconfirmed` is distinct from `Absent` (`remote.go:98-141`), and upload-that-skips-what-exists (`remote.go:144-184`) — which is precisely the flow `emit` needs, already written. Three sha256 assumptions must become per-instance: 1. **Key layout.** `BlobKey()` (`manifest.go:26-28`) hardcodes `sha256/<a>/<b>/<sha>`; NWSync needs `data/sha1/<h0h1>/<h2h3>/<sha1>`. Same shape, different prefix — so a `keyFn func(string) string` field on `httpBackend`, not a second backend type. 2. **Verify hash.** `Get()` (`remote.go:209-221`) tees through `sha256.New()` and deletes on mismatch, which would reject every NWSync blob. Same fix, a `newHash func() hash.Hash` field. 3. **The `Checksum` header, which is the real trap.** `remote.go:171` sends `Checksum: strings.ToUpper(sha)`. Bunny documents that header as **SHA-256** and validates the body against it — handing it a 40-char SHA-1 will make Bunny reject the upload. Either drop the header for NWSync blobs, or compute a SHA-256 of the same bytes in the same pass with `io.MultiWriter` and send that. The latter costs almost nothing and keeps server-side integrity checking on a 71k-object upload, which is worth having. Roughly 20 lines across `remote.go` and `backend.go`, and `emit` inherits the retry logic, IPv4 pinning, tri-state probe, skip-if-present upload and `Sweep()`. A separate NWSync-only client would duplicate every one of those hard-won behaviours and drift from them. The one genuinely new need: `Put(ctx, sha, src string)` takes a filesystem path, but NWSync blobs come from *inside* a hak and must never be staged to disk. So `emit` needs `PutReader(ctx, key string, r io.Reader, size int64)`, with `Put` reimplemented on top of it. Folded into #60. ### Idempotency and concurrency Re-PUT is a plain overwrite returning `201`; Bunny documents no conflict or precondition response, and both the bash and Go paths treat re-upload as safe. Two producers writing the same key is harmless because the key *is* the content hash — both bodies are byte-identical, so whichever wins is correct. A reader seeing a partially-written object mid-overwrite is **unverified** (Bunny documents nothing either way), but the `Checksum` header rejects a truncated upload and `Get()` re-hashes on download and deletes on mismatch (`remote.go:217-221`). The practical risk is throttling, not correctness. The design doc's field notes (`:53-56`) record `428`/`000` under concurrent probing — which is exactly why `ProbeJobs` is 16 and why `Unconfirmed` exists. `emit` keeps the same ceiling and the same discipline: a throttled probe must never be read as "missing, re-upload" **or** as "present, skip".

archvillainette closed this issue

2026-07-27 18:59:40 +00:00

archvillainette referenced this issue

2026-07-27 19:00:31 +00:00

The emit upload path and NWCompressedBuffer round-trip #60

archvillainette referenced this issue

2026-07-27 19:01:14 +00:00

Map: NWSync publishing — emit at artifact birth, assemble at module release #54

archvillainette referenced this issue

2026-07-28 07:34:50 +00:00

Where the nwsync verbs live in the command surface #56

archvillainette referenced this issue