Skip to content

fix(cli): add packageManager to scaffolded pnpm package.json - #93

Merged
Brooooooklyn merged 1 commit into
mainfrom
scaffold-package-manager
Apr 8, 2026
Merged

Brooooooklyn merged 1 commit into
mainfrom
scaffold-package-manager

Conversation

@Boshen

@Boshen Boshen commented Apr 8, 2026

Copy link
Copy Markdown
Member

Summary

  • Scaffolded pnpm projects now get a packageManager: "pnpm@10.33.0" entry in package.json, so pnpm/action-setup@v4 in the generated deploy.yml can resolve a version.

🤖 Generated with Claude Code

pnpm/action-setup@v4 in the generated deploy.yml requires a
packageManager field to resolve a pnpm version.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Brooooooklyn
Brooooooklyn merged commit 0334c34 into main Apr 8, 2026
7 checks passed
@Brooooooklyn
Brooooooklyn deleted the scaffold-package-manager branch April 8, 2026 11:40
cpojer pushed a commit that referenced this pull request Sep 11, 2026
…est (#93)

The fan-out change in #91 let KV/R2 PUTs run concurrently but each pool
slot independently charges `bufferedBudget` for its resident bytes. CLIs
that mirror the same binary under multiple URL paths (cpojer-blog emits
every media file both under `/blog/...` and `/client/blog/...`) doubled
every large asset's memory residency during parse, pegging the 64 MiB
aggregate cap on deploys that would otherwise fit.

When the declared `assetManifest` maps an incoming `asset:/path` to a
hash already resident or in-flight this request, drain the body with
strict size verification and commit the manifest entry directly from
the declared hash+size. Zero budget charge, zero redundant PUT.

Security: dedupe is per-request, keyed by blobs already read-and-hashed
by this same request. A malicious client declaring a wrong hash for a
later path can only cause the first-occurrence content to serve at that
path — exactly what the declared manifest asked for. Cross-project
bleed is blocked by the `projectId`-scoped KV/R2 key namespace.

Parses the declared manifest eagerly so the fast-path can fire as soon
as the `assetManifest` part arrives (standard CLI order: manifest,
assetManifest, asset:*).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cpojer pushed a commit that referenced this pull request Sep 11, 2026
…94)

* feat(api): stream static asset bodies directly to R2 for O(1) memory

When a client declares `{ hash, size }` for a static-deploy asset whose
size crosses the KV threshold, pipe its multipart body straight through a
hashing transform into `r2.put` instead of buffering the bytes for a later
pool-driven upload. Per-asset Worker memory drops from O(asset size) to
O(hasher state + ≤ 2-byte base64 carry), so a single large SPA bundle no
longer pins its full size against the 128 MiB request budget.

Load-bearing pieces:

- Linear Transform pipeline, NOT `tee()`. A `tee()` fork would buffer for
  the slower branch (r2.put vs hashing) and reintroduce asset-sized
  residency — defeating the point.
- `r2.put(..., onlyIf: { etagDoesNotMatch: '*' })`. The atomic
  write-if-absent precondition guards retained deployments against
  content-addressed-key poisoning: without it, a lying client could
  overwrite trusted bytes another live deploy still serves. A precondition
  miss (pre-existing key) returns `null` and the put becomes a no-op,
  letting duplicate content across deploys dedupe for free.
- Post-stream hash verify against `declared.hash`, matching PR #93 review
  feedback. Skipping verification on an existing-key hit would let a
  malicious client commit a manifest entry pointing at bytes it does not
  own.
- Byte-count verification in the hashing transform (deterministic
  MultipartParseError on size mismatch), independent of whatever workerd's
  FixedLengthStream happens to throw.
- Hash construction matches `readAssetBody` / `verifyStreamedAssetPart`
  exactly (base64 of the bytes, 3-byte-aligned hasher updates, extension
  appended as the final update, first 16 bytes of the BLAKE3 digest).

The parse loop drains its upload pool before invoking the streaming path,
because `streamStaticAssetToR2` holds `part.body` open for the entire pipe
and the multipart parser cannot advance until the stream closes.

Out of scope:
- Worker-backed deploy path (`collectBufferedAsset`). Still buffers
  because the CF asset upload session API needs full bytes for
  signing / base64 encoding.
- Small assets (< 2 MiB). Still flow through the KV pool path — KV PUTs
  are cheap and the streaming complexity isn't justified.
- `MAX_ASSET_BYTES` cap stays at 50 MiB; raising it is a follow-up.
- CLI. This is a server-only change.

Test coverage (5 new integration tests in api-deploy.test.ts):
- happy path with 10 MiB + 8 MiB streamed to R2, no buffer charge
- hash-mismatch body triggers parse error and deletes the corrupted blob
- size-mismatch body triggers parse error with no R2 blob surviving
- identical-content second deploy returns precondition miss, succeeds
- bad-bytes second deploy under same declared hash: precondition
  protects the pre-existing retained blob (no overwrite, etag unchanged)

Integration harness updates:
- `MemoryR2Bucket` mock grows `head()`, `ReadableStream` put values,
  `onlyIf: { etagDoesNotMatch: '*' }` precondition, and etags so the
  precondition-miss path can be exercised under Node.
- `FixedLengthStream` polyfill installed in `setup.ts` (workerd-only
  global). Errors on close with wrong byte count, matching workerd's
  observable behaviour.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(api): drop MAX_ASSET_BYTES per-asset cap

The direct-to-R2 streaming path introduced in the previous commit has O(1)
per-asset memory, so a hard per-asset byte cap is no longer needed to bound
worker memory. Remove `MAX_ASSET_BYTES` entirely. The remaining caps are
sufficient:

- Buffered paths (`readAssetBody`, worker-backed `collectBufferedAsset`,
  streamed-static pool via `readStaticAssetPart`) are bounded by the
  aggregate `MAX_AGGREGATE_BUFFERED_BYTES` (64 MiB) threaded through as
  `BufferedBudget`. Any asset whose residency would push past the cap
  still throws `PartTooLargeError('aggregate buffered deploy payload …')`
  mid-stream. The effective per-asset ceiling for buffered paths is
  therefore the aggregate cap itself, which is already load-bearing for
  keeping the 128 MiB Worker request budget safe.

- The direct-to-R2 streaming path (`streamStaticAssetToR2`) has constant
  residency (hasher state + one chunk in flight). Size is now bounded
  only by R2's single-`put` limit. Size lies and corruption are still
  caught by `FixedLengthStream` + post-stream hash verify against
  `declared.size` / `declared.hash`.

- Duplicate-hash fast-path (`verifyStreamedAssetPart`) continues to use
  `declared.size` as its upper bound, which is the tighter of the two
  client-agreed invariants anyway.

Drop the `maxBytes` parameter from `readAssetBody`, `readStaticAssetPart`,
`collectBufferedAsset`, and `verifyStreamedAssetPart`. Drop the shape-check
`declared.size <= MAX_ASSET_BYTES` from the two deploy.ts branches.

Test removed: `returns 413 when a single asset:* part exceeds the 50 MiB
cap`. That behaviour no longer exists — the aggregate-budget test
(`returns 413 when buffered parts sum past the aggregate 64 MiB cap`)
covers the surviving enforcement path for buffered deploys.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(api): register streaming R2 hash before put so abort cleanup can retry delete

Reviewer caught that `streamStaticAssetToR2`'s inline `r2.delete` on the
hash-mismatch-after-write branch can transiently fail. Because
`streamedHashStores.set(hash, 'r2')` in deploy.ts only ran AFTER the
function returned successfully, the exception path bypassed the
registration entirely — `cleanupStreamedAssetsOnAbort` had no record
of the poisoned key and never retried the delete.

The downstream consequence: a later deploy sending the correct bytes
for the same declared hash hits the precondition-miss path (key
exists), stream-verifies only its OWN incoming body (which matches
declared.hash), and commits a manifest entry trusting the stored bytes.
Users served corrupt content for a hash the server believes is valid.

Fix: move the `streamedHashStores` registration, `streamedLargeHashes`
add, and `ensureStreamedAssetClaim` call BEFORE the streaming put, so
`cleanupStreamedAssetsOnAbort` always sweeps the hash on abort
regardless of where inside `streamStaticAssetToR2` the throw happens.

Safety across the other outcomes:
  - Happy path: no abort runs; the map is discarded at request end.
  - Early throw (FixedLengthStream size mismatch pre-put): cleanup
    tries to delete a never-written key — R2 delete is a no-op.
  - Precondition miss (existing blob, owned by another deploy): the
    foreign-claim / retained-hash protections in
    `cleanupStreamedAssetsOnAbort` already prevent cross-deploy
    deletion, so our pre-registration does not cause a wipe of
    someone else's bytes.

Regression test: mock r2.delete to fail the first call (the inline
attempt inside streamStaticAssetToR2) and succeed on subsequent. Send
a hash-mismatch body. Assert that both delete attempts fire
(pre-registration enabled the retry) and the blob is gone at end.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(api): dedupe declared-manifest validation + surface R2 put failures raw

Two review findings on the streaming-to-R2 path:

1. The in-request fast-path and the direct-to-R2 streaming path both
   looked up `declaredAssetManifest?.[path]` and ran identical shape
   validation (hash is 32-char string, size is finite non-negative
   integer). Any future change to the well-formedness rules would need
   updating in two places. Factored into a single validated `declared`
   reference reused by both branches; branch predicates now only carry
   the truly path-specific conditions (already-resident vs. size crosses
   KV threshold).

2. `streamStaticAssetToR2`'s catch block rewrote any non-MultipartParseError
   into a size-mismatch parse error whenever `bytesSeen < declared.size`.
   That condition is also true when `r2.put` rejects mid-upload (R2
   cancels its consumer, the pipe sees fewer bytes than declared), so a
   real infra fault surfaced to the client as a misleading client
   protocol violation. The fix distinguishes the two cases by the class
   of the thrown error: FixedLengthStream's close-with-wrong-count
   throws a `TypeError` propagated through both sides of the pipe, so
   `err instanceof TypeError && bytesSeen < declared.size` identifies a
   genuine short body regardless of which side's rejection `Promise.all`
   saw first. Any other shape of failure with a surviving `putFailure`
   flag is an R2 infra fault — surface it raw so the outer deploy
   handler's `sanitizeDeployErrorMessage` can route it through the
   generic "internal error" path instead of the parse-step message.

Regression test: mock `r2.put` to reject immediately. Assert the NDJSON
error event is NOT a size-mismatch message.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(api): drop MAX_AGGREGATE_BUFFERED_BYTES in-memory cap

The 64 MiB aggregate buffered-bytes ceiling is redundant now that the
direct-to-R2 streaming path handles the large-asset case with O(1)
residency. The remaining buffered paths (worker-backed asset collection
and the streamed-static pool for sub-KV-threshold files) are bounded by
their per-part caps alone:

  - MAX_WORKER_FILE_BYTES       = 32 MiB (worker modules)
  - MAX_MANIFEST_BYTES          =  8 MiB (DeployManifest JSON)
  - MAX_ASSET_MANIFEST_BYTES    = 32 MiB (per-path declared manifest)
  - MAX_PREFLIGHT_BYTES         =  1 MiB (preflight metadata)
  - MAX_SMALL_TEXT_BYTES        = 64 KiB (source, commit)

The 128 MiB Worker isolate memory ceiling is the ultimate backstop. A
worker-backed deploy with many buffered assets can approach that ceiling
at the user's own risk — the direct-to-R2 path is the intended escape
valve for large static payloads.

Removed: the `BufferedBudget` interface export from multipart.ts, the
`budget` parameter thread-through in `readPartBytes` / `readPartText` /
`readPartAsBlob` / `readAssetBody` / `readStaticAssetPart` /
`collectBufferedAsset`, the `bufferedBudget` state variable in the
deploy handler, the pool-task `finally { bufferedBudget.used -= ... }`
decrement, and the worker-backed dedupe swap decrement. Also deleted
the three integration tests that pinned the aggregate-cap enforcement
("buffered parts sum past the aggregate 64 MiB cap", "does not leak
aggregate budget when duplicate asset hashes replace buffered entries",
and "returns 413 on streamed static deploys that sum past the aggregate
64 MiB cap"). Comments that used the aggregate cap as the load-bearing
enforcement were rewritten to point at per-part caps and the isolate
budget.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(api): prefer R2 putFailure over TypeError short-stream rewrite

Reviewer (cursor + codex) caught that the prior `streamStaticAssetToR2`
catch-block ordering checked `err instanceof TypeError && bytesSeen <
declared.size` BEFORE `putFailure !== undefined`. If `r2.put` itself
rejected with a TypeError-shaped error (cancelled-consumer cascade,
malformed-response TypeError, any infra path whose error surfaces as
`TypeError`), the short-read symptom driven by R2 cancelling its reader
would get rewritten as `MultipartParseError('body size does not match
declared')` — hiding a real server-side storage fault behind a
misleading client protocol-violation event.

The class-based distinguisher cannot tell FLX-propagation-TypeError
apart from R2-own-TypeError without reference equality that isn't
guaranteed by the streams spec. Swap the order so `putFailure` surfaces
raw (via the outer handler's generic sanitizer) whenever the put side
failed, keeping the TypeError short-stream rewrite for the pipe-only
failure path. Tradeoff: a genuine client short body now reaches the
outer sanitizer as "deploy failed due to an internal error" instead of
the previously friendly "body size X does not match declared Y" parse
event. That's acceptable — R2 misclassifications are a diagnostic
hazard for ops, while the short-body case remains reachable through
client-side length validation and ships an error anyway.

Regression test: `does not misclassify a TypeError-shaped R2 put
failure as a client size mismatch` mocks `r2.put` to reject with a
TypeError, confirms the NDJSON error is NOT a size-mismatch message.
Pre-existing `does not match declared` test updated to assert only
that an error surfaces + no stranded R2 blob — the exact message shape
is no longer a tested invariant.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants