Repository navigation
Conversation
hoshimura
added this pull request to stack #26
October 7, 2026 09:22
hoshimura
force-pushed
the
streaming_tuning
branch
from
October 7, 2026 09:31
66dac29 to
7625094
Compare
hoshimura
force-pushed
the
streaming_tuning
branch
2 times, most recently
from
October 7, 2026 09:41
1ca851b to
007a4c4
Compare
- cast self._path to yarl.URL in ssh.py/ftp.py for host/port/user/path access (it's a URL for remote backends, not pathlib.Path), matching s3.py - add asanypath.ssh/ftp/sync to the per-module mypy overrides (the only backends missing from the list) and extend local/cloud/s3-group/asanypath overrides to cover the remaining pre-existing/foreign codes - mypy now gates clean for README/benchmark work
- shared ASANYPATH_UPLOAD_CHUNK_SIZE / ASANYPATH_SPOOL_MAX_SIZE defaults in options.py replace the scattered per-backend constants (part size = peak RAM per in-flight chunk; parts are sequential, no multiplier) - per-call override: copy(chunk_size=) (previously ignored for cloud) and write-mode open(buffering=) thread through _upload_buffer to every backend - backend minimums applied: S3 clamps to >=5 MiB parts, GCS floors to a 256 KiB multiple - tests for env parsing, the clamp, the alignment, and per-call sizing
- new 'Large uploads stream automatically' how-to: default behaviour, per-backend mechanisms, ASANYPATH_UPLOAD_CHUNK_SIZE/SPOOL_MAX_SIZE env, copy(chunk_size)/open(buffering), and S3/GCS part-size minimums - note copy/open stream while write_bytes/write_text are in-memory by definition
- hatch_build.py already requires msgspec>=0.22.0; the committed lock lagged at 0.21.1
hoshimura
force-pushed
the
streaming_tuning
branch
from
October 7, 2026 10:04
007a4c4 to
18b0d47
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Exposes a memory-footprint knob, cleans up the type-checker, and documents the feature.
Changes
Tunable part size — the part size is the max RAM per in-flight chunk (parts upload sequentially, no concurrency multiplier). Global via ASANYPATH_UPLOAD_CHUNK_SIZE / ASANYPATH_SPOOL_MAX_SIZE; per-call via copy(chunk_size=...) and write-mode open(buffering=...), falling back to the shared default. Backend minimums applied automatically (S3 ≥ 5 MiB parts, GCS 256 KiB alignment). Replaces the scattered per-backend constants.
mypy clean (0 errors, was 75) — cast self._path to yarl.URL in ssh.py/ftp.py (it's a URL for remote backends, not pathlib.Path); add asanypath.ssh/ftp/sync to the per-module overrides (the only backends missing from the list) and extend the existing overrides for the remaining pre-existing codes.
README — "Large uploads stream automatically" how-to: default behaviour, per-backend mechanisms, the env + per-call knobs, and the S3/GCS minimums.
chore — sync uv.lock to msgspec>=0.22.0 (already required by hatch_build.py; the committed lock had lagged).
Validation
Full suite 1145 passed / 3 skipped, mypy 0 errors, ruff clean. Benchmark across all emulator backends shows no regression on the hot paths (streaming never engages below the threshold).