Repository navigation
Conversation
hoshimura
added this pull request to stack #26
October 7, 2026 09:22
hoshimura
force-pushed
the
streaming_backends
branch
3 times, most recently
from
October 7, 2026 09:41
6ad7ee8 to
231a7b8
Compare
- az_put_block (stage) + az_put_block_list (commit) primitives, SharedKey-signed with comp/blockid query canonicalization - AzurePath._upload_buffer streams the spool as 8 MiB blocks above an 8 MiB threshold (equal-length base64 block ids), bounding memory to one block - validated end-to-end against Azurite (single-PUT, multi-block 16 MB, 10 MB); unit tests mock the primitives
- gcs_start_resumable (init session, returns Location URI) + gcs_upload_chunk (PUT Content-Range ranges; 308 Resume-Incomplete accepted) primitives - GCSPath._upload_buffer streams the spool as 8 MiB (256 KiB-aligned) resumable chunks above an 8 MiB threshold, bounding memory to one chunk - validated end-to-end against fake-gcs (single-PUT, 16 MB resumable, 10 MB); unit tests mock the primitives
- Artifactory has no multipart/chunked-commit API; bound client memory by streaming a single PUT request body from a file (reqwest wrap_stream + explicit Content-Length framing), via new http::do_request_stream_file and art_put_stream - ArtifactoryPath._upload_buffer streams from a real source path when available (zero-copy for local->cloud copy), else spills the anonymous spool to a NamedTemporaryFile - adds tokio fs + tokio-util io + reqwest stream features - validated end-to-end against real Artifactory (small single-PUT, 12 MB open(wb) spool stream, 12 MB local->art copy stream)
- SSH: new ssh_write_chunk native primitive using OpenFlags APPEND (first chunk truncates, rest append) so chunked writes need no offset tracking; SSHPath._upload_buffer streams >8 MiB objects chunk-by-chunk - FTP: FTPPath._upload_buffer streams to the aioftp data connection in chunks (pure Python, no native change) - both previously buffered whole files via the default _upload_buffer; the win lands on large local->remote copies (open() stays unsupported) - validated end-to-end against atmoz/sftp (APPEND concatenation) and alpine-ftp (20 MB, sha-verified)
- SyncPath.copy's remote-file/tree paths now hand the source file handle to the dest's _upload_buffer (run via the sync runner) instead of read_bytes()+write_bytes(), so the sync adapter gets multipart/block/resumable/streaming for free, matching the async copy
hoshimura
force-pushed
the
streaming_backends
branch
from
October 7, 2026 10:04
231a7b8 to
dd9efbc
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends bounded-memory uploads to every remaining backend, each using what it natively offers. Builds on the _upload_buffer hook from PR 1.
Changes
Azure — block blobs: az_put_block (stage) + az_put_block_list (commit), equal-length base64 block ids, 8 MiB blocks.
GCS — resumable upload: gcs_start_resumable (session URI) + gcs_upload_chunk (Content-Range, 256 KiB-aligned, accepts 308 Resume Incomplete).
Artifactory — no multipart API, so the single PUT body is streamed from disk (reqwest wrap_stream + explicit Content-Length); streams directly from a local source path, else spills the spool to a temp file. Adds tokio fs + tokio-util + reqwest stream features.
SSH — ssh_write_chunk using SFTP OpenFlags::APPEND (first chunk truncates, the rest append — no offset tracking).
FTP — streams to the aioftp data connection in chunks (pure Python).
Sync copy — SyncPath.copy now routes through _upload_buffer too, so the sync API gets streaming for free (the async proxy already did).
Benefit lands on large open("wb") and local→remote copy() (SSH/FTP open() stays unsupported; direct write_bytes(data) is in-memory by definition).
Validation
Unit tests per backend (mocked primitives); validated end-to-end against Azurite, fake-gcs, real Artifactory, and atmoz/sftp + alpine-ftp emulators (20 MB round-trips, sha256-verified, incl. SSH APPEND concatenation and sync copies).