Skip to content

S3 AWS credential chain + cloud listing fast-path + build/docs groundwork #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hoshimura merged 7 commits into from
Oct 2, 2026
Merged

S3 AWS credential chain + cloud listing fast-path + build/docs groundwork #18

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
675d6ab
feat(s3): resolve credentials via the AWS SDK provider chain
hoshimura Sep 30, 2026
0b2fc66
build: strip symbols from the release binary
hoshimura Sep 30, 2026
e4e4c6d
docs: bold per-operation benchmark winners
hoshimura Sep 30, 2026
f4f7380
docs: add ssh/ftp(s) and unsupported paths to How It Works
hoshimura Sep 30, 2026
66f19ae
perf(cloud): fast-path child construction in iterdir/glob/walk
hoshimura Oct 1, 2026
a0d91f8
style: rustfmt s3 credential helper
hoshimura Oct 1, 2026
40a2d6a
Merge branch 'main' into pr1-s3-creds-and-cloud-fastpath
hoshimura Oct 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 37 additions & 27 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ locally before connecting to a cloud account.

- Python 3.10 through 3.14.
- Access credentials for each cloud service that your application uses.

S3 authentication accepts explicit keys or uses the AWS SDK credential provider
chain (including shared profiles and IAM roles) when keys are not supplied.
- A POSIX-compatible shell such as Bash or Zsh for the terminal commands below.
Windows users can use Windows Subsystem for Linux or adapt the commands for
PowerShell.
Expand Down Expand Up @@ -285,55 +288,55 @@ uv run --extra bench python scripts/bench.py --backend art --rounds 3 --concurre

| Operation | asanypath | s3fs | aiobotocore | cloudpathlib |
|-----------|-----------|------|-------------|--------------|
| write | 8.47 ms | 9.52 ms | 43.39 ms | 48.02 ms |
| read | 4.03 ms | 9.43 ms | 20.16 ms | 13.30 ms |
| exists | 3.58 ms | 4.54 ms | 20.49 ms | 4.48 ms |
| iterdir | 38.91 ms | 39.33 ms | 57.76 ms | 41.95 ms |
| write | **8.47 ms** | 9.52 ms | 43.39 ms | 48.02 ms |
| read | **4.03 ms** | 9.43 ms | 20.16 ms | 13.30 ms |
| exists | **3.58 ms** | 4.54 ms | 20.49 ms | 4.48 ms |
| iterdir | **38.91 ms** | 39.33 ms | 57.76 ms | 41.95 ms |

### Azure — Azurite emulator (median of 20 rounds, 10 objects, c=1)

| Operation | asanypath | azure-sdk | adlfs | cloudpathlib |
|-----------|-----------|-----------|-------|--------------|
| write | 1.77 ms | 1.86 ms | 1.93 ms | 8.29 ms |
| read | 0.68 ms | 1.15 ms | 2.50 ms | 3.46 ms |
| exists | 0.56 ms | 0.83 ms | 1.04 ms | 1.02 ms |
| iterdir | 1.34 ms | 2.21 ms | 4.37 ms | 2.59 ms |
| write | **1.77 ms** | 1.86 ms | 1.93 ms | 8.29 ms |
| read | **0.68 ms** | 1.15 ms | 2.50 ms | 3.46 ms |
| exists | **0.56 ms** | 0.83 ms | 1.04 ms | 1.02 ms |
| iterdir | **1.34 ms** | 2.21 ms | 4.37 ms | 2.59 ms |

### GCS — fake-gcs-server emulator (median of 20 rounds, 10 objects, c=1)

| Operation | asanypath | gcloud-aio | cloudpathlib | gcsfs |
|-----------|-----------|------------|--------------|-------|
| write | 0.22 ms | 0.24 ms | 2.88 ms | 0.33 ms |
| read | 0.12 ms | 0.14 ms | 1.34 ms | 0.19 ms |
| exists | 0.12 ms | 0.14 ms | 0.44 ms | 0.41 ms |
| iterdir | 0.20 ms | 0.16 ms | 0.55 ms | 0.24 ms |
| write | **0.22 ms** | 0.24 ms | 2.88 ms | 0.33 ms |
| read | **0.12 ms** | 0.14 ms | 1.34 ms | 0.19 ms |
| exists | **0.12 ms** | 0.14 ms | 0.44 ms | 0.41 ms |
| iterdir | 0.20 ms | **0.16 ms** | 0.55 ms | 0.24 ms |

### Artifactory — live server (median of 20 rounds, 10 objects, c=1)

| Operation | asanypath | aiohttp | requests |
|-----------|-----------|---------|----------|
| write | 39.77 ms | 38.00 ms | 36.67 ms |
| read | 4.29 ms | 4.75 ms | 5.43 ms |
| exists | 2.97 ms | 3.02 ms | 4.52 ms |
| iterdir | 4.59 ms | 4.18 ms | 6.26 ms |
| write | 39.77 ms | 38.00 ms | **36.67 ms** |
| read | **4.29 ms** | 4.75 ms | 5.43 ms |
| exists | **2.97 ms** | 3.02 ms | 4.52 ms |
| iterdir | 4.59 ms | **4.18 ms** | 6.26 ms |

### SSH / SFTP — localhost emulator (median of 20 rounds, 10 objects, c=1)

| Operation | asanypath | asyncssh |
|-----------|-----------|----------|
| write | 0.32 ms | 0.27 ms |
| read | 0.34 ms | 0.33 ms |
| exists | 0.11 ms | 0.11 ms |
| iterdir | 0.52 ms | 0.40 ms |
| write | 0.32 ms | **0.27 ms** |
| read | 0.34 ms | **0.33 ms** |
| exists | **0.11 ms** | **0.11 ms** |
| iterdir | 0.52 ms | **0.40 ms** |

### FTP — localhost emulator (median of 20 rounds, 10 objects, c=1)

| Operation | asanypath | aioftp |
|-----------|-----------|--------|
| write | 0.40 ms | 0.34 ms |
| read | 0.36 ms | 0.32 ms |
| exists | 1.12 ms | 0.92 ms |
| iterdir | 1.10 ms | 0.92 ms |
| write | 0.40 ms | **0.34 ms** |
| read | 0.36 ms | **0.32 ms** |
| exists | 1.12 ms | **0.92 ms** |
| iterdir | 1.10 ms | **0.92 ms** |

Latencies are per-operation medians over 20 rounds. S3 and Artifactory ran
against live services; Azure, GCS, SSH/SFTP, and FTP against local emulators
Expand All @@ -355,9 +358,16 @@ API across all backends.
| `az://`, `azure://` | `AzurePath` | reqwest + SharedKey |
| `art://` | `ArtifactoryPath` | reqwest + Bearer |
| `http://`, `https://` | `HTTPPath` / `HTTPSPath` | reqwest |

All network I/O is handled by `asanypath-native` (Rust/PyO3 + reqwest) with
jittered exponential backoff retry for transient errors.
| `ssh://` | `SSHPath` | `asyncssh` (SFTP) |
| `ftp://` | `FTPPath` | `aioftp` |
| `ftps://` | `FTPSPath` | `aioftp` + TLS |
| any other scheme | `UnsupportedProtocolPath` | none — pure-path only |

Cloud and HTTP network I/O is handled by `asanypath-native` (Rust/PyO3 + reqwest)
with jittered exponential backoff retry for transient errors; SSH/SFTP and
FTP/FTPS use `asyncssh` and `aioftp` respectively. Unknown schemes construct an
`UnsupportedProtocolPath`: pure-path operations (`.name`, `.parent`, joins) work,
while any backend operation raises `UnsupportedProtocolError`.

## API Reference

Expand Down
Loading
Loading