Skip to content

Commit fa0265f

Browse files
berndverstBernd VerstCopilot
authored
Add v2 blob payload tokens and explicit read scopes (#310)
* Align direct blob downloads with payload token grammar Require the complete v1 payload token format for direct downloads, matching automatic resolution. Add offline destination checks and document token and storage-location compatibility. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Preserve changelog line endings Keep the existing changelog formatting while adding the payload token compatibility note. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add v2 blob payload tokens with v1 read compatibility Write self-describing blob URLs by default and read both token formats from the configured storage endpoint and container. Account for URL token length in worker action-size estimates and document reader upgrades and storage migration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add explicit Azure Blob payload read scopes Support independent primary-account container access and listed account endpoints while retaining configured-container reads by default. Preserve named v1/v2 references and legacy v2 URL suffix handling. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Return immutable snapshots of additional payload account URIs Keep getter results independent from the stored endpoint list and verify returned snapshots remain immutable after subsequent option updates. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Bernd Verst <beverst@microsoft.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 parent dcd243b commit fa0265f

17 files changed

Lines changed: 1349 additions & 71 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
11
## Unreleased
22

3+
* Write self-describing v2 blob payload tokens by default while retaining v1 and v2 reads from the configured storage endpoint and container. Require complete token formats for direct downloads and account for URL token length in worker action-size estimates. Upgrade readers before deploying v2 writers; changing storage locations requires explicit migration or a store configured for the original location.
4+
* Add explicit primary-account-wide and additional-account read scopes for Azure Blob payloads while retaining configured-container access by default. Additional accounts require identity authentication and exact HTTPS account service endpoints. Preserve named and nested v1/v2 references and ignore legacy v2 URL query and fragment suffixes.
5+
36
## v1.10.0
47
* Enable custom exception providers for Durable Functions Java activities ([#294](https://github.com/microsoft/durabletask-java/pull/294))
58
* Recover verified ambiguous payload uploads on HTTP 409 `BlobAlreadyExists` as well as HTTP 412, while rejecting unrelated conflicts.

‎azure-blob-payloads/README.md‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,111 @@ LargePayloadStorageOptions payloadOptions = new LargePayloadStorageOptions()
5252

5353
The storage identity needs permission to create the container and read and write blobs. Payload blobs are retained after orchestration processing, so configure an Azure Storage lifecycle policy appropriate for the application's retention requirements.
5454

55+
## Payload references and storage location
56+
57+
The built-in store now writes self-describing `blob:v2:<absolute-blob-URL>` tokens, matching the .NET token
58+
format. It reads both v2 tokens and legacy `blob:v1:<container>:<blob-name>` tokens, including named and nested
59+
blobs. New blob names remain 32 lowercase hexadecimal characters. New v2 URLs retain the blob's location but
60+
never include user information, query parameters, fragments, or credentials. Upgrade all readers before
61+
deploying writers that emit v2 tokens.
62+
63+
Both formats default to the configured storage endpoint **and container**. A reference to another container in
64+
the same account or to another account is rejected, even when the configured identity has Storage permissions
65+
there. Recognizing a self-describing URL does not authorize its destination. Direct downloads and automatic
66+
resolution use the same token parsing and destination policy; blob-name shape does not authorize access.
67+
Token-formatted strings are treated as references even when their blob names are not generated UUIDs.
68+
69+
### Explicit read scopes
70+
71+
Configure read scopes on `LargePayloadStorageOptions` before constructing the store:
72+
73+
| Configuration | Permitted reads |
74+
|---|---|
75+
| Default: `allowAllContainersInConfiguredAccount = false`, no additional account URIs | Configured account endpoint and configured container only |
76+
| `setAllowAllContainersInConfiguredAccount(true)` | All containers at the configured account endpoint |
77+
| `setAdditionalAccountUris(...)` | All containers at each explicitly listed additional account endpoint |
78+
79+
These settings are independent. Listing additional accounts does not widen primary-account container access.
80+
To allow every container across the primary and additional accounts, enable the boolean and list those accounts.
81+
Listing the primary endpoint is rejected during construction; use `setAllowAllContainersInConfiguredAccount(true)`
82+
to widen its container scope. Equivalent additional endpoints are deduplicated. Uploads and upload
83+
verification always use the configured container, regardless of read scope.
84+
85+
For primary-account-wide reads using the existing authentication mode:
86+
87+
```java
88+
payloadOptions.setAllowAllContainersInConfiguredAccount(true);
89+
```
90+
91+
For additional-account reads using an explicitly configured Storage identity:
92+
93+
```java
94+
LargePayloadStorageOptions payloadOptions = new LargePayloadStorageOptions()
95+
.setAccountUri(URI.create("https://primary.blob.core.windows.net"))
96+
.setCredential(new DefaultAzureCredentialBuilder().build())
97+
.setAdditionalAccountUris(Arrays.asList(
98+
URI.create("https://archive.blob.core.windows.net")));
99+
```
100+
101+
An account URI is an exact service endpoint, not a hostname allowlist, container URL, or path-prefix grant.
102+
Authorization compares scheme, hostname (case-insensitive), effective port (including default ports), and the
103+
complete account path. Container-scoped access additionally compares the container. Account-wide access relaxes
104+
only that container check: it does not permit different ports, schemes, path-style accounts, or subpaths.
105+
For Azurite, an account endpoint includes the account path, for example `http://127.0.0.1:10000/devstoreaccount1`.
106+
107+
Endpoint validation and the final SDK account/container/name check run before HTTP requests or credential
108+
acquisition. Additional entries must be genuine Blob service endpoints, not container/blob URLs or arbitrary
109+
URL prefixes. Endpoint query parameters, fragments, user information, traversal segments, encoded separators,
110+
double-encoded account paths, duplicate account-path slashes, and control characters are rejected.
111+
Escaped account paths must use the same spelling as the configured URI; alternate escaped spellings are not
112+
treated as aliases. Endpoints whose Azure Blob SDK parsing changes the account boundary are rejected during
113+
construction. One trailing slash on an account endpoint is accepted. Policy, account endpoints, configured
114+
container, and selected authentication are captured when a store is constructed; construct a new store to
115+
change its grants.
116+
117+
V2 reads discard legacy query and fragment suffixes before selecting an endpoint or constructing a client.
118+
These suffixes never supply SAS credentials, snapshots, or version selection. User information remains invalid.
119+
Account selection uses only configured service endpoints, then decodes the container and remaining blob name
120+
once, preserving escaped and nested names. Encoded separators cannot change the selected account or container.
121+
V1 names are literal names passed through the SDK: percent sequences are not pre-decoded. Traversal segments
122+
and control characters are rejected. Denied destinations and SDK destination mismatches propagate as errors.
123+
124+
### Authentication and Storage permissions
125+
126+
Default and primary-account-wide reads support connection strings using account keys or SAS credentials, and
127+
identity authentication using `AccountUri` plus `TokenCredential`. Same-account routing reuses the configured
128+
authentication pipeline, including its SAS restrictions. Storage can still reject another container when a
129+
container SAS is used; opting in does not expand the SAS's permissions.
130+
131+
Additional-account reads require `AccountUri` **and** `TokenCredential`, and additional endpoints must use HTTPS.
132+
Account keys and SAS credentials are not reused across accounts. Unsupported combinations fail during store
133+
construction. As with existing configuration, complete identity configuration takes precedence if a connection
134+
string is also supplied. Every permitted destination uses that same selected authentication pipeline and
135+
identity; there is no credential discovery, mode switching, or fallback after Storage rejects a request.
136+
The identity must independently have read permission on the relevant account and containers. Storage redirects
137+
are not followed by the configured SDK pipeline, and retries do not select a primary-account secondary host.
138+
139+
Keep the configured container dedicated to payloads shared by the same trusted clients and workers; a known
140+
blob name within it can be resolved using the store's configured credentials. Custom `PayloadStore`
141+
implementations are responsible for their own destination checks and can override `getEstimatedTokenSizeBytes()`
142+
when their upload tokens exceed the default size estimate. The built-in store supplies an estimate based on
143+
its configured URL length so worker action-size validation accounts for v2 tokens.
144+
145+
Changing the configured storage location requires explicit migration of the payload blobs while preserving their
146+
names and updating v2 references to the new URLs, or a store configured for the original location. V1 tokens
147+
do not record the original account, so changing the account does not locate or migrate existing payloads
148+
automatically, and v1 references are never searched for in additional accounts. V2 tokens retain the original
149+
location: either migrate blobs and update their references, keep a store configured for the original location,
150+
or explicitly authorize the original account/container through the scopes above. These scopes do not migrate
151+
payloads, rewrite URLs, or locate missing blobs.
152+
153+
This module does not implement dashboard reads or delete payload blobs during instance purge. Purge operations
154+
are forwarded to the scheduler; payload retention is managed separately through Storage lifecycle policies.
155+
Client and worker interceptors share this store for inputs, activity and orchestration results, events,
156+
streamed history, custom status, failure traces, and entity state. The history-export module obtains resolved
157+
history through that client path; its separate archive writer is not a payload-reference reader. There is no
158+
service implementation or delegated dashboard credential flow in this repository to modify.
159+
55160
## Upload failures
56161

57162
Transient upload failures (including HTTP 408, 429, and 5xx responses) propagate without sending a worker
@@ -77,6 +182,10 @@ in-memory HTTP transport that commits a PUT and loses its response, exercising S
77182
and off. In-process gRPC tests cover `$root` tokens through client, worker, and streamed history paths and resolved
78183
nested failure traces in decoded JSON and JSONL exports. An orchestration replay regression also verifies that
79184
entity callers receive the original nested failure diagnostics after work-item payload resolution.
185+
Access-scope tests use the real Azure SDK with an offline HTTP transport and instrumented credentials. They
186+
cover the restrictive default, both opt-in scopes, exact endpoint/path boundaries, v1/v2 resolution across
187+
client and worker surfaces, selected authentication modes, and failures before HTTP or token acquisition.
188+
A loopback-only HTTP test exercises the default SDK transport and verifies that a Storage redirect is not followed.
80189

81190
## Integration tests
82191

0 commit comments

Comments
 (0)