Skip to content

Commit b3eef29

Browse files
ThorFuchsclaude
andauthored
Add RFC 7464 JSON text sequences to JSONStreaming (#98)
* Add RFC 7464 JSON text sequences to JSONStreaming JSONTextSequence reads application/json-seq texts framed by the record separator with the NDJSON pull model and budgets. Unterminated top-level scalars count as possibly truncated (RFC 7464 section 2.4). Malformed texts fail by default or are skipped and counted on request; resource failures are never skipped. JSONTextSequenceWriter writes RS, strict JSON and LF per value, as one sequence or in awaited bounded chunks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Use plain code for the cross-module JSONWritingOptions reference Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 97bfd7d commit b3eef29

11 files changed

Lines changed: 436 additions & 7 deletions

File tree

‎AGENTS.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,8 @@ store/follow API on 2026-11-10; later API changes stay additive and human-gated.
5757
one invocation-owned ledger per operation. Positions remain unforgeable and
5858
source-instance-local. No persistence or mutable store is exposed. See
5959
Documentation/Architecture/StructuralIntelligence/BoundedWorkflows1.8.md.
60-
Wider retrieval/persistence and JSON text-sequence framing remain separate work.
60+
Wider retrieval/persistence remains separate work. RFC 7464 JSON text sequences
61+
are 1.10 development in JSONStreaming (`JSONTextSequence`, `JSONTextSequenceWriter`).
6162

6263
JSONTransform uses one private compiler/VM and the CLI's accounted adapter.
6364
Preserve its accepted 42 authored declarations and exact semantics. The original

‎CHANGELOG.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,13 @@
2828
complete the page. Its CSP authorizes the two stylesheets by hash and sets
2929
`script-src 'none'`. The comparison is a package walker in JSON shared with
3030
`JSONPatch.difference`, so JSONHTML still depends on JSON only.
31+
- Add RFC 7464 JSON text sequences (`application/json-seq`) to JSONStreaming.
32+
`JSONTextSequence` reads texts framed by the record separator byte with the
33+
same pull model and budgets as NDJSON. Unterminated top-level scalars are
34+
treated as possibly truncated (RFC 7464 section 2.4). Malformed texts fail by
35+
default; `JSONTextSequenceOptions.MalformedTexts.skip` skips them and counts
36+
them on the iterator, but never skips resource failures. `JSONTextSequenceWriter`
37+
writes RS, strict JSON and LF per value, as one sequence or in awaited chunks.
3138

3239
## Unreleased — 1.9.1 hygiene
3340

‎Documentation/Qualification/TestDiscovery.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
"JSONTests": 198,
55
"JSONSchemaTests": 79,
66
"JSONRegexRuntimeTests": 16,
7-
"JSONStreamingTests": 36,
7+
"JSONStreamingTests": 42,
88
"JSONPatchTests": 22,
99
"JSONIntegrationTests": 45,
1010
"JSONHTMLTests": 103,
@@ -76,7 +76,7 @@
7676
"JSONTests": 198,
7777
"JSONSchemaTests": 79,
7878
"JSONRegexRuntimeTests": 16,
79-
"JSONStreamingTests": 36,
79+
"JSONStreamingTests": 42,
8080
"JSONPatchTests": 22,
8181
"JSONIntegrationTests": 45,
8282
"JSONHTMLTests": 87,
@@ -92,7 +92,7 @@
9292
"JSONTests": 198,
9393
"JSONSchemaTests": 79,
9494
"JSONRegexRuntimeTests": 16,
95-
"JSONStreamingTests": 36,
95+
"JSONStreamingTests": 42,
9696
"JSONPatchTests": 22,
9797
"JSONIntegrationTests": 45,
9898
"JSONHTMLTests": 87,

‎README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ Choose the products your application needs:
9090
| **JSON** | Exact values, strict parsing/writing, native Codable, Pointer, presence, checked construction, source documents and diagnostics |
9191
| **JSONSchema** | Draft 2020-12 compilation/evaluation, offline registries, bounded regex, annotations and raw declaration inspection |
9292
| **JSONSchemaExamples** | Opt-in deterministic exact candidates, conservative typed construction and native schema certification, introduced in 1.6.0 |
93-
| **JSONStreaming** | Pull syntax events and values, strict NDJSON, root-array elements and awaited output chunks |
93+
| **JSONStreaming** | Pull syntax events and values, strict NDJSON, RFC 7464 text sequences, root-array elements and awaited output chunks |
9494
| **JSONHTML** | Exact HTML and immutable exports; structured forms/cards and bounded previews in 1.6.0 |
9595
| **swift-json** | Bounded command-line JSON workflows (available since 1.5.0) |
9696
| **JSONBridges** | Opt-in native integer, Double and Foundation Decimal conversions (available in the 1.4.0 tag and 1.5.0 release) |
@@ -471,6 +471,8 @@ producer queue. Root-array elements and syntax events remain provisional until
471471
completion. A document completes only after its closing syntax, trailing whitespace
472472
and EOF. `JSONValueSequence` waits for its framing boundary before yielding a value;
473473
`NDJSONValueSequence` rejects blank/malformed physical records and never skips them.
474+
`JSONTextSequence` reads RFC 7464 `application/json-seq` texts; it treats unterminated
475+
top-level scalars as possibly truncated and skips malformed texts only when asked.
474476
Since 1.8.0, `JSONLocatedTopLevelArraySequence` drives the same root-array reader and
475477
also yields each element's `JSONSourceSpan`: zero-based half-open UTF-8 byte offsets
476478
in the complete stream, excluding whitespace and separators. It retains no original

‎Sources/JSONCLIEngine/CLIBuildIdentity.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,6 @@
22
// Generated by Scripts/generate_cli_identity.py; excludes this file.
33
// SHA-256 of the ordered Package.swift + Sources content manifest (DocC catalogs excluded), not a Git commit.
44
package enum CLIBuildIdentity {
5-
package static let revision = "sha256:dafab040289a71fed6924ffe8aa7a7273e6315c5701d1fd35fd5d1b812a52ccd"
5+
package static let revision = "sha256:c0c4442af4a6e52b5e232fc41405acbf92616b366832ae0df0bfb843e18a3773"
66
package static let kind = "source-sha256"
77
}

‎Sources/JSONStreaming/Documentation.docc/JSONStreaming.md‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# ``JSONStreaming``
22

3-
Pull-driven events and values, strict NDJSON, root-array elements and awaited sinks.
3+
Pull-driven events and values, strict NDJSON, RFC 7464 text sequences, root-array elements and awaited sinks.
44

55
Sources are caller-owned AsyncSequences of owned ``JSONByteChunk`` values.
66
Each iterator has exclusive mutable state, performs one pull at a time and
@@ -19,6 +19,12 @@ blank/malformed records and accepts a valid final record without a final LF.
1919
There is no skipped-record recovery. Error locations include absolute byte and
2020
physical record positions.
2121

22+
``JSONTextSequence`` reads RFC 7464 JSON text sequences (`application/json-seq`):
23+
each text starts with a record separator byte. Unterminated top-level scalars
24+
are treated as possibly truncated. Malformed texts fail by default and can be
25+
skipped explicitly. ``JSONTextSequenceWriter`` writes the same framing. See
26+
<doc:TextSequences>.
27+
2228
``JSONTopLevelArraySequence`` emits indexed elements followed by one explicit
2329
document-completed summary. Earlier elements remain provisional with respect
2430
to the enclosing document. A malformed/truncated tail produces no completion.
@@ -43,6 +49,7 @@ publication must stage bytes in its own sink.
4349
### Guides
4450

4551
- <doc:CompletionAndBackpressure>
52+
- <doc:TextSequences>
4653

4754
### Pull sequences
4855
- ``JSONByteChunk``
@@ -55,10 +62,13 @@ publication must stage bytes in its own sink.
5562
### Record framing and completion
5663
- ``NDJSONValueSequence``
5764
- ``NDJSONOptions``
65+
- ``JSONTextSequence``
66+
- ``JSONTextSequenceOptions``
5867
- ``JSONTopLevelArraySequence``
5968
- ``JSONTopLevelArrayEvent``
6069
- ``JSONLocatedTopLevelArraySequence``
6170
- ``JSONLocatedTopLevelArrayEvent``
6271

6372
### Awaited output
6473
- ``JSONAsyncWriter``
74+
- ``JSONTextSequenceWriter``
Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# JSON text sequences
2+
3+
Read and write RFC 7464 `application/json-seq` streams.
4+
5+
## Overview
6+
7+
A JSON text sequence frames each JSON text with a leading record separator byte
8+
(RS, `0x1E`) and, when written, a trailing LF. Because RS can never appear
9+
unescaped inside JSON, a reader can find the next text even after a malformed
10+
one. That makes the format useful for logs and long-running streams where one
11+
damaged record should not hide the rest.
12+
13+
``JSONTextSequence`` reads texts from caller-owned chunks with the same pull
14+
model, cumulative iterator budget and per-text value quotas as
15+
``NDJSONValueSequence``:
16+
17+
```swift
18+
for try await value in JSONTextSequence(chunks) {
19+
handle(value)
20+
}
21+
```
22+
23+
Each text runs from an RS to the next RS or the end of input and is parsed by
24+
the strict grammar, so surrounding whitespace is allowed and a text may span
25+
several lines. Consecutive RS bytes are not empty texts.
26+
27+
### Truncated and malformed texts
28+
29+
A top-level number, `true`, `false` or `null` that does not end in whitespace
30+
may have been cut short ("123" could be the start of "1234"), so RFC 7464
31+
section 2.4 treats it as possibly truncated. The reader reports it as an
32+
`unexpectedEOF` syntax failure. Bytes before the first RS are malformed as well.
33+
34+
By default a malformed text ends the sequence with its failure, located by
35+
absolute byte offset, text number (`record`) and line and column within the
36+
text. With ``JSONTextSequenceOptions/MalformedTexts/skip`` the reader skips
37+
malformed texts and continues with the next RS, as RFC 7464 recommends;
38+
``JSONTextSequence/Iterator/skippedTexts`` counts them. Resource failures,
39+
cancellation and source errors are never skipped.
40+
41+
### Writing
42+
43+
``JSONTextSequenceWriter`` writes each value as RS, the strict writer's bytes
44+
and LF. ``JSONTextSequenceWriter/encode(_:)`` produces a complete sequence under
45+
one ledger. ``JSONTextSequenceWriter/write(_:to:)`` writes one text in awaited,
46+
bounded chunks, like ``JSONAsyncWriter``; call it once per value. Bytes a sink
47+
has accepted cannot be recalled, but a reader recognizes a partially written
48+
text as malformed.
49+
50+
## Topics
51+
52+
- ``JSONTextSequence``
53+
- ``JSONTextSequenceOptions``
54+
- ``JSONTextSequenceWriter``
Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
// SPDX-License-Identifier: Apache-2.0
2+
// Original RFC 7464 (application/json-seq) reader over the shared stream input.
3+
import JSON
4+
/// A pull reader for RFC 7464 JSON text sequences (`application/json-seq`).
5+
///
6+
/// Each text starts with the record separator byte (RS, `0x1E`) and runs to the
7+
/// next RS or the end of input. Consecutive RS bytes are not empty texts. Each text
8+
/// is parsed by the strict UTF-8 grammar, so surrounding whitespace, including the
9+
/// LF that writers append, is allowed.
10+
///
11+
/// A text that is a top-level number, `true`, `false` or `null` must end in
12+
/// whitespace; otherwise it may have been truncated and is malformed (RFC 7464
13+
/// section 2.4). Bytes before the first RS are a malformed text too. Malformed
14+
/// texts end the sequence by default; ``JSONTextSequenceOptions/MalformedTexts/skip``
15+
/// continues with the next text instead.
16+
///
17+
/// Failures report the absolute byte offset, the one-based text number as `record`,
18+
/// and the line and column within the text.
19+
public struct JSONTextSequence<Source: AsyncSequence>: AsyncSequence where Source.Element == JSONByteChunk {
20+
/// Each complete JSON text, in order.
21+
public typealias Element = JSON
22+
let source: Source
23+
let options: JSONTextSequenceOptions
24+
/// Wraps a caller-owned source of `application/json-seq` bytes.
25+
///
26+
/// - Parameters:
27+
/// - source: The caller-owned asynchronous source of owned UTF-8 chunks.
28+
/// - options: Limits and the malformed-text policy.
29+
public init(_ source: Source, options: JSONTextSequenceOptions = .init()) { self.source = source; self.options = options }
30+
/// Creates an independent iterator with exclusive mutable state and no prefetch task.
31+
public func makeAsyncIterator() -> Iterator { .init(source: source.makeAsyncIterator(), options: options) }
32+
/// Exclusive pull iterator; never call next concurrently.
33+
///
34+
/// Each iterator owns its budget and parsing state. Conditional Sendable of the
35+
/// sequence does not make this mutable iterator safe for concurrent access.
36+
public struct Iterator: AsyncIteratorProtocol {
37+
private var source: Source.AsyncIterator?
38+
private let options: JSONTextSequenceOptions
39+
private var driver: JSONTextSequenceDriver<Source.AsyncIterator>?
40+
fileprivate init(source: Source.AsyncIterator, options: JSONTextSequenceOptions) { self.source = source; self.options = options }
41+
/// The number of malformed texts skipped so far; always zero with the `fail` policy.
42+
public var skippedTexts: Int { driver?.skipped ?? 0 }
43+
/// Pulls at most one value through exclusively owned iterator state.
44+
///
45+
/// An error or cancellation makes an initialized iterator terminal and prevents later pulls.
46+
/// - Returns: The next JSON text's value, or nil at the end of input.
47+
/// - Throws: Source, syntax or resource errors, or cancellation.
48+
public mutating func next() async throws -> JSON? {
49+
if driver == nil { driver = try JSONTextSequenceDriver(upstream: source!, options: options); source = nil }
50+
return try await driver!.next()
51+
}
52+
}
53+
}
54+
extension JSONTextSequence: Sendable where Source: Sendable {}
55+
56+
private final class JSONTextSequenceDriver<Upstream: AsyncIteratorProtocol> where Upstream.Element == JSONByteChunk {
57+
static var separator: UInt8 { 0x1E }
58+
let input: JSONStreamInput<Upstream>
59+
let skipMalformed: Bool
60+
var terminal = false, resetOnNext = false, afterSeparator = false
61+
var offset: UInt64 = 0, texts: UInt64 = 0
62+
var skipped = 0
63+
init(upstream: Upstream, options: JSONTextSequenceOptions) throws {
64+
input = try JSONStreamInput(upstream: upstream, limits: options.limits); skipMalformed = options.malformedTexts == .skip
65+
}
66+
func next() async throws -> JSON? {
67+
guard !terminal else { return nil }
68+
do {
69+
try input.meter.checkpoint(); try input.meter.require(.pendingValues, 1)
70+
while true {
71+
if resetOnNext { try input.resetRecord(); resetOnNext = false }
72+
let introduced = afterSeparator, start = offset
73+
var text: [UInt8] = [], ended = false
74+
while true {
75+
guard try await input.ensureBytes() else { ended = true; break }
76+
let byte = input.bytes[input.index]; input.index += 1; offset += 1
77+
try input.meter.work()
78+
if byte == Self.separator { break }
79+
try input.meter.require(.inputBytes, UInt64(text.count) + 1); try input.meter.retain(1); text.append(byte)
80+
}
81+
afterSeparator = !ended
82+
// Empty segments: input that starts with RS, consecutive RS bytes, or a final RS.
83+
if text.isEmpty {
84+
if ended { terminal = true; return nil }
85+
continue
86+
}
87+
texts += 1; resetOnNext = true
88+
do {
89+
let value = try parse(text, introduced: introduced)
90+
try input.meter.charge(.totalRecords)
91+
if ended { terminal = true }
92+
return value
93+
} catch let failure as JSONFailure {
94+
let relative = failure.location?.byteOffset ?? 0
95+
let located = JSONFailure(failure.category, code: failure.code,
96+
location: .init(byteOffset: start + relative, line: failure.location?.line,
97+
column: failure.location?.column, record: texts),
98+
resource: failure.resource)
99+
guard skipMalformed, failure.category == .syntax else { throw located }
100+
try input.meter.charge(.totalRecords); skipped += 1
101+
if ended { terminal = true; return nil }
102+
}
103+
}
104+
} catch { terminal = true; input.failed = true; throw error }
105+
}
106+
/// Parses one text; a missing leading RS or a possibly truncated scalar is a syntax failure.
107+
private func parse(_ text: [UInt8], introduced: Bool) throws -> JSON {
108+
guard introduced else { throw JSONFailure(.syntax, code: .invalidSyntax, location: .init(byteOffset: 0, line: 1, column: 1)) }
109+
let value = try JSONParser().parse(text, meter: &input.meter)
110+
switch value {
111+
case .object, .array, .string: return value
112+
default:
113+
// RFC 7464 section 2.4: a top-level scalar without trailing whitespace may be truncated.
114+
guard let last = text.last, last == 0x20 || last == 0x09 || last == 0x0A || last == 0x0D else {
115+
throw JSONFailure(.syntax, code: .unexpectedEOF, location: .init(byteOffset: UInt64(text.count)))
116+
}
117+
return value
118+
}
119+
}
120+
}
Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
// SPDX-License-Identifier: Apache-2.0
2+
import JSON
3+
/// Finite limits and the malformed-text policy for RFC 7464 JSON text sequences.
4+
public struct JSONTextSequenceOptions: Sendable {
5+
/// What a reader does with a text that is not one complete JSON value.
6+
public enum MalformedTexts: Sendable, Equatable {
7+
/// Throw the syntax failure and end the sequence (the default).
8+
case fail
9+
/// Skip the text and continue with the next record separator, as RFC 7464
10+
/// recommends. Resource failures, cancellation and source errors still end
11+
/// the sequence. The iterator counts skipped texts.
12+
case skip
13+
}
14+
/// Finite capacities and work quotas applied to library-owned operations.
15+
public var limits: JSONLimits
16+
/// The policy for texts that fail to parse or are possibly truncated.
17+
public var malformedTexts: MalformedTexts
18+
/// Creates the configuration; finite limits are checked at operation entry.
19+
///
20+
/// - Parameters:
21+
/// - limits: Finite inclusive capacities and work quotas, per text and for the iterator lifetime.
22+
/// - malformedTexts: Whether malformed texts end the sequence or are skipped.
23+
public init(limits: JSONLimits = .balanced, malformedTexts: MalformedTexts = .fail) {
24+
self.limits = limits; self.malformedTexts = malformedTexts
25+
}
26+
}

0 commit comments

Comments
 (0)