Improve arrow-avro decoding for one-record messages

[jordepic]

Which issue does this PR close?

• Closes Improve arrow-avro decoding of transport-bounded datums and null runs #10712.

Rationale for this change

Some messaging systems deliver records one at a time. For example, Kafka gives a consumer the complete byte array for one message, so the consumer already knows where that message begins and ends. If that message contains one Avro record, the decoder does not need an Avro header to identify the record boundary.

This is not limited to schema-registry-framed Avro. Apache Flink supports both format = 'avro-confluent', where each Kafka message includes a Confluent header containing its schema ID, and format = 'avro', where each message contains a bare Avro datum with no Confluent header. For the latter, Flink derives the writer schema from the table definition, so the consumer already knows the schema even though the payload carries no schema ID or framing. Supporting this format with the existing decoder currently requires manufacturing a synthetic prefix and copying every message.

In Avro terminology, one encoded value is called a datum. A datum can be a primitive value or a complete record.

The existing Decoder::decode method expects each record to include an Avro framing prefix. That prefix identifies the Avro format and the writer schema. A caller that already has one complete Kafka message and already knows its schema must therefore add a temporary prefix and copy the payload before decoding it, or separately inspect the schema to determine how many bytes belong to the record.

Nullable nested records have a separate performance cost. Consider an event with three optional nested records where only one is populated on each row. For every absent record, the decoder currently walks through all of its child fields and immediately appends placeholder values. Long sequences of absent records repeat the same work row by row.

What changes are included in this PR?

• Add Decoder::decode_datum, which decodes one complete Avro value directly from the supplied bytes using the writer schema already selected on the decoder. It returns the number of bytes used by that value, leaving any remaining bytes untouched.
• Record consecutive null values immediately in the validity bitmap, but postpone adding their child placeholders. When a non-null value arrives, or the batch is flushed, add the accumulated placeholders as one run.
• Add a benchmark representing three optional nested event records, with the populated record changing every 100 rows.

The second change only affects how the decoder builds arrays internally. The returned Arrow arrays are unchanged, and all child arrays are brought to the correct length before a batch is returned.

Are these changes tested?

Yes.

• cargo test -p arrow-avro --all-features: 479 unit tests passed; 26 documentation tests passed; 1 documentation test ignored.
• cargo clippy -p arrow-avro --all-targets --all-features -- -D warnings
• cargo fmt --all -- --check

The benchmark below decodes all 10,000 rows into one batch. It was run on an Apple M1 Max against a saved main baseline.

Benchmark	main median	this PR median	Change
Three sparse optional nested records	820.16 us	581.44 us	29.2% less time, or 41.3% more throughput

Command:

cargo bench -p arrow-avro --bench decoder -- 'SparseNested\(Struct\)/10000' --baseline main

Are there any user-facing changes?

Yes. Decoder gains a new, non-breaking decode_datum method for callers that already have the complete bytes for one Avro value and have already selected its writer schema. Existing decoding methods and framed input behavior are unchanged.

AI assistance disclosure: Codex was used to help port the implementation, draft tests and the benchmark, and prepare the issue and PR text. The resulting code and all reported outputs were reviewed before submission, and I take responsibility for the contribution.

[alamb]

FYI @jecsand838 -- could you help review this PR?

[jecsand838]

FYI @jecsand838 -- could you help review this PR?

Absolutely! I'll have time tonight / tomorrow morning to review this.

[kinshuk-bb]

@jordepic is there currently a way to convert multiple Avro Datums in a single .avro to a RecordBatches directly? A Raw Binary Encoded Avro

[jordepic]

@kinshuk-bb Yes, for raw binary Avro containing multiple back-to-back datums encoded with the same known writer schema, this PR supports decoding them directly into RecordBatches without adding framing. Configure a Decoder with that writer schema, repeatedly call decode_datum on the remaining byte slice, advance by the returned consumed-byte count, and call flush whenever batch_is_full (plus once at the end). Each call decodes exactly one datum and leaves subsequent datums untouched.

If by .avro you mean a standard Avro object-container file, that is already supported today: ReaderBuilder::build reads the container header/schema and iterates over RecordBatches. The new method is specifically for raw datum streams/messages without an object-container header or per-record framing.

I am adding explicit coverage for concatenated raw datums and batch boundaries to make this use case clear.

[jordepic]

@jecsand838 if you have any time to take a look in the next few days I'd greatly appreciate it! This is a really big win for StreamFusion and any other system trying to beat out avro processing from the JVM!

[kinshuk-bb]

@jordepic thank you! i was relying on adding framing just to decode to be sure i wouldn't face issues. will try this out now. An addition to the cargo docs that this is directly possible now would be great.

[jordepic]

@kinshuk-bb Addressed in 92a4210: the crate-level documentation now includes a runnable example decoding consecutive bare Avro datums directly with Decoder::decode_datum, and the reader/module/builder docs plus README now explicitly list unframed binary datums as a supported format.