Repository navigation
Add Avro and ORC parsers (data-lake siblings of Parquet) - #46
Conversation
Two parsers in ax-normalize, both lowering to the same engine-independent Columns as the Parquet/Arrow parsers so no library type escapes the contract. AvroParser (apache-avro): each record in the object-container file is a row; record fields become typed columns via avro_to_value (bool/int/long/float/double /string/enum mapped; bytes/fixed -> hex Str; date/time logical types -> their integer value; unions unwrap; nested records/arrays/maps/decimals -> Null, honest absence for v1's flat-scalar lowering). Magic Obj\x01; extension .avro. OrcParser (orc-rust -> Arrow): the file is read into Arrow record batches; each cell is rendered and run through infer_scalar so numbers/bools become typed columns, nulls preserved. Magic ORC; extension .orc. arrow pinned to the major orc-rust uses so the RecordBatch types unify. Both behind the default-on datalake feature (binary formats), so the text-only build stays lean. Roundtrip tests write a tiny Avro file (apache-avro Writer) and a tiny ORC file (orc-rust Arrow writer) in-memory — no committed binaries; avro_to_value is unit-tested across all handled variants; non-format input is a clean Parse error. Mutation gate: 0 surviving mutants on the new file. Deleting avro_to_value's explicit Null arm is a documented equivalent (it and the catch-all both return Null). Closes #23 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Warning Review limit reached
More reviews will be available in 20 minutes and 15 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (5)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Implements the Avro / ORC plugins — issue #23 (the last of the 23 format issues).
Both lower to the same engine-independent
Columns as the Parquet/Arrowparsers, so no library type escapes the contract.
AvroParser(apache-avro)Each record in the object-container file is a row; record fields become typed
columns via
avro_to_value: bool/int/long/float/double/string/enum mapped;bytes/fixed→ hexStr; date/time logical types → their integer value;unions unwrap to their held value; nested records/arrays/maps/decimals →
Null(honest absence for v1's flat-scalar lowering). Magic
Obj\x01; extension.avro.OrcParser(orc-rust→ Arrow)The file is read into Arrow record batches; each cell is rendered and run through
infer_scalar, so numbers/bools become typed columns and nulls are preserved.Magic
ORC; extension.orc.arrowis pinned to the majororc-rustuses sothe
RecordBatchtypes unify.Feature gating
Both behind the default-on
datalakefeature (binary formats); the text-only--no-default-featuresbuild stays lean. Both builds verified.Testing
Roundtrip tests write a tiny Avro file (apache-avro
Writer) and a tiny ORC file(orc-rust Arrow writer) in-memory — no committed binaries;
avro_to_valueisunit-tested across every handled variant (incl. union unwrap, date/time, bytes);
non-format input is a clean
AxError::Parse.Gates
fmt/clippy -D warnings/ full workspace tests green (default and--no-default-features).avro.rs. Deletingavro_to_value'sexplicit
Nullarm is a documented equivalent (it and the catch-all bothreturn
Null).Closes #23
🤖 Generated with Claude Code