[kafka] Add schema-aware Produce transcoding - #4189
Draft
gyang94 wants to merge 4 commits into
Draft
Conversation
Introduce request dispatch, ApiVersions, metadata discovery, and topic lifecycle operations as the reviewable foundation for subsequent Produce support. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5.6-sol AI-Contributed/Feature: 2698/2698 AI-Contributed/UT: 1160/1160
Support raw and string Produce requests, validate acks=all semantics, and wake delayed follower fetches after successful appends. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5.6-sol AI-Contributed/Feature: 1204/1204 AI-Contributed/UT: 2066/2066
Authenticate Kafka protocol connections with SASL/PLAIN and propagate the authenticated principal through request dispatch and authorization. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5.6-sol AI-Contributed/Feature: 1503/1503 AI-Contributed/UT: 1514/1514
Resolve Fluss table schemas and transcode raw, string, and JSON Kafka records into typed rows for Produce requests. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5.6-sol AI-Contributed/Feature: 2201/2201 AI-Contributed/UT: 1230/1230
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: #4185
Depends on #4188.
This is PR 4 of 4 for the Basic Produce compatibility milestone. The intended review delta is the single tip commit
dc276fe8d; the earlier stack commits remain visible until their PRs are merged and this branch is rebased.Brief change log
Tests
API and Format
The change defines Kafka mapping through table custom properties and writes existing Fluss indexed or Arrow log formats. It does not introduce a new storage format or change the Fluss public client API.
Documentation
Adds design and research notes for Kafka JSON-to-Fluss schema mapping, including complex types and rescue behavior.
Stack
Generative AI disclosure: Codex (GPT-5.6) assisted with implementation and commit organization. Human review is required before merge.