Skip to content

feat: Apache Fluss data connector with CDC refresh (connector-fluss) - #2

Draft
J0hnG4lt wants to merge 7 commits into
trunkfrom
feat/fluss-connector-v2
Draft

feat: Apache Fluss data connector with CDC refresh (connector-fluss)#2
J0hnG4lt wants to merge 7 commits into
trunkfrom
feat/fluss-connector-v2

Conversation

@J0hnG4lt

@J0hnG4lt J0hnG4lt commented Aug 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

📝 Summary

Adds an Apache Fluss data connector (connector-fluss) with efficient CDC-based refresh, modeled on the DynamoDB Streams connector pattern.

  • Log tables → refresh_mode: append: continuous ingestion via RecordBatchLogScanner.
  • Primary-key tables → refresh_mode: changes: the table's changelog streams in as Debezium-style CDC ops (c/u/d, -U skipped), applied by primary key.
  • Checkpointing: per-(partition, bucket) offsets persist via the generic spice_sys::checkpoint_store sidecar (spice_sys_fluss_log); offsets ride each ChangeEnvelope's committer so they commit only after durable apply — exact resume on graceful restart, at-least-once on crash. Zero changes to the runtime crate.
  • Readiness: high-watermark accounting seeded from the checkpoint; caught-up starts announce readiness up front with a zero-row ready-signal envelope.
  • Fault recovery: poll errors back off 1s; a checkpoint stranded past truncated log segments (LogOffsetOutOfRangeException) recovers — CDC replays from EARLIEST_OFFSET (idempotent ops converge), append rejoins the live tail with a loud data-loss warning.
  • Feature flag fluss (in spiced defaults), per the CONTRIBUTING feature-flag convention; linkme registration; MetricsProvider with consume/error counters.

Verification: E2E suite fully green — 20/20 assertions across 7 scenarios against a live Fluss 0.9.1 cluster (all podman; see e2e/fluss/): bootstrap replay (both modes), realtime append (1s visibility), live CDC ins/upd/del, graceful restart (exact, no re-ingest), SIGKILL crash (zero loss), tablet-server fault and coordinator-pause chaos under continuous load (exact convergence: 478/478, 871/871). The suite caught two real defects fixed in this PR: a NotReady deadlock on caught-up stream start, and a hot error loop on source log truncation.

Unit tests cover CDC batch construction (op mapping, -U filtering, per-row primary keys), readiness accounting, and checkpoint serialization.

🔗 Related

🚨 Breaking Changes

None. New connector behind a new fluss feature (added to spiced defaults, matching other connectors).

📚 Docs

  • In-repo: docs/features/fluss-connector.md (setup, both modes, delivery semantics, known limitations, troubleshooting).
  • For upstream submission: spiceai/docs connector-table entry + cookbook recipe still to be filed (Alpha criteria) — deferred while this PR is fork-internal.

Request in the official repo

spiceai#9423

👀 Notes for Reviewers

Alpha criteria checklist (docs/criteria/connectors/alpha.md):

  • Basic functionality of the native source (log + PK streaming, both verified E2E)
  • Common use cases execute with a low error rate (20/20 E2E assertions; unit tests)
  • Known limitations logged (docs page): changelog-retention-dependent bootstrap (table.log.ttl; KV-snapshot read not yet in fluss-rs), partition set fixed at stream start, no auth params, git-pinned fluss-rs until an Arrow ≥58 release
  • Documentation with setup steps + limitations (docs/features/fluss-connector.md)
  • Cookbook recipe + spiceai/docs table entry (deferred until upstream submission)
  • DRI sign-off (needs a proposal issue upstream — not filed yet by design; this PR is fork-internal)

Process notes for the eventual upstream submission (per CONTRIBUTING.md):

  1. File the proposal issue first (sets expectations, gets a DRI assigned).
  2. make signoff on a Linux dev machine to attest the pushed commit (fmt/clippy/test for the changed crates were run in-container for this draft: cargo fmt --check, cargo clippy --all-targets -D warnings, cargo test -p connector-fluss).
  3. Decide the fluss-rs dependency story with the DRI (git pin vs waiting for a crates.io release).

Review focus: stream.rs offset/readiness accounting (watermark-exclusive vs last-offset-inclusive), the truncation-recovery semantics per mode, and the committer ordering contract in lib.rs.

🤖 Generated with Claude Code

J0hnG4lt and others added 5 commits July 30, 2026 23:38
@J0hnG4lt @claude
feat(fluss): scaffold connector-fluss crate for SpiceAI 2.x
d85bf61 
Port of the v1-era Fluss connector (feat/fluss-connector-v1) onto the 2.x
connector architecture: standalone crate under crates/data-connectors,
linkme registration, generic spice_sys blob checkpoint sidecar, framework
ready-signal envelopes.

- Log tables -> refresh_mode: append (RecordBatchLogScanner)
- PK tables -> refresh_mode: changes (LogScanner changelog, upstream spiceai#639)
- Per-bucket offset checkpoints committed per envelope (at-least-once);
  CDC path now checkpoints for real via upstream ScanRecord::offset()
- Readiness via high-watermark accounting seeded from the checkpoint
- fluss-rs git-pinned to J0hnG4lt/fluss-rust arrow-58 (= apache main +
  Arrow 57->58 bump) until upstream ships an Arrow 58-compatible release

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@J0hnG4lt @claude
test(fluss): podman E2E suite — cluster, producer, scenario runner
a348b1f 
All-podman harness for the Fluss connector: Fluss 0.9.x cluster (zk +
coordinator + 2 tablets, host networking), deterministic Rust producer
(setup/append/cdc/mixed with retry), spiced built from source with the
connector, and a scenario runner asserting through POST /v1/sql:

  s1 bootstrap replay (append + CDC upserts/deletes before spiced starts)
  s2 realtime append latency
  s3 live CDC insert/update/delete exact state
  s4 graceful restart -> exact checkpoint resume
  s5 SIGKILL crash -> at-least-once appends, exact PK state
  s6 tablet-server fault under load -> convergence
  s7 chaos (coordinator pause + tablet restart) -> convergence

Cargo.lock: add fluss-rs (git, arrow-58) + connector-fluss resolution.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@J0hnG4lt @claude
fix(fluss-e2e): alternate spiced ports, MSYS path fix, libclang for b…
70e3758 
…uilder

- spiced on 18090/15051 so the suite coexists with another spiced on the
  machine's shared host-network port namespace
- hand podman-compose a Windows-style path under Git Bash (cygpath -m)
- clang + libclang-dev in the builder image (custom-labels bindgen)
- run-e2e.sh executable bit

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@J0hnG4lt @claude
fix(fluss): DataFusion 54 API drift — TableProvider downcast, builder…
201a52c 
… deps

TableProvider no longer declares as_any (the trait inherits Any); drop the
impl and downcast Arc<dyn TableProvider> directly, matching connector-kafka.
Builder image gains libprotobuf-dev (substrait well-known types) — spiced
now compiles and links with the connector.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@J0hnG4lt @claude
fix(fluss-e2e): sql body needs parameters field; decouple spiced life…
f018253 
…cycle from compose deps

- /v1/sql JSON requires 'parameters' — without it every assert read as empty
- drop spiced depends_on: podman-compose 'up -d spiced' recreates dependency
  containers, wiping ZooKeeper (and all Fluss tables) mid-suite
- restart scenarios use plain podman stop/start/kill
- clean force-removes containers by name (compose down ordering fails)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 1, 2026 •
edited
Loading

Copy link
Copy Markdown

✅ Pull with Spice Passed

Passing checks:

  • ✅ Title meets minimum length requirement (10 characters)
  • ✅ No banned labels detected
  • ✅ Has a label from required category kind/
  • ✅ Has a label from required category area/
  • ✅ Has at least one assignee: J0hnG4lt

@J0hnG4lt @claude
fix(fluss): announce readiness on caught-up start; recover from log t…
10c5eda 
…runcation

Two defects found by the E2E suite:

- A stream that starts already caught up (empty table, or a resume whose
  checkpoint covers the watermarks) emitted no envelope until new data
  arrived, leaving the dataset NotReady and unqueryable indefinitely.
  Announce readiness up front with a zero-row ready-signal envelope.

- A checkpoint pointing past the server's surviving log segments (tablet
  loses its unflushed tail) hot-looped LogOffsetOutOfRangeException. Poll
  errors now back off 1s, and out-of-range recovers: the CDC path replays
  the changelog from EARLIEST (idempotent PK ops converge), the append
  path rejoins at the live tail with a loud data-loss warning.

E2E: tablet restarts get a 30s graceful stop so Fluss flushes acked rows.
Suite: 20/20 scenarios pass (bootstrap, realtime, CDC, graceful+crash
resume, tablet fault, chaos).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@J0hnG4lt J0hnG4lt self-assigned this Aug 2, 2026
@J0hnG4lt @claude
feat(fluss): upstream contribution conformance — feature flag, unit t…
27a3242 
…ests, docs

Per CONTRIBUTING.md and the Alpha connector criteria:

- 'fluss' feature flag in spiced (optional dep, in defaults), matching the
  connector feature convention; cfg-gated force-link
- unit tests: CDC batch construction over real ScanRecords (op mapping,
  UpdateBefore filtering, per-row primary keys), watermark readiness
  accounting, checkpoint serde roundtrip (9 tests)
- docs/features/fluss-connector.md: setup, both modes, delivery semantics,
  known limitations, troubleshooting
- clippy strict pass (-D warnings, --no-deps): doc backticks, if-let over
  single-pattern match, struct-update init, expect(implicit_hasher)
- Containerfile BUILD_CMD arg so the cached builder runs the
  signoff-equivalent checks (fmt/clippy/test)

Checks (in-container, rust 1.96.1): cargo fmt --check clean; cargo clippy
--all-targets --no-deps -D warnings clean; cargo test 9 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@J0hnG4lt J0hnG4lt

Labels

area/config area/docs kind/dependencies kind/feature size/xl

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@J0hnG4lt