Problem
Spark Structured Streaming cannot advance past an initial full snapshot that produces no splits. Every trigger restores the same full snapshot, so later appended snapshots are never reached.
Reproduction
- Create a Paimon table, insert one row, then make the current snapshot empty with
INSERT OVERWRITE ... WHERE false.
- Start a Spark streaming read with the default
latest-full mode.
- Append one row in a new snapshot.
- The query still reads nothing.
DataTableStreamScan advances its checkpoint to the next snapshot, but StreamHelper creates an offset only from the last returned split. An empty full plan therefore loses the advanced cursor, and the next trigger restores the same full snapshot again.
Expected: persist progress for a completed zero-split full snapshot and continue with later delta snapshots.
This also occurs without consumer-id. A fix must distinguish a truly completed empty snapshot from ReadMinRows, which may intentionally defer a non-empty plan.
Problem
Spark Structured Streaming cannot advance past an initial full snapshot that produces no splits. Every trigger restores the same full snapshot, so later appended snapshots are never reached.
Reproduction
INSERT OVERWRITE ... WHERE false.latest-fullmode.DataTableStreamScanadvances its checkpoint to the next snapshot, butStreamHelpercreates an offset only from the last returned split. An empty full plan therefore loses the advanced cursor, and the next trigger restores the same full snapshot again.Expected: persist progress for a completed zero-split full snapshot and continue with later delta snapshots.
This also occurs without
consumer-id. A fix must distinguish a truly completed empty snapshot fromReadMinRows, which may intentionally defer a non-empty plan.