Skip to content

[spark] Streaming source cannot advance past an empty initial full snapshot #9308

Description

@LsomeYeah

Problem

Spark Structured Streaming cannot advance past an initial full snapshot that produces no splits. Every trigger restores the same full snapshot, so later appended snapshots are never reached.

Reproduction

  1. Create a Paimon table, insert one row, then make the current snapshot empty with INSERT OVERWRITE ... WHERE false.
  2. Start a Spark streaming read with the default latest-full mode.
  3. Append one row in a new snapshot.
  4. The query still reads nothing.

DataTableStreamScan advances its checkpoint to the next snapshot, but StreamHelper creates an offset only from the last returned split. An empty full plan therefore loses the advanced cursor, and the next trigger restores the same full snapshot again.

Expected: persist progress for a completed zero-split full snapshot and continue with later delta snapshots.

This also occurs without consumer-id. A fix must distinguish a truly completed empty snapshot from ReadMinRows, which may intentionally defer a non-empty plan.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions