[Bug] IcebergCommitCallback.expireAllBefore fails permanently on a manifest list a previous rebuild already deleted

[lucasgameiroborges]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

1.4.1 (but code seems identical on current master 4a346f450)

Compute Engine

Flink 2.0

Minimal reproduce step

Unit test for reproduction will be included in the PR. Not sure we can reliably reproduce in a live system but the process was:

1. Writing continuously to a Paimon table with Iceberg REST metadata sync enabled.
2. Get a committer recovery that replays a committable more than one snapshot behind the catalog, ie createMetadataWithoutBase runs.
3. expireAllBefore(N) deletes the manifest lists of every version < N. Then at some point we need to have v(N-1).metadata.json retained while the manifest lists it references have just been deleted.
4. Again get to a committer recovery where createMetadataWithoutBase runs. expireAllBefore now walks v(N-1).metadata.json and calls manifestList.read(listName) on a list that was already deleted in step 3. This will lead to:

java.lang.RuntimeException: Failed to read snap-1-1f878b70-445c-4332-98a4-73ea0c549437.avro
    at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:131)
    at org.apache.paimon.iceberg.IcebergCommitCallback.expireAllBefore(IcebergCommitCallback.java:985)
    at org.apache.paimon.iceberg.IcebergCommitCallback.createMetadataWithoutBase(IcebergCommitCallback.java:386)
    at org.apache.paimon.iceberg.IcebergCommitCallback.retry(IcebergCommitCallback.java:249)
    at org.apache.paimon.operation.FileStoreCommitImpl.filterCommitted(FileStoreCommitImpl.java:277)
    at org.apache.paimon.flink.sink.RestoreCommittableStateManager.recover(RestoreCommittableStateManager.java:81)
    at org.apache.paimon.flink.sink.CommitterOperator.initializeState(CommitterOperator.java:147)
Caused by: java.io.FileNotFoundException: File '…/metadata/snap-1-1f878b70-….avro' not found

What doesn't meet your expectations?

expireAllBefore exists to delete the manifests and manifest lists of expired versions. It deletes them with deleteQuietly (tolerating absence) but reads the manifest list with a hard manifestList.read(...) that throws when the file is already gone. If the two cleanup steps lead to a retained metadata JSON referencing an already-deleted list, the task dies during Flink state initialisation, the committable is never dropped, and recovery replays it until the retained JSON happens to be deleted by later churn.

Skipping a manifest list that no longer exists seems safe since the loop's only purpose in reading it is to delete what it references, and those manifests were deleted by the same earlier pass.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!