Search before asking
Description
Follow-up to #4119 and #4120. This also follows the design discussion in PR #4120.
Historical partition writes keep local KV state for records that have not yet been incorporated into the lake table. This state currently has no incremental reclamation mechanism and can grow indefinitely for continuously written historical buckets.
An all-or-nothing cleanup was considered in #4120: wait until the lake log end offset exactly matches the local log end offset, then drop and rebuild the whole RocksDB instance. That approach was removed because a continuously written bucket may never observe exact equality, cleanup depends on several asynchronous steps completing together, and rebuilding the entire KV state creates a reclamation cliff and a rebuild-failure window.
Implement incremental cleanup using the RocksDB compaction-filter infrastructure:
- Tag historical values and tombstones with the WAL offset that produced them.
- Maintain a monotonic cleanup watermark for each historical bucket from confirmed lake log end-offset progress. Since the lake log end offset is exclusive and recovery starts from that offset, entries tagged with offsets lower than the watermark are covered by lake storage and are eligible for removal.
- Reuse or generalize the existing TTL compaction-filter support, which already accepts a server-controlled callback, so the cleanup decision is driven by the lake watermark instead of wall-clock time.
- Trigger RocksDB compaction after the watermark advances. Coalesce and rate-limit compaction requests because a compaction filter only physically removes eligible entries while compaction runs.
- Preserve lookup correctness for local values and tombstones while allowing keys already covered by lake to fall back to the refreshed lake lookuper.
- Restore the watermark safely across leader changes and restarts without requiring the lake and local log end offsets to become equal.
Add coverage for continuously written buckets, values and tombstones, repeated lake-progress notifications, leader changes/recovery, compaction rate limiting, and lookup correctness before and after cleanup.
This follow-up does not need to block the 1.0 release.
Willingness to contribute
Search before asking
Description
Follow-up to #4119 and #4120. This also follows the design discussion in PR #4120.
Historical partition writes keep local KV state for records that have not yet been incorporated into the lake table. This state currently has no incremental reclamation mechanism and can grow indefinitely for continuously written historical buckets.
An all-or-nothing cleanup was considered in #4120: wait until the lake log end offset exactly matches the local log end offset, then drop and rebuild the whole RocksDB instance. That approach was removed because a continuously written bucket may never observe exact equality, cleanup depends on several asynchronous steps completing together, and rebuilding the entire KV state creates a reclamation cliff and a rebuild-failure window.
Implement incremental cleanup using the RocksDB compaction-filter infrastructure:
Add coverage for continuously written buckets, values and tombstones, repeated lake-progress notifications, leader changes/recovery, compaction rate limiting, and lookup correctness before and after cleanup.
This follow-up does not need to block the 1.0 release.
Willingness to contribute