[Feature] Optimize partition discovery with projected manifest scans

[zhoulii]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Partition discovery only requires a small subset of manifest fields, but readPartitionEntries() currently reads complete manifest entries. This causes unnecessary decoding and object allocation for fields such as key/value statistics and embedded indexes.

Although partition entries are created eagerly, the scan still builds an intermediate list for each manifest before merging the partition statistics.

Solution

Introduce a projected, streaming manifest scan for partition discovery:

• Project only the fields required for partition aggregation and structural filtering.
• Push partition and bucket filters into the manifest reader.
• Merge projected entries directly into the partition map without per-manifest intermediate lists.
• Fall back to complete manifest entries when active filters require additional fields.
• Preserve the existing cache path for cache-eligible manifests.

This change does not modify the manifest format or partition scan semantics.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!