Core: Add Puffin files metadata table#17192
Open
yangshangqing95 wants to merge 1 commit into
Open
Conversation
yangshangqing95
force-pushed
the
feat/puffin-metadata-tables
branch
from
9d29903 to
7e27ad9
Compare
yangshangqing95
force-pushed
the
feat/puffin-metadata-tables
branch
from
7e27ad9 to
928a796
Compare
ebyhr
reviewed
Jul 13, 2026
Comment on lines
+946
to
+953
| required(1, "snapshot_id", Types.LongType.get()), | ||
| required(2, "statistics_path", Types.StringType.get()), | ||
| required(3, "file_size_in_bytes", Types.LongType.get()), | ||
| required(4, "file_footer_size_in_bytes", Types.LongType.get()), | ||
| required(5, "blob_count", Types.IntegerType.get()), | ||
| optional(6, "blob_types", Types.ListType.ofRequired(7, Types.StringType.get())), | ||
| optional(8, "field_ids", Types.ListType.ofRequired(9, Types.IntegerType.get())), | ||
| optional(10, "field_names", Types.ListType.ofOptional(11, Types.StringType.get()))) |
Member
There was a problem hiding this comment.
Puffin spec currently supports two blob types apache-datasketches-theta-v1 & deletion-vector-v1: https://iceberg.apache.org/puffin-spec/#blob-types Exposing NDV-specific columns is weird to me. We should rename the metadata table, or change these columns.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a new puffin_files metadata table that exposes registered Puffin statistics files from Iceberg table metadata.
The table reads only
Table#statisticsFiles()and does not open Puffin files, read Puffin footers, or parse blob payload bytes.The new metadata table returns one row per registered statistics file with file-level summary information:
Example Spark SQL output:
.../xxxxxx....stats["apache-datasketches-theta-v1"][1,2]["id","data"]Motivation
Iceberg tables can register Puffin statistics files in table metadata, but there is currently no metadata table that makes those registered statistics files easy to inspect through SQL.
Exposing this information through a metadata table makes it easier to debug and observe Puffin-backed table statistics, such as NDV statistics written by compute_table_stats, without requiring users to inspect metadata JSON manually.