Core: Refactor ContentStats and FieldStats by rdblue · Pull Request #17159 · apache/iceberg

rdblue · 2026-07-10T21:46:24Z

This updates the content stats API and implementations used for columnar stats in v4.

API changes:

Removes StructLike used for serialization from the interfaces
Adds copy methods to support copying TrackedFile with stats
Aligns interface conventions (use Iterable, expose full struct type as type, etc.)

StatsUtil changes:

Reduce complexity of ID methods, reduce number of constants
Remove FieldStatistic enum; not all offsets were represented as FieldStatistic and it also embedded util methods. All utils are now in StatsUtil.
Avoid double switch statement to resolve position to enum and enum to field in FieldStatsStruct get and set
Simplified field stats struct generation
Replaced content stats schema visitor with a loop
Added stats schema generation for read path based on field IDs
Added stats schema generation for write path based on MetricsConfig

ContentStats implementation changes:

Rename to ContentStatsStruct to align with other v4 types
Implement StructLike methods without copying data to allow object reuse, do not rebuild field stats
Use setCustomType instead of copying via builder in set
Keep field stats in a map rather than syncing between list and map
Remove unnecessary builder
Fix raw types suppressions

FieldStats implementation changes:

Rename to FieldStatsStruct to align with other v4 types
Remove SupportsIndexProjection that is bypassed
Implement get and set projection directly based on offset, do not rely on mapping passed to constructor
Use internal get/set to avoid duplicated javaClass.cast
Remove equals and hashCode
Remove unnecessary builder

Test changes:

Separated StatsUtil test values from production constants
Aligned implementation tests with other v4 object tests (test accessors, get, set, projection, serialization, and copy)
Updated tests that depend on stats classes

rdblue · 2026-07-10T21:47:41Z

+  Types.StructType type();
+
+  /** Returns a copy of these {@link ContentStats}, deep-copying the contained field stats. */
+  ContentStats copy();


Needed to implement copy in TrackedFile correctly. We also need to follow up with methods to project by field ID.

rdblue · 2026-07-10T21:49:32Z

+   * <p>Note: This type may be a projection of the stats stored in manifest files.
   */
-  Type type();
+  Types.StructType type();


This changed to the actual schema. The lower and upper bounds types do not always match (for geo types) so this is the best way to expose types. It is also more consistent with existing conventions.

rdblue · 2026-07-10T21:49:58Z


  /** The total NaN value count */
-  Long nanValueCount();
+  long nanValueCount();


These changes are conservative. We may want to widen later, but I want to make a conscious choice.

Curious what's driving this, from a spec perspective is there an open question about if these stats should be required? My understanding of the parquet spec is that value counts are required but null/NaN counts are optional

I think these accessors should reflect the API that we want for building on top of stats. So the decision should be made when we introduce classes that consume these.

I think that primitives are more locked down. They will throw a NullPointerException if called when there is no value, which is good for catching calls that should not be made. For example, IS NaN and IS NOT NaN predicates cannot be bound to fields that are not floating points (throws ValidationException) so accessing the NaN count should not happen if the field is missing from the stats struct.

On the other hand, these values may just be missing for a specific data file because they weren't collected. In this case, we may want to signal that the value is missing by returning the underlying null. This is why I left the avg_value_size_in_bytes column as an Integer.

If there is a case where we know we always have a value, then it makes sense to use the primitive. But it may also be better to have a method to check whether the value is known and still use a primitive when we may not have a value. I'm undecided at the moment and I'm waiting for actual uses. I'm also currently writing the evaluator and, while it isn't done, I think we may want to add more methods and keep the types more restrictive.

rdblue · 2026-07-10T21:50:30Z

+
+  private Object getOffset(int offset) {
+    return switch (offset) {
+      case StatsUtil.LOWER_BOUND_OFFSET -> lowerBound();


Uses offsets to avoid a switch by position immediately followed by a switch by enum symbol.

rdblue · 2026-07-10T21:52:12Z


  private static final Types.StructType CONTENT_STATS_TYPE =
-      StatsUtil.contentStatsFor(TABLE_SCHEMA).type().asStructType();
+      StatsUtil.statsReadSchema(TABLE_SCHEMA, ImmutableList.of(1, 2));


The read schema method is easiest to call for testing since it creates full stats structs for each field ID. The write schema method will omit lower and upper bounds for MetricsMode.Counts.

rdblue · 2026-07-10T21:53:16Z

+          10L,
+          3L,
+          null);
+  private static final ContentStatsStruct CONTENT_STATS =


I left this in to continue testing the MetricsUtil conversion, but I suspect that those conversions will be done lazily.

Those are the only place where fieldId is needed so I think we can avoid needing to recover field ID from base ID if we rewrite them.

rdblue · 2026-07-10T21:53:39Z

    assertThat(dataFile.equalityFieldIds()).isNull();
    assertThat(dataFile.columnSizes()).isNull();
-    assertThat(dataFile.valueCounts()).containsOnly(Map.entry(1, 100L), Map.entry(2, 200L));
+    assertThat(dataFile.valueCounts()).containsOnly(Map.entry(1, 100L), Map.entry(2, 100L));


Updated to align with what stats would actually look like.

amogh-jahagirdar · 2026-07-14T15:09:24Z


  /** The total NaN value count */
-  Long nanValueCount();
+  long nanValueCount();


Curious what's driving this, from a spec perspective is there an open question about if these stats should be required? My understanding of the parquet spec is that value counts are required but null/NaN counts are optional

amogh-jahagirdar · 2026-07-14T15:12:46Z

+    return tightBounds;
+  }
+
+  @Override
+  public long valueCount() {
+    return valueCount;


Based on the comment below the change to make this API a long was a conscious one but given the additional change to MetricsUtil that's made wouldn't this NPE in cases where value / null / nan counts are null?

I added a comment about MetricsUtil conversion, but it was in tests so it wasn't obvious. I don't think that we actually want to convert between the two representations and that it was premature to add that logic.

Conversion is expensive. It allocates lots of buffers and copies data into them, but we don't really know what is going to be accessed. I think we should have a simple Map implementation that translates when values are accessed, or just return null for the old APIs.

I think we're seeing another place where you get in trouble when production code is introduced just for testing.

amogh-jahagirdar · 2026-07-14T15:21:17Z

+  /** Container struct type containing tracked field-level stats structs. */
+  Types.StructType type();
+
+  /** Returns a copy, deep-copying all field stats. */


nit: Returns a deep copy of all field stats?

amogh-jahagirdar · 2026-07-14T15:30:32Z

+
+  @Override
+  public <T> T get(int pos, Class<T> javaClass) {
+    return javaClass.cast(idToFieldStats.get(posToId[pos]));


BaseContentStats get used to check pos against the struct size, and it looks like the old code returned null in case the pos was "out of bounds" of the struct. It looks like that was done for Avro because on the write path Avro calls get() for every field in the writer schema. I don't think we're fully plumbed through which is why we don't see any issues but are we handling that case differently in this PR, or we're just deferring that? If I recall correctly we wanted to use this in memory representation regardless of format version (because we theoretically could adapt the existing stats structure to this)

Yes, that behavior deviated from what StructLike instances are intended to do. If the indexes don't match then the reader is using a different schema than the class and we want that to fail. Part of the problem with the old implementation was not implementing the StructLike interface or using the SupportsIndexProjection base class correctly.

This implementation should bring the struct classes in line with what SupportsIndexProjection does.

github-actions Bot added the core label Jul 10, 2026

rdblue commented Jul 10, 2026

View reviewed changes

rdblue force-pushed the fix-content-stats branch from b35b148 to f996392 Compare July 10, 2026 21:56

rdblue commented Jul 10, 2026

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/TrackedFileStruct.java Outdated

rdblue added this to V4: metadata tree Jul 13, 2026

rdblue mentioned this pull request Jul 13, 2026

Core: Use mocks in TrackedFile tests #17133

Merged

rdblue added 6 commits July 13, 2026 15:20

Core: Refactor ContentStats and FieldStats.

1845a0a

Implement filtered stats copy.

a291d02

Fix checkstyle and unnecessary whitespace changes.

0ba1da1

Core: Fix checkstyle imports and misc.

51b6c11

Fix test failure in TestTrackedFile.

f722f12

Core: Add geo and variant tests for field stats structs.

33128e1

rdblue force-pushed the fix-content-stats branch from 297cd9b to 33128e1 Compare July 13, 2026 22:22

Fix new copy tests.

1f81ee6

amogh-jahagirdar self-requested a review July 14, 2026 15:04

amogh-jahagirdar reviewed Jul 14, 2026

View reviewed changes

rdblue mentioned this pull request Jul 14, 2026

API: Extract superclass from InclusiveMetricsEvaluator #17201

Open

Uh oh!

Conversation

rdblue commented Jul 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants