release-notes.md

dataprof 0.12.0 · Profile, then gate

0.12.0 turns dataprof from something you run once on unfamiliar data into a checkpoint that stays in the pipeline. A report can now say what deserves attention (findings()), and a declared policy turns it into a verdict (check(), QualityPolicy, and python -m dataprof.check for CI). The verdict has three values, and inconclusive is a real answer: a sampled or partial scan never passes a claim about the whole source. On a large file read in full, sampled quality scores now carry an interval, and the gate decides on it.

A gate is only as good as the numbers under it, so most of this release is about the numbers. Of more than a hundred commits, 48 are fixes, many found by profiling real files and checking the output against something independent: a CSV of the same values, the previous release, or the planted truth. Several were silent and plausible, which is the worst failure for a profiler: polars and Arrow-backed pandas frames profiled from their first chunk only, Parquet decimal columns reported 10^scale times too large, a small spread on a large offset reported the wrong standard deviation, and a US date column was read day-first value by value. Each is fixed, and the same data now gives the same serialized numbers on every engine and input path.

Numbers move because of that, and some APIs changed shape. The upgrade checklist below says where, and what to re-baseline.

Install

# Python
pip install --upgrade dataprof==0.12.0

# Rust
cargo add dataprof@0.12.0

Wheels cover standard CPython 3.10 to 3.14 on Linux, macOS and Windows, with no Python dependencies. Rust 1.96 remains the minimum supported version. dataprof ships libraries and Python packages; there is no CLI binary.

Highlights

• A quality gate, in Rust, Python and CI (#725, #732). State a policy as data and get pass, fail or inconclusive, with the evidence behind each check. python -m dataprof.check data.csv --min-quality 90 exits 0, 1 or 2.
• Findings (#770). report.findings() lists what deserves attention, each with a stable code, a severity and its evidence, never a raw value, and lists the rules it could not evaluate instead of reading them as clean.
• Gates that decide on large files (#789, #820). When quality was computed over the retained sample, the report carries a 99.9% interval per score, and the gate passes or fails on it. Past a million distinct rows, uniqueness is bounded with certainty from what the exact counts held.
• Column projection (#655). Profile only the columns you name, on every input path; Parquet skips the unselected column chunks entirely.
• Arrow C Stream inputs (#706). profile() accepts PyArrow RecordBatchReader objects and DuckDB relations without collecting a table.
• Reports that record how they were made. quality_status says why a report has no quality assessment (#722), recovery_events records engine fallback (#735), metric_semantics records the definitions in force (#765), and JSON saves carry the complete report in both languages (#759).
• Honest about what it cannot read. Numbers written as 1.234,56 or 10,50 are counted and reported (#807), a CSV that ends inside an open quote is reported (#791), and nested columns report their counts instead of statistics about a display string (#769).

Upgrade checklist

If you rely on…	What changed	What to do
flat Python quality accessors (quality.missing_values_ratio, …)	The 16 accessors deprecated in 0.9 are removed (#733). Reading one raises AttributeError naming its nested replacement.	Read the nested dimension, after checking it is not None: quality.completeness["missing_values_ratio"].
the layout of to_json() / JSON save()	Both write the complete canonical report, the same document Rust serializes: data_source, column_profiles, quality under quality.metrics (#759). to_dict() keeps the flat summary.	Parse to_dict() if you consumed the flat layout. Loaders accept both layouts.
overall_quality_score() / overall_score	A report where nothing was assessed returns None (serialized null), not 0.0 (#629). Rust overall_score() and QualityAssessment::score() return Option<f64>; MetricConfidence gains NotAssessed.	Decide what an unassessable report means for you; branch on assessed_dimensions().
dimension evidence dicts (quality.validity, …)	A dimension that assessed nothing returns None and is omitted from serialized output, instead of ratios computed from zero inputs (#640).	Check for None before reading keys.
variance, std_dev, quartiles.iqr	A spread that overflows f64 is null, not 0.0 or inf (#800, #803). Rust fields are Option<f64>.	Treat null as "too large to represent", not "no spread".
text lengths	min_length, max_length, avg_length count Unicode scalar values, not UTF-8 bytes (#641): "東京" is 2, not 6. ASCII is unchanged. Saved reports record this as metric_semantics.text_length_unit (#765).	Re-profile non-ASCII text rather than comparing across the 0.11/0.12 boundary.
struct, list and map columns	Typed nested (DataType::Nested) with counts only; no length statistics, distinct count or patterns (#769).	Add a Nested arm to exhaustive matches; compare nested columns within a release.
text most_frequent / least_frequent (Rust)	Newly computed profiles leave them None on every path (#758).	Compute them explicitly with dataprof_metrics::stats::text::{calculate_most_frequent, calculate_least_frequent}.
Rust struct literals	QualityAssessment is built with new, exact or approximate (#759). Public structs gain public fields: ProfileReport gains quality_status (#722) and metric_semantics (#765); ExecutionMetadata gains recovery_events (#735), sampled_row_ranges (#721) and unterminated_quote (#791); ColumnProfile gains locale_number_count (#807) and unique_count_lower_bound (#820); DateTimeStats gains slash_date_order (#817); RowDuplicateSummary gains max_duplicate_rows (#820). Lower-level BifurcatedResult and ColumnProfileInput gain score_bounds and unique_count_lower_bound.	Use the constructors (ProfileReport::new, ExecutionMetadata::new), or add the new fields to literals.
RobustCsvParser::parse_csv() / parse_csv_with_recovery() (Rust)	Return CsvParseOutput { headers, records, recovery_events } instead of a tuple (#735).	Read the named fields.
Python interpreters	Wheels are published for standard, GIL-enabled CPython 3.10 to 3.14 only (#707). PyPy, preview and free-threaded wheels are no longer built.	Use a supported CPython.
quoted policy thresholds	check() and findings() reject strings and booleans as thresholds; a quoted number in a policy file is an input error (#772).	Write thresholds as numbers.
hand-edited flat documents	from_dict() rejects a malformed quality.score_weights instead of substituting defaults (#764).	Fix the document.

Numbers that move

Re-baseline stored reports and gates that read these.

• Rows and everything after them, on chunked DataFrames (#662). Multi-chunk polars frames and Arrow-backed pandas frames profiled their first chunk only.
• Parquet and Arrow decimals (#828): statistics were 10^scale times too large, and decimal256 columns had no statistics and a fabricated distinct count.
• Variance and standard deviation (#677, #806): values on a large offset (IDs, epoch timestamps, 1e12 + x) lost precision; the mean of values whose sum overflows is now reported (#804).
• Distinct counts are exact up to a million values on every engine, and an estimate never exceeds the values seen (#786, #779). Duplicate rows are exact up to a million distinct rows, including JSON records whose keys appear late (#658).
• Timeliness: start/end pairs are compared only within the same row (#792), and a US date column is read month-first as a whole (#817). Offsets in RFC 3339 timestamps normalize to UTC (#660), and timestamps in named time zones profile instead of failing (#669).
• Patterns: ten-digit IDs and epoch seconds are no longer reported as US phone numbers or flagged sensitive (#818).
• Whitespace-padded numbers enter numeric statistics (#628); columns with no parsed value report no statistics rather than zeros (#682).
• Capped Parquet profiles (max_rows) sample 32 ranges across the file instead of its prefix (#721), recorded in execution.sampled_row_ranges.
• Empty inputs report an empty quality assessment on every path (#724).
• In-memory profiles of the same data are deterministic (#774).
• Parquet schema inference (infer_schema(), analyze_structure()) types text columns from their values, as profile() does (#699).

Known limitations

Shipping with these; each has an issue.

• Common null markers are values (#813). Only empty, null and nan are null, so NA, N/A, #N/A, \N and None count as values and completeness can read 100% on data that is mostly missing.
• Zero-padded codes are numbers (#814). 00123 and 20240115 are typed integer and averaged.
• Locale-formatted numbers are reported, not parsed (#433). locale= still tunes pattern detection only. 10,50 can also match the Geographic Coordinates pattern (#808), and amounts with a currency or percent sign are not counted (#809).
• Very large sources can stay inconclusive (#830). The certain uniqueness bound widens with row count; past about ten million rows a high min_quality_score may not be decidable.
• Sensitive-data coverage is narrow (#790): two national IDs and two phone formats. International and E.164 phone numbers are not detected.
• --baseline is not supported yet (#749); the flag exits 2.
• Database connectors need a source build (#588); they are not in the wheel.
• Durations report statistics in their storage unit, which the report does not name (#823).

Detailed changes

The generated changelog lists every commit. This section groups the user-visible ones.

Quality gate and findings

• ProfileReport.check() (Python) and QualityPolicy (Rust) evaluate minimum quality and dimension scores, per-column null limits, a duplicate-row limit and required metrics, with scope="full_source" (default) or "observed". Each check records what it expected, what it observed, the evidence and why it could not be evaluated when it could not (#725).
• python -m dataprof.check wraps the gate for CI: JSON policy files and threshold flags, --json to stdout, human summaries to stderr, exit codes 0/1/2 (#732). See the CI entrypoint guide.
• findings() reports all_null, null_heavy, mixed_types, duplicate_rows, future_dates, temporal_order_violations, ragged_rows, unterminated_quote, records_skipped, locale_numbers, constant_column, sensitive_pattern and partial_scan, identically in Rust and Python (#770, #791, #807).
• quality.score_bounds / report.quality_score_bounds: a 99.9% interval per sampled score, and the gate's decision rule on it (#789). Uniqueness gets a certain interval past a million distinct rows (#820).

Correctness

• One stable numeric accumulator for every path: Welford on shifted values and a compensated sum (#677, #806); overflow reported as null (#800, #803); the mean of an overflowing sum (#804).
• Exact distinct counts to a million on every engine, using 64-bit fingerprints; exact sets spill under memory pressure on the incremental, columnar and Arrow batch paths, and the columnar engine honours memory_limit_mb (#786, #792).
• Accumulator merge produces the single-pass profile, including one-sided columns, completeness and reservoir weighting (#650).
• Parquet columns are profiled by value, not physical encoding (dictionary, run-end and view types), and an unreadable column is an error rather than a fabricated full-cardinality profile (#661).
• Memory-mapped CSV chunks end on record boundaries; headerless configuration is honoured by both CSV engines (#628). CSV decoding stops at the row cap (#755), and max_bytes stops at the first record past the budget (#776).
• Slash dates are read in one day/month order per column, recorded as DateTimeStats::slash_date_order (#817).

Python

• Every chunk of a pandas or polars DataFrame is profiled (#662); zero-row Arrow sources profile over their declared columns (#666).
• Arrow C Stream producers, including DuckDB relations, are accepted (#706).
• Native and restored reports share one set of read-only views (#746), and the bindings are split into private modules behind a small facade (#705).
• The async database helpers are coroutine functions (#778); engine aliases are documented (#680); progress callbacks hear from every file route (#777).

Removed flat quality accessors

The 16 flat DataQualityMetrics accessors deprecated in 0.9 are gone (#733). Reading one raises AttributeError naming its replacement; the names no longer appear in dir() or the type stubs. Read the nested dimension only after checking it is not None, since an unassessed dimension must not become a fabricated zero or perfect score:

quality = report.quality
completeness = quality.completeness if quality is not None else None
missing_ratio = completeness["missing_values_ratio"] if completeness is not None else None

Dimension	Removed flat names (replacement keys)
completeness	missing_values_ratio, complete_records_ratio, null_columns
consistency	data_type_consistency, format_violations, encoding_issues
uniqueness	duplicate_rows, key_uniqueness, high_cardinality_warning
accuracy	outlier_ratio, range_violations, negative_values_in_positive
timeliness	future_dates_count, stale_data_ratio, temporal_violations, invalid_date_values

Nested values, report serialization and the schema version are unchanged, and saved reports still load through the nested API.

Reports and schema

• JSON saves write the canonical report; to_dict() stays the flat summary (#759). The report schema stays v1: every new field is optional.
• New fields: quality_status (#722), execution.recovery_events (#735), metric_semantics (#765), execution.sampled_row_ranges (#721), execution.unterminated_quote (#791), quality.score_bounds (#789), locale_number_count (#807), DateTimeStats::slash_date_order (#817), and the nested data type (#769). See the schema guide.
• The cross-engine numeric equality contract is scoped to serialized, rounded metrics and enforced by comparing complete serialized column documents (#708).

Structure, Parquet and databases

• analyze_structure() reports whole-file counters for Parquet from the footer (#702), types text columns from their values (#699), and refuses duplicate column names (#701).
• Database connectors decode temporal, decimal, UUID and unsigned columns (#642) and MySQL TIME as a time of day (#646).
• AgentGuard.llm_context() redacts host paths (#703).

Performance and benchmarks

• Peak-memory sampling is throttled, so small chunk sizes no longer pay a process-table walk per chunk (#775); the columnar engine reads its ragged-row count from arrow-csv instead of pre-scanning (#754).
• The benchmark suites share one home, one CI artifact and one page, with a locked four-tool comparison (dataprof, pandas, polars, ydata-profiling), energy and peak-memory collection, and Python/Arrow boundary costs (#736, #747, #748, #750). The page reports what each run measured and does not claim a general ranking.