Python API Guide

Complete reference for the dataprof Python package (v0.12.0).

For upgrade-sensitive changes to sampling, execution controls, quality scores, parser behavior, semantic hints, and exception types, read the 0.12.0 release notes and migration guide.

The Python API is built for quick inspection and follow-up analysis: point it at a file, DataFrame, Arrow batch, ad-hoc notebook data, or database query and get back a report you can slice, export, and wire into notebooks or checks.

Installation

uv pip install dataprof
# or
pip install dataprof

Supports standard, GIL-enabled CPython 3.10–3.14. PyPy, preview Python and free-threaded builds are outside the supported wheel set; see the interpreter policy. The package ships pre-built wheels for Linux, macOS, and Windows, and declares no Python dependencies. The base API needs nothing else: local file profiling, DataFrame and Arrow inputs, ad-hoc dict/bytes inputs, and report exports. Install the pandas extra only for pandas-typed exports (to_dataframe(), describe() as a DataFrame).

The wheel also carries the async API: dataprof.asyncio, HTTP URL profiling, and remote Parquet all work on a bare pip install dataprof.

Database profiling is the one documented feature the wheel does not contain. Connectors are a compile-time feature rather than a Python dependency, so no extra can install them; they need a build from source, either with pip from the published sdist or from a checkout:

pip install dataprof --no-binary dataprof   --config-settings="build-args=--features python,python-async,async-streaming,parquet-async,database,sqlite"

The feature list has to be complete, not just the extra parts. See Database Profiling for why, and for the checkout equivalent.

Inspect the current installation without importing optional packages or trying a network/database operation:

import dataprof as dp

features = dp.capabilities()
print(features)

if features.database and "sqlite" in features.database_connectors:
    # Database helpers are available in this build.
    ...

if features.pandas_interop and features.pandas_installed:
    # Both compiled interoperability and the optional Python package are present.
    ...

Quick Start

import dataprof as dp

# Profile a file
report = dp.profile("data.csv")
print(f"{report.rows} rows, {report.columns} columns")
print(f"Quality score: {report.quality_score}")

# Access columns directly
col = report["age"]
print(f"mean={col.mean}, nulls={col.null_percentage}%")

# Profile a pandas DataFrame
import pandas as pd
df = pd.read_csv("data.csv")
report = dp.profile(df)

# Profile ad-hoc notebook data
report = dp.profile({"age": [31, 42, 29], "city": ["Rome", "Milan", "Rome"]})
report = dp.profile([{"age": 31, "city": "Rome"}, {"age": 42, "city": "Milan"}])
report = dp.profile(b"age,city\n31,Rome\n", format="csv")

# Profile a PyArrow table
import pyarrow.parquet as pq
table = pq.read_table("data.parquet")
report = dp.profile(table)

`profile()` -- Primary Entry Point

dp.profile(
    source,                          # str, Path, DataFrame, Arrow, dict, rows, or bytes
    *,
    engine="auto",                   # "auto", "incremental", "columnar"
    chunk_size=None,                 # int -- bytes per streaming chunk
    memory_limit_mb=None,            # int -- memory cap
    format=None,                     # str -- file format override
    max_rows=None,                   # int -- row cap; Parquet samples across the file
    name=None,                       # str -- label for DataFrame/Arrow sources
    csv_delimiter=None,              # str -- override auto-detection (e.g. ";")
    csv_flexible=None,               # bool -- allow variable column counts
    sampling=None,                   # SamplingStrategy
    stop_condition=None,             # StopCondition
    on_progress=None,                # Callable[[ProgressEvent], None]
    progress_interval_ms=None,       # int -- ms between progress events
    metrics=None,                    # list[str] -- "schema", "statistics", "patterns", "quality"
    quality_dimensions=None,         # list[str] -- subset of dimensions to compute
    columns=None,                    # list[str] -- profile only these columns
    locale=None,                     # str -- "CA"|"DE"|"FR"|"GB"|"IT"|"US"
    positive_columns=None,           # list[str] -- columns expected to be non-negative
    identifier_columns=None,         # list[str] -- semantic IDs, not measures
    temporal_columns=None,           # list[str] -- columns assessed for timeliness
) -> ProfileReport

For file profiling, engine accepts the case-insensitive names auto, incremental (compatibility alias: streaming), and columnar (compatibility alias: arrow). This applies to dp.profile(..., engine=...), dp.profile_file(..., engine=...), and dp.Profiler().engine(...).profile(...). CSV reports use the selected canonical name, incremental or columnar, in report.engine and report.to_dict()["execution"]["engine"]; aliases are never emitted, and auto reports the engine it selected.

Source types:

Type	Description
`str` or `Path`	File path (CSV, JSON, JSONL, Parquet)
pandas `DataFrame`	In-memory DataFrame
polars `DataFrame`	In-memory Polars DataFrame
PyArrow `Table` or `RecordBatch`	Zero-copy via PyCapsule interface
Arrow `RecordBatchReader` / `__arrow_c_stream__` producer	One-shot, incremental batch profiling; includes DuckDB relations
`dict[str, list]`	Columns of cells; profiled natively, no dependencies
`list[dict]`	Row-oriented notebook data; rows may omit keys, which read as nulls
`bytes` or `io.BytesIO`	In-memory file contents; requires `format="csv"`, `"json"`, `"jsonl"`, or `"parquet"`

Dict, row-dict, and byte inputs are profiled by the Rust core directly, so they need no third-party package. That includes format="parquet": byte buffers go through the same compiled Arrow reader as Parquet files, so the two report the same types, column order, and statistics, and capabilities().local_parquet predicts both.

A cell is missing when it is None, NaN, or a null-like token -- the same rule the CSV, JSON, Parquet and Arrow paths use. The tokens are empty or whitespace, "null" and "nan" in any case, and exactly "NA", "N/A", "n/a", "#N/A", "\N" and "None"; metric_semantics["null_tokens"] records this vocabulary. Note that a dict is not round-tripped through pandas, so an integer column containing a null stays integer rather than being widened to float.

Dicts, row lists, pandas, polars and Arrow inputs apply StopCondition.max_rows and row-only |/& compositions. They accept StopCondition.never() and SamplingStrategy.none() as no-ops. Other stop conditions and sampling strategies raise ValueError because those routes cannot apply them. max_rows and stop_condition remain mutually exclusive.

StopCondition.confidence_threshold(f) means the fraction of the estimated record count read. CSV files and materialized async CSV buffers use an exact logical-record count from a bounded-memory scan before profiling. This preflight also supplies progress percentages; scans without fraction stops or progress skip it. The preflight reads the whole CSV before a profiling row or byte cap takes effect; bytes_consumed describes the profiling pass, excluding this preflight. A true async stream with a size hint estimates its total from complete records and bytes already read, excluding CSV header bytes; changing row widths can bias that estimate. For a true stream without a size hint the condition is inert. Schema stability starts after the first chunk establishes a baseline, and each leaf in an |/& tree retains its own threshold.

Column order follows the source on every input and transport: the CSV header, the Parquet/Arrow schema, the dict or DataFrame key order, and for JSON/JSONL the field order of the first record, with fields that only appear in later records appended where they were first seen. Converting a dataset between formats therefore does not reshuffle the report.

Column projection narrows a profile to the columns you name. columns takes a list of source column names and is accepted by dp.profile(), dp.profile_file(), dp.Profiler().columns([...]), and every dataprof.asyncio entry point — profile_file(), profile_bytes() and profile_url() forward it through ProfilerConfig, which also carries it into analyze_database_async():

report = dp.profile("orders.csv", columns=["amount_eur", "city"])
list(report)        # ['city', 'amount_eur']
report.rows         # every row is still read

Projection selects columns, never rows: report.rows is unchanged, and each retained column reports exactly the values it reports in an unprojected profile of the same source — same types, statistics, patterns, and lengths. It applies identically on every input and transport (CSV, JSON, JSONL, Parquet, bytes, dicts, row dicts, DataFrames, and Arrow) and on both engines.

The projected report keeps source order, not the order you listed the names in, so a projection is a filter on the report rather than a reordering of it.

Names are validated rather than silently ignored. A name absent from the source raises ValueError listing the unmatched names, and so does a repeated name. columns=[] is a projection onto no columns, which is distinct from columns=None (no projection): it reports columns == 0 while rows still counts the rows that were read.

Quality is assessed over the projected columns only, so quality_score generally differs from the score for the whole dataset. Two dimensions are withheld rather than narrowed: completeness and uniqueness both report row-level measurements (complete_records_ratio, total_cells, duplicate_rows, rows_checked) that mean something different once a row has been projected, and the report schema cannot mark only those fields as projected. Rather than publish plausible numbers under full-row names, a projected report leaves both dimensions None and out of assessed_dimensions():

dp.profile("orders.csv").quality.completeness
# {'complete_records_ratio': 73.8, 'total_cells': 5000, ...}

dp.profile("orders.csv", columns=["amount_eur", "city"]).quality.completeness
# None

The overall score is therefore an average over fewer dimensions, and is often higher than the unprojected score even for the same columns — dropping completeness drops the dimension that penalizes nulls. Compare projected scores against other projected scores, not against whole-dataset ones.

If a projection leaves no assessable dimension — quality_dimensions selecting only completeness and/or uniqueness — the report carries no quality object at all: report.quality is None and so is quality_score, which is the "not analyzed" answer rather than a zero.

Semantic hints are resolved against the projection too — a positive_columns hint naming a column that projection excluded raises, since that column is not in the report it would describe.

Synchronous byte inputs use the in-memory columnar path. They support max_rows, metric/quality selection, semantic hints, CSV delimiters, and JSONL error policy, but reject streaming-only controls (chunk_size, memory_limit_mb, stop_condition, progress callbacks, and flexible CSV recovery) instead of silently ignoring them. For those controls, use dataprof.asyncio.profile_bytes(). JSON and JSONL byte buffers follow RFC 8259: the non-standard NaN and Infinity constants are malformed input.

JSON record policy. Only JSON objects are profileable records — they are the only JSON value with named fields to become columns. A record that is valid JSON but not an object (a scalar, an array, null) is never silently dropped: with jsonl_on_error="skip" (the default) it is counted in error_count and the records after it still profile, and with jsonl_on_error="strict" the first one raises ValueError naming its position and JSON kind. Input whose every record is non-object fails, exactly as all-malformed input does. File, byte, and async byte transports apply this identically.

Records with no fields. A JSON object with no fields ({}) is a record: it was read and analysed, and nothing was found in it. It profiles as a row against zero columns, so [{}, {}] reports rows == 2, columns == 0 on every transport — file, bytes, async bytes, URL, and a Python list of dicts. That keeps three shapes distinct:

Shape	Meaning	Example
`rows > 0`, `columns == 0`	rows were read; none of them had fields	`[{}, {}]`
`rows == 0`, `columns == 0`	the input held no records	`[]`
`rows == 0`, `columns > 0`	a known schema with no rows under it	a CSV header line

A zero-column report is a normal report: it serialises, round-trips, and renders. len(report) counts columns, so it is 0 while report.rows is not. A record with no fields is well-formed, so it is clean under both error policies and never counted in error_count.

Engine options:

Engine	When to use
`"auto"`	Let dataprof choose based on file size and format (recommended)
`"incremental"`	True streaming with bounded memory -- large files, streams
`"columnar"`	Arrow-based batch processing -- Parquet, in-memory data

`ProfileReport`

Returned by profile() and all analysis functions.

Properties:

Property	Type	Description
`source`	`str`	Source identifier (file path, table name, etc.)
`source_type`	`str`	`"file"`, `"bytes"`, `"query"`, `"dataframe"`, `"stream"`
`engine`	`str \| None`	Engine or parser that produced the report
`rows`	`int`	Number of rows processed
`columns`	`int`	Number of columns detected
`column_profiles`	`dict[str, ColumnProfile]`	Per-column statistics (by name)
`quality_score`	`float \| None`	Overall quality score (0--100)
`quality`	`DataQualityMetrics \| None`	Detailed quality breakdown
`quality_status`	`str`	Why `quality` is or is not there (see below)
`quality_error`	`str \| None`	The error a failed quality computation reported
`quality_sampled_dimensions`	`list[str] \| None`	Metric components computed from a retained sample rather than every scanned row; `None` when there is no assessment or a loaded document does not record it
`quality_score_bounds`	`dict \| None`	Where the scores over every scanned row lie when some were computed over a retained sample: `confidence_level`, `overall_score` and `dimension_scores`, each interval `{"lower", "upper"}` or `None` when unbounded. `None` when every score is exact (see quality gates)
`metric_semantics`	`dict[str, str] \| None`	How the measurements were defined, e.g. `{"text_length_unit": "unicode_scalar", "null_tokens": "common_markers", "pattern_match_count": "validated", "shape_moments": "adjusted_sample"}`; `None` for a report written before 0.12, whose definitions are unknown. A 0.12 report records only `text_length_unit`: it counted only empty, `null` and `nan` as missing, counted every regex match as a pattern match, and used another skewness and kurtosis estimator
`execution_time_ms`	`int`	Total processing time
`throughput`	`float \| None`	Rows per second
`memory_peak_mb`	`float \| None`	Peak memory usage
`truncation_reason`	`str \| None`	Why processing stopped early
`source_exhausted`	`bool`	Whether the entire source was read
`ragged_row_count`	`int`	Rows whose field count differed from the header (`0` = no ragged rows). Reported for file, async and columnar CSV inputs
`unterminated_quote`	`bool \| None`	`True` when a CSV source ended inside a quoted field: a quote was never closed, so the last record holds every row after it. `None` for non-CSV input and for a scan stopped before the end of the source. `csv_flexible=False` refuses such a source, and CSV bytes passed to `profile()` always do
`recovery_events`	`list[dict[str, str]] \| None`	Ordered failed attempts and retries; `[]` means no recovery, `None` means history was not recorded
`sampling_applied`	`bool`	Whether sampling was used
`sampling_ratio`	`float \| None`	Fraction of data sampled
`sampled_row_ranges`	`list[list[int]] \| None`	Exact zero-based, half-open source row intervals when recorded; `[]` means zero selected rows

For Parquet files, byte buffers and HTTP URLs, max_rows=N selects up to 32 contiguous ranges spread across the file, with exactly N rows when the file is larger than the cap. The first and last rows are included for N ≥ 2; N = 1 selects the middle row and N = 0 selects none. Selection is deterministic and independent of row-group sizes, batch sizes and transport. The exact ranges appear in sampled_row_ranges and to_dict()["execution"]["sampled_row_ranges"]. sampling_applied is true, sampling_ratio is selected rows / file rows, and source_exhausted is false with the usual max_rows(N) truncation reason. A cap at or above the file row count reads the whole file without sampling.

Statistics and row-level quality metrics describe the selected population. Duplicate detection includes matches across separate ranges. This deterministic sample does not establish full-source uniqueness or provide an unbiased estimate of full-source ratios. Before 0.12, local and byte-buffer Parquet caps selected a prefix, while HTTP reads ignored the cap. Older reports have no recorded ranges and load with sampled_row_ranges=None. Re-baseline capped Parquet comparisons when upgrading.

What happened to the quality computation:

quality is None answers two different questions, so every report carries quality_status. It is computed exactly when quality holds an assessment; the other five states are the reasons it does not:

`quality_status`	meaning
`computed`	quality was computed; `quality` holds the assessment
`not_requested`	the quality pack was deselected (`metrics=["schema"]`)
`no_data`	requested, but no quality sample was supplied to the assembler
`withheld_by_projection`	requested, but every requested dimension measures whole rows and `columns=` selected a subset
`failed`	requested and attempted; the computation failed, and `quality_error` says how
`unrecorded`	loaded from a report saved before 0.12 that carried no assessment

An analyzed empty source reports computed with an empty assessment and quality_score=None, whether or not it declares columns. This is distinct from an unavailable quality sample (no_data).

A gate deciding on a report should branch on this before reading quality_score: a failed computation and a run that never asked for quality both leave the score None, and they are not the same verdict.

report = dataprof.profile("data.csv")
if report.quality_status != "computed":
    raise SystemExit(f"no quality to gate on: {report.quality_status}")

The same object is in to_dict()["quality_status"], as {"state": ...} plus an "error" key on failed only.

Dict-like column access:

col = report["column_name"]          # -> ColumnProfile
"column_name" in report              # -> True/False
for name in report: print(name)      # iterate column names
len(report)                          # number of columns

Export methods:

report.to_dict()                  # historical flat summary (rounded values)
report.to_json(indent=2)         # complete canonical Rust report as JSON
report.to_dataframe()            # pandas DataFrame -- all stats (requires pandas)
report.to_polars()               # polars DataFrame -- all stats (requires polars)
report.to_arrow()                # PyArrow Table -- all stats (requires pyarrow)
report.describe()                # transposed summary like pandas describe()
report.quality_summary()         # single-row dict for quality tracking
report.to_html()                 # embeddable HTML (same as the notebook display)
report.to_markdown()             # GitHub-flavored markdown table
report.compare(other)            # dict of quality/schema/null deltas vs another report
report.save("report.json")      # save to JSON
report.save("report.csv")       # save column profiles to CSV
report.save("report.parquet")   # save column profiles to Parquet (requires pyarrow)

# Round-trip a saved report without re-profiling (read-only view)
reloaded = dp.ProfileReport.load("report.json")   # from a saved .json file
reloaded = dp.ProfileReport.from_json(report.to_json())  # from a JSON string
reloaded = dp.ProfileReport.from_dict(report.to_dict())  # from a dict

Rounding

All floating-point values in exported data are rounded. The precision is chosen by what the number is, not by which object it lives on:

Kind	Precision	Examples
`0..100` percentage	2dp	`null_percentage`, `coefficient_of_variation`, every float in the quality dimension dicts
statistic	4dp	`mean`, `std_dev`, `variance`, `skewness`, `kurtosis`, `avg_length`
data value	4dp	`min`, `max`, `median`, `mode`
`0..1` ratio	4dp	`uniqueness_ratio`, `true_ratio`

A 0..1 ratio takes 4dp so it carries the same resolution as the equivalent percentage at 2dp. Data values take 4dp because a profiler must not report a min the column never contained. Quartiles are the deliberate exception: they stay at 2dp, being distribution landmarks rather than exact values.

Ties round the stored float, away from zero. This is the value actually held, not the shortest decimal string that prints for it -- 23 / 4000 * 100 prints as 0.575 but is stored just below it, and so rounds to 0.57.

The Rust and Python layers implement the same convention and are held to shared fixtures (tests/fixtures/rounding_parity.json and report_rounding_parity.json), so the same data profiled through either gives the same numbers. Raw property access on ColumnProfile returns unrounded Rust values; use the export methods for rounded output. Cross-engine numeric equality is defined on those serialized metrics, with no extra tolerance. Native attributes may differ in their final digits with accumulation order; loading a report retains the precision that was saved. Compare the same data, schema semantics, analysis options and analyzed population, and exclude source and execution provenance such as timing and memory from equality checks. See the numeric contract for scope, sampling and absence semantics.

Round-trip fidelity: a report reloaded with from_dict, from_json, or load reports the same values as the report it was saved from, at the precision above. Live and reloaded reports use the same read-only column, pattern, and quality accessors. They read native values or the saved document respectively; reloading does not recompute metrics or invent values for missing fields. Both expose dataprof.ColumnProfile and dataprof.DataQualityMetrics objects, so type checks and accessor behavior no longer depend on how a report was loaded.

`ColumnProfile`

Per-column profiling statistics.

Field	Type	Description
`name`	`str`	Column name
`data_type`	`str`	Inferred type: `"string"`, `"identifier"`, `"integer"`, `"float"`, `"date"`, `"boolean"`, `"nested"`. A `nested` column (struct, list or map) reports its counts only: `unique_count`, `type_homogeneity`, `locale_number_count`, `stats` and `patterns` are `None`
`total_count`	`int`	Total number of values
`null_count`	`int`	Number of null/missing values
`unique_count`	`int \| None`	Distinct value count
`invalid_count`	`int \| None`	Non-null values that did not parse as a finite number (parse failures and non-finite tokens like `inf`/`NaN`) and are excluded from the statistics. `None` = check did not run (non-numeric column, or statistics skipped); `0` = every non-null value parsed
`type_homogeneity`	`dict[str, int] \| None`	Non-null values counted by lexical class: `{"numeric", "date", "boolean", "text"}`. Tells a `string` column of ordinary text from one that defeated type inference, which `data_type` cannot. All four keys are always present; all-zero = classified with nothing to classify (all-null or no rows); `None` = the classification did not run. Counted over the values the profiler retained, so sum them against `total_count - null_count` to tell an exact count from one bounded by the 10k reservoir sample
`locale_number_count`	`int \| None`	Of the `text` values in `type_homogeneity`, how many are numbers written with a decimal comma or digit-group separators (`10,50`, `1.234,56`, `1,234.56`, `1'234.56`, `1 234,56`). dataprof does not parse them as numbers, so they are in no numeric statistic: a column of them is typed `string`, and a numeric column counts them in `invalid_count`. `None` = the count did not run (a `nested` column, or a report saved before 0.12); `0` = none found
`null_percentage`	`float`	Null ratio (0.0--100.0)
`uniqueness_ratio`	`float`	Unique values / total values
`min`	`float \| None`	Minimum (numeric columns)
`max`	`float \| None`	Maximum (numeric columns)
`mean`	`float \| None`	Mean (numeric columns)
`std_dev`	`float \| None`	Standard deviation
`variance`	`float \| None`	Variance
`median`	`float \| None`	Median
`mode`	`float \| None`	Mode
`skewness`	`float \| None`	Skewness
`kurtosis`	`float \| None`	Kurtosis
`coefficient_of_variation`	`float \| None`	CV
`quartiles`	`dict \| None`	`{"q1", "q2", "q3", "iqr"}`; `iqr` is `None` when `q3 - q1` overflows `f64`
`is_approximate`	`bool \| None`	Whether stats were estimated from a sample
`min_length`	`int \| None`	Shortest value, in Unicode scalar values (0.11: UTF-8 bytes)
`max_length`	`int \| None`	Longest value, in Unicode scalar values (0.11: UTF-8 bytes)
`avg_length`	`float \| None`	Mean length, in Unicode scalar values (0.11: UTF-8 bytes)
`true_count`	`int \| None`	Number of `True` values (boolean columns)
`false_count`	`int \| None`	Number of `False` values (boolean columns)
`true_ratio`	`float \| None`	Ratio of `True` values (0.0--1.0)
`patterns`	`list[Pattern] \| None`	List of detected value patterns

Changed in 0.12 (#667). Numeric columns with no finite parsed values and boolean columns with no parsed booleans have no statistics block. Their statistic accessors return None, including boolean counts and true_ratio. The declared column type, row/null counts, and numeric invalid_count remain available. A measured numeric zero or an all-false column still reports 0.0. CSV columns containing only nulls infer as text because CSV carries no declared type.

This uses the existing absent-statistics representation (ColumnStats::None in Rust); numeric fields do not become optional within NumericStats, and the report schema version is unchanged. Saved reports remain readable as written: loading an older report does not replace its recorded zeros. Re-profile the source to obtain the corrected absence semantics.

Changed in 0.12. Text lengths count Unicode scalar values, not UTF-8 bytes and not grapheme clusters. "東京" has a length of 2 and "🙂" a length of 1. In 0.11 and earlier the same values reported 6 and 4, because every accumulator used Rust's byte-valued str::len() under a field named length (#627). ASCII text is unaffected, and reports written by different releases are not comparable for non-ASCII text.

A combining sequence counts each scalar, so the decomposed spelling of e-acute (e followed by U+0301) has a length of 2 while the precomposed spelling (U+00E9) has a length of 1. The two render identically and profile differently. Segmenting grapheme clusters instead would need a Unicode segmentation table and a policy for which version of it; encoded size is a property of the encoding rather than of the value, and is reported at the source level instead.

`Pattern`

Represents a pattern found in a column. Matches pass both the regular expression and the semantic validator, when the detector has one. The counts and percentages describe the analyzed non-null values, which may be a sample of the column.

Field	Type	Description
`name`	`str`	Name of the pattern (e.g. `"Email"`, `"URL"`)
`regex`	`str`	Regular expression used for matching
`match_count`	`int`	Number of analyzed non-null values passing validation
`match_percentage`	`float`	Percentage of analyzed non-null values passing validation (0.0--100.0)

Configuration

`ProfilerConfig`

Reusable configuration object:

config = dp.ProfilerConfig(
    engine="incremental",
    chunk_size=65536,                # bytes per chunk
    memory_limit_mb=512,
    max_rows=100000,
    csv_delimiter=";",
    quality_dimensions=["completeness", "uniqueness"],
    positive_columns=["pressure"],
    identifier_columns=["order_id", "customer_id"],
)

`SamplingStrategy`

Controls how data is sampled during profiling:

from dataprof import SamplingStrategy

SamplingStrategy.none()                              # process everything
SamplingStrategy.random(size=10000)                  # uniform sample of 10000 rows
SamplingStrategy.reservoir(size=10000)               # same guarantee, Algorithm R
SamplingStrategy.systematic(interval=10)             # every Nth row

Sampling applies to CSV sources on the auto and incremental engines and to every dataprof.asyncio entry point. The columnar engine and the JSON/Parquet readers cannot sample row by row and raise ValueError rather than silently returning a full profile.

random and reservoir give the same guarantee — a uniform sample of exactly size rows — and both hold size rows in memory, because which rows belong in the sample is not settled until the source ends. The other strategies decide each row as it arrives and add no memory. Sampling bounds the cost of analysis, not of reading: use a StopCondition to stop early.

`StopCondition`

Composable early-termination conditions. Combine with | (any) or & (all):

from dataprof import StopCondition

# Stop after 10k rows or 50 MB
stop = StopCondition.max_rows(10000) | StopCondition.max_bytes(50_000_000)

# Stop when schema stabilizes AND confidence exceeds 95%
stop = StopCondition.schema_stable(500) & StopCondition.confidence_threshold(0.95)

# Built-in presets
StopCondition.schema_inference()   # fast schema-only mode
StopCondition.quality_sample()     # enough rows for quality assessment
StopCondition.never()              # process everything (default)

report = dp.profile("huge.csv", stop_condition=stop)

`ProgressEvent`

Track progress with a callback:

def on_progress(event):
    if event.percentage is not None:
        print(f"{event.percentage:.1f}% ({event.rows_processed} rows)")

report = dp.profile("data.csv", on_progress=on_progress)

Every file route reports at least a started and a finished event. A CSV on the default or incremental engine also reports schema_detected and chunk_processed events while it reads; the columnar engine, JSON and Parquet read without reporting, so they send the two bracketing events only.

Event fields: kind, rows_processed, bytes_consumed, elapsed_ms, processing_speed, percentage, column_names, total_rows, total_bytes, truncated, message, estimated_total_rows, estimated_total_bytes.

`DataQualityMetrics`

Quality metrics informed by ISO 8000 and ISO/IEC 25012, accessible via report.quality. The aggregate score is dataprof's formula, not an ISO-defined or certified score:

q = report.quality

# Overall score. None when no dimension was assessable — a header-only file, or
# one whose every dimension had a zero denominator. `assessed_dimensions()` is
# empty for exactly those reports, and `report.quality_score` agrees.
print(q.overall_quality_score())
print(q.assessed_dimensions())
print(q.score_weights)  # relative weights used by dataprof's aggregate formula

# Nested dimension evidence (None when the dimension was not assessed)
print(q.completeness)    # {"missing_values_ratio": ..., "complete_records_ratio": ..., "null_columns": [...]}
print(q.consistency)     # {"data_type_consistency": ..., "format_violations": ..., "encoding_issues": ...}
print(q.uniqueness)      # {"duplicate_rows": ..., "key_uniqueness": ..., "high_cardinality_warning": ...}
print(q.accuracy)        # {"outlier_ratio": ..., "range_violations": ..., "negative_values_in_positive": ...}
print(q.timeliness)      # {"future_dates_count": ..., "stale_data_ratio": ..., "temporal_violations": ...}
print(q.validity)        # {"valid_values_ratio": ..., "invalid_values": ..., "values_checked": ...}
print(q.precision)       # {"decimal_places_consistency": ..., "inconsistent_precision_values": ..., "numeric_values_checked": ...}

A dimension's evidence is present only when it assessed something. None covers both "not computed" and "computed with nothing to divide by": no cells, no non-null values, no numeric values, no dates, no confidently detected pattern. The dimensions holding evidence are exactly the ones whose dimension_scores() entry is a number.

assessed_dimensions() is that set narrowed once more, to the dimensions carrying weight in the overall score. The two coincide under the default weights; a dimension given a weight of 0.0 still holds evidence and a score while sitting outside assessed_dimensions(), because it contributes nothing to the aggregate. Branch on dimension_scores() to ask "was this measured", and on assessed_dimensions() to ask "is this behind the overall score".

That distinction is the point rather than a technicality. validity reporting valid_values_ratio: 100.0 from values_checked: 0 reads as a clean bill of health for something nobody looked at, and any file without a pattern-bearing column used to report exactly that. Serialized reports follow the same rule: an unassessed dimension has no key, so a stored report cannot be read as a perfect score either.

Flat DataQualityMetrics accessors, deprecated in 0.9, are removed in 0.12. The old names raise AttributeError naming the replacement key. Use nested dimensions so skipped dimensions are explicit (see the migration table):

if q.completeness is not None:
    q.completeness["missing_values_ratio"]

negative_values_in_positive is driven by explicit positive_columns; dataprof does not infer positive-only domains from column names. identifier_columns marks numeric-looking IDs as semantic strings so numeric stats and outlier metrics do not treat them as measures. Timeliness scoring assesses confidently inferred date columns by default. temporal_columns adds columns that inference cannot identify confidently, such as mixed-format date strings.

Hints are validated, never silently dropped. A hint that names a missing column raises ValueError listing the unmatched names and the available columns; a positive_columns hint on a column with no numeric values, or a temporal_columns hint on a column with no dates, is likewise rejected. report.semantic_hint_bindings records how each hint bound — column, kind, checked_values, matched_values, and exact (whether the counts covered every row or a sample):

report = dp.profile("readings.csv", positive_columns=["pressure"])
report.semantic_hint_bindings
# [{"column": "pressure", "kind": "positive",
#   "checked_values": 1000, "matched_values": 1000, "exact": True}]

Selective dimensions -- compute only what you need:

report = dp.profile("data.csv", quality_dimensions=["completeness", "uniqueness"])
# report.quality.consistency will be None
# report.quality.completeness will have values

Export Methods

`to_dataframe()` / `to_polars()` / `to_arrow()`

All three return an enriched table of column profiles with rounded values:

# pandas DataFrame
df = report.to_dataframe()

# polars DataFrame (no pandas dependency needed)
pl_df = report.to_polars()

# PyArrow Table (no pandas dependency needed)
table = report.to_arrow()

Columns included: name, data_type, total_count, null_count, null_percentage, unique_count, uniqueness_ratio, locale_number_count, dominant_type, dominant_type_share, min, max, mean, std_dev, variance, median, mode, skewness, kurtosis, coefficient_of_variation, q1, q2, q3, iqr, is_approximate, min_length, max_length, avg_length, top_pattern, top_pattern_pct.

`describe()`

Transposed summary similar to pandas.DataFrame.describe():

desc = report.describe()
#          col_a   col_b   col_c
# count    1000    1000    1000
# null%    0.0     2.1     0.0
# unique   45      800     3
# mean     34.5    None    None
# std      12.1    None    None
# min      1.0     None    None
# 25%      25.0    None    None
# 50%      33.0    None    None
# 75%      44.0    None    None
# max      99.0    None    None

Returns a pandas DataFrame if available, otherwise a dict-of-dicts.

`quality_summary()`

Single-row dict for easy aggregation across multiple reports:

qs = report.quality_summary()
# {"source": "data.csv", "rows": 1000, "quality_score": 92.3,
#  "completeness": 98.0, "consistency": 95.0, ...}

# Track quality over time
import pandas as pd
rows = [dp.profile(f).quality_summary() for f in files]
history = pd.DataFrame(rows)

`to_html()` / `to_markdown()`

Render the report for sharing outside a notebook:

html = report.to_html()          # same rich table Jupyter shows, as a string
open("report.html", "w").write(html)

md = report.to_markdown()        # GitHub-flavored markdown table
# Paste straight into a PR comment, issue, or Slack message

`load()` / `from_json()` / `from_dict()`

Rebuild a report from previously exported data without re-profiling. The reconstructed report is a read-only view backed by the exported values, but all export methods (to_json, to_markdown, to_dataframe, describe, quality_summary, mapping access, …) work as usual.

load(path) is the path-based entry point — the natural counterpart to save(). Only .json files carry a full report; .csv / .parquet store column profiles only and cannot round-trip:

report.save("report.json")
# ...later...
reloaded = dp.ProfileReport.load("report.json")
reloaded.quality_score          # == the original report's quality_score
reloaded["email"].null_percentage

from_json(text) and from_dict(data) take an in-memory JSON string or dict instead of a file path:

reloaded = dp.ProfileReport.from_json(report.to_json())
reloaded = dp.ProfileReport.from_dict(report.to_dict())

Report schema versioning

Saved reports are durable artifacts — CI baselines, drift references, agent inputs — so the document carries its own schema version, independent of the package version:

report.to_dict()["schema_version"]   # == dp.REPORT_SCHEMA_VERSION

The compatibility policy when loading:

Document	Behavior
No `schema_version` field	Legacy pre-0.10 report; loads through a compatibility path
`schema_version` ≤ `dp.REPORT_SCHEMA_VERSION`	Loads normally; unknown additive fields from newer writers are ignored
`schema_version` > `dp.REPORT_SCHEMA_VERSION`	Raises `ValueError` immediately — an incompatible report is never partially decoded

The version only increments when the document format itself changes incompatibly, not on every dataprof release. The same field with the same semantics appears in reports serialized from Rust (serde), where readers enforce the identical policy.

The committed JSON Schema 2020-12 contract can validate to_dict(), to_json(), and JSON save() output in CI or another consumer. Version 1 includes both the high-level Python export shape and the complete Rust serialization shape; see the schema notes for compatibility and regeneration rules.

In 0.12, to_json() and JSON save() write the complete Rust document, with data_source, column_profiles, id, timestamp, and quality.metrics / quality.confidence. to_dict() keeps the historical summary keys source, source_type, columns, and flattened quality scores. It is a convenience projection, so json.loads(report.to_json()) is the way to obtain the complete document as a dictionary. All three loaders accept both layouts. An old flat document stays flat on resave because its missing provenance cannot be inferred.

When quality metrics are present, the summary's quality block always carries a low_sample_warning boolean (true when the profiled sample was below the recommended minimum of 10 rows, false otherwise). It round-trips through to_dict()/from_dict(); treat quality_score and the per-dimension ratios as directional rather than reliable whenever it is true.

`findings()` -- what deserves attention

A report holds every metric and says nothing about which ones matter. findings() turns the metrics already computed into a short, deterministic list, so the call site does not have to invent thresholds:

report = dp.profile("orders.csv")
result = report.findings()

for finding in result:
    print(finding.severity, finding.code, finding.column, finding.evidence)
# warning null_heavy amount {'null_percentage': 25.0, 'threshold': 20.0}
# info constant_column channel {'non_null_count': 4, 'unique_count': 1}
# info sensitive_pattern email {'category': 'contact', 'match_percentage': 100.0, 'pattern': 'Email'}

Each finding has a stable code, a severity ("warning" or "info"), the column it concerns (None for the whole report), the evidence that caused it, and a fixed summary sentence. Evidence is metric values, thresholds, and names; no finding carries a raw cell value.

Code	Severity	Reported when
`all_null`	warning	Every value of a column is null
`null_heavy`	warning	A column's null percentage is at least `null_heavy_percentage` (default 20)
`mixed_types`	warning	Values outside a column's dominant lexical type are at least `mixed_types_percentage` (default 5) of those classified. `dominant_type` in the evidence says which way the mix leans. Identifier columns are exempt
`locale_numbers`	warning	At least half of a column's text values are numbers written with a decimal comma or digit grouping (`locale_number_count`), which no numeric statistic includes. Identifier columns are exempt
`duplicate_rows`	warning	The source holds exact duplicate rows
`future_dates`	warning	Date values lie after the time the report was produced
`temporal_order_violations`	warning	Start dates fall after their paired end dates
`ragged_rows`	warning	Rows had a different field count from the header and were recovered
`unterminated_quote`	warning	A CSV source ended inside a quoted field, so its last record may hold several source rows
`records_skipped`	warning	Errors were counted while reading, e.g. JSONL lines skipped under `jsonl_on_error="skip"`
`constant_column`	info	Every non-null value of a column is the same one, seen more than once
`sensitive_pattern`	info	A column confidently matches a contact or financial pattern, a US SSN, or an Italian codice fiscale
`partial_scan`	info	The scan stopped early or sampled rows (`reason` is `"truncated"` or `"sampled"`)

Findings sort by severity, then code, then report-level before column-level, then column position. The default thresholds are the ones to_llm_context() flags at. Findings compare at the report's 2dp precision and the flags do not, so a share within rounding of a threshold (4.9992% against 5) can be a finding without being a flag. Both thresholds take a percentage above 0 and at most 100, and are applied at 2dp, so the evidence states exactly the threshold that was compared. Anything else, including a value that rounds to 0, raises ValueError:

report.findings(null_heavy_percentage=50, mixed_types_percentage=10)

Absence is not a clean result. A rule whose input the report does not carry produces no finding and is listed in result.not_evaluated, so an empty result.findings means "looked, found nothing" only for the rules not listed there. The result has no len(), and bool(result) raises TypeError, so if not report.findings(): cannot silently read an unevaluated rule as clean.

dp.profile("orders.csv", metrics=["schema"]).findings().not_evaluated
# ({'code': 'duplicate_rows', 'reason': 'quality_unavailable', 'quality_status': 'not_requested'},
#  ...
#  {'code': 'sensitive_pattern', 'reason': 'not_computed', 'columns': ['order_id', 'email', ...]})

`reason`	Meaning
`quality_unavailable`	The report carries no quality assessment; `quality_status` says why
`not_assessed`	Quality was computed, but the dimension the rule reads had nothing to assess
`estimated`	The duplicate count is an estimate, which witnesses nothing
`sampled`	The count is zero, but it came from the retained quality sample, so it rules nothing out for the rows the sample left behind. This is the ordinary state for timeliness on a source larger than the sample. A nonzero count is still reported
`unrecorded`	The document was written before dataprof recorded what the rule needs: ragged rows before 0.10, quality sample coverage and locale-formatted numbers before 0.12
`not_computed`	The metric was not computed for the listed `columns` (pack not selected, or a nested column)
`no_values`	The listed `columns` had no values to look at

Findings are derived, not stored. They are not part of the saved report, and a report loaded with ProfileReport.load() or from_dict() yields the same findings as the one that was saved. Findings describe the rows the report read: a truncated or sampled scan adds partial_scan rather than withholding the rest. For a pass/fail decision about the whole source, use check().

result.to_dict() / to_json() serialize the whole result, identical to what the Rust FindingPolicy writes for the same report and thresholds.

`check()` -- quality gates

State a policy as data and get a structured verdict back. Nothing is printed, the process is not exited, and the report is not modified, so this composes into Airflow, Dagster, a GitHub Action, or a shell script -- no CLI involved.

report = dp.profile("daily_drop.csv")
result = report.check(
    min_quality_score=90,
    max_null_percentage={"customer_id": 0, "*": 20},
    max_duplicate_rows=0,
    require_metrics=["quality"],
)

if not result.passed:
    for check in result.violations:
        print(check.code, check.column, check.message, check.observed)

Keyword	Type	Meaning
`min_rows`	`int`	Floor for the number of rows the scan read
`min_quality_score`	`float`	Floor for the overall score, 0--100
`min_dimension_scores`	`dict[str, float]`	Floor per ISO 25012 dimension, 0--100
`max_null_percentage`	`dict[str, float] \| float`	Ceiling for a column's null percentage, 0--100. `""` sets the limit for every column without its own entry; a bare number means the same as `{"": number}`
`max_duplicate_rows`	`int`	Ceiling for the duplicate-row count
`require_metrics`	`list[str]`	Metrics that must have been analyzed: `"quality"`, or a dimension name
`scope`	`str`	`"full_source"` (default) or `"observed"`

Thresholds are percentages on the same 0--100 scale the report reports, not 0--1 ratios. A policy that cannot be evaluated as written -- a threshold outside the range, an unknown dimension, no requirement at all -- raises ValueError rather than failing the dataset: a misconfigured gate is not a bad extract. A threshold must be a real number: text is refused even when it spells one ("90"), and so is a number too large for a float. Numpy scalars, Decimal and Fraction are accepted.

The gate is evaluated in Rust, the same code a Rust caller uses. It reads a profiled report or one loaded from the JSON save() and to_json() write. A report loaded from a flat to_dict() summary raises ValueError: the summary does not record the provenance the gate decides on.

The verdict has three values, not two.

`result.verdict`	Meaning
`"fail"`	At least one requirement was conclusively violated
`"inconclusive"`	Nothing was violated, and something could not be checked
`"pass"`	Every requirement was evaluated and met

result.passed is True only for "pass", so an unanswerable gate never reads as a green one. "fail" wins over "inconclusive": a witnessed violation is a decision.

A requirement goes unevaluated -- and lands in result.unevaluated -- when:

the report carries no quality assessment (reason: quality_unavailable, carrying the report's own quality_status, so "you did not ask for this" is distinguishable from "this broke");
the dimension or column had nothing to measure (reason: not_assessed);
the report holds no profile for a named column (reason: column_not_profiled) -- it was projected away or it is not in the source, and a report does not record which, so it is not decided either way;
the evidence does not reach as far as the requirement does (reason: evidence_incomplete).

require_metrics is how a caller turns the first of those into a failure, which is usually what a pipeline wants when quality is supposed to be configured on.

Scope and evidence. scope="full_source" states the requirements about the entire source. A ratio or an average computed over a truncated scan, a sampled scan, or a bounded quality sample bounds nothing about the rows that were not read, so such a requirement is left unevaluated rather than passed. The exception is a witnessed count: rows already seen as duplicates do not stop being duplicates when more rows are read, so an exact duplicate count above the allowance fails conclusively even on a partial scan -- while a count at or below it is still not a pass, and an estimated count settles neither direction.

scope="observed" states the requirements about whatever the metrics actually measured, which is always evaluable and says nothing about the rest of the source.

Each check records its own evidence, which can be weaker than the scan's: report.quality_sampled_dimensions names the components computed from a retained sample, and a fully read file can still have them. Provenance is resolved per component, so a completeness score from exact column counters is decided on its value, and the overall score only counts as sampled when a sampled component actually reaches it (a dimension the score weights exclude cannot move the aggregate).

Sampled scores on a fully read source (0.12+). Most quality dimensions are computed over a uniform sample of up to 10,000 values per column, so on any larger source the scores are estimates. The report bounds them: report.quality_score_bounds gives an interval per score that holds, all at once, with probability confidence_level (0.999). The sample fixes each check (a column's type, dominant form, detected pattern, decimal scale and outlier fences); the interval covers what those same checks give over every value the scan read. When the scan read the whole source and the only gap is the quality sample, a min_quality_score or min_dimension_scores requirement is decided on that interval: it passes when the whole interval meets the minimum, fails when none of it does, and stays evidence_incomplete only when the minimum falls inside it. The check records the interval it used as bounds next to evidence, which still says quality_sampled. On a clean column of 10,000 sampled values the interval is about 0.15 points wide; at 5% failures, about 2 points.

Past a million distinct values the key count and the duplicate-row count are estimates rather than samples. They are bounded with certainty instead of at 0.999: when an exact count is dropped, the distinct values it held are a floor the true count cannot go under. Uniqueness then gets an interval from those floors, wide on a source far past a million rows (the floor is a million of its rows), and the overall score is bounded again. A max_duplicate_rows requirement on an estimated count stays unevaluated, since the estimate can neither witness duplicates nor rule them out.

min_rows reads the report's execution.rows_processed. Rows already read do not go away when more are read, so a scan that read at least min_rows rows passes even when it stopped early or sampled. One that read fewer fails when it covered the whole source, or under scope="observed"; under "full_source" a scan that stopped early leaves it evidence_incomplete, since the rows it did not read may make up the difference.

A duplicate-row scan over a sample, on engines without a row tracker, has no bound, and a requirement that reads it stays unevaluated. Start/end date ordering is compared only between columns whose samples hold the same rows, so a pair where either date column has nulls is not compared. When no pair is compared, the temporal_order_violations finding reads not_assessed. scope="observed" is unchanged and decides on the sampled score itself.

A report loaded from a document written before dataprof recorded this says nothing about how its numbers were obtained. Unknown coverage is a third answer rather than "nothing was sampled", so quality_sampled_dimensions is None and a full-source check reports coverage_unrecorded. Both language layers answer the same way for such a report.

result.to_dict() / to_json() serialize the whole result, identical to what the Rust QualityPolicy writes for the same report and policy:

{
  "verdict": "fail",
  "scope": "full_source",
  "evidence": {"coverage": "complete"},
  "checks": [
    {
      "code": "max_null_percentage",
      "column": "customer_id",
      "expected": {"comparison": "at_most", "value": 0.0},
      "observed": 16.67,
      "scope": "full_source",
      "evidence": {"coverage": "complete"},
      "status": "failed",
      "message": "this column's null percentage is above the allowance",
    }
  ],
}

A score decided on its interval also carries bounds:

{
  "code": "min_quality_score",
  "expected": {"comparison": "at_least", "value": 90.0},
  "observed": 97.41,
  "scope": "full_source",
  "evidence": {"coverage": "incomplete", "reason": "quality_sampled"},
  "bounds": {"lower": 96.87, "upper": 97.93, "confidence_level": 0.999},
  "status": "passed",
  "message": "the overall quality score was computed over a sample, and its whole-source interval meets the required minimum",
}

message carries no numbers on purpose: observed and expected hold those, so nothing depends on how a float was formatted into a sentence.

See python/examples/etl_quality_gate.py for a runnable gate over four drops.

`python -m dataprof.check` -- CI entrypoint (0.12+)

Evaluate the same policy from a shell or CI job:

python -m dataprof.check daily_drop.csv --min-quality 90 --max-null 'customer_id=0' --json
python -m dataprof.check daily_drop.csv --policy quality-policy.json --json > verdict.json

To keep the full profile behind the verdict, add --report PATH (0.13+):

python -m dataprof.check daily_drop.csv --min-quality 90 --report profile.json --json > verdict.json

The path must end in .json; reload it with dp.ProfileReport.load(). It is the report the verdict was computed from, written on pass, fail and inconclusive and not when profiling fails. An existing file is overwritten; a write failure exits 3 and names the path. --json stdout is unchanged. The report path must not refer to the source or policy file, including through symbolic links, hard links or case-insensitive path aliases.

The policy file is a UTF-8 JSON object containing the check() keywords above; a leading byte-order mark (BOM) is accepted. Thresholds are JSON numbers; a quoted one such as "90" is a policy error (exit 3). Commit it alongside the pipeline so threshold changes can be reviewed:

{
  "min_rows": 1,
  "min_quality_score": 90,
  "max_null_percentage": {"customer_id": 0, "*": 20},
  "max_duplicate_rows": 0,
  "require_metrics": ["quality"],
  "scope": "full_source"
}

Flags replace the corresponding policy-file key in full. For example, --max-null '*=10' replaces the entire max_null_percentage mapping, including any column-specific entries. Repeat --max-null COLUMN=PERCENT, --min-dimension NAME=PERCENT, or --require-metric NAME to supply multiple entries; the last occurrence of a repeated mapping key wins. --min-rows, --min-quality, --max-duplicate-rows, and --scope cover the other policy keywords. Empty policies, unknown keywords, and duplicate JSON keys (including inside dimension or column mappings) are policy errors (exit 3).

--engine, --format, --max-rows, and repeatable --metric PACK pass through to profile_file(). As with the library, capped or sampled evidence may leave a full-source requirement inconclusive. --scope observed explicitly limits the claim to the analyzed population. A threshold on an unavailable metric stays unevaluated; an explicit --require-metric requirement fails if it is absent.

Exit code	Meaning
`0`	Every requirement passed; also used by `--help` and `--version`
`1`	The gate found a proven violation, even if other checks are unevaluated
`2`	The gate is inconclusive: a threshold could not be evaluated
`3`	An argument, policy, or source could not be read or used, or the report could not be written

Human summaries go to stderr. With --json, stdout contains only the existing QualityGateResult.to_json() document, including inconclusive results. Errors (exit 3) write an error object instead of a result document: {"error": {"kind": "argument" | "policy" | "input" | "output", "message": "...", "path": "..."}}, where path is present when the error names a file. A policy error without path comes from a command-line flag (for example --min-quality 150), not from the policy file. Without --json, errors go to stderr and stdout stays empty. --help prints usage to stdout and lists every flag and code. --version prints dataprof <version> to stdout and exits 0.

Baseline-relative gates await support in ProfileReport.check(). --baseline PATH currently exits 3 with an explicit unsupported-operation message, whether or not the path exists. It never ignores the requested comparison or substitutes an absolute check.

For a project with daily_drop.csv and quality-policy.json checked in, this GitHub Actions job runs the gate and retains its result even when it fails:

name: Data quality
on: [push, pull_request]
jobs:
  quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7.0.1
      - uses: actions/setup-python@v7
        with:
          python-version: '3.12'
      - run: python -m pip install dataprof==0.12.0
      - name: Check incoming data
        run: python -m dataprof.check daily_drop.csv --policy quality-policy.json --json > verdict.json
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: quality-verdict
          path: verdict.json

This example targets the 0.12 release. Before it is published, build this checkout with uv sync and uv run maturin develop, then invoke the module with uv run --no-sync python -m dataprof.check ....

`compare()`

Detect quality drift or schema changes between two profiles (e.g. the same dataset before and after a pipeline run):

before = dp.profile("data_v1.csv")
after = dp.profile("data_v2.csv")
delta = before.compare(after)
# {
#   "quality_score": {"a": 92.3, "b": 88.1, "abs": -4.2, "rel_pct": -4.55},
#   "dimensions": {"completeness": {...}, "consistency": {...}, ...},
#   "columns": {"email": {"null_pct_a": 1.0, "null_pct_b": 6.5, "null_pct_delta": 5.5}, ...},
#   "schema": {"added": ["phone"], "removed": [], "common": ["id", "email", ...]},
#   "metric_semantics": {"a": {...}, "b": {...}, "comparable": True},
# }

metric_semantics.comparable is True when both reports record every measurement definition this release knows, with the same values, and None when either side does not, which includes every report written before 0.12. Text lengths counted UTF-8 bytes through 0.11, so across that boundary a changed max_length on non-ASCII text can be the unit rather than the data.

The compare() result shape is provisional and will align with the Rust-side QualityDelta type once it lands.

`save()`

report.save("report.json")      # full report as JSON
report.save("profiles.csv")     # column profiles as CSV (no extra deps)
report.save("profiles.parquet") # column profiles as Parquet (requires pyarrow)
report.save("report.html")      # HTML fragment (same as to_html())
report.save("report.md")        # markdown table (same as to_markdown())

Partial Analysis

Fast operations that don't require a full profile:

`infer_schema()`

result = dp.infer_schema("data.csv")
print(f"{result.num_columns} columns, {result.rows_sampled} rows sampled")
for col in result.columns:
    print(f"  {col['name']}: {col['data_type']}")

Every format runs the same inference profile() does. On a Parquet file that means the metadata answers for every column it types, and text columns are sampled: only their values say whether they hold dates, integers or booleans, which is what a writer that did not type its input leaves behind. rows_sampled reports what that cost -- 0 for a fully typed file.

The sample is bounded in every format, so the type matches a full profile() whenever schema_stable is true -- nothing was left unread that could move it, either because the scan reached the end of the file or because the Parquet metadata typed every column and no scan was needed. Once inference stops at the cap, a later row can still move it.

`analyze_structure()`

structure = dp.analyze_structure("data.parquet")
for column in structure.columns:
    print(f"{column.name}: {column.data_type} nulls={column.null_count} ({column.provenance})")

Every counter it reports describes the whole file or is absent -- it never returns a partial count. For CSV and JSON that means checking truncated; for Parquet the counts come from the file footer, so they cover every row even when the type sample stopped at max_rows. Distinct counts are absent for Parquet, which has no whole-file source for one.

`quick_row_count()`

result = dp.quick_row_count("data.parquet")
print(f"{result.count} rows ({'exact' if result.exact else 'estimated'})")
print(f"Method: {result.method}, took {result.count_time_ms}ms")

CSV counts scan every logical record, including quoted multiline fields, and return exact=True, method full_scan, and no relative-error estimate. This costs a complete read even for large CSV files. Parquet uses footer metadata; large JSONL files retain their sampled density estimate.

Async API

The dataprof.asyncio module provides async variants for use in web frameworks, stream processors, and other async contexts. These helpers ship in the published wheels.

from dataprof.asyncio import profile_file, profile_bytes, profile_url

# Async file profiling
report = await profile_file("data.csv", max_rows=10000)

# Profile raw bytes (e.g. from an HTTP request body)
report = await profile_bytes(csv_bytes, format="csv")

# Profile a remote file over HTTP
report = await profile_url("https://example.com/data.parquet")

Remote Parquet is read with several ranged requests, all tied to the ETag the server sent first. If the object is replaced while it is being read, profile_url raises OSError naming the URL instead of profiling a mix of the two versions. A server that sends no ETag leaves nothing to check against.

Async byte streams use the incremental engine; engine="columnar" is rejected. Local async Parquet profiling honors max_rows and row-limit stop conditions, and rejects stop conditions or sampling strategies the Parquet reader cannot apply.

Additional async utilities:

from dataprof.asyncio import infer_schema_stream, quick_row_count_stream

schema = await infer_schema_stream(csv_bytes, format="csv")
count = await quick_row_count_stream(csv_bytes, format="csv")

Database Profiling (Source Build Only)

Async database functions for PostgreSQL, MySQL, and SQLite are not in the published wheels, and there is no extra that installs them. On a wheel install they exist as stubs that raise ImportError when called, and dp.capabilities().database is False. They need a source build with python-async, database, and the connector features you want.

From the published sdist, with pip

No checkout required:

pip install dataprof --no-binary dataprof   --config-settings="build-args=--features python,python-async,async-streaming,parquet-async,database,sqlite"

From a checkout

uv run maturin develop --features "python,python-async,async-streaming,parquet-async,database,sqlite"

List every feature you want, not only the extra ones

--features replaces the [tool.maturin] features list in pyproject.toml; it does not add to it. This is the part that bites, because the result looks like it worked:

# Wrong: builds database support and silently drops async, URL profiling,
# and remote Parquet — surface that a plain `pip install dataprof` includes.
--features "python,python-async,database,sqlite"

That command yields capabilities().database == True alongside async_streaming == False, url_profiling == False, and remote_parquet == False. Nothing errors; the package is simply smaller than the published wheel. The lists above are the shipped set plus the database features, which is why they are long.

database on its own is also not enough: the connectors are reported only when python-async is compiled in as well.

A source build needs a Rust toolchain (1.96 or later; CI compiles the extension crate on exactly 1.96, so that floor is tested rather than assumed). postgres and mysql are pure Rust; sqlite compiles libsqlite3-sys, which is C.

Check what you actually got before relying on it:

import dataprof as dp

caps = dp.capabilities()
assert caps.database and "sqlite" in caps.database_connectors
assert caps.async_streaming  # still present, because the list above kept it

Then the following APIs become available:

import asyncio
import dataprof as dp

async def main():
    # Test connection
    ok = await dp.test_connection_async("postgres://user:pass@localhost/mydb")

    # Profile a query
    report = await dp.analyze_database_async(
        "postgres://user:pass@localhost/mydb",
        "SELECT * FROM users",
        batch_size=10000,
    )
    print(f"{report.rows} rows, quality: {report.quality_score}")

    # Get table schema
    columns = await dp.get_table_schema_async(
        "postgres://user:pass@localhost/mydb", "users"
    )

    # Count rows
    count = await dp.count_table_rows_async(
        "postgres://user:pass@localhost/mydb", "users"
    )

asyncio.run(main())

batch_size must be greater than zero. Query result column names must be unique; duplicate aliases are rejected before values can be merged.

Arrow Interop

profile() also accepts one-shot producers implementing the Arrow C Stream capsule protocol:

import pyarrow as pa
import dataprof

batches = [pa.record_batch({"id": [1, 2]}), pa.record_batch({"id": [3, 4]})]
reader = pa.RecordBatchReader.from_batches(batches[0].schema, iter(batches))
report = dataprof.profile(reader, max_rows=3)
assert report.rows == 3
assert report.source_type == "stream"

# With DuckDB installed, pass a relation directly:
# report = dataprof.profile(duckdb.sql("select * from my_table"), max_rows=1000)

dataprof consumes and releases each batch before fetching another; it does not call to_arrow_table() or collect the stream. Producer batch buffers and the bounded profiling accumulators determine memory use. A producer may itself materialize data when exporting; dataprof cannot control that allocation. PyArrow and DuckDB remain optional producer dependencies.

max_rows stops at the requested row boundary, including zero, without asking for a subsequent batch. Reaching the cap sets truncation_reason even if that row happened to be the stream's last: end-of-stream was not checked. An uncapped or shorter stream is complete only after its producer signals the end. The report uses engine columnar, source type stream, and source system arrow_c_stream; the name identifies one profiling operation (batch_id="0"). The stream is one-shot: use a fresh reader for another profile.

Metric packs, quality dimensions, locale, semantic hints, and column selection work as on Arrow tables. Unsupported file/transport controls raise ValueError before capsule export. Duplicate names are rejected before any batch is read, even with column selection. Selected nested types (structs, lists, maps, unions) raise TypeError; select flat columns before exporting, or with columns=. An empty stream with a schema retains its columns and unassessed quality. Batch/schema failures abort the call; no partial report is returned. The C interface carries producer error text, which is retained as __cause__, rather than the original Python exception object. Existing Table, C Array, pandas, and polars adapters retain their behavior.

The RecordBatch class supports zero-copy exchange via the Arrow PyCapsule interface:

import dataprof as dp
import pyarrow as pa

# Profile a PyArrow table directly
report = dp.profile(table)

# RecordBatch properties
batch.num_rows
batch.num_columns
batch.column_names

# Convert to other formats
df = batch.to_pandas()
pl_df = batch.to_polars()

# PyCapsule protocol for zero-copy exchange
schema_capsule = batch.__arrow_c_schema__()
array_capsule = batch.__arrow_c_array__()

Navigation Menu

Repository navigation

README.md

README.md

Python API Guide

Installation

Quick Start

`profile()` -- Primary Entry Point

`ProfileReport`

Rounding

`ColumnProfile`

`Pattern`

Configuration

`ProfilerConfig`

`SamplingStrategy`

`StopCondition`

`ProgressEvent`

`DataQualityMetrics`

Export Methods

`to_dataframe()` / `to_polars()` / `to_arrow()`

`describe()`

`quality_summary()`

`to_html()` / `to_markdown()`

`load()` / `from_json()` / `from_dict()`

Report schema versioning

`findings()` -- what deserves attention

`check()` -- quality gates

`python -m dataprof.check` -- CI entrypoint (0.12+)

`compare()`

`save()`

Partial Analysis

`infer_schema()`

`analyze_structure()`

`quick_row_count()`

Async API

Database Profiling (Source Build Only)

From the published sdist, with pip

From a checkout

List every feature you want, not only the extra ones

Arrow Interop

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

Python API Guide

Installation

Quick Start

profile() -- Primary Entry Point

ProfileReport

Rounding

ColumnProfile

Pattern

Configuration

ProfilerConfig

SamplingStrategy

StopCondition

ProgressEvent

DataQualityMetrics

Export Methods

to_dataframe() / to_polars() / to_arrow()

describe()

quality_summary()

to_html() / to_markdown()

load() / from_json() / from_dict()

Report schema versioning

findings() -- what deserves attention

check() -- quality gates

python -m dataprof.check -- CI entrypoint (0.12+)

compare()

save()

Partial Analysis

infer_schema()

analyze_structure()

quick_row_count()

Async API

Database Profiling (Source Build Only)

From the published sdist, with pip

From a checkout

List every feature you want, not only the extra ones

Arrow Interop

`profile()` -- Primary Entry Point

`ProfileReport`

`ColumnProfile`

`Pattern`

`ProfilerConfig`

`SamplingStrategy`

`StopCondition`

`ProgressEvent`

`DataQualityMetrics`

`to_dataframe()` / `to_polars()` / `to_arrow()`

`describe()`

`quality_summary()`

`to_html()` / `to_markdown()`

`load()` / `from_json()` / `from_dict()`

`findings()` -- what deserves attention

`check()` -- quality gates

`python -m dataprof.check` -- CI entrypoint (0.12+)

`compare()`

`save()`

`infer_schema()`

`analyze_structure()`

`quick_row_count()`