The versioned
profile-report.v1.schema.json document is the
machine-readable contract for serialized ProfileReport values. It uses JSON
Schema 2020-12, carries a stable project-owned $id, and keeps every $ref
self-contained under $defs.
Schema versions are independent of package versions. The filename and $id
must use the same version as Rust's REPORT_SCHEMA_VERSION and Python's
dataprof.REPORT_SCHEMA_VERSION.
Starting in 0.12, the canonical persisted document is Rust's complete runtime
report: data_source, column_profiles, report id and timestamp, and the
confidence-wrapped quality assessment. Python to_json() and JSON save() use
the same Rust serializer. ProfileReport::to_json() sorts object keys recursively
for byte-stable output; arrays retain their order. Ordinary Rust Serde output
has the same document values, without a promise about map-key byte order.
Version 1 deliberately continues to accept two shapes:
- Rust's complete runtime document (
data_source,column_profiles, and the confidence-wrapped quality assessment). - The historical Python summary (
source,source_type,columns, and its flattened quality summary), still available through Pythonto_dict().
Both producers now live in Rust, and the summary's producer uses the very types
that generate its schema. Python no longer builds a parallel serialized report.
The summary is a convenience projection, not a lossless persistence format;
json.loads(report.to_json()) is the full document as a Python dictionary.
Python from_dict(), from_json() and load() accept either shape. A loaded
flat summary stays flat when resaved: it never recorded report identity, full
source metadata or quality confidence, so manufacturing those fields would
misrepresent the original run. Missing provenance markers stay missing; the
resave does add schema_version, because it is written by a v1 build. Rust's ProfileReport reader reads the canonical shape; the flat
compatibility loader remains a Python API. check() refuses a loaded flat summary,
because the gate runs on Rust's report.
This preserves the published v1 validation contract instead of silently
narrowing it. New profiles save only the canonical shape. Consumers of the old
JSON layout should use to_dict() for its summary keys, or migrate to the
canonical paths above. A future schema v2 may remove the summary branch from
the persistence schema; that requires an explicit migration for incomplete
legacy provenance, not relabeling a flat document as a complete runtime report.
The canonical quality block now also retains scores.overall_score and
scores.dimension_scores, at the existing two-decimal score precision. These
are the aggregate values the Python summary already persisted. They must travel
with the rounded input metrics: deriving them again from rounded ratios can
change the saved score beyond its documented rounding. Older canonical documents without scores
remain readable and derive scores from the metrics they recorded. Rust callers
that edit a restored assessment's public metrics invalidate its saved scores;
subsequent score access and serialization use the edited metrics.
Both dialects accept unknown additive object properties. This matches the v1 reader policy and lets compatible fields be added without invalidating stored reports. Required fields, schema version, known enum values, and primitive types remain enforced.
execution.sampled_row_ranges is an optional additive v1 field in both
dialects. Each [start, end] pair identifies a zero-based, half-open source
row interval selected for analysis, in source order. Capped Parquet reads
record at most 32 ranges. Absence means the selection was not recorded, while
[] means a recorded selection of zero rows. Existing prefix-capped reports
remain readable and do not gain inferred ranges. The existing truncation reason
and sampling fields describe the cap and coverage; these ranges identify its
analyzed population exactly.
execution.recovery_events is optional execution provenance in both v1
dialects. New reports record an ordered array; [] means no recovery occurred.
An absent field means history is unknown, including when loading older reports,
and remains absent on reserialization. It does not change metric semantics or
participate in cross-engine metric comparisons.
Each event contains kind (engine_fallback or csv_auto_recovery),
attempted (the engine or parse strategy that failed), retry (the next engine
or strategy), and error (diagnostic text). Engine names are columnar and
incremental, matching execution.engine, which still identifies the engine
that produced the report. Strategy names and error messages are diagnostics,
not stable codes. The last retry in a successful result succeeded; preceding
retries may have failed and led to further events. Row-level error and ragged-row
counts retain their existing meaning and are not incremented for failed attempts.
Python exposes the same history through report.recovery_events. No current
engine records csv_auto_recovery: it came from the standalone CSV recovery
parser, removed before 1.0, and stays in the schema so reports written by earlier
versions still load.
skewness uses adjusted Fisher-Pearson sample skewness and requires at least
three finite numeric values. kurtosis uses adjusted Fisher sample excess
kurtosis and requires at least four. These match pandas
Series.skew
and Series.kurt.
A constant column reports zero once the minimum count is met. When a reservoir
is used, its mean, sample standard deviation and higher moments all describe
that same retained sample. is_approximate: false records exact numeric
statistics; true records sampling, and absence means the approximation status
was not analyzed or is missing from an older report.
The coefficient of variation is 100 * std_dev / abs(mean). Its zero-mean
guard is relative to the spread (abs(mean) <= f64::EPSILON * abs(std_dev)),
so changing measurement units does not make the statistic disappear.
Timeliness compares whole dates against the freshness cutoff. A configured
max_data_age_years uses the Gregorian mean year of 365.2425 days, including
fractional years; the age budget is rounded down to whole days before it is
subtracted from the reference date. A date equal to the cutoff remains fresh.
For numeric columns, mean is always a finite number, including when the sum
of the input values overflows f64: the mean of finite values lies between
their minimum and maximum. std_dev and variance remain required fields in the
canonical report. They are null when the spread of finite input values cannot
be represented as a finite f64; a true constant column reports 0.0. A
variance too large for f64, as for values near ±1e155, is null while its
standard deviation, about the size of the values, fits: it is taken without
forming the variance and reported with the skewness and kurtosis that follow
from it, and with the coefficient of variation wherever the mean allows one;
values balanced around a mean of 0 have none (#1150). Only when the sum of
squared deviations passes about 2e609, a standard deviation past about 4e304
divided by the square root of the row count, is the standard deviation null
too. A variance too small for f64, as for values near 1e-200, is null too
rather than 0.0, and its standard deviation is reported the same way (#1088).
Values below about 1e-306, at the bottom of f64's range, square to products
that lose their digits even so: their std_dev is
null as well rather than a number with too few digits. Statistics derived
from an unrepresentable standard deviation, including the coefficient of
variation, skewness and kurtosis, are absent. The same rule
covers quartiles.iqr: it stays required and is null when q3 - q1 cannot be
represented, while q1, q2 and q3 are still reported. outlier_count is
then 0, which is exact: fences beyond the f64 range leave no finite value
outside them. This distinction survives report save/load and applies to the
Python summary as well. A document that omits any of the three keys is
malformed and fails to load; it is not read as null.
Integers of 2^53 or more in magnitude, up to the 64-bit range, are not exact as
f64. variance, std_dev, skewness and kurtosis are computed from the
exact integers on every input path, so epoch nanoseconds a few units apart keep
their spread (#877). mean sums the exact integers in a compensated f64
sum. mode is counted over the exact values, so distinct values that round
alike do not tie for it, and is reported as the nearest f64 of the exact
mode. quartiles.iqr and the fences outlier_count uses come from exact
differences between neighbouring values, so a spread of a few units has an IQR
of a few units (#1038). min, max, median, q1, q2 and q3 are taken from
the nearest f64 of each value, so iqr can differ from q3 - q1 as
reported.
stats.exact carries the order statistics themselves, as canonical decimal
strings (no exponent, no trailing zeros in a fraction): min, max, median,
q1, q2, q3 and mode, each present when the f64 field of the same name
is, over the same values. It is present on every integer column and every
decimal-typed column (an Arrow decimal read from Arrow, Parquet, pandas or
polars), and absent on a float column, whose values are not held exactly. A
CSV or JSON column of long decimal text is typed float and has no exact
although its spread is exact, and so is a database numeric column, which
arrives as text. A median or quartile between two integers is a decimal, such
as "9007199254740997.5". exact is absent on a report written before it
existed (#1038).
Decimals with more digits than f64 keeps get the same treatment (#1149):
variance, std_dev, skewness and kurtosis come from their exact
deviations, so cents on 10^15 keep their spread, mode, quartiles.iqr and
outlier_count are exact as above, and the other statistics are taken in
f64. A column typed as a decimal (an Arrow decimal of any
width, from Decimal32 to Decimal256, read from Arrow, Parquet, pandas or
polars) is held exactly from 11 significant digits. Text is held exactly
from 18: the shortest form of an f64, what Python and Rust write, has at most 17 digits, so shorter text reads
as the float it may be, as the same column typed Float64 does. A CSV of a
decimal column therefore serializes what the typed column does when its values
have at most 10 significant digits or at least 18, and may differ in between. A
decimal too wide to hold exactly, past 38 digits, keeps a spread only where it
stands well above the rounding of its nearest f64 (2^-32 of the magnitude);
below that, a constant column of such values included, the spread statistics
are null.
The cross-engine identical-numbers contract governs serialized, rounded
metric values (#547): Rust's Serde report and Python's to_dict(),
to_json(), and JSON save() output. JSON persistence now uses the same document
layout; to_dict() retains the summary projection described above.
For the same logical values, schema semantics, analysis options and analyzed
population, serialized metrics must compare exactly, with no additional
tolerance or rounding by the consumer. Counts, types and absence must also
agree: None/missing means not analyzed, while empty means analyzed with
nothing found. Source names, engine identifiers, timestamps, elapsed time,
throughput, memory and input-byte measurements describe execution and are not
cross-path equality targets. Whole report documents need not be identical.
| Metric kind | Serialized precision |
|---|---|
| 0–100 percentages, including coefficient of variation | 2 decimal places |
| Moment statistics: mean, standard deviation, variance, skewness, kurtosis | 4 decimal places, at most 12 significant digits |
| Data values: extrema, median, mode | 4 decimal places |
| 0–1 ratios and average text length | 4 decimal places |
| Quartiles | 2 decimal places |
Rounding uses the stored binary float, with ties away from zero. The Rust
serde_helpers and Python _rounding modules implement the same convention.
Native Rust fields and Python ColumnProfile attributes retain full precision;
their final digits may depend on accumulation order. Raw values are not
required to be bit-identical across engines, or to equal the rounded values
restored from JSON. Parity tests supplement exact serialized comparisons with
raw diagnostics using relative tolerance 1e-9 and absolute tolerance 1e-12;
those tolerances do not weaken the serialized contract.
Moment statistics are accumulated, so their last bits depend on the order the values were added in, which differs between routes. Four decimal places absorb that drift only while they are finer than it, below about 1e11, so moment statistics also keep at most 12 significant digits (#1178). Below 1e8 that is the 4dp value; above it the digits where the drift lives are dropped, and the rounded value is the float nearest its decimal form. Data values are read, not accumulated, and keep 4 decimal places at any magnitude. Rounding alone cannot mathematically guarantee equality for every possible floating-point input, especially near rounding boundaries. A difference that survives serialization remains a parity defect to investigate, rather than an allowed raw-precision difference. The known nested JSON/Arrow representation mismatch (#637) is one such defect.
Text frequencies are unassessed in newly computed profiles (#709):
most_frequent and least_frequent are None and omitted by Rust Serde on
every path, including analyze_column and the database connectors. Python
exports already omit these fields. This keeps absence consistent without
presenting retained-sample frequencies as full-column measurements. The fields
remain optional in schema v1; Rust deserialization and reserialization preserve
historical frequency measurements, including measured-empty lists. Callers that
need explicit frequency analysis can use
dataprof_metrics::stats::text::{calculate_most_frequent, calculate_least_frequent}
over their chosen population; these helpers count exactly the supplied values,
so callers choose which null-like values to exclude.
Sampling and approximate statistics must retain their provenance. Different row selections or retained samples are different analyzed populations; compare like with like rather than treating sampling differences as numeric drift. Producer-side conversions also matter: pandas widening nullable integers to floats changes the schema before dataprof sees it.
Derived threshold decisions should use serialized precision so that saving and
loading cannot change the verdict, as established for to_llm_context() in
#526. That is not yet true of every consumer: _dominant_pattern compares the
native confidence against its 0.5 threshold while serialization rounds that
value to four decimals, so a confidence just under the boundary can be
suppressed natively and emitted after a round trip.
Structural claims remain exact: all-null is based on null and total counts,
not a percentage rounded to 100. This decision does not define historical
metric-unit compatibility, tracked separately in #675.
Run from the repository root:
cargo run --example generate_profile_schema
cargo test --test profile_report_schemaThe generator explicitly selects SchemaSettings::draft2020_12() and the
Serde serialization contract. CI reruns it and fails if the committed artifact
drifts. Rust tests validate the artifact against the bundled Draft 2020-12
meta-schema, compile it without network resolution, and validate representative
reports. Python tests validate to_dict(), to_json(), and JSON save()
output against the same file.
When changing a serialized report field, review compatibility first. Additive
fields with reader defaults stay in the current schema version; an incompatible
change requires incrementing REPORT_SCHEMA_VERSION and committing a new
versioned schema without deleting schemas that supported releases still read.
The schema is also useful to consumers that store reports outside dataprof. The published URL is stable across package releases:
https://andreabozzo.github.io/dataprof/schema/profile-report.v1.schema.json
For example, create a full JSON report and validate it with the optional
jsonschema package. The package is a validation-tool dependency only; it is
not required by the base dataprof wheel.
Because the schema allows unknown additive properties, a successful validation
confirms required fields and primitive types but does not reject extra keys.
New Python JSON saves write the ProfileReport shape, so the examples
below pin validation to that branch of the versioned schema. This keeps errors
focused on the fields that matter to Python consumers instead of reporting the
whole document as failing a top-level anyOf.
python -m pip install jsonschema
python - <<'PY'
import copy
import json
import os
from pathlib import Path
from urllib.request import urlopen
from jsonschema import Draft202012Validator
SCHEMA_URL = (
"https://andreabozzo.github.io/dataprof/schema/"
"profile-report.v1.schema.json"
)
REPORT_PATH = Path("report.json")
schema_source = os.environ.get("DATAPROF_SCHEMA", SCHEMA_URL)
if schema_source.startswith(("http://", "https://")):
with urlopen(schema_source, timeout=30) as response: # noqa: S310 - project default or caller-supplied URL
schema = json.load(response)
else:
schema = json.loads(Path(schema_source).read_text(encoding="utf-8"))
document_schema = {key: value for key, value in schema.items() if key != "anyOf"}
document_schema["$ref"] = "#/$defs/ProfileReport"
Draft202012Validator.check_schema(document_schema)
report = json.loads(REPORT_PATH.read_text(encoding="utf-8"))
validator = Draft202012Validator(document_schema)
errors = sorted(
validator.iter_errors(report),
key=lambda error: [str(path_part) for path_part in error.path],
)
if errors:
for error in errors:
print(f"{REPORT_PATH}: {error.json_path}: {error.message}")
raise SystemExit(1)
print(f"{REPORT_PATH}: valid")
# A deliberately invalid document fails for a useful reason, rather than
# merely failing because the file could not be read.
invalid_report = copy.deepcopy(report)
invalid_report["schema_version"] = "not-an-integer"
invalid_errors = list(validator.iter_errors(invalid_report))
assert invalid_errors, "the invalid schema_version should be rejected"
print(
"invalid example: rejected "
f"({invalid_errors[0].json_path}: {invalid_errors[0].message})"
)
PYThe report.json input above can be produced by the normal Python API:
import dataprof as dp
dp.profile("data.csv").save("report.json")For offline or project CI validation, save the Python block above as
validate_report.py and use the checked-in artifact instead of the network URL:
DATAPROF_SCHEMA=docs/schema/profile-report.v1.schema.json python validate_report.pyA minimal GitHub Actions job can validate every report committed under
reports/ without adding anything to the runtime package:
name: Validate dataprof reports
on: [push, pull_request]
jobs:
reports:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.x"
- run: python -m pip install jsonschema
- name: Validate saved reports
run: |
python - <<'PY'
import json
from pathlib import Path
from jsonschema import Draft202012Validator
schema_path = Path("docs/schema/profile-report.v1.schema.json")
schema = json.loads(schema_path.read_text(encoding="utf-8"))
document_schema = {
key: value for key, value in schema.items() if key != "anyOf"
}
document_schema["$ref"] = "#/$defs/ProfileReport"
Draft202012Validator.check_schema(document_schema)
validator = Draft202012Validator(document_schema)
report_paths = sorted(Path("reports").rglob("*.json"))
if not report_paths:
raise SystemExit("no JSON reports found under reports/")
failures = []
for report_path in report_paths:
report = json.loads(report_path.read_text(encoding="utf-8"))
failures.extend(
f"{report_path}: {error.json_path}: {error.message}"
for error in validator.iter_errors(report)
)
if failures:
raise SystemExit("\n".join(failures))
print(f"validated {len(report_paths)} report(s)")
PYVersion 1.1 (additive) introduced the "bytes" value in PythonSourceType for
byte-buffer inputs (CSV/JSON/JSONL/Parquet bytes). Previously they were labeled
"dataframe". Consumers matching on "dataframe" must also accept "bytes"
for byte inputs; stored v1 reports are unchanged.
Version 1.2 (additive) widened two value domains for reports where no quality dimension had anything to assess — a header-only file, or one whose every dimension has a zero denominator:
overall_scorein the Python dialect acceptsnullas well as a number. Previously the empty set of dimension scores was averaged to0.0, which reads as "this data is terrible" rather than "there was nothing to assess", and contradictedreport.quality_score, which already returnedNone.MetricConfidenceaccepts a"NotAssessed"value in the Rust dialect'squality.confidence."Exact"claimed certainty about a score that was never computed.
Both are widenings, so stored v1 reports remain valid. Consumers that read
overall_score as a number must now handle null, and consumers matching on
MetricConfidence must accept "NotAssessed"; in both cases the signal is
"not assessed", never zero. assessed_dimensions is empty for exactly these
reports and is the authority to branch on.
min_length, max_length and avg_length on a text column counted UTF-8 bytes
through 0.11, under names that disclose no encoding. They now count Unicode
scalar values, so a column of CJK or emoji values reports smaller numbers than a
stored 0.11 report does for the same data.
No property was added, removed or retyped, and ASCII values are unchanged, so
the schema's validation shape and version are unchanged and stored reports
remain valid. The document itself does change: both dialects' min_length,
max_length and avg_length gain description annotations naming the unit.
A consumer comparing a report across the 0.11/0.12 boundary is comparing two
units for non-ASCII text, which schema_version cannot signal because nothing
about validation changed. Compare within a release, or re-profile.
The seven quality dimension objects (completeness, consistency,
uniqueness, accuracy, timeliness, validity, precision) were already
optional in both dialects, and are now genuinely absent when the dimension
assessed nothing. Previously a dimension was emitted whenever its metric struct
existed, which is whenever it was requested, so a file with no pattern-bearing
column published validity.valid_values_ratio: 100.0 beside
values_checked: 0.
No property was added, removed or retyped, so the schema document is unchanged
and stored reports remain valid. What changed is which optional properties
appear: a consumer that assumed every dimension key was present must read it as
optional. A dimension key is present exactly when that dimension had a positive
denominator, which is exactly when its dimension_scores entry is a number.
That is a slightly wider set than assessed_dimensions, which additionally
filters on a positive score weight. Under the default weights the two agree; a
dimension configured with a weight of 0.0 is assessed and serialized while
contributing nothing to overall_score, so it is absent from
assessed_dimensions. Read a dimension's presence as "this was measured", not
as "this is behind the overall score".
Both dialects now carry a top-level quality_status object naming what
happened to the quality computation. It has a state string and, on failed
only, an error message:
state |
meaning |
|---|---|
computed |
quality was computed; quality holds the assessment |
not_requested |
the quality pack was deselected for this run |
no_data |
requested, but no quality sample was supplied to the assembler |
withheld_by_projection |
requested, but every requested dimension measures whole rows and the run profiled a subset of columns |
failed |
requested and attempted; the computation failed, and error says how |
unrecorded |
read back from a document written before this field existed |
quality alone could not answer this. A run that never asked for quality and a
run whose quality computation failed both left it absent, so absence itself was
the plausible value that hid the failure, and the only record of the failure was
a log::warn! line that no consumer reads back.
The field is additive and not required: stored v1 reports remain valid. A
document without it reads back as computed when it carries an assessment and
unrecorded when it does not — a stored assessment proves the computation ran,
and nothing else about an older document is knowable.
quality_status is provenance, not a metric, so it is outside the numeric
equality contract above. It is assigned in one place, ReportAssembler, which
every input path builds through, and agrees across engines and paths for a
source with the same analysis options, including an empty source.
A successfully analyzed zero-row source is analyzed, nothing found, whether
it declares columns or has no columns at all. With quality enabled, CSV, JSON,
JSONL, Parquet, queries and Arrow/dataframe inputs report computed and carry
an assessment with no assessable dimensions. Python serializes
overall_score: null, assessed_dimensions: [] and null dimension scores;
Rust serializes the corresponding empty quality metrics. File, byte-buffer,
async and HTTP transports follow the same rule. Header-only CSV and zero-row
Arrow/Parquet schemas retain their columns.
Empty CSV streams and zero-column Arrow RecordBatches, which previously raised
errors, now produce reports too.
Previously, empty CSV files on the direct parser, incremental and columnar
paths withheld quality and reported no_data, while CSV buffers and most
other inputs emitted an empty assessment. Those CSV file paths, and the
database assembler's no-column branch, now emit the assessment too. This is a
behavior correction within schema v1: both shapes were already valid. Stored
reports keep their original status and quality presence when loaded; consumers
comparing empty extracts should regenerate older baselines for parity.
Deselecting quality still yields absent quality and not_requested, and
projection withholding still applies to the requested dimensions. no_data
remains available for an assembler that receives no quality sample, distinct
from an explicitly supplied empty sample. It no longer denotes a successfully
analyzed empty source.
The Python dialect's quality object now carries score_weights, the relative
weights behind overall_score, with the same rule the Rust dialect applies to
quality.metrics.score_weights: custom weights are written and default weights
are omitted (#760). The next section supersedes the omission: since 0.13 both
dialects write default weights too.
Previously the summary dropped them, so a report assessed with custom weights
came back from ProfileReport.from_dict(report.to_dict()) with the default
weights next to an overall_score those defaults did not produce.
The field is additive and not required: stored v1 reports remain valid.
Summaries of reports that used the default weights were unchanged by #760. A summary
written before this change omits the field whatever weights were used, so there
its absence does not show that the defaults applied; the loader still reads an
absent field as the defaults, as it did before. The canonical document from
to_json() or JSON save() has kept custom weights since #714.
Both dialects now write score_weights whatever the weights are, defaults
included (quality.metrics.score_weights in the canonical document,
quality.score_weights in the Python dialect). A stored overall_score is kept
rather than recomputed, so a document that omits default weights only names
the defaults of whichever release reads it; if a later release changed them,
the score and its decoded weights would disagree (#749).
The field stays optional in the schema, so stored v1 reports remain valid, and
an absent field is still read as the defaults. The default weights have not
changed since weights became configurable in 0.10, so on a canonical document
an absent field did mean the defaults. On a flat summary written before 0.12 it
does not, because those dropped custom weights; re-exporting such a summary
with to_dict() keeps the field absent rather than claim the defaults.
Both dialects now carry a top-level metric_semantics object naming how the
report's measurements were defined. It has one field per intentionally changed
definition:
| field | values | meaning |
|---|---|---|
text_length_unit |
"unicode_scalar" |
unit of min_length, max_length and avg_length on text columns |
schema_version answers whether a document validates. It cannot answer whether
two valid documents measured the same way. Text lengths are the case in point:
they counted UTF-8 bytes through 0.11 and count Unicode scalar values since
(see above), with no change to the document's shape (#675).
Every report this release produces records the object. It is additive and not
required, so stored v1 reports remain valid. A document without it was written
before dataprof recorded it: its definitions are unknown, not the current
ones, and loading and saving it keeps the object absent rather than stamping the
reader's definitions on it. An explicit null, for the object or for a
definition inside it, is malformed: it fails to load in both dialects and the
schema rejects it. Definition names this build does not know are ignored, so a later
release can add one; a value this build does not know fails to load.
Python's compare() reports both sides and comparable: True when both
record every definition this release knows, with the same values, and None
otherwise, including when either side records an empty object. Only True makes a difference
in a measurement such as max_length a difference in the data.
The object is provenance, not a metric, so it is outside the numeric equality contract above, and every engine and input path records the same value.
A column of structs, lists or maps is now typed Nested in the Rust dialect and
"nested" in the Python dialect, and reports its counts only: total_count,
null_count and null_percentage. unique_count, invalid_count,
type_homogeneity, stats and patterns are absent, meaning not analyzed
(#637).
Through 0.11 such a column was typed String and measured on a text rendering
of its values. That rendering was Arrow's display string on the Parquet and
Arrow paths and compact JSON on the JSON paths, so the same records reported
different lengths, distinct counts and patterns depending on the file format.
A JSON column is Nested when every non-null value is an object or an array; a
column mixing containers with scalars has no typed counterpart and stays text.
The enum gains a value, so stored v1 reports remain valid. A reader built
before this change rejects a report that contains the new value, as it would
any enum value it does not know. Consumers matching on data_type must accept
nested. A nested column typed String in a 0.11 report and nested in a
0.12 report is the same data measured under a different definition. Compare
within a release, or re-profile.
Both dialects carry an optional boolean execution.unterminated_quote for CSV
input (#782). true means the source ended inside a quoted field: a quote was
opened and never closed, and the CSV parser read every following row into the
last field of one record. The report then counts one row where the source held
several, and nothing else in it shows the difference. false means the scan
reached the end of the source outside any quoted field.
Absence means the check did not run: non-CSV input, a scan that stopped before the end of its source (a row cap, a byte budget), or a report written before 0.12. The quote that swallows the rest of the file is on the last record, so a scan that did not reach the end did not read it.
Under csv_flexible=False such a source is refused instead, as a ragged row is.
The field is additive and not required, so stored v1 reports remain valid.
It is provenance about the source, not a metric, and every CSV engine and input
path (file, bytes, async) reports the same value for the same source.
The Rust dialect's DateTime statistics carry an optional slash_date_order
(#811). It records how the column's NN/NN/YYYY values were read, decided once
per column from its own values:
month_first: some values only parse month-first (12/31/2024) and none only day-first, so01/02/2024is 2 January.day_first: the reverse.assumed_day_first: every value parses both ways, so nothing decides it; the values were read day-first.mixed: values that only parse day-first and values that only parse month-first are both present. Each was read in the order it parses, and the dates cannot all be right.
Absence means the column holds no slash dates, or the report predates 0.12.
Through 0.11 each value was parsed on its own: month-first when day-first
failed, day-first otherwise. A US column therefore read its ambiguous values
day-first and reported wrong minimums, month distributions and timeliness
counts. Dates, statistics and timeliness checks derived from slash dates in
0.12 reports can differ from 0.11 reports of the same data. The field is
additive and not required, so stored v1 reports remain valid.
Both dialects carry an optional integer locale_number_count on each column
(#433). Of the values type_homogeneity counted as text, it is how many are
numbers written with a decimal comma or digit-group separators: 10,50,
1.234,56, 1,234.56, 12,34,567, 1'234.56, 1 234,56. dataprof does not
parse these as numbers, so they are in no numeric statistic. A column made of
them is typed string, and a numeric column counts them in invalid_count.
The count does not decide which convention a value follows: 1,234 is a
thousand in one locale and a fraction in another.
It is counted over the same values as type_homogeneity and is present
whenever that field is. Absence means the count did not run: a nested column,
or a report written before 0.12. 0 means the column was checked and none were
found. The locale_numbers finding reads it, and lists a column from an older
report as unrecorded rather than clean.
The field is additive and not required, so stored v1 reports remain valid.
Every engine and input path reports the same count for the same values.
metric_semantics gains a second definition (#813):
| field | values | meaning |
|---|---|---|
null_tokens |
"common_markers" |
the text values counted as null |
Under common_markers a value is null when it is empty or whitespace, null
or nan in any case, or exactly one of NA, N/A, n/a, #N/A, \N and
None, ignoring surrounding whitespace. The markers match only in that
spelling, so Na and none stay values. Typed nulls (an Arrow null, SQL
NULL, Python None, a float NaN) are null on every path, as before.
Through 0.12 only empty, null and nan were nulls, and the markers were
counted as values: completeness read 100% on columns that were mostly missing,
and a numeric column with markers in it could fall below the inference
threshold and be typed string. Null counts, completeness, inferred types and
every statistic of a column holding these markers can differ from a 0.12 report
of the same data.
A 0.12 report records text_length_unit and not null_tokens, so its null
vocabulary reads back as unknown and Python's compare() reports
comparable: None against a current report. Every engine and input path
records the same value. The field is additive and not required, so stored v1
reports remain valid; an explicit null is malformed, as for
text_length_unit.
metric_semantics gains two definitions:
| field | values | meaning |
|---|---|---|
pattern_match_count |
"validated" |
what every pattern's match_count and match_percentage count |
shape_moments |
"adjusted_sample" |
the estimator behind skewness and kurtosis |
Under validated a pattern match is a value that passes the pattern's regular
expression and, where the pattern has one, its semantic validator (checksum,
range or calendar check). Through 0.12 every regular expression match counted,
so random 16-digit ids were 100% Credit Card.
Under adjusted_sample skewness is the adjusted Fisher-Pearson sample skewness,
from three values, and kurtosis the adjusted Fisher sample excess kurtosis, from
four, as pandas computes them. Through 0.12 both divided population moments by
the sample standard deviation.
A report that records neither reads back with both unknown, and Python's
compare() reports comparable: None against a current report. Every engine
and input path records the same values. The fields are additive and not
required, so stored v1 reports remain valid; an explicit null is malformed,
as for text_length_unit.
SemanticHintKind, the kind of a semantic_hint_bindings entry, gains the
value temporal_pair in both dialects. Each column named by a temporal_pairs
entry is recorded once with this kind.
The enum gains a value, so stored v1 reports remain valid. A reader built
before this change rejects a report that contains the new value, as it would
any enum value it does not know. Consumers matching on kind must accept
temporal_pair.
Both dialects gain a top-level analysis_settings object recording the
settings a report was profiled with that move its scores on unchanged data and
are recorded nowhere else in it (#1010):
| field | values | meaning |
|---|---|---|
pattern_locale |
"none", "CA", "DE", "FR", "GB", "IT", "US" |
the locale detected patterns were ranked with, which the validity score reads; "none" when no locale was given |
columns |
{"selection": "all"} or {"selection": "listed", "names": [...]} |
the columns the run profiled: every column, or a columns= projection in the order given. A projection that names a missing column is refused, so with "all" a column without a profile was not in what the run read |
metric_semantics records how a release defines its measurements;
analysis_settings records what the caller chose. Score weights live in the
quality assessment and semantic hints in the hint bindings, so neither is
repeated here.
The object is absent from documents written before 0.13, whose settings are
unknown rather than the defaults, and from a report built directly with
ProfileReport::new rather than through a profiling run. An explicit null,
an object without pattern_locale or columns, an unknown locale value, and a
columns object of another shape are malformed.
A field a later release adds is ignored by this reader. Stored v1 reports
remain valid.
metric_semantics gains one definition:
| field | values | meaning |
|---|---|---|
range_rule_count |
"violating_values" |
what the range and sign rules behind the accuracy score count |
quality.metrics.accuracy gains the two counts it defines (#1005):
| field | meaning |
|---|---|
rule_values_checked |
numeric values, a literal past the f64 range included, of the columns a range rule (age, percentage, count, year names) or the sign rule (positive_columns) applies to |
rule_violating_values |
of those, the values breaking at least one rule, each counted once |
The accuracy score subtracts 100 * rule_violating_values / rule_values_checked
from 100 - outlier_ratio (9 violating values of 20 take off 45 points);
columns under no rule add nothing to either side,
and with no such column the rules take nothing off. Through 0.12 every rule
failure counted, a negative percentage hinted positive twice, and the failures
were divided by numeric_values_checked, the numeric values of every column,
so clean columns beside a broken one raised its score. range_violations and
negative_values_in_positive keep counting failures per rule, as diagnostics.
A report that records no range_rule_count reads back with it unknown, and a
quality gate leaves the accuracy and overall score rules over it
definition_not_comparable. Both counts default to 0 when absent, and such a
document still scores its range and sign failures over
numeric_values_checked, as the release that wrote it did. The fields are
additive and not required, so stored v1 reports remain valid; an explicit
null for range_rule_count is malformed, as for text_length_unit.
metric_semantics gains one definition:
| field | values | meaning |
|---|---|---|
key_uniqueness_units |
"non_null_values" |
what key_uniqueness divides the key column's distinct values by |
quality.metrics.uniqueness gains key_values_checked, the non-null values of
key_column (#1005). key_uniqueness is the key's distinct values over them,
so a nullable key no longer loses uniqueness for the nulls completeness already
counts: 100 distinct ids and 100 nulls score 100, where 0.12 reported 50. A key
column holding no value is still named, with key_values_checked 0, and its
key_uniqueness carries no signal: the uniqueness score then rests on the
duplicate-row component alone, or the dimension is unassessed without one.
A report that records no key_uniqueness_units reads back with it unknown, and
a quality gate leaves the uniqueness and overall score rules over it
definition_not_comparable. key_values_checked is absent when no key column
was identified, and on documents written before it, whose key_uniqueness
divided by every row and still scores as recorded. Both fields are additive and
not required, so stored v1 reports remain valid; an explicit null for
key_uniqueness_units is malformed, as for text_length_unit.
metric_semantics gains one definition:
| field | values | meaning |
|---|---|---|
outlier_units |
"tested_values" |
what outlier_ratio is measured over |
quality.metrics.accuracy gains outlier_values_checked, the finite numeric
values of the columns that reached outlier_min_samples (#1005). With none, no
outlier test ran: outlier_ratio is 0 and carries no signal, and the component
takes nothing off the score. Through 0.12 that 0 read as a measurement.
The accuracy dimension is now assessed when one of its components had units: an
outlier test ran, or a range or sign rule applied to a value
(rule_values_checked). Numeric values neither component examined no longer
assess it, so a source whose only numeric columns are shorter than
outlier_min_samples and fall under no rule reports accuracy as unassessed,
where 0.12 scored it 100. The overall score then renormalizes over the other
dimensions, as for any unassessed one.
A report that records no outlier_units reads back with it unknown, and a
quality gate leaves the accuracy and overall score rules over it
definition_not_comparable. outlier_values_checked is absent on documents
written before it; those are assessed and scored as their release did. The
field is additive and not required, so stored v1 reports remain valid; an
explicit null for outlier_units is malformed, as for text_length_unit.
metric_semantics gains one definition:
| field | values | meaning |
|---|---|---|
key_column_source |
"declared" |
which column key_uniqueness may measure |
key_uniqueness now measures only a declared key: the first identifier_columns
entry the source has (#1065). Through 0.12, with none declared, the first column
whose name had an id, key or pk word was the key. A name cannot tell a
table's key from a foreign key, so a customer_id listed before order_id was
scored as the table key, and the same clean orders table scored 100 or 94.38
depending on column order. Without a declared key, key_column and
key_values_checked are absent, key_uniqueness carries no signal, and the
uniqueness score rests on the duplicate-row component alone, or the dimension
is unassessed without one. Declare the key to have it scored:
profile(..., identifier_columns=["order_id"]).
A report that records no key_column_source reads back with it unknown, and a
quality gate leaves the uniqueness and overall score rules over it
definition_not_comparable. Documents written before it keep their key and
score as recorded. The field is additive and not required, so stored v1
reports remain valid; an explicit null for key_column_source is malformed,
as for text_length_unit.
metric_semantics gains statistic_rounding, with the one value
"four_decimals_twelve_significant": every column's mean, std_dev,
variance, skewness and kurtosis are serialized at 4 decimal places and at
most 12 significant digits (#1178). Through 0.13 they kept 4 decimal places at
any magnitude, where the order values were accumulated in could change the last
serialized digits between input routes.
A report that does not record it reads back with the definition unknown, and
Python's compare() reports comparable: None against a current report. Every
engine and input path records the same value. The field is additive and not
required, so stored v1 reports remain valid; an explicit null is malformed,
as for text_length_unit.
Both dialects gain a top-level generator object naming the dataprof build
that produced the report (#907):
| field | values | meaning |
|---|---|---|
version |
string | the dataprof package version |
commit |
string | the full git hash the build was compiled from |
dirty |
boolean | whether the compiled Rust sources (Cargo.toml, Cargo.lock, crates/) had uncommitted changes when it was built |
A development build carries the version of the release it follows, so
version alone cannot tell them apart; commit and dirty can. Both are
absent where the build did not record them: a crate built from a package, an
sdist, or a build without a usable git. The Python extension records them;
a Rust program built against the crates records version alone.
A stored report keeps the generator it was written with. The object is absent
from documents written before 0.13. An explicit null, for the object or for
commit or dirty, a missing version, and a value of the wrong type are
malformed. A field a later release adds is ignored by this reader. Stored v1
reports remain valid.
Both dialects gain a top-level source_identity object recording which data
the report describes (#866). data_source and source name a location; the
data behind a path or a table name can change between the check and the
promotion, so a pipeline needs an identity of the data itself to compare with
what it promotes.
| field | values | meaning |
|---|---|---|
version |
string | the version of the source the caller supplied (source_version=, --source-version), recorded verbatim: a table snapshot ID, a commit version, a load or batch ID |
dataprof neither interprets nor verifies version. No profiling path records
an identity on its own, so the object is absent unless the caller supplied
one, and it is absent from every document written before 0.14. Each member
names its kind of identity; a later release may add others, such as a digest
of the bytes read, and this reader ignores members it does not know, so an
object holding none it knows reads back as {}. An explicit null, for the
object or for version, and a value of the wrong type are malformed.
The identity is provenance, not a metric: two input paths over the same
logical data carry different identities, so like data_source it stays
outside the serialized-value parity comparison. Stored v1 reports remain
valid.