qsv 22.0.1

[BrewTestBot]

Created by brew bump

---

Created with brew bump-formula-pr.

Details

release notes

## [22.0.1] - 2026-08-09 📐 The "Data Schematic" Release 📊





qsv's biggest release ever with 580+ commits since v21.1.0. The headliner is viz - an entirely new command that turns a CSV into interactive plotly charts and maps, with viz smart auto-designing a Data Schematic while bundling a DataTables viewer as well. Schematics are self-contained, offline-capable HTML with static PNG/SVG/PDF export via viz_static. See the gallery.
A Data Schematic is our take on a modern, interactive, storytelling Data Dictionary in the Age of AI.
The name is descriptive rather than decorative: a schematic is the visual form of a metadata schema, and viz smart renders the editable JSON Schema describegpt drafts, together with the statistics that back it. Where a dictionary lists fields, a Schematic shows components and how they connect - correlation, process order, hierarchy, temporal pacing, spatial pairing - and every claim it makes is checkable against the data it describes.
A  Data Schematic is neuro-symbolic by construction.
Statistics, heuristics and algorithms are deterministic and reproducible, so they decide what gets drawn and what the numbers are. LLMs handle what computation cannot - classifying each field against a shared, catalog-wide ontology; general world knowledge; and language translations - and because that drafted schema is saved as an editable sidecar (see example JSON Schema), a human in the loop (data steward, librarian, SME) can ratify or correct anything the LLM proposed and re-render from the corrected schema. The format is defined in docs/DATA_SCHEMATIC.md.
Three more new commands land alongside it: denull (detect the null sentinels that silently degrade typing), fixedwidth (convert fixed-width text to CSV) and clean (remove qsv-generated cache files).
Highlights

viz: a whole new visualization command. Interactive plotly charts and maps from CSV, with 20+ standalone chart subcommands and a viz smart mode that auto-designs an entire Data Schematic from qsv's existing stats & frequency caches. Output is self-contained, offline-capable HTML, with static PNG/SVG/PDF export via viz_static. See the gallery (#302; #4019).
The Data Schematic is explorable, not just viewable. An embedded DataTables data viewer drawer puts the underlying rows beside the charts, cross-linked with map points in both directions, alongside a browsable Data Dictionary drawer - all in one shareable file (#4283; #4284, #4306).
denull: detect the null sentinels that silently corrupt typing. Literal NULL/N/A text makes stats type a numeric column as String, quietly degrading viz smart, schema and describegpt downstream (#4175).
fixedwidth: convert fixed-width text to CSV, with positions auto-detected from a header comment so qsv table --align leftfwf output round-trips (#4168).
clean: remove qsv-generated cache files, with verify-before-delete safety and --dry-run as the default (#3373; #4015).
Your Schematic and dictionary speak your data's language. describegpt detects the dataset's content language locally with whatlang (zero tokens) and viz smart renders its entire UI, chart strings and coverage notes in it (#4301, #4310, #4313).
⚠️ Three breaking changes. minijinja 2.23 changes rendered template output (booleans now render True/False, none renders None) across template, apply, fetchpost, describegpt and profile. The cached 2 → 3 migration swaps the on-disk cache backend from sled to redb, invalidating existing on-disk caches and inverting the meaning of a TTL of 0 (was "immediately stale", now "cache indefinitely"). describegpt's bundled prompt file is bumped 8.0.0 → 9.0.0.


Windows prebuilts: viz yes, viz_static no. The Windows binaries ship viz/viz smart with full interactive HTML output, but not the viz_static static image export tier (-o chart.png/.svg/.pdf/.jpeg/.webp). viz_static pulls in a headless-browser/webdriver stack whose release build does not complete inside GitHub's 6h Action Runner job limit on Windows. Linux x86_64 prebuilts are unaffected. Build natively with --features viz_static on Windows if you need static export. qsvmcp also now builds with plain viz on every platform - no behavior change there, since viz over MCP was already HTML-only by design.

Detailed MCP Server and Cowork Plugin changes are documented in the MCP Server/Cowork Plugin CHANGELOG.

Added


viz: new command that generates interactive charts and maps from CSV using plotly - the headline feature of this release. Standalone subcommands cover bar, line, scatter, histogram, box, violin, pie, heatmap, candlestick/ohlc, sankey, radar, geo, map, choropleth, contour, scatter3d, treemap, sunburst, icicle, splom, parcats and bubble. viz smart auto-designs a Data Schematic - a whole one-page rendering of the dataset's schema and statistics - by reusing qsv's existing stats & frequency caches rather than re-scanning: continuous numeric columns become box plots drawn from precomputed quartiles, low-cardinality columns become frequency bars, and the panel set, chart types, axis scaling and ordering are all chosen statistically. Output is self-contained, offline-capable interactive HTML (no network needed to view), with static PNG/SVG/PDF/JPEG/WebP export via the viz_static feature and --open to launch a viewer. Built on the plotly crate with no polars dependency, sidestepping the plotlars/polars version conflict. Gated behind a new viz feature (+ viz_static sub-feature); included in distrib_features/all_features/qsvmcp, not in qsvlite or qsvdp. See the visualizations gallery (#302, #4019).


fixedwidth: new command that converts fixed-width text (fields at fixed byte-column positions, with no delimiters) to CSV. Column positions are given explicitly via --positions/--widths, or auto-detected from a #1,10,15-style header comment - the same format qsv table --align leftfwf emits, so fixed-width output round-trips back to CSV (#4168).


clean: new command that removes qsv-generated cache & sidecar files to reduce clutter and simplify data packaging. Cleans the index (<file>.idx), stats and frequency caches by default; user outputs (--schema, --validate, --moarstats, --all) are opt-in and never touched unless named. Every deletion is verified before it happens, never matched on filename alone: the stats family is anchored on its .stats.csv.json args sidecar (so your own report.stats.csv survives), the frequency cache is validated by its first-line qsv_version/arg_input metadata, and a .idx must pass a csv-index size-multiple-of-8 magic check and have a sibling source. --dry-run is the safe default (and wins over --force); --stale limits cleaning to caches older than their source or orphaned entirely, and --recursive walks a directory (#3373, #4015).


denull: new command that detects null sentinels - literal text like NULL or N/A standing in for a missing value, which makes stats type a numeric column as String (nullcount stays 0, no quartiles), silently degrading viz smart, schema and describegpt downstream. Scans each column once with bounded memory (a 414 MB, 86-column file peaks at ~39 MB). A column is CONFIRMED only when every non-numeric value it holds is a known sentinel and at least two distinct numeric values remain; otherwise it is REJECTED with the reason (off-vocab value, zero-padded codes, too many distinct non-numeric values). --apply rewrites the data, blanking sentinels per column - something a single qsv replace pass cannot do, since it takes one regex across all selected columns. Numeric sentinels (-999, 9999) are deliberately NOT detected: they parse as valid numbers and no scan can distinguish them from real data (#4175).


viz: viz smart now explains why it skipped a column instead of dropping it silently, pointing at qsv denull when the column looks like numeric data held back by a non-numeric token (#4175).


validate: new --split-ragged flag quarantines wrong-column-count rows instead of aborting. validate previously stopped dead on the first ragged row; --split-ragged streams well-formed rows to <input>.valid, diverts ragged rows to <input>.invalid, appends a report to <input>.validation-errors.tsv, and still exits non-zero. Works in both RFC 4180 and JSON Schema modes (#4231, #4237).


stats: new --jsonl and --pretty-json flags emit the per-column statistics to stdout (or --output) as JSON, instead of only to the --stats-jsonl sidecar. --jsonl writes NDJSON (one object per column, machine-friendly); --pretty-json writes a single pretty-printed JSON array of the same objects. Both are mutually exclusive with each other and with --stats-jsonl (#4112, #4195).


extsort & sortcheck: stats-cache aware short-circuit. When a valid stats cache exists, both consult its per-column sort_order statistic to skip work when the relevant single column is already sorted in the requested order - extsort (CSV mode) streams the input straight through and skips the external merge sort entirely (honoring --reverse), and sortcheck answers instantly on the exit-code path (#2116, #4006).


get: new cache-fetch subcommand exports an already-cached entry's decompressed contents to --output <file> or stdout. It is offline - it reads the content-addressed blob directly and never re-fetches the source - making it a first-class alternative to qsv input dc:<name>. Accepts and ignores a leading dc: prefix, and errors cleanly when the name is not cached (#4124).


describegpt: new --infer-null-values flag has the LLM propose each field's null sentinels, emitted into the JSON Schema dictionary's per-property x-qsv object. The LLM proposes a single list; qsv - not the model - splits it into null_values (independently confirmed to occur as a literal value in a String column, listing every observed casing, spelled as found in the data) and null_candidates (everything else, each stamped confirm_required: true). A numeric or date placeholder can never reach the confirmed list: -999 parses as a valid integer and no scan can distinguish it from a real reading, so an LLM is the only thing that can propose one - and a human the only thing that can confirm it. This is the complement of denull, which detects only sentinels in columns that would promote to a numeric type once blanked; a purely categorical column (status = ok/pending/NULL) is confirmed here and ignored there. Sentinels are reported, never applied (#4175).


describegpt: --context-file now accepts non-text files in addition to text/Markdown. The file type is sniffed from its contents (reusing sniff's detectors): CSV and Excel/ODS spreadsheets are extracted to text locally, while PDF and image files (JPEG/PNG/WebP/GIF) are base64-encoded and sent to the LLM as a multimodal Chat Completions attachment (needs a multimodal model & endpoint; ~32 MB max). Word/PowerPoint are rejected with guidance to convert to PDF/text. The context is now injected into the user message instead of the system prompt.


describegpt: new --format okf output emits the Data Dictionary as an Open Knowledge Format document - a leaner, vendor-neutral plain-markdown-plus-YAML-frontmatter sibling of SemanticMd (type/title/description/resource/timestamp/tags frontmatter + a Column | Type | Description Schema table). New --okf-type flag sets the required type key (default "CSV Table"); --ds-source/--ds-updated also map to OKF's resource/timestamp (#4018).


viz: new chart subcommands pie, heatmap (correlation matrix or category×category pivot), candlestick/ohlc, sankey, and radar (polar). viz smart now adds a correlation-heatmap panel when the dataset has 2+ numeric columns (#302).


viz: new chart subcommands geo (token-free projection point map via ScatterGeo, with --projection), contour (2D density of two numeric columns, binned via --bins), and scatter3d (3D scatter over --x/--y/--z). geo and scatter3d honor the --color/--size marker encodings and --series (#302).


viz smart now auto-wires the new chart types: a 3D scatter of the strongest-correlation triple when there are 3+ numeric columns; a 2D density contour instead of the correlated-pair scatter for large datasets (where a scatter overplots); and an offline ScatterGeo projection world-overview instead of mapbox tiles when the coordinates span a continental/global extent (#302).


viz smart box plots now overlay sample points via a size-based heuristic - all points for small data, Tukey outliers for medium, none for large (a fast cache-only quartile box) - overridable with --box-points (now accepted by smart, not just box) (#302).


viz smart frequency bar charts now show a (NULL) bar for empty cells and an Other (N) aggregate bar for the categories beyond --limit (N = the count of distinct categories rolled up), matching qsv frequency's default output. Both aggregate bars are drawn in a muted grey so they read as summaries rather than real categories. New --no-nulls and --no-other flags suppress them (#302).


viz choropleth & viz smart can now build a choropleth from a user-supplied GeoJSON by point-in-polygon binning: each row's --lat/--lon is tested directly against the GeoJSON polygons (even-odd ray casting, handling holes & MultiPolygon) and the matched feature id becomes the location - exact, works for any country/admin level, and needs no geocoding or GeoNames lookup. Points outside every region snap to the nearest feature by default (--no-snap drops them instead); either way a coverage note reports how many points missed every polygon. Wired into the viz smart Data Schematic as the "Regions" panel when a --geojson is supplied. Zero new dependencies (#302).


viz choropleth & viz smart choropleths now have richer hover tooltips. Instead of a bare feature id and value, each region shows a human-readable name + id (e.g. Kagoshima (JP46)), the value labeled with its measure (count: 65), the share of total (15.6% of total, for count/sum aggregations only), and the rank (rank 1 of 47). Region names are read from the GeoJSON via the new --feature-name-key flag, or auto-detected from common name properties (properties.name, etc.) when omitted. Applies to all paths (point-in-polygon, literal --locations, geocoded) and both the geo and MapLibre (--map) basemaps (#302).


viz smart now draws summary choropleth(s) from a region-code column - no --lat/--lon needed. When a --geojson is supplied and a column is tagged as a geographic region code (e.g. concept: geo.zip_code, from a describegpt/curated Data Dictionary), viz smart aggregates the rows by that code, auto-matches the codes to the GeoJSON feature ids (zero-padding short zips to 5 digits), and leads the Data Schematic with a count-per-region choropleth plus, when the dataset has a measure column, a median-of-measure-per-region choropleth (median resists the heavy right-skew of things like prices). The region column's frequency bar is suppressed in favor of the map. Tile vs. offline-projection basemap is chosen from the matched regions' geographic span. Zero new dependencies (#302).


viz smart now adds a judicious animated panel - at most one per Data Schematic, and only when the motion is genuinely informative. In priority order it picks: a Gapminder entity-bubble chart (one bubble per low-cardinality category, tracing a path through two measures over time, sized by a third least-redundant measure) when an entity column has dense enough per-time-bucket coverage; an animated geo map (dated points accumulating on a ScatterGeo projection) for continental/global extents; or an animated scatter pair whose per-time-bucket centroid path bends the most (a trailing-window reveal) - deliberately NOT the strongest-correlated pair, since a near-perfect line (e.g. open≈close, packages∝weight) has no 2-D shape to evolve and animating it adds nothing. --slider auto (default) fires only on a strong signal (a date column + enough time buckets); on lowers the bar; off suppresses. Reuses the native viz geo --slider animation core (#4212).


viz smart: Data Schematics gain an interactive data viewer drawer - the data behind the charts, in the same file. Next to the rowcount in the metadata frontmatter, an "(Explore)" link (all rows embedded) or "(Preview)" link (first N rows) opens the underlying table in a bottom drawer, mirroring the Data Dictionary drawer. New --preview-threshold <n> (default 50,000; 0 disables) decides which: at or under the threshold every row embeds. Built on vendored DataTables running vanilla - no jQuery - with CSV export, correct date sorting, and per-column search widgets via DataTables ColumnControl (#4283, #4284, #4288, #4302, #4320).


viz smart: data viewer rows and map points are now cross-linked. Clicking a row highlights its point on the map and clicking a point reveals its row - each scrolling the counterpart into view - and rows with no plottable coordinates are pinned rather than silently unreachable. Region-click on a summary choropleth filters the data viewer to that region (#4306, #4324, #4338).


viz smart: Data Schematics are now localized. The UI renders in the dataset's language, resolved once with explicit intent winning over inference: --language <lang> > the dictionary's x-qsv.detected_language_code > English. --language accepts a BCP-47/ISO 639-1 tag, an ISO 639-3 code (what describegpt's detection pass emits), or an English language name. Coverage extends past the UI chrome to the chart strings and coverage notes, across seven languages (#4310, #4313, #4317).


describegpt: auto-detects the dataset's content language when compiling a data dictionary - locally via whatlang, deterministic and costing zero tokens (no LLM round trip). Emitted as detected_language/detected_language_code when confidence clears the threshold (default 80%, tunable via --language <float>), and the detected language then drives the LLM's output language in every inference phase, so a Spanish dataset gets a Spanish dictionary without being asked (#4301, #4305).


viz smart: new --photos renders a column of image URLs as hover-dwell map imagery - resting the pointer on a map point for two seconds opens that row's photo in a preview card beside the marker. Detection is stats-cache-only (each column's min/max are sniffed for an http(s) URL ending in a known image extension), so it works on a bare qsv viz smart data.csv with no Data Dictionary (#4271).


viz smart: skew-aware encoding feeds the stats cache's distribution-shape signals (skew, zero-share, Gini) into --smarter's axis and chart choices. Additive-amount measures with a high Gini get a Lorenz curve alongside the box, labeled with the cached gini_coefficient so curve and number always agree; heavily-skewed measures get log-scaled boxes with an explicit log-scale cue. Candidate detection blends the dictionary's additive-vs-intensive aggregation with the Gini gate, adding no extra data pass (#4219, #4224).


viz smart: a dictionary-declared pipeline funnel panel. A funding/processing pipeline (planned → committed → spent) was previously visible only as % zeros annotations scattered across three unrelated box titles; the funnel states it once, as the actionable signal it is. Detection requires both a vocabulary match and row-wise containment, and bridges declared pipelines whose stages do not strictly nest (#4222, #4281, #4282).


viz scatter: the animated Gapminder-style bubble chart is now directly requestable, not just auto-selected by viz smart. viz scatter --slider previously hard-errored on --color/--size; it now accepts them together with --series to animate bubbles through time (#4217, #4319).


viz: new QSV_VIZ_CDN environment variable loads the plotly.js runtime from its CDN (<script src>, pinned to the same version qsv would otherwise embed) instead of embedding it. This shrinks a chart to a few KB and a smart Data Schematic by ~1.9 MB, at the cost of requiring network access when the page is viewed - intended for charts published to the web. It governs only the plotly.js bundle; figure payloads still follow QSV_VIZ_NO_COMPRESS. Default output remains self-contained and offline-capable (#302).


viz smart --dict-info: the embedded Data Dictionary page now offers a download row - the dictionary as JSON Schema (as before), the frequency counts the Data Schematic actually charted, and every generated sidecar the run consumed: the stats cache (.stats.csv.data.jsonl), its parsing-options metadata (.stats.csv.json), the frequency cache (.freq.csv.data.jsonl) and the --bivariate stats CSV. Every file is bundled into the HTML as a base64 data: URI on a plain <a download> - send someone the Data Schematic and they can download the underlying files with no access to your machine, and the page stays a single self-contained, script-free file. A sidecar is offered only when it was genuinely USED, never merely present: the frequency cache appears only when viz actually read it (it silently falls back to a full recompute when the cache is stale or option-incompatible), the bivariate CSV only when this run freshly wrote it, and the stats pair is located the way get_stats_records locates it, dc: handles resolved. The human-readable <stem>.stats.csv is deliberately NOT offered: viz never reads it, so nothing can show it describes the same computation the Data Schematic used. The charted-frequency CSV (<stem>.viz-frequency.csv) is generated in memory - nothing is written to disk - and is always available, since it is exactly what the panels show (capped at --limit, aggregate (NULL)/Other (N) buckets flagged) whether the bars came from a cache hit or a recompute. Absolute local paths (canonical_input_path, canonical_stats_path, arg_input) are stripped to bare file names before embedding, so sharing a Data Schematic never discloses your directory layout. Sidecars over 4 MB are skipped with a note (#302).


viz: a --dict-info Data Schematic now bundles every sidecar the run consumed as an in-page download, not just the dictionary: the JSON Schema export, the frequency counts the panels actually charted (<stem>.viz-frequency.csv, generated in memory), and the stats/freq/bivariate sidecars. The HTML was self-contained; the data behind it wasn't - send someone the Data Schematic and they could read the charts but not get at the numbers (#4256).


viz smart: the correlation drill-down panels are now legibility- and skew-aware for heavy-tailed, zero-inflated numeric columns - the case where money-like measures (e.g. 45–60% zeros, Gini > 0.9) previously collapsed the 2D density panel into a single dark cell at the origin and the 3D scatter into an unreadable blob. The density contour is dropped when its mass concentrates into one bin, first retried as a log-scaled contour over the strictly-positive rows (with the omitted-zeros share stated in the title) when enough points survive; the strongest-pair scatter now scales each axis logarithmically when that column spans orders of magnitude and holds no zeros (the axis is named in the panel title); and the 3D scatter is skipped when its points collapse onto the origin in any 2D projection. Extends the skew-aware axis work from #4219 to the relationship panels; --log-scale on|off still overrides (#4223).


viz smart: now uses robust statistics on measures whose mean is set by their tail rather than a typical row. Such a column (mean at least 2x the median, or a zero-inflated non-negative column whose median is 0) is summed rather than averaged in the "measure by dimension" bar - "top 10 by mean" otherwise ranks whoever holds the single biggest item instead of the biggest total - unless its name marks it intensive (a rate, ratio, index, per-capita figure), which is never summed. When a majority of a correlation matrix's columns are tail-dominated, the matrix is computed with Spearman's rank rho instead of Pearson's r, whose covariance those same extreme rows would otherwise dominate. Both heatmaps now NAME their coefficient in the panel title (Correlation (Pearson r) / Correlation (Spearman ρ - rank, robust to outliers)) instead of a bare "Correlation", and the grouped bar states η² as a plain explained-variance share ("explains 17% of variance") rather than a bare coefficient. On the reporting dataset this flips the headline panel from (mean, η²=0.17) to (sum, explains 17% of variance) and lifts the two near-duplicate spend columns from an outlier-driven r=0.80 to their true rank agreement of ρ=0.94 (#4220).


viz smart: now detects 1:1 categorical columns from the data and charts only one of them. A code/label pair like magencyacro ↔ magencyname (DPR ↔ Department of Parks & Recreation) otherwise produces byte-identical frequency bars and spends two parcats axes on the same variable, with trivial straight-through ribbons between them. The shorter-valued member is kept - it fits a bar label or treemap tile where the long form truncates - and the drop is reported so the mapping stays discoverable. This is the data-driven counterpart of the existing name-based subject/subject_code rule, which needs both a <base>_code spelling and a data dictionary and so could not see this pair; both feed the same suppression set, so the per-column panels, the grouped bar, parcats and the hierarchy panel all inherit the verdict. Detected by partition refinement - two columns are 1:1 exactly when they induce the same grouping of rows, so candidate columns are split apart on the first row their groupings disagree, which also means detection stops reading as soon as no candidates remain. Strict 1:1 only, no "near-1:1" tolerance, and guarded on support: at least 4 populated rows per category, so a small-sample coincidence cannot delete a panel. An empty cell is its own category for this purpose, so columns blank on different rows are never treated as the same variable (#4221).


viz smart: the "measure by dimension" bar now picks its dimension deterministically. η² is accumulated over a HashMap's values, so its float rounding depended on iteration order and two mathematically tied dimensions (1:1 columns explain exactly the same variance) swapped places between runs of the same binary on the same file - the panel title flipped from by magencyname to by magencyacro and back. Ties now resolve to the lowest column index (#4221).


viz smart: the correlated-pair scatter now judges legibility on the axes it will actually render. It previously tested for a degenerate cloud on the raw values before resolving its log axes, so a strictly-positive heavy-tailed pair - squashed against the origin on linear axes but spread cleanly once logged - was dropped in exactly the case log scaling exists to rescue, and --log-scale on could not override the drop. The pair is now tested in the space it will be drawn in, mirroring the density contour's linear → log retry from #4223. --log-scale off still drops such a cloud: with log declined the linear axes are what will be drawn, and absent beats unreadable (#4276).


moarstats: new --bivariate-batch <n> bounds the peak memory of --bivariate, which was otherwise unbounded in the COLUMN count. #4356 cut the constant factor hard, but peak memory stayed O(pairs) = O(columns²), so a wide-enough CSV re-crossed any threshold regardless of its size: a 160-column, 100k-row input is only 60 MB yet needed 21.4 GiB with mi/nmi/u enabled - more than twice what the 41-column, 1M-row (539 MB) benchmark needs, and well past a GitHub hosted runner's 15.57 GiB. --bivariate-batch <n> processes at most n field pairs per pass over the indexed file, so peak memory is O(n × chunks) - tunable and hard-bounded. Measured on that 160-column fixture (12,720 pairs; debug build, 16 cores, macOS; RSS varies ~2% run to run): **~21 GiB unbatc...

View the full release notes at https://github.com/dathere/qsv/releases/tag/22.0.1.

---

[github-actions]

🤖 An automated task has requested bottles to be published to this PR.

Caution

Please do not push to this PR branch before the bottle commits have been pushed, as this results in a state that is difficult to recover from. If you need to resolve a merge conflict, please use a merge commit. Do not force-push to this PR branch.