viz smart: dictionary-declared pipeline funnel (#4222)

Cargo.toml

@@ -463,16 +463,17 @@ csvs_convert = { git = "https://github.com/jqnatividad/csvs_convert", branch = "
 # use our patched fork of whatlang to bump hashbrown until our PR is merged
 whatlang = { git = "https://github.com/jqnatividad/whatlang-rs", branch = "bump-hashbrown-to-16" }
 
-# Pinned to our stacked branch `deps/webdriver-downloader-0.17` (PR
+# Pinned to our stacked branch `deps/webdriver-downloader-0.17+funnel` (PR
 # https://github.com/plotly/plotly.rs/pull/428), a strict superset of the earlier pins:
 #   - MapLibre `map`-subplot traces qsv's `viz` maps use (ChoroplethMap, DensityMap,
 #     ScatterMap; upstream PRs #406/#410/#415/#416) — now merged into plotly:main
 #   - plotly.js 3.7 attribute backfill (legendrank/zorder/fillgradient/... ; PR #421)
 #   - new trace types Indicator, Histogram2d, Icicle + layout-level `uirevision` (PR #423)
 #   - new trace types Splom, Parcats (PR #425) — used by qsv `viz splom`/`viz parcats`
 #   - webdriver-downloader 0.17 (PR #427) — used by qsv `viz_static` for headless Chromium/Firefox export
-# When #421/#423/#425/#428 merge upstream, repoint this to a plotly.rs release/main rev.
-plotly = { git = "https://github.com/dathere/plotly", branch = "deps/webdriver-downloader-0.17" }
+#   - Funnel trace + layout funnelmode (PR #432) — used by qsv `viz smart` pipeline funnels
+# When #421/#423/#425/#428/#432 merge upstream, repoint this to a plotly.rs release/main rev.
+plotly = { git = "https://github.com/dathere/plotly", branch = "deps/webdriver-downloader-0.17+funnel" }
 
 # TRACKING: remove this patch when upstream resolves it (see exit conditions below).
 #

docs/help/describegpt.md

@@ -228,7 +228,7 @@ qsv describegpt --help
 |--------|------|-------------|--------|
 |  `‑‑num‑examples`  | integer | The number of Example values to include in the dictionary. | `5` |
 |  `‑‑truncate‑str`  | integer | The maximum length of an Example value in the dictionary. An ellipsis is appended to the truncated value. If zero, no truncation is performed. Bounds the prompt sent to the LLM, and applies to the csv/json/semantic-md dictionary output. The jsonschema output is machine-facing, so it always carries the full, untruncated Example values. | `25` |
-| &nbsp;`‑‑infer‑content‑type`&nbsp; | flag | Also have the LLM classify each field's semantic "Content Type", mapped to a curated, documented vocabulary (e.g. email, city, category, name, credit card, etc.) see <https://github.com/dathere/qsv/blob/master/src/cmd/synthesize/faker_map.rs>. Adds a "Content Type" column/field to the Data Dictionary output. Fields where cardinality equals the row count (i.e. every row has a distinct non-null value - primary keys, surrogate keys, sequence numbers) are deterministically classified as "unique_id", overriding any token the LLM returned for that field. For Date/DateTime fields, the LLM also infers the column's strftime date format (e.g. "date:%m/%d/%Y"); the Markdown, JSON & JSON Schema dictionaries then render Min/Max AND Examples in that inferred format so they match how the dates actually appear in the data, instead of qsv's normalized form. (TSV output keeps Min/Max & Examples in qsv's raw normalized form.) For a CONTINUOUS numeric measure on a canonical scale (a percentage, a 0-1 ratio/probability, a bounded index), the LLM also proposes an "x-qsv.gauge_range" [min, max]; qsv keeps it only when the field is a numeric measure AND the observed data lies within it, so that a "viz smart" dictionary-driven dashboard draws that KPI tile as a GAUGE. (A KPI "vs target" delta uses "x-qsv.target", which is a GOAL you hand-author - never inferred.) |  |
+| &nbsp;`‑‑infer‑content‑type`&nbsp; | flag | Also have the LLM classify each field's semantic "Content Type", mapped to a curated, documented vocabulary (e.g. email, city, category, name, credit card, etc.) see <https://github.com/dathere/qsv/blob/master/src/cmd/synthesize/faker_map.rs>. Adds a "Content Type" column/field to the Data Dictionary output. Fields where cardinality equals the row count (i.e. every row has a distinct non-null value - primary keys, surrogate keys, sequence numbers) are deterministically classified as "unique_id", overriding any token the LLM returned for that field. For Date/DateTime fields, the LLM also infers the column's strftime date format (e.g. "date:%m/%d/%Y"); the Markdown, JSON & JSON Schema dictionaries then render Min/Max AND Examples in that inferred format so they match how the dates actually appear in the data, instead of qsv's normalized form. (TSV output keeps Min/Max & Examples in qsv's raw normalized form.) For a CONTINUOUS numeric measure on a canonical scale (a percentage, a 0-1 ratio/probability, a bounded index), the LLM also proposes an "x-qsv.gauge_range" [min, max]; qsv keeps it only when the field is a numeric measure AND the observed data lies within it, so that a "viz smart" dictionary-driven dashboard draws that KPI tile as a GAUGE. (A KPI "vs target" delta uses "x-qsv.target", which is a GOAL you hand-author - never inferred.) The LLM also infers dataset-level INTER-COLUMN RELATIONSHIPS, emitted as a "relationships" array (top-level in the JSON dictionary, and in the dataset-level "x-qsv" object of the JSON Schema one). Each entry has a "kind" of "joint", "ordered", "correlated" or "pipeline", plus a "members" list of the field names involved. A "pipeline" is a process whose stages narrow monotonically - planned to committed to spent, impressions to clicks to conversions. Its stages may be separate COLUMNS (listed in "members", upstream first - the opposite direction from "ordered", which ascends) or the VALUES of one category column (via "stage_column", an ordered "stages" list, and an optional "value_column" to sum). "qsv viz smart --dictionary" reads these to draw its pipeline funnel panel; edit them in the saved schema to correct or add one. |  |
 | &nbsp;`‑‑infer‑null‑values`&nbsp; | flag | Also have the LLM propose each field's null sentinels - literal values that stand in for "missing" (e.g. NULL, N/A, -999, 9999-12-31). Emitted into the JSON Schema dictionary's per-property "x-qsv" object, split into two lists that carry different warranties.<ul><li>"null_values" - proposed by the LLM AND independently confirmed by qsv to occur as a literal value in that String column. Every observed casing is listed.</li><li>"null_candidates" - proposed by the LLM but NOT confirmable by any scan; numeric/date placeholders that parse as valid values of the column's own type (-999 is a legal integer), or tokens never seen in the data. Each carries "confirm_required": true.</li></ul> A proposal that is IMPOSSIBLE is dropped rather than reported: a non-numeric token in a numeric column (no cell could hold it, or the column would not have typed as numeric), and an echo of the null placeholder the Frequency Distribution displays for cells qsv already counts as empty. A confirmed "null_values" entry means only that the literal is PRESENT in the column; the judgment that it MEANS "missing" remains the LLM's. This complements "qsv denull", which reports a narrower set - only the columns that would promote to a numeric type once their sentinels are blanked. A purely categorical column (status = ok/pending/NULL) is outside denull's remit. Sentinels are REPORTED, never applied - qsv does not modify your data on an LLM's say-so. Only "qsv denull --apply" masks values, and only in the columns it independently confirmed. Only rendered by "--format JSONSchema"; the other formats ignore them. |  |
 | &nbsp;`‑‑two‑pass`&nbsp; | flag | Run a second LLM call that takes the full first-pass Data Dictionary as JSON context and refines each field's Label, Description and (when --infer-content-type is set) Content Type using cross-field awareness. The LLM can then relate fields that belong together (e.g. street_no + street_name + city + state + zip describing a single mailing address; first_name + last_name naming a single person; lat + lng forming a coordinate pair). The refined dictionary becomes the emitted output and is also what downstream Description, Tags and Prompt inference phases see as dictionary context. Roughly doubles dictionary LLM cost and latency, so opt-in. Most useful when combined with --infer-content-type. Allowed with the --dictionary, --all and --prompt inference flags. Mutually exclusive with --prepare-context and --process-response (MCP sampling is single-turn per inference phase). |  |
 | &nbsp;`‑‑addl‑cols`&nbsp; | flag | Add additional columns to the dictionary from the Summary Statistics. |  |