viz smart: bridge declared pipelines whose stages do not nest (#4222)

Cargo.toml

@@ -472,6 +472,8 @@ whatlang = { git = "https://github.com/jqnatividad/whatlang-rs", branch = "bump-
 #   - new trace types Splom, Parcats (PR #425) — used by qsv `viz splom`/`viz parcats`
 #   - webdriver-downloader 0.17 (PR #427) — used by qsv `viz_static` for headless Chromium/Firefox export
 #   - Funnel trace + layout funnelmode (PR #432) — used by qsv `viz smart` pipeline funnels
+#   - Waterfall trace (PR #432) — used by qsv `viz smart` when a declared pipeline's stages do
+#     NOT nest, so the panel bridges the signed gaps instead of asserting containment
 # When #421/#423/#425/#428/#432 merge upstream, repoint this to a plotly.rs release/main rev.
 plotly = { git = "https://github.com/dathere/plotly", branch = "deps/webdriver-downloader-0.17+funnel" }
 

docs/help/viz.md

@@ -109,18 +109,26 @@ that contains the value, and gains a "vs target" DELTA when it supplies an
 with no headline measure. (Overall dataset completeness - the share of
 non-empty cells - is a quiet "Completeness:" line in the header metadata
 table, not a KPI tile.)
-- pipeline funnel, when the dictionary DECLARES one (see --dictionary). Which
+- pipeline panel, when the dictionary DECLARES one (see --dictionary). Which
 columns are process stages, and in which direction, is semantics rather than
 a statistic - no column-name vocabulary settles it and no statistic does
-either - so a funnel is drawn ONLY from an explicit declaration, never
+either - so the panel is drawn ONLY from an explicit declaration, never
 guessed. Both encodings are supported: stages held in separate measure
 columns, and stages held as values of one category column. Costs one extra
 data pass over the declared stages only.
+The declaration fixes WHICH columns and in WHAT order; the numbers decide the
+FORM. A funnel's band widths are a containment claim, so one is drawn only
+while the stage totals never grow. If any stage outruns the one before it, the
+same declaration is drawn as a BRIDGE instead: the signed difference between
+consecutive totals, each step labelled as the arithmetic difference it is
+rather than as a flow. A funnel there would render a band wider than the one
+above it, asserting the opposite of the data. The subtitle says which form was
+used and why.
 Stage order is the declared order and is never re-sorted by size. For the
 column encoding, row-wise containment (does each stage nest inside the one
-before it?) is MEASURED and disclosed in the subtitle, but never refuses the
-panel: a pipeline whose stages overrun is a finding to name, not a reason to
-show nothing. Totals sum over the rows complete across every declared stage,
+before it?) is MEASURED and disclosed in the subtitle - separately from the
+form, since rows can overrun while the totals still shrink, or nest while the
+totals grow. Totals sum over the rows complete across every declared stage,
 so they do NOT match `stats.sum`; the subtitle always discloses that
 denominator. See also the standalone `qsv viz funnel` chart type, which takes
 its stage order from the file and needs no dictionary.
@@ -447,7 +455,7 @@ qsv viz --help
 |  `‑‑no‑other`  | flag | Omit the "Other (N)" aggregate bar from frequency bar charts. It collects the categories beyond --limit (N = how many distinct categories were rolled up) and is shown by default. When just ONE category is left over, it is charted under its own name instead of as an opaque "Other (1)" bar. |  |
 | &nbsp;`‑‑smarter`&nbsp; | flag | Before building the dashboard, run `qsv moarstats --advanced` to enrich the stats cache with distribution-shape statistics (bimodality, entropy, skewness, outlier share, Gini). This unlocks histograms for bimodal columns, frequency bars for concentrated high-cardinality columns, skew/outlier hints on box panels, and Lorenz curves for the most unequal additive measures (high Gini). Costs one extra pass over the data and writes <stem>.stats.csv, its sidecars, and an .idx index (like running `qsv moarstats` manually). On geocode-enabled builds it also enriches map point hovers with the US FIPS code and annotates the spatial-extent summary with the country's continent; the county is always shown in map hovers, with or without --smarter. Only affects `smart`. Applied only with default parsing; inputs using --no-headers or a custom --delimiter fall back to the standard dashboard. |  |
 | &nbsp;`‑‑hierarchy‑style`&nbsp; | string | For `smart`, the chart used for the categorical part-to-whole hierarchy panel (built when 2+ low-cardinality dimensions exist). One of: auto (default), treemap, sunburst, icicle. auto follows best practice — a treemap for a shallow 2-level hierarchy (accurate size comparison) and a sunburst for a deep 3-level one (parent child structure); icicle is an opt-in level-aligned alternative. Only affects `smart`. |  |
-| &nbsp;`‑‑dictionary`&nbsp; | string | Use a describegpt Data Dictionary to guide panel selection from each field's semantic role/concept (falling back to its content type) instead of relying on column statistics alone: dimensions and numeric codes (ward, census_tract, zone) become bars, measures get box/correlation/trend panels, date/datetime columns feed the time-series panel (not noisy frequency bars), identifiers / PII / free-text are skipped, and lat/lon feed the map. Field labels are shown as panel subtitles beneath the field-name titles. Columns the dictionary cannot classify still use the statistical heuristic. <src> is one of: "infer" to run describegpt on the input now (with description, infer-content-type, two-pass and jsonschema output; requires an LLM configured) and use its output; or a path to an existing describegpt dictionary file (jsonschema or json). With "infer", the generated dictionary is saved beside the input as <stem>.schema.json so you can fine-tune it; if that file already exists, it is reused as-is (skipping the LLM) - edit it to fine-tune, or delete it to force a fresh re-infer. Generation/read failures soft-fall back to the stats-only dashboard. The dictionary also drives the KPI overview row via two optional per-field hints in a property's "x-qsv" object (edit them in the saved schema to fine-tune). A "gauge_range" of [min, max] on a continuous numeric measure renders its KPI tile as a GAUGE on that canonical scale (e.g. [0,1] for a ratio, [0,100] for a percent); qsv keeps it only when the observed data lies within the range, so a mis-scaled range can't draw a misleading dial, and "infer" emits it for canonical-scale measures. A "target" number on a measure renders a "vs target" DELTA against that goal (value minus target) - a GOAL you supply, never a fabricated prior-period baseline, so "infer" never emits it; hand-author it. The dictionary is also the ONLY source of the pipeline funnel panel, declared in the dataset-level "x-qsv" object as a "relationships" entry with "kind": "pipeline". Two encodings, both hand-editable: stages as COLUMNS - "members" lists the stage columns in process order, WIDEST/UPSTREAM FIRST (note this is the opposite direction from a "kind":"ordered" group, which ascends), e.g. {"kind":"pipeline", "members":["planned_amt","committed_amt","spent_amt"]} stages as ROW VALUES - "stage_column" names the category column, "stages" lists its values in process order, and an optional "value_column" names the measure to sum per stage (omit it to count rows), e.g. {"kind":"pipeline","members":["stage","revenue"], "stage_column":"stage", "stages":["Impression","Click","Lead","Conversion"], "value_column":"revenue"} Declared order is authoritative and is never re-sorted by size, so a stage that outruns its predecessor stays visible instead of being quietly reordered away. A declaration naming a missing column, or a stage that is an average/rate rather than a summable amount, is skipped with a note rather than erroring. Only affects `smart`. |  |
+| &nbsp;`‑‑dictionary`&nbsp; | string | Use a describegpt Data Dictionary to guide panel selection from each field's semantic role/concept (falling back to its content type) instead of relying on column statistics alone: dimensions and numeric codes (ward, census_tract, zone) become bars, measures get box/correlation/trend panels, date/datetime columns feed the time-series panel (not noisy frequency bars), identifiers / PII / free-text are skipped, and lat/lon feed the map. Field labels are shown as panel subtitles beneath the field-name titles. Columns the dictionary cannot classify still use the statistical heuristic. <src> is one of: "infer" to run describegpt on the input now (with description, infer-content-type, two-pass and jsonschema output; requires an LLM configured) and use its output; or a path to an existing describegpt dictionary file (jsonschema or json). With "infer", the generated dictionary is saved beside the input as <stem>.schema.json so you can fine-tune it; if that file already exists, it is reused as-is (skipping the LLM) - edit it to fine-tune, or delete it to force a fresh re-infer. Generation/read failures soft-fall back to the stats-only dashboard. The dictionary also drives the KPI overview row via two optional per-field hints in a property's "x-qsv" object (edit them in the saved schema to fine-tune). A "gauge_range" of [min, max] on a continuous numeric measure renders its KPI tile as a GAUGE on that canonical scale (e.g. [0,1] for a ratio, [0,100] for a percent); qsv keeps it only when the observed data lies within the range, so a mis-scaled range can't draw a misleading dial, and "infer" emits it for canonical-scale measures. A "target" number on a measure renders a "vs target" DELTA against that goal (value minus target) - a GOAL you supply, never a fabricated prior-period baseline, so "infer" never emits it; hand-author it. The dictionary is also the ONLY source of the pipeline panel (drawn as a funnel, or as a bridge when the stage totals do not nest), declared in the dataset-level "x-qsv" object as a "relationships" entry with "kind": "pipeline". Two encodings, both hand-editable: stages as COLUMNS - "members" lists the stage columns in process order, WIDEST/UPSTREAM FIRST (note this is the opposite direction from a "kind":"ordered" group, which ascends), e.g. {"kind":"pipeline", "members":["planned_amt","committed_amt","spent_amt"]} stages as ROW VALUES - "stage_column" names the category column, "stages" lists its values in process order, and an optional "value_column" names the measure to sum per stage (omit it to count rows), e.g. {"kind":"pipeline","members":["stage","revenue"], "stage_column":"stage", "stages":["Impression","Click","Lead","Conversion"], "value_column":"revenue"} Declared order is authoritative and is never re-sorted by size, so a stage that outruns its predecessor stays visible instead of being quietly reordered away - it switches the panel to a bridge rather than drawing a funnel that widens. A declaration naming a missing column, or a stage that is an average/rate rather than a summable amount, is skipped with a note rather than erroring. Only affects `smart`. |  |
 | &nbsp;`‑‑dictionary‑context`&nbsp; | string | Path to a file with extra context about the dataset (a glossary, README, data dictionary, PDF, etc.) forwarded to describegpt as --context-file when `--dictionary infer` generates the dictionary. Better context yields better role/concept/label/grain tags, hence a better dashboard. Ignored unless `--dictionary infer` is used (it does not apply when reading an existing dictionary file). Only affects `smart`. |  |
 | &nbsp;`‑‑dict‑info`&nbsp; | flag | When a usable Data Dictionary is available (per --dictionary), add a "Data Dictionary" link beneath the dashboard title and an info icon on each panel title: hovering shows that column's dictionary description; clicking opens a human-friendly rendering of the dictionary in a side drawer NEXT TO the plots (embedded in the dashboard file - no extra file is written), scrolled to and highlighting that column's entry. The drawer is open by default on load and can be dismissed with its close button or Esc. The dictionary page carries a role-tinted table of contents and per-column "View chart" links back to the panels, plus a row of download buttons: the dictionary itself as JSON Schema, the frequency counts the dashboard actually charted, and every generated sidecar this run read (the stats cache and its metadata, the frequency cache when it was reused, and the bivariate stats CSV when freshly written). A sidecar qsv wrote but viz never read - the human-readable <stem>.stats.csv - is NOT offered, since nothing can show it describes the same computation the dashboard used. Every file is BUNDLED into the HTML, so anyone you send the dashboard to can download them with no access to your machine; absolute local paths are stripped from the embedded metadata so sharing a dashboard doesn't disclose your directory layout. Sidecars over 4 MB are skipped with a note. The drawer's popout button opens the same document in its own browser tab instead (needs a browser that allows user-initiated pop-ups). HTML output only; ignored with a note when no dictionary is available or when exporting an image. Only affects `smart`. |  |
 | &nbsp;`‑‑dataset‑pid`&nbsp; | string | A persistent identifier (PID) for the dataset - typically a full URL such as a DOI (<https://doi.org/10.1234/abc>) or other citable link. When set, a "PID" row is added to the metadata table at the top of the dashboard. http(s) and mailto values become a clickable link (opened in a new tab); any other scheme is shown as plain text rather than linked. HTML output only. Only affects `smart`. |  |