Stage-1 confidence is a free 0-1 field with no rubric and no decision consumer, yet it ships to the report and CSV as stage1_confidence — across 213 Stage-2-verified findings it barely separates agreed from refuted (AUC 0.57) — define it, consume it, or drop it

[gadievron]

Summary (enhancement — no wrong line; a ruling request)

Stage-1 confidence is a free 0-1 field with no rubric and no decision
consumer, yet it is emitted to the report record and the CSV as
stage1_confidence — presented as triage-meaningful. Across the campaign's
213 Stage-2-verified findings it barely separates agreed from refuted
(AUC 0.57). The ask is a ruling among: (a) give it a rubric, (b) give it a
consumer (verify priority, report sort), or (c) drop it from the report and
CSV.

Mechanism (at ad2bb7e)

1. prompts/vulnerability_analysis.py:254 — the reply shape asks for
   "confidence": 0.0-1.0 with no rubric, while the adjacent severity and
   cwe_id fields each get one.
2. Nothing at ad2bb7e thresholds or compares it: no confidence <,
   min_confidence, or consumer outside the report path (core/scanner.py:141
   is the llm-reachability signal field, a different confidence). But Confirmed findings carry almost no structured metadata, and no severity field exists #215
   deliberately threads it into the report record (core/reporter.py:613)
   and the CSV exporter names it stage1_confidence
   (report/csv_export.py:260,278).

Executed (the discriminability measurement)

The AUC computation, self-verifying on an inline example when run without
arguments:

$ python3 conf_auc.py
inline example: AUC = 1.000 (3 agreed at 0.9, 3 refuted at 0.5 — perfect separation)

conf_auc.py — full script

#!/usr/bin/env python3
"""Mann-Whitney AUC of a confidence score against a boolean outcome.
Feed it JSON lines {"confidence": x, "agree": true|false}; with no arguments,
self-verifies on an inline example."""
import json, sys

def auc(pairs):
    pos = [c for c, a in pairs if a]
    neg = [c for c, a in pairs if not a]
    if not pos or not neg:
        return None
    ranked = sorted(c for c, _ in pairs)
    ranks, i = {}, 0
    while i < len(ranked):
        j = i
        while j < len(ranked) and ranked[j] == ranked[i]:
            j += 1
        ranks[ranked[i]] = (i + j + 1) / 2
        i = j
    R1 = sum(ranks[c] for c in pos)
    U1 = R1 - len(pos) * (len(pos) + 1) / 2
    return U1 / (len(pos) * len(neg))

if len(sys.argv) > 1:
    pairs = [(float(json.loads(l)["confidence"]), bool(json.loads(l)["agree"]))
             for l in open(sys.argv[1])]
    print(f"AUC = {auc(pairs):.3f} (n={len(pairs)})")
else:
    pairs = [(0.9, True), (0.9, True), (0.9, True), (0.5, False), (0.5, False), (0.5, False)]
    print("inline example: AUC = %.3f (3 agreed at 0.9, 3 refuted at 0.5 — perfect separation)" % auc(pairs))

The campaign measurement (a private committed corpus; numbers quoted, method
above): over every results_verified.json — 213 VULNERABLE rows with a
Stage-2 verification and a Stage-1 confidence (112 agreed / 101 refuted):
agreed mean confidence 0.713 vs refuted 0.686; 12 distinct values
(0.35-0.92); AUC 0.569. Stage-2 agreement sits at 50-67% at six of the seven
values with n>=7 (0.6 / 0.7 / 0.75 / 0.8 / 0.85 / 0.9; n=178); the
seventh, 0.55 (n=28), sits at 32% — the one value that behaves as a
confidence should. The remaining five rows are 0.5 (n=3, 67%) and four
singletons — 0.35, 0.82 and 0.88 at 0%, 0.92 at 100%. Two disclosed limits: Stage-2
agreement is a proxy label, not ground truth, and it is context-sensitive
(the no-context arm skews it — the context-fallback issue); and the sample
is findings that reached verification, not all findings.

Falsifier

A rubric or a decision consumer for Stage-1 confidence at a future HEAD, or
a labeled corpus on which it discriminates (AUC >= 0.7). Authored at filing
time; no falsifier existed in the source record.

Prior art

#345 (closed — the llm-reachability promotion threshold, a different
confidence field); #215 (threaded this field into the report record); #720
(the context-sensitivity of the Stage-2 proxy). No open issue names Stage-1
confidence.

Fix-direction: neither — a ruling request; the field gains meaning or
disappears from the emitted surfaces, and no finding count changes.