Skip to content

fix(triage): harden duplicate clustering against test renaming and order Pareto ties by earliest PR - #3059

Open
gttps wants to merge 1 commit into
limitlessprojects:mainfrom
gttps:fix/triage-bot-dup-evasion-and-pareto-order
Open

gttps wants to merge 1 commit into
limitlessprojects:mainfrom
gttps:fix/triage-bot-dup-evasion-and-pareto-order

Conversation

@gttps

@gttps gttps commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Hardens tools/triage_bot.py against two duplicate-evasion and line-golfing loopholes observed when a later copycat PR (#2836) duplicated an earlier compiler fix (#2614 for issue #2613):

  1. Test-Renaming Duplicate Evasion (Jaccard = 1/3 < 0.60):

    • Previously, classify() only grouped PRs into a duplicate cluster when jaccard(sets[a], sets[b]) >= cfg["dup_jaccard"] (0.60).
    • For a surgical 2-file fix (1 compiler source file + 1 new integration test file), copying the compiler fix in loans.rs while renaming the test file drops the 2-file Jaccard similarity to 1/3 = 0.333 < 0.60, bypassing DUP detection entirely.
    • classify() now clusters two PRs together if any of the following hold:
      • jaccard(sets[a], sets[b]) >= cfg["dup_jaccard"], OR
      • Normalized PR titles match (stripping conventional commit prefixes like fix(hy-check): and trailing (#NNN) / (fixes #NNN) suffixes), OR
      • Both PRs reference the same issue number (#NNN via fixes/closes/resolves #NNN or (#NNN) in title/body) and share at least one changed non-test source file.
  2. Canonical Survivor Priority & Line-Golfing Tie-Breaking:

    • Inside a duplicate cluster, members were previously sorted by (-x["additions"], x["number"]) rather than earliest PR number (x["number"], -x["additions"]).
    • Outside duplicate clusters, render_board() (Pareto NOW) and merge_gate() previously broke score ties by x["additions"] ascending, which rewarded copycat PRs that minified multi-case regression tests into \n-escaped one-liners to jump ahead of the original PR.
    • Both cluster survivor selection and LAND-CANDIDATE tie-breaking now prioritize the earliest PR number (x["number"] ascending) first.

Verification

  • Added offline unit test suite tools/test_triage_bot.py (registered in .github/workflows/ci.yml and verified via tools/check_test_registration.py) covering:
    1. Copycat PR renaming a test file (Jaccard = 0.333) with same issue/title as earlier PR #2614 is caught as DUP of #2614, preserving #2614 as the canonical LAND-CANDIDATE.
    2. Copycat PR paraphrasing the title while referencing the same issue #2613 and touching the same source file (loans.rs) is caught as DUP of #2614.
    3. Unrelated PRs touching loans.rs for distinct issues with distinct test files (Jaccard = 0.333) remain independent LAND-CANDIDATEs.
    4. Pareto NOW orders score ties by earliest PR number (x["number"] ascending) rather than fewest additions.

@github-actions github-actions Bot added the bot:land-candidate Shepherd: fresh, gate-clean, verifiable (set from classify()) label Nov 1, 2026
@gttps
gttps force-pushed the fix/triage-bot-dup-evasion-and-pareto-order branch 2 times, most recently from 05b7f1c to 7bbb964 Compare November 2, 2026 01:33
@github-actions github-actions Bot added the bot:dup-of:#2510 Shepherd W4: superseded by cluster survivor label Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Shepherd (Phase A — propose only): this PR's changed-file set overlaps >=0.6 Jaccard with #2510, which carries the larger take. Proposing close as superseded.

This is a proposal, not a close — nothing auto-closes while this comment is fresh. If this PR has unique content (a different rule, extra tests, a different fix), say so or push, and the label comes off. Cluster signal is a file-set heuristic; the verdict stays human.

@github-actions github-actions Bot added the bot:stale-base Shepherd: behind main beyond threshold label Nov 2, 2026
@gttps
gttps force-pushed the fix/triage-bot-dup-evasion-and-pareto-order branch from 7bbb964 to b3b7366 Compare November 4, 2026 04:48
@gttps

gttps commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Not a Duplicate of #2510 (False-Positive File-Set Overlap)

@limitlessprojects — #3059 is not a duplicate of #2510. Shepherd flagged #3059 solely because both PRs modify the exact same 3 files (tools/triage_bot.py, tools/test_triage_bot.py, and .github/workflows/ci.yml), giving a 3/3 = 1.00 >= 0.60 whole-PR file-set Jaccard overlap, even though they fix completely orthogonal issues:

  1. What #3059 fixes (Duplicate Clustering Evasion & Pareto NOW FIFO Tie-Breaking):
    • Hardens classify() duplicate clustering against copycat test-renaming evasion (where a 2-file PR touching 1 compiler source file + 1 renamed test file drops file-set Jaccard to 1/3 = 0.333 < 0.60) by also clustering PRs that share a normalized title or share both a Fixes #N issue reference and non-test source files.
    • Changes DUP cluster canonical survivor election and Pareto NOW tie-breaking at equal score (score == 9, behind_by == 0) from lowest line count (x["additions"] ascending, which rewards test minification) to earliest PR number ((x["number"], x["additions"])).
  2. What #2510 fixes (tests_included Path-Detection Regexes, #2508):
    • Fixes tests_included path matching so tools/test_*.py and related test paths are recognized when scoring PRs.

Both PRs are complementary and address distinct parts of tools/triage_bot.py.

@gttps
gttps force-pushed the fix/triage-bot-dup-evasion-and-pareto-order branch 21 times, most recently from 50f3bfb to 931d390 Compare November 9, 2026 09:45
@gttps
gttps force-pushed the fix/triage-bot-dup-evasion-and-pareto-order branch 4 times, most recently from 8086582 to 1cafcd0 Compare November 12, 2026 13:00
@gttps
gttps force-pushed the fix/triage-bot-dup-evasion-and-pareto-order branch from 1cafcd0 to 1d93b86 Compare November 12, 2026 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:dup-of:#2510 Shepherd W4: superseded by cluster survivor bot:land-candidate Shepherd: fresh, gate-clean, verifiable (set from classify()) bot:stale-base Shepherd: behind main beyond threshold

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant