Skip to content

kev-finetune: compare complete questions when detecting label conflicts - #240

Open
aakash-kharb wants to merge 1 commit into
jaredpalmer:mainfrom
aakash-kharb:fix/compare-complete-question-definitions
Open

aakash-kharb wants to merge 1 commit into
jaredpalmer:mainfrom
aakash-kharb:fix/compare-complete-question-definitions

Conversation

@aakash-kharb

Copy link
Copy Markdown
Contributor

The fine-tuning splitter detects conflicting labels by state, question ID and instructions, but does not compare the question's type or criteria. For the same state and instructions, score criteria ["bad", "good"] with label 1 and ["bad", "neutral", "good"] with label 2 both mean “good”. The current key treats them as contradictory and drops every record for that state. A fixture containing 60 such states loses all 120 valid records.

Include type and criteria in the question identity used for conflict detection. Sorted object keys keep equivalent choice criteria comparable, while array order stays significant for score levels. Exact duplicates and contradictory answers to the same complete question are still removed. State grouping is unchanged, so variants of one state stay in a single partition.

This fixes record retention in the local splitter; it does not change model behavior or make an accuracy claim.

Test plan

  • At dd375f27fd61, the added tests reproduce 4 failures; after the fix, uv run python -m pytest tests/test_skill_scripts.py -q passes all 27 tests.
  • Check differing score scales, choice descriptions, and question types, plus a genuine conflict with reordered dictionary keys.
  • The CLI retains all 120 fixture records in train/calibration/development partitions of 84/18/18; no state appears in multiple partitions.
  • Ordinary inputs produce byte-identical train, calibration, development and summary files before and after the fix.
  • Full Python CI test list in a CPU environment on macOS (Python 3.13, optional MLX packages omitted): 564 passed, 26 skipped, 2 warnings in 244.49s (0:04:04).

The default Apple Silicon serving environment exposes four pre-existing device-test failures from a fake Checkpoint without hybrid_base; the same failures reproduce on untouched upstream. The CPU environment passes all 63 device tests. Upstream Linux CI is green at the tested base. No test assertions or skip conditions were changed to obtain these results.

evidence

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant