Repository navigation
kev-finetune: compare complete questions when detecting label conflicts - #240
Open
aakash-kharb wants to merge 1 commit into
Open
aakash-kharb wants to merge 1 commit into
aakash-kharb wants to merge 1 commit into
Conversation
9 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The fine-tuning splitter detects conflicting labels by state, question ID and instructions, but does not compare the question's type or criteria. For the same state and instructions, score criteria
["bad", "good"]with label1and["bad", "neutral", "good"]with label2both mean “good”. The current key treats them as contradictory and drops every record for that state. A fixture containing 60 such states loses all 120 valid records.Include type and criteria in the question identity used for conflict detection. Sorted object keys keep equivalent choice criteria comparable, while array order stays significant for score levels. Exact duplicates and contradictory answers to the same complete question are still removed. State grouping is unchanged, so variants of one state stay in a single partition.
This fixes record retention in the local splitter; it does not change model behavior or make an accuracy claim.
Test plan
dd375f27fd61, the added tests reproduce 4 failures; after the fix,uv run python -m pytest tests/test_skill_scripts.py -qpasses all 27 tests.The default Apple Silicon serving environment exposes four pre-existing device-test failures from a fake
Checkpointwithouthybrid_base; the same failures reproduce on untouched upstream. The CPU environment passes all 63 device tests. Upstream Linux CI is green at the tested base. No test assertions or skip conditions were changed to obtain these results.