Repository navigation
kev-finetune: two rubrics on one state are not a label conflict - #242
Open
jaredpalmer wants to merge 4 commits into
Open
jaredpalmer wants to merge 4 commits into
jaredpalmer wants to merge 4 commits into
Conversation
Conflict detection already treats type, instructions and criteria as the question. The histogram and the missing-label warning still pooled every record with that id, so a 2-level score and a 3-level score shared one index. One form per id prints the same line as before. Co-authored-by: aakash-kharb <akgdsc7@gmail.com>
jaredpalmer
marked this pull request as ready for review
October 15, 2026 22:40
A generated name such as "answer (2 options)" replaced that question id's own counts, and sorting ids changed summary.json key order for a file with one form per id. A partition that contained only one of two forms also fell back to the bare id. Names are chosen once from every record the run writes. An id with one form keeps that id, in first-seen order. A taken name gets a #n suffix, and every partition uses it. Co-authored-by: Jared Palmer <hello@jaredpalmer.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
split_data.pydrops a state when one question has two labels. It decided the question from the id and the instructions alone. The same ticket scored on a 2-level scale and again on a 3-level scale was a conflict, and both records were deleted. Reordered choice keys were already the same question. Criteria that actually differ were not.This continues #240. The conflict key is the question id plus
question_form: type, instructions and criteria, dumped withsort_keysso key order does not matter. The label histogram and the missing-label warning still added every record with that id into one counter, so index 1 on the short scale and index 1 on the long scale were one bucket, and the warning read the criteria off whichever record came first.label_groupscounts each form on its own. The summary name is chosen once from every record the run writes, the training file and the holdout together, and every partition reuses it. A file with one form per id keeps the question id, in the order those ids first appear, which is the key ordersummary.jsonstored before this change. A second form is namedrating (2 levels). When two forms would share that name, or the name is already some other question id, the next one isanswer (2 options) #2, and that other id keeps its own counts.The assumption is that a normal fine-tune has one type, one instructions string and one criteria object per question id. That file's report does not change. Two forms are two questions that share an id, and their level indices are counted apart.
data-format.mdsays this. A model that will be served one question should still be trained on one definition. The script no longer deletes the second rubric to enforce that.Test plan
uv run --extra serve python -m pytest tests/test_skill_scripts.py -q: 52 passedrating (2 levels)andrating (3 levels)insummary.jsoné/naïve, with different labels are still a conflict. Omitted noul criteria and"criteria": nullare the same formlabel_tableon a single-definition record still keysdepartment,escalateandfrustration. A file whose ids appear aszebrathenapplekeeps that order insummary.json#2and#3names when the file is reversed. A question idanswer (2 options)is not overwrittenwarnings_forreports missing level2only on the 3-level scale, and names a noul that has criteriaanswer (noul)even when that criteria object has two keys