Skip to content

kev-finetune: two rubrics on one state are not a label conflict - #242

Open
jaredpalmer wants to merge 4 commits into
mainfrom
cursor/question-identity-873c
Open

jaredpalmer wants to merge 4 commits into
mainfrom
cursor/question-identity-873c

Conversation

@jaredpalmer

@jaredpalmer jaredpalmer commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

split_data.py drops a state when one question has two labels. It decided the question from the id and the instructions alone. The same ticket scored on a 2-level scale and again on a 3-level scale was a conflict, and both records were deleted. Reordered choice keys were already the same question. Criteria that actually differ were not.

This continues #240. The conflict key is the question id plus question_form: type, instructions and criteria, dumped with sort_keys so key order does not matter. The label histogram and the missing-label warning still added every record with that id into one counter, so index 1 on the short scale and index 1 on the long scale were one bucket, and the warning read the criteria off whichever record came first. label_groups counts each form on its own. The summary name is chosen once from every record the run writes, the training file and the holdout together, and every partition reuses it. A file with one form per id keeps the question id, in the order those ids first appear, which is the key order summary.json stored before this change. A second form is named rating (2 levels). When two forms would share that name, or the name is already some other question id, the next one is answer (2 options) #2, and that other id keeps its own counts.

The assumption is that a normal fine-tune has one type, one instructions string and one criteria object per question id. That file's report does not change. Two forms are two questions that share an id, and their level indices are counted apart. data-format.md says this. A model that will be served one question should still be trained on one definition. The script no longer deletes the second rubric to enforce that.

Test plan

  • uv run --extra serve python -m pytest tests/test_skill_scripts.py -q: 52 passed
  • Two score lengths on one state stay in the file, share a partition, and appear as rating (2 levels) and rating (3 levels) in summary.json
  • The same two scales on different states keep those names in a partition that contains only one of them. A holdout on the other scale does too
  • Reordered choice keys, including é / naïve, with different labels are still a conflict. Omitted noul criteria and "criteria": null are the same form
  • Different instructions, a reversed score scale, and noul criteria text are not conflicts
  • label_table on a single-definition record still keys department, escalate and frustration. A file whose ids appear as zebra then apple keeps that order in summary.json
  • Three choice forms with the same option count keep the same #2 and #3 names when the file is reversed. A question id answer (2 options) is not overwritten
  • A score with no criteria, and a choice whose criteria is a list, are invalid lines and do not enter the histogram
  • warnings_for reports missing level 2 only on the 3-level scale, and names a noul that has criteria answer (noul) even when that criteria object has two keys
Open in Web Open in Cursor 

aakash-kharb and others added 2 commits October 14, 2026 22:44
Conflict detection already treats type, instructions and criteria as the question. The histogram and the missing-label warning still pooled every record with that id, so a 2-level score and a 3-level score shared one index. One form per id prints the same line as before.

Co-authored-by: aakash-kharb <akgdsc7@gmail.com>
@jaredpalmer
jaredpalmer marked this pull request as ready for review October 15, 2026 22:40
jaredpalmer and others added 2 commits October 15, 2026 18:41
A generated name such as "answer (2 options)" replaced that question id's own counts, and sorting ids changed summary.json key order for a file with one form per id. A partition that contained only one of two forms also fell back to the bare id. Names are chosen once from every record the run writes. An id with one form keeps that id, in first-seen order. A taken name gets a #n suffix, and every partition uses it.

Co-authored-by: Jared Palmer <hello@jaredpalmer.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants