[jaredpalmer]

split_data.py, convert_data.py, and generate_data.py read JSONL with str.splitlines(). Python also breaks a line on U+2028, U+2029, and U+0085. json.dumps with ensure_ascii=False writes those characters raw inside a string, and the line is still valid JSON. split_data.py reports both halves as not JSON, drops the record, and exits 0. Each of those characters also pushes every later line number up by one. On a JSONL file, convert_data.py raises JSONDecodeError. generate_data.py raises the same error when it reads records already in --out, and when it reads --examples. kev.train --data accepts the file. kev.suite.read_jsonl has split on "\n" only since #101.

The skill scripts now split on "\n" only, the separator they write. generate_data.py reads --out and --examples through one function, read_jsonl. upload_data in kev_modal.py counted lines with the same splitlines() call, so one record became two in the count printed before training. That count splits on "\n" as well. The upload was not run.

The scripts are standard library only, so each reader copies the read_jsonl split. The file is assumed to use LF endings. The scripts write LF. A CRLF row still parses, because json.loads treats the leftover carriage return as whitespace.

The second convert_data.py in the issue's repro exits 1 with 2 rows skipped. That file is already Kev records, and it has no department column for the label map. A flat JSONL file, the input convert_data.py maps, converts and keeps the character. extract_workload.py still uses splitlines(), because it numbers source lines.

Fixes #276. Reported by @luc-pimentel, with the repro and the diagnosis.

Test plan

• test_jsonl_readers_keep_unicode_line_separators failed before the change with read_records returning [], and passes after. It covers U+2028, U+2029, and U+0085, the line number of a following broken line, convert_data.rows_from, existing_records, --examples via --dry-run, upload_data's count with batch_upload replaced, and a CRLF row.
• Issue repro before the change, from skills/kev-finetune. CSV convert writes 2 records. split_data.py prints invalid: line 1 and invalid: line 2, 1 valid records, exit 0. JSONL convert, existing_records, and --examples raise JSONDecodeError: Unterminated string. U+2029 and U+0085 match. A following not json line is reported as line 4. The count expression on that file is 3.
• Same repro after the change. split_data.py prints 2 valid records (0 invalid lines...), exit 0. existing_records returns 2. --examples exits 0 and the prompt contains U+2028. U+2029 and U+0085 read as 2 records. The following broken line is line 3. Flat JSONL with one record per character converts to 3 records. The count expression is 2.
• python3 -m pytest tests/test_skill_scripts.py -q: 58 passed.