[jaredpalmer]

Kev's order tooling covers Choice only: /v1/systemone/permute, --perm_kl, the Permutation and IIA checks in kev.evaluate, and kev.benchmark --rotations. Score had no measurement of its own. The two numbers in #161 — a first-slot rate of 0.334 over all 6 orders of three urgency levels on jaredpalmer/kev-0.8b, and an identical-option control of −0.299 in raw logits — were run by hand and cannot be reproduced from this repo.

scripts/score_presentation_checks.py turns both checks into one JSON report. It scores through the predictor kev.benchmark already uses (a request record in, probabilities or logits out), so --run (LocalPredictor) and --remote (RemotePredictor) are the same measurement.

A checkpoint is loaded at kev.suite.SERVING_CONTEXT (65,536 state tokens, the window kev.serve admits and the one #161 measured in). LocalPredictor's own default is the 384-token training context. Left there, a state the endpoint would score raises ContextOverflow before the first question, and --run and --remote stop being the same measurement. The report records that context (max_state, max_branch, max_packed) on a --run.

The first-slot rate is the share of win-mass on slot 0. The same Score question is asked over --states states and over orders of its K levels: every permutation when K! is at most --orders-max, otherwise that many permutations drawn one at a time with random.Random(0). The sample path does not build the K! list. No positional preference means 1/K (0.333 at K=3).

An exact tie for the maximum splits that one win evenly across the tied slots. tie_rate, the share of answers whose maximum was shared, is reported next to the rate. The rule is in the first_slot_rate docstring. A flat head is the 1/K null: every slot is equal, so the rate is 1/K and tie_rate is 1. On the #232 tip, max(..., key=) gave every tie to the lowest index, and a flat head measured 1.0. A unique argmax still takes the whole win. The decision is read from probabilities when the predictor reports them, and from logits otherwise. A near-tie that is not an exact equality still goes to the unique maximum, including when the other space would have called it a tie. A NaN or an infinity is refused: max() does not order NaN (it either divides by an empty winner list or drops the NaN and splits the rest), and an infinity makes the identical-option difference NaN.

The identical-option control gives every level the same text and reports slot 0 minus the mean over slots. It uses logits when the predictor has them (--raw-logits scores a checkpoint at head temperature 1.0, the "raw" column in #161) and probabilities otherwise. Identical texts make every permutation of the levels the same request: the positions are the slots inside that one answer. The control is one predictor call per state. On the #232 tip it called level_orders(k, K!), which builds every permutation and then scores that same request K! times. A deterministic predictor returns the same number either way. Both of Kev's predictors are deterministic given the request. That is the assumption the one-call control relies on. LocalPredictor runs the pointer head under torch.no_grad after model.eval() (checkpoint.load), so the LoRA dropout is off and nothing is sampled. RemotePredictor posts to /v1/systemone, which is that same readout. /v1/systemone/permute shuffles Choice options; this script does not call it. If a remote model drew a fresh sample on every call, one call would be a noisier estimate of the same quantity, and this script would be the wrong shape. --raw-logits together with --remote is an error: the endpoint returns probabilities, and the flag would otherwise print that number under a name that means raw logits.

--levels is the count when --level-texts is omitted. The default is still the three urgency words. --levels 4 asks four levels named "level 1" through "level 4", which is how #161 varied the identical-option control over K = 3, 4 and 5. Previously --level-texts defaulted to low,medium,high, so --levels was ignored and --levels 5 measured K = 3. Passing both flags with different counts is an error. --level-texts on its own still names the texts.

--states is the number of states in both modes, and with --data it is the first that many rows (rows[:states], fewer when the file is shorter). An empty file, including one of only blank lines, exits before a checkpoint is loaded. Rows are read with kev.suite.read_jsonl. Option keys come from kev.api.question_keys. A Score question is capped at kev.api.MAX_OPTIONS (255).

Score levels are an ordered scale, which is why --rotations leaves Score alone. Reordering "low, medium, high" changes how the question reads as well as where the answer sits, so a high first-slot rate mixes position bias with order-as-meaning. The report carries that caveat (ORDERED_SCALE_CAVEAT) next to the numbers. Only the identical-option control, whose texts are interchangeable, isolates position.

This is report-only. It does not grow kev.benchmark (#98) and it is not a gate. It names what a checkpoint or an endpoint does.

The script, the two checks, and the ordered-scale caveat are @zhongerdeajie's, from #232 (6756115b). This branch ports that tip and applies the corrections from a review of it: one call per state for the identical-option control, sampling that does not materialise K!, ties split with tie_rate beside the slot rate, --states honoured for --data, the canonical helpers for keys and JSONL, the serving window, and --levels actually counting. #232 can be closed once this lands.

I split a tie instead of dropping it. Dropping a tie leaves a flat head with no rate, and the null this check is built around is 1/K. tie_rate is what says the 1/K was made of ties. I did not re-run #161's kev-0.8b numbers: bench_en is not in this repo, and the script sets no shipping threshold.

Test plan

• uv run --extra serve python -m pytest tests/test_unit.py tests/test_compare.py tests/test_device.py tests/test_research.py tests/test_generators.py tests/test_conventions.py tests/test_documents_tools.py tests/test_hard_v1.py tests/test_devtools_v1.py tests/test_breadth_v1.py tests/test_rounds.py tests/test_skill_scripts.py -q — 555 passed, 26 skipped. One test_resume_is_bit_identical case missed bit-identity under local CPU contention and passed when re-run alone; that test trains a tiny model and is not in this diff.
• Flat head: first_slot_rate == 1/K and tie_rate == 1 (test_score_presentation_checks_ties_split_the_win_and_a_flat_head_is_the_null)
• K=1 is one order and a zero control; a tie of the last two slots leaves the first-slot rate at 0; a near-tie is a unique maximum; probabilities decide when logits disagree (test_score_presentation_checks_edges_k_ties_and_which_space_decides)
• A NaN in the space being read is refused (test_score_presentation_checks_non_finite_scores_are_refused)
• Identical-option control is one call per state; level_orders(12, 3) does not call itertools.permutations (test_score_presentation_checks_identical_option_is_one_call_per_state; 12! is 479,001,600)
• --states 2 --data keeps the first two rows: 14 predictor calls for K=3 (2 states × 6 orders, plus 2 control calls). The report's context is SERVING_CONTEXT.
• --levels 4 asks four "level N" texts; an empty --data, a --levels that disagrees with --level-texts, and --raw-logits --remote all exit before a checkpoint is loaded. build_predictor passes SERVING_CONTEXT, not the 384-token training context (test_score_presentation_checks_levels_and_empty_data_fail_before_the_checkpoint)
• A checkpoint read (--run jaredpalmer/kev-0.8b) was not run here. The tests use fake predictors and load no weights.

  

[zhongerdeajie]

Thanks for the port and the credit — and for fixing a genuine bug in my version: max(range(k), key=...) hands every tie to the lowest index, so a flat head would have measured a first-slot rate of 1.0 where the 1/K null predicts 1/3. Splitting the win across tied slots and reporting tie_rate next to the rate is the right shape; I verified _split_win standalone (flat → [1/K]*K + tied, unique → full win, partial tie → 1/len(winners) each). The one-call identical-option control is also strictly better than my version's K! repeats of the same request, and --states applying to --data closes a real inconsistency.

Two things I checked and found faithful to the original: the ordered-scale caveat survives verbatim next to the numbers, and the request record is unchanged (score_request still validates against the serving schema the way the tests here assert).

🤖 Generated with Claude Code

[zhongerdeajie]

Ran the merged presentation checks (scripts/score_presentation_checks.py) against jaredpalmer/kev-0.8b on CPU (raw logits) and swept the number of score levels K=2..6 with neutral level texts, 40 built-in states. The first-slot check enumerates all K! orders at K≤5 and 120 sampled orders at K=6; the identical-option control always enumerates all orders. This extends the two hand-run numbers from #161 (first-slot 0.334, control −0.299, urgency levels): neutral texts take order-as-meaning out of the first-slot rate, so the sweep reads closer to pure position bias.

K	first-slot rate (expected 1/K)	identical-option control (logits)	last-slot share (expected)
2	0.500 (0.500)	−0.014	0.500 (0.500)
3	0.342 (0.333)	+0.083	0.292 (0.333)
4	0.330 (0.250)	+0.187	0.190 (0.250)
5	0.335 (0.200)	+0.273	0.122 (0.200)
6	0.282 (0.167)	+0.346	0.073 (0.167)

Findings:

1. The identical-option control grows almost linearly with K — about +0.09 logits per added level, from −0.014 at K=2 to +0.346 at K=6. Flat at two levels; by six levels the first-listed slot carries over a third of a logit of pure position advantage.
2. Late slots starve. The last slot's decision share falls from 0.500 (K=2) to 0.073 at K=6 — less than half of the 1/K = 0.167 it should get. The excess mass concentrates in the first two positions (slot 1 is the max at K=6 with 0.367).
3. Semantic labels change the shape, not just the numbers. With the default low,medium,high the first-slot rate drops below chance (0.183) and the control's per-slot means put the middle level +0.21 logits above the mean (+0.33 over slot 0) — order-as-meaning mixed with position, exactly the caveat the script prints, so the neutral-text sweep is the cleaner read of pure position.
4. Read the control at raw logits. At the checkpoint's shipped temperature the same semantic-level control reads −0.053 instead of −0.123; across checkpoints that confounds calibration with bias.

Reproduction (per table row; take the first K texts):

python scripts/score_presentation_checks.py --run jaredpalmer/kev-0.8b --raw-logits \
    --states 40 --level-texts "opt a,opt b,opt c,opt d,opt e,opt f" \
    --orders-max 120 --out kev-0.8b-K6.json

JSON reports for all rows available if useful. Happy to run the same sweep on 4B/9B/27B — 4B is feasible on CPU overnight; 9B/27B would want --remote or a GPU.