PUBLIC-READY
ETB-15 (artifact-blind rebind) with the R01E1-M1 positional-rebinding mechanism: a cache joined on a positional key that is not unique across the two index spaces that produce it
What happens
LightevalTask._get_docs_from_split assigns doc.id = str(ix) where ix comes from an enumerate() that restarts for every split in evaluation_splits (src/lighteval/tasks/lighteval_task.py, _get_docs_from_split, around line 314). So when a task evaluates more than one split, the same id exists once per split and refers to different questions.
SampleCache stores and serves cached model responses under the key (task full name, task hash, sampling method, doc.id) (src/lighteval/utils/cache_management.py, get_samples_to_process_and_cache, cache_samples, get_samples_from_cache). Nothing in the key or in the stored row identifies the split or the content of the question.
Two consequences, both reproduced at pin 932e1f2:
-
Crash plus persistent cache corruption. When both owners of a colliding id are processed in the same call, cache_samples writes two rows with the same sample_id, and get_samples_from_cache then does dataset_df.loc[doc.id], which returns a two-row frame instead of a row. _load_sample feeds that to ModelResponse(**...) and the run dies with TypeError: ModelResponse() got multiple values for keyword argument '2'. The crash happens after the model has already produced its outputs, and the duplicated rows stay in the parquet, so every later run on the same cache keeps crashing until the cache directory is deleted by hand.
-
Silent cross-split rebind. When the set of selected ids changes between two runs (in the PoC: the dataset content changed upstream, which changes the seeded shuffle and the selection; a different --max_samples has the same effect), an id that was cached while owned by split A is served for the question that currently sits at that position in split B. The PoC shows a validation question being served the answer the model produced for a test question, with one model call for four documents and no warning anywhere.
Tasks shipped in this repository use multi-split evaluation, for example lsat_qa (5 task variants with evaluation_splits=["validation", "test"]), lextreme, and legal_summarization, so this is reachable with built-in tasks plus caching enabled.
Repro
PoC: poc_cache_split_collision.py (deterministic, local datasets, no network, no model inference; uses the real @cached decorator, the real SampleCache, and real parquet read/write). Driver: REPRO-E1-F1.sh runs it twice and diffs stdout.
PYTHONPATH=<deps>:<lighteval src> HF_HUB_OFFLINE=1 /usr/bin/python3 poc_cache_split_collision.py
Key output:
== A.RUN2: CRASHED after model execution with TypeError: ... got multiple values for keyword argument '2'
== PART A persisted cache file ... sample_ids=['3', '0', '1', '4', '2', '2', '5'] (duplicates present: True)
== B.RUN2-dataset-v2: {"doc_id": "1", "split": "validation", "question": "VAL-Q1", "served_text": "ANSWER_TO::TEST-Q1", "matches_own_question": false}
== PART B verdict: VULNERABLE (1 silent mismatches of 4)
Impact
For multi-split tasks with the prediction cache enabled: wasted full model runs that crash at the end and a cache directory that must be deleted manually; and, when the selected subset changes between runs, scores computed from responses produced for different questions than the ones reported, with zero signal in the output. The stored detail rows then also misattribute answers, since the doc and the served response no longer correspond.
Suggested fix
Make the cache key content-unique per document. Either include the split name in doc.id (for example f"{split}/{ix}"), or key the cache on a hash of the doc's query and choices rather than the positional id. Additionally, cache_samples should refuse to write two rows with the same sample_id in one batch (that is where the persistent corruption is created), and get_samples_from_cache should fail with a clear message instead of passing a DataFrame slice into ModelResponse(**...).
PUBLIC-READY
ETB-15 (artifact-blind rebind) with the R01E1-M1 positional-rebinding mechanism: a cache joined on a positional key that is not unique across the two index spaces that produce it
What happens
LightevalTask._get_docs_from_splitassignsdoc.id = str(ix)whereixcomes from anenumerate()that restarts for every split inevaluation_splits(src/lighteval/tasks/lighteval_task.py,_get_docs_from_split, around line 314). So when a task evaluates more than one split, the same id exists once per split and refers to different questions.SampleCachestores and serves cached model responses under the key (task full name, task hash, sampling method,doc.id) (src/lighteval/utils/cache_management.py,get_samples_to_process_and_cache,cache_samples,get_samples_from_cache). Nothing in the key or in the stored row identifies the split or the content of the question.Two consequences, both reproduced at pin 932e1f2:
Crash plus persistent cache corruption. When both owners of a colliding id are processed in the same call,
cache_sampleswrites two rows with the samesample_id, andget_samples_from_cachethen doesdataset_df.loc[doc.id], which returns a two-row frame instead of a row._load_samplefeeds that toModelResponse(**...)and the run dies withTypeError: ModelResponse() got multiple values for keyword argument '2'. The crash happens after the model has already produced its outputs, and the duplicated rows stay in the parquet, so every later run on the same cache keeps crashing until the cache directory is deleted by hand.Silent cross-split rebind. When the set of selected ids changes between two runs (in the PoC: the dataset content changed upstream, which changes the seeded shuffle and the selection; a different
--max_sampleshas the same effect), an id that was cached while owned by split A is served for the question that currently sits at that position in split B. The PoC shows a validation question being served the answer the model produced for a test question, with one model call for four documents and no warning anywhere.Tasks shipped in this repository use multi-split evaluation, for example
lsat_qa(5 task variants withevaluation_splits=["validation", "test"]),lextreme, andlegal_summarization, so this is reachable with built-in tasks plus caching enabled.Repro
PoC:
poc_cache_split_collision.py(deterministic, local datasets, no network, no model inference; uses the real@cacheddecorator, the realSampleCache, and real parquet read/write). Driver:REPRO-E1-F1.shruns it twice and diffs stdout.Key output:
Impact
For multi-split tasks with the prediction cache enabled: wasted full model runs that crash at the end and a cache directory that must be deleted manually; and, when the selected subset changes between runs, scores computed from responses produced for different questions than the ones reported, with zero signal in the output. The stored detail rows then also misattribute answers, since the doc and the served response no longer correspond.
Suggested fix
Make the cache key content-unique per document. Either include the split name in
doc.id(for examplef"{split}/{ix}"), or key the cache on a hash of the doc's query and choices rather than the positional id. Additionally,cache_samplesshould refuse to write two rows with the samesample_idin one batch (that is where the persistent corruption is created), andget_samples_from_cacheshould fail with a clear message instead of passing a DataFrame slice intoModelResponse(**...).