Skip to content

Add ranked-list retrieval benchmark mode#7

Open
lilyjge wants to merge 3 commits into
castorini:mainfrom
lilyjge:codex/ranked-list-retrieval-benchmarks
Open

Add ranked-list retrieval benchmark mode#7
lilyjge wants to merge 3 commits into
castorini:mainfrom
lilyjge:codex/ranked-list-retrieval-benchmarks

Conversation

@lilyjge

@lilyjge lilyjge commented Jul 16, 2026

Copy link
Copy Markdown
Member

Summary

  • add an OUTPUT_MODE=ranked_list path that prompts agents for one deduplicated ranked docid list, persists ranked_docids, enforces the requested depth, and writes TREC run files
  • propagate ranked-list options through single, shared, and sharded launchers, including merged TREC output
  • add query-set-specific trec_eval metrics and full MS MARCO v1 passage dev setup alongside DL19 and DL20
  • document the GPT-5.5 Pyserini REST 2-tool runs and comparison protocol

Recorded results

Query set AP nDCG@10 RR@10 R@1K
DL19 0.3780 0.7381 n/a 0.4811
DL20 0.4492 0.7412 n/a 0.5376
MS MARCO passage dev n/a n/a 0.3300 0.6725

These runs use openai-codex/gpt-5.5, medium thinking, the Pyserini REST msmarco-v1-passage index, and the pyserini-rest-2tool interface (search and read_document). Raw run and evaluation artifacts remain ignored; the protocol and final values are recorded in docs/msmarco-v1-passage-ranked-list-results.md.

The agent emits roughly 23-27 documents per query while the reference systems retrieve 1,000. AP, nDCG@10, and RR@10 are useful ranking comparisons; R@1K is reported but is not depth-matched and should not be treated as a fair recall comparison.

Verification

  • npm run check
  • npm test (230 passing)
  • bash -n scripts/benchmarks/msmarco_v1_passage/setup.sh scripts/benchmarks/msmarco_v1_passage/generate_query_slices.sh
  • npm run prek

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant