Skip to content

Support Castorini prebuilt indexes, topics, and qrels#8

Open
lilyjge wants to merge 1 commit into
mainfrom
prebuilt-indexes-support
Open

Support Castorini prebuilt indexes, topics, and qrels#8
lilyjge wants to merge 1 commit into
mainfrom
prebuilt-indexes-support

Conversation

@lilyjge

@lilyjge lilyjge commented Jul 19, 2026

Copy link
Copy Markdown
Member

Summary

  • load the live Castorini prebuilt-index and Anserini topics/qrels catalogs instead of maintaining a hard-coded asset list
  • add piika prebuilt discovery and setup commands with topic normalization, checksum verification, atomic downloads, mirror fallback, and safe archive extraction
  • register installed benchmark manifests dynamically and preserve upstream source provenance in run snapshots
  • document the workflow, including MS MARCO V2 Passage with DL21

Validation

  • npm run check
  • npx tsx --test tests/prebuilt_catalog.test.ts tests/prebuilt_setup.test.ts tests/benchmarks/registry.test.ts (19/19 passing)
  • live catalog dry-run for msmarco-v2-passage + dl21 + dl21-passage
  • real two-query smoke run against the installed MS MARCO V2 Passage index using the Pyserini REST two-tool interface, openai-codex/gpt-5.5, and medium thinking; zero tool failures

The complete npm test suite was also run. Its fixture-dependent tests fail in this checkout because ignored benchmark assets are absent under data/benchmark-template, data/browsecomp-plus, and data/msmarco-v1-passage; the scoped and static checks above pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant