Python monorepo hosting two packages for LLM edge-evaluation:
| Package | Role |
|---|---|
pipette-scores |
HTTP service that serves eval samples and scores client completions |
pipette-calibration |
Offline pipeline for building and calibrating eval datasets |
Plus shared infrastructure:
datasets/— the committedifstructdataset (LFS). Builder scripts create the other eval datasets in the same layout.vendor/ifstruct/— submodule: standalone IFStruct evaluator (Liquid4All/ifstruct).vendor/ifbench/— submodule: upstream IFBench test set (allenai/IFBench).vendor/ifbench-checkers/— the IFBench instruction checkers (allenai/IFBench@1a902f9, the revision liquid_evals scores with), vendored with small service patches.docker/pipette-scores.Dockerfile— production image for the scores service.
| Eval | Type | Description |
|---|---|---|
ifbench |
Generation | Instruction-following benchmark (test set from vendor/ifbench, checkers from vendor/ifbench-checkers) |
ifstruct |
Generation | Structured format (JSON/YAML) generation (backed by vendor/ifstruct) |
gpqa_diamond |
Generation | Graduate-level science MCQ (A–D), generative + regex extraction |
math_500 |
Generation | Competition math, \boxed{} answer graded by the grader that the dataset version records |
| VLM evals | Vision | mme, mmstar, ocrbench, mmmu_val, realworldqa, mmbench_en_dev, pope (see datasets) |
| ASR evals | Speech | asr_esb_* (Open ASR Leaderboard English sets) and asr_fleurs_* (FLEURS locales), scored by word error rate (see ASR evals) |
# Clone with submodules + LFS
git clone --recurse-submodules https://github.com/Liquid4All/pipette-scores.git
cd pipette-scores
git lfs pull
# Install the workspace
uv sync --extra dev
# Run the pipette-scores service (from workspace root)
uv run --package pipette-scores uvicorn pipette_scores.api.app:app --reload
# Run all tests
uv run pytest
# Run lint / format
uv run ruff check
uv run ruff format --checkThe full test suite runs with no credentials and no network access. The
live ASR reference tests are skipped unless PIPETTE_ASR_REFERENCE_ENV is set:
uv sync --extra dev
uv run pytestNo test requires a HuggingFace token. The suite never contacts the HuggingFace
Hub: packages/pipette-scores/tests/conftest.py materializes small local
parquet/jsonl fixtures in tmp_path for the text evals, and the GPQA fixture builds
a synthetic two-row stand-in whose content_sha256 matches what the loader
verifies — so the gated upstream dataset is never downloaded. The VLM and ASR
tests build their fixtures in their own test files. The tests in
test_integration_scoring.py are "integration" only in the sense that they
exercise the real loaders and scorers end to end; their inputs are the checked-in
fixtures under tests/fixtures/, the LFS-tracked datasets/ tree, and
vendor/ifbench/data/IFBench_test.jsonl from the submodule.
HuggingFace access is needed only outside the test suite, to build the gated datasets for an actual eval run. For example, GPQA-Diamond:
HF_TOKEN=... uv run --project packages/pipette-scores \
python packages/pipette-scores/scripts/build_gpqa_diamond_dataset.pyThat script downloads the gated
Idavidrein/gpqa dataset.
Accept the dataset's terms on HuggingFace first — see
datasets. The ASR builder
(scripts/build_asr_datasets.py) also reads gated sets, such as GigaSpeech.
Contributors who only run tests, lint, or the service against already built
datasets do not need a token.
Per-package docs:
- Scores: api reference · docker deployment · scoring logic · datasets · ASR evals · eval packages
- Calibration: architecture · creating a dataset · representative selection · verification · slurm runs
This repository's original work is licensed under the Apache License, Version 2.0. See LICENSE and NOTICE for details and third-party attributions.