Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,17 @@ datasets/math_500/

# IFBench — keep the third-party test set out of the image; materialized at runtime.
datasets/ifbench/

# The 7 VLM datasets are builder-only (docs/dataset-licenses.md) — never bake
# them into the image; materialized + mounted at runtime.
datasets/mme/
datasets/mmstar/
datasets/ocrbench/
datasets/mmmu_val/
datasets/realworldqa/
datasets/mmbench_en_dev/
datasets/pope/

# ASR datasets (ESB, FLEURS: ~20 GB, SPGISpeech non-commercial, GigaSpeech
# gated) — never bake them into the image; materialized + mounted at runtime.
datasets/asr_*/
14 changes: 14 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Code owners for pipette-scores.
#
# GitHub requests a review from these accounts on every pull request. A merge
# to the default branch requires an approval from one of them. These same
# accounts can merge their own pull requests without an approval.
#
# The maintainers accountable for the codebase hold ownership. This set is
# narrower than the accounts that hold admin through corporate team
# membership.
#
# Several accounts on one line mean that any single one of them satisfies the
# requirement.

* @m3m0ryh0l3 @ykhrustalev @ovenmitt @iamstuffed
5 changes: 2 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,12 +69,10 @@ jobs:
cache-suffix: ${{ github.job }}
- run: uv sync --extra dev
- run: uv run pytest -x -q
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}

# The installed wheel loads IFBench from the bundled
# `scoring/ifbench/_upstream` copy (force-included at build time); the editable
# installs above only hit the submodule fallback. Do a non-editable install
# installs above only hit the vendor/ifbench-checkers fallback. Do a non-editable install
# (mirrors the Docker build) and import through the bundled path, so a
# packaging regression in the force-include block is caught.
wheel-smoke:
Expand All @@ -99,6 +97,7 @@ jobs:
f = evaluation_lib.__file__.replace("\\", "/")
assert "site-packages" in f, f"not from installed pkg: {f}"
assert "scoring/ifbench/_upstream" in f, f"not the bundled copy: {f}"
assert "pipette-scores:" in open(f).read(), f"not the patched vendor/ifbench-checkers copy: {f}"
assert callable(test_instruction_following_loose)
print("bundled IFBench _upstream OK:", f)
PY
Expand Down
20 changes: 20 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,23 @@ datasets/math_500/
# others. The builder (scripts/build_ifbench_dataset.py) materializes it from the
# vendored vendor/ifbench submodule. Not committing it means we don't redistribute.
datasets/ifbench/

# The 7 VLM benchmark datasets are builder-only: a license audit concluded none
# may be redistributed in transformed form from this repo (RealWorldQA is
# CC-BY-ND; the rest have non-commercial / uncleared-image terms — see
# docs/dataset-licenses.md). The builders (scripts/build_<eval>_dataset.py)
# materialize them locally / at deploy from VLMEvalKit's pinned upstream TSVs;
# they must NEVER be committed.
datasets/mme/
datasets/mmstar/
datasets/ocrbench/
datasets/mmmu_val/
datasets/realworldqa/
datasets/mmbench_en_dev/
datasets/pope/

# The 16 ASR datasets (Open ASR Leaderboard, FLEURS) are builder-only too: SPGISpeech is
# non-commercial and GigaSpeech gated (docs/dataset-licenses.md). The builder
# (scripts/build_asr_datasets.py) refuses an output inside this checkout; this
# is the backstop for one copied in. They must NEVER be committed.
datasets/asr_*/
2 changes: 1 addition & 1 deletion CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ Assume good intent, keep technical disagreements focused on the work, and help m

Project maintainers may moderate issues, pull requests, discussions, or other project spaces to uphold these expectations.

To report conduct concerns, contact the project maintainers privately.
To report conduct concerns, contact the project maintainers privately. The maintainers are the accounts listed in [`.github/CODEOWNERS`](.github/CODEOWNERS).
47 changes: 43 additions & 4 deletions NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,48 @@ This repository bundles and adapts third-party code. Full license texts are in
THIRD_PARTY_LICENSES.md.

Vendored verbatim
- vendor/ifbench — the upstream IFBench repository, from allenai/IFBench
(Apache-2.0), vendored as a git submodule (LICENSE at vendor/ifbench/LICENSE).
Only its test set is used.
- packages/pipette-scores/pipette_scores/scoring/asr/open_asr_normalizer/ — the Open
ASR Leaderboard's English text normalizer (normalizer.py, english_abbreviations.py),
from huggingface/open_asr_leaderboard @ 67e8bd6acea240819ad67080f6f31e15d4a90da5
(Apache-2.0, (c) 2022 The OpenAI team and The HuggingFace Team; mostly copied, per
its header, from openai/whisper, MIT, (c) 2022 OpenAI), and its multilingual
normalizer and compound-pair alignment (multilingual.py: unchanged excerpts of
normalizer/data_utils.py and normalizer/eval_utils.py @
62a457c5c32d0de4df77e9470308aae7d45b055f, Apache-2.0).
Provenance and hashes of the ASR files: scoring/asr/NOTICE.md.

Vendored with modifications
- packages/pipette-scores/pipette_scores/scoring/math_500/prm800k/grading/ — the PRM800K
mathematical equivalence grader, from openai/prm800k (MIT, (c) 2023 OpenAI). Its
mathematical equivalence grader (grader.py, math_normalize.py), from openai/prm800k
(MIT, (c) 2023 OpenAI). The changes are a relative import and raw-string regex
literals; the behavior is the same. Its __init__.py is Liquid AI's. Its
math_normalize.py is, per its header, largely copied from Hendrycks' MATH
`math_equivalence` (hendrycks/math, MIT, (c) 2021 Dan Hendrycks).
- vendor/ifbench — the upstream IFBench scoring source, from allenai/IFBench
(Apache-2.0), vendored as a git submodule (LICENSE at vendor/ifbench/LICENSE).
At build time it is bundled into the wheel under
- vendor/ifbench-checkers — the IFBench scoring source (evaluation_lib.py,
instructions.py, instructions_registry.py, instructions_util.py), from
allenai/IFBench @ 1a902f9 (Apache-2.0, (c) Allen Institute for AI and The Google
Research Authors). Liquid AI changed instructions.py, instructions_util.py and
evaluation_lib.py; each carries a notice, and the changes are listed in
vendor/ifbench-checkers/README.md.
At build time these files are bundled into the wheel under
pipette_scores/scoring/ifbench/_upstream/ (including the LICENSE).

Adapted
- packages/pipette-scores/pipette_scores/dataset_catalog/_asr_audio.py —
`resample_16k_mono` follows the decoding sequence of huggingface/datasets
2.19.0's `Audio.decode_example` (Apache-2.0, (c) The HuggingFace Datasets
Authors) and librosa's `to_mono`, `resample` (res_type "soxr_hq") and
`util.fix_length` (librosa/librosa, ISC, (c) 2013--2026 librosa development team).
- packages/pipette-scores/pipette_scores/scoring/vlm_upstream/, scoring/profile_answers.py,
dataset_catalog/prompt_profiles.py, dataset_catalog/_vlm_build.py and _vlm_common.py, and
the seven VLM dataset builders in packages/pipette-scores/scripts/ — answer extraction and
metrics, a model wrapper's answer extraction, model-specific prompts and the datasets'
default prompts and pinned sources, from open-compass/VLMEvalKit @ 8eca8e5 (Apache-2.0,
(c) 2023 VLMEvalKit Authors), modified by Liquid AI (deterministic only: no LLM judge, no
random choice). Each file names what it ports; THIRD_PARTY_LICENSES.md lists them.
- packages/pipette-scores/pipette_scores/scoring/math_500/math_generic.py — the robust-boxed
numeric normalization (`normalize_numeric_answer`) is copied from groq/openbench
`robust_boxed.py` (MIT, (c) 2025 Groq, Inc.).
Expand All @@ -30,6 +62,13 @@ Liquid AI original work — vendored from a separate Liquid repo
- packages/pipette-scores/pipette_scores/scoring/ifstruct.py and the vendored
`vendor/ifstruct` validator are Liquid AI's own (Liquid4All/ifstruct); they carry
no third-party license and are covered by this repository's license.
- packages/pipette-scores/pipette_scores/scoring/math_500/liquid_evals_parser.py holds
unchanged excerpts of Liquid AI's Liquid4All/liquid_evals (liquid_evals/parser.py
@ d5534cf). Its sympy grader derives from openai/prm800k (MIT, (c) 2023 OpenAI)
and its mathd normalization from hendrycks/math (MIT, (c) 2021 Dan Hendrycks);
both licenses are in THIRD_PARTY_LICENSES.md.
- packages/pipette-scores/pipette_scores/scoring/completion_text.py
(`remove_thinking_tags`) is vendored from the same liquid_evals/parser.py.

This repository's original work is licensed under the Apache License, Version 2.0.
See LICENSE for the full license text.
38 changes: 22 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,19 +9,22 @@ Python monorepo hosting two packages for LLM edge-evaluation:

Plus shared infrastructure:

- **`datasets/`** — LFS-tracked canonical eval datasets (parquet/jsonl).
- **`datasets/`** — the committed `ifstruct` dataset (LFS). Builder scripts create the other eval datasets in the same layout.
- **`vendor/ifstruct/`** — submodule: standalone IFStruct evaluator (`Liquid4All/ifstruct`).
- **`vendor/ifbench/`** — submodule: upstream IFBench instruction checkers (`allenai/IFBench`).
- **`vendor/ifbench/`** — submodule: upstream IFBench test set (`allenai/IFBench`).
- **`vendor/ifbench-checkers/`** — the IFBench instruction checkers (`allenai/IFBench` @ `1a902f9`, the revision liquid_evals scores with), vendored with small service patches.
- **`docker/pipette-scores.Dockerfile`** — production image for the scores service.

## Supported evals

| Eval | Type | Description |
|------|------|-------------|
| `ifbench` | Generation | Instruction-following benchmark (backed by `vendor/ifbench`) |
| `ifbench` | Generation | Instruction-following benchmark (test set from `vendor/ifbench`, checkers from `vendor/ifbench-checkers`) |
| `ifstruct` | Generation | Structured format (JSON/YAML) generation (backed by `vendor/ifstruct`) |
| `gpqa_diamond` | Generation | Graduate-level science MCQ (A–D), generative + regex extraction |
| `math_500` | Generation | Competition math, `\boxed{}` answer graded by `score_math_generic` |
| `math_500` | Generation | Competition math, `\boxed{}` answer graded by the grader that the dataset version records |
| VLM evals | Vision | `mme`, `mmstar`, `ocrbench`, `mmmu_val`, `realworldqa`, `mmbench_en_dev`, `pope` (see [datasets](packages/pipette-scores/docs/datasets.md)) |
| ASR evals | Speech | `asr_esb_*` (Open ASR Leaderboard English sets) and `asr_fleurs_*` (FLEURS locales), scored by word error rate (see [ASR evals](packages/pipette-scores/docs/asr-evals.md)) |

## Quick start

Expand All @@ -47,8 +50,8 @@ uv run ruff format --check

### Running the tests

The full test suite runs with **no credentials and no network access**. All 152
tests pass offline:
The full test suite runs with **no credentials and no network access**. The
live ASR reference tests are skipped unless `PIPETTE_ASR_REFERENCE_ENV` is set:

```bash
uv sync --extra dev
Expand All @@ -57,33 +60,36 @@ uv run pytest

No test requires a HuggingFace token. The suite never contacts the HuggingFace
Hub: `packages/pipette-scores/tests/conftest.py` materializes small local
parquet/jsonl fixtures in `tmp_path` for every eval, and the GPQA fixture builds
parquet/jsonl fixtures in `tmp_path` for the text evals, and the GPQA fixture builds
a synthetic two-row stand-in whose `content_sha256` matches what the loader
verifies — so the gated upstream dataset is never downloaded. The tests in
verifies — so the gated upstream dataset is never downloaded. The VLM and ASR
tests build their fixtures in their own test files. The tests in
`test_integration_scoring.py` are "integration" only in the sense that they
exercise the real loaders and scorers end to end; their inputs are the checked-in
fixtures under `tests/fixtures/`, the LFS-tracked `datasets/` tree, and
`vendor/ifbench/data/IFBench_test.jsonl` from the submodule.

HuggingFace access is needed only **outside** the test suite, to materialize the
gated GPQA-Diamond dataset for an actual eval run:
HuggingFace access is needed only **outside** the test suite, to build the gated
datasets for an actual eval run. For example, GPQA-Diamond:

```bash
HF_TOKEN=... uv run --project packages/pipette-scores \
python packages/pipette-scores/scripts/build_gpqa_diamond_dataset.py
```

That script is the only consumer of `HF_TOKEN` in the repository. It downloads
the gated [`Idavidrein/gpqa`](https://huggingface.co/datasets/Idavidrein/gpqa)
dataset, which requires accepting the dataset's terms on HuggingFace first — see
[datasets](packages/pipette-scores/docs/datasets.md). Contributors who only need
to run tests, lint, or the service against the other evals do not need it.
That script downloads the gated
[`Idavidrein/gpqa`](https://huggingface.co/datasets/Idavidrein/gpqa) dataset.
Accept the dataset's terms on HuggingFace first — see
[datasets](packages/pipette-scores/docs/datasets.md). The ASR builder
(`scripts/build_asr_datasets.py`) also reads gated sets, such as GigaSpeech.
Contributors who only run tests, lint, or the service against already built
datasets do not need a token.

## Documentation

Per-package docs:

- Scores: [api reference](packages/pipette-scores/docs/httpapi.md) · [docker deployment](packages/pipette-scores/docs/docker.md) · [scoring logic](packages/pipette-scores/docs/scoring.md) · [datasets](packages/pipette-scores/docs/datasets.md)
- Scores: [api reference](packages/pipette-scores/docs/httpapi.md) · [docker deployment](packages/pipette-scores/docs/docker.md) · [scoring logic](packages/pipette-scores/docs/scoring.md) · [datasets](packages/pipette-scores/docs/datasets.md) · [ASR evals](packages/pipette-scores/docs/asr-evals.md) · [eval packages](packages/pipette-scores/docs/packages.md)
- Calibration: [architecture](packages/pipette-calibration/docs/architecture.md) · [creating a dataset](packages/pipette-calibration/docs/creating-a-dataset.md) · [representative selection](packages/pipette-calibration/docs/representative-selection.md) · [verification](packages/pipette-calibration/docs/verification.md) · [slurm runs](packages/pipette-calibration/docs/slurm.md)

## License
Expand Down
Loading
Loading