Skip to content

feat: sync with internal-pipette-scores - #5

Merged
m3m0ryh0l3 merged 2 commits into
mainfrom
kyle/sync-internal-2026-11-09
Nov 9, 2026
Merged

m3m0ryh0l3 merged 2 commits into
mainfrom
kyle/sync-internal-2026-11-09

Conversation

@m3m0ryh0l3

@m3m0ryh0l3 m3m0ryh0l3 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Problem
The public repository stopped at the August snapshot of internal-pipette-scores and lacks every eval and scoring change made since.

Solution

  • Adds the VLM accuracy evals (7 benchmarks) and the ASR evals (Open ASR Leaderboard English, FLEURS), with their scorers and dataset builders
  • Adds the eval packages that pipette-mgmt serves, and prompt profiles and reasoning parsing for more VL models
  • Adds MATH-500 2026.10.1, ifbench scoring with the liquid_evals checkers (vendor/ifbench-checkers), and the ifstruct bumps
  • Adds CODEOWNERS
  • Corrects docs that were wrong or stale against the code, dockerignores the VLM datasets, and drops the unused HF_TOKEN from the test job
  • Keeps internal deploy config, job logs, and the names of internal repos, buckets, and tools out of the public tree; the public GHCR workflow stays

Testing

  • uv run pytest: 1883 passed, 7 skipped
  • uv run ruff check, uv run ruff format --check, uv lock --check

Bring over the internal changes since the public snapshot (internal
e67656a), internal PRs #2-#15:

- VLM accuracy evals (7 benchmarks): scoring and dataset builders
- ASR evals (Open ASR Leaderboard English, FLEURS): datasets, prompt
  profiles, WER scoring
- eval packages that pipette-mgmt serves
- prompt profiles and reasoning parsing for more VL models
- MATH-500 2026.10.1 graded as liquid_evals' math500 task
- ifbench scored with the liquid_evals checkers (vendor/ifbench-checkers)
- ifstruct bumps for nested JSON and oversized numbers
- CODEOWNERS

Kept private:
- ECR publishing in ci.yml (AWS account, IAM role); the public GHCR
  workflow stays, with only the wheel-smoke check for the patched
  ifbench-checkers copy ported.
- logs/ (Slurm job output)
- names of the private CDK repo and the eval-datasets bucket, replaced
  with "the organization's private storage"
- names of internal tools and repos (eval scheduler, other harnesses,
  the VLMEvalKit fork), replaced with generic descriptions
Fix statements that were wrong or stale against the current code: the
GHCR image and tag format, the full eval list (text, VLM, ASR), scorer
return shapes and context keys, dataset defaults and committed sets,
license records for modified vendored files, and the calibration
workflow (dataset names, removed annealing, Slurm resources and
failure handling). Replace internal cluster paths with a generic
shared root.

Also dockerignore the VLM datasets so a local build cannot bake them
into an image, and stop passing HF_TOKEN to the test job, which does
not use it.
@m3m0ryh0l3
m3m0ryh0l3 merged commit 8f868fe into main Nov 9, 2026
8 checks passed
@m3m0ryh0l3
m3m0ryh0l3 deleted the kyle/sync-internal-2026-11-09 branch November 9, 2026 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant