Repository navigation
feat: sync with internal-pipette-scores - #5
Merged
Merged
Conversation
Bring over the internal changes since the public snapshot (internal e67656a), internal PRs #2-#15: - VLM accuracy evals (7 benchmarks): scoring and dataset builders - ASR evals (Open ASR Leaderboard English, FLEURS): datasets, prompt profiles, WER scoring - eval packages that pipette-mgmt serves - prompt profiles and reasoning parsing for more VL models - MATH-500 2026.10.1 graded as liquid_evals' math500 task - ifbench scored with the liquid_evals checkers (vendor/ifbench-checkers) - ifstruct bumps for nested JSON and oversized numbers - CODEOWNERS Kept private: - ECR publishing in ci.yml (AWS account, IAM role); the public GHCR workflow stays, with only the wheel-smoke check for the patched ifbench-checkers copy ported. - logs/ (Slurm job output) - names of the private CDK repo and the eval-datasets bucket, replaced with "the organization's private storage" - names of internal tools and repos (eval scheduler, other harnesses, the VLMEvalKit fork), replaced with generic descriptions
Fix statements that were wrong or stale against the current code: the GHCR image and tag format, the full eval list (text, VLM, ASR), scorer return shapes and context keys, dataset defaults and committed sets, license records for modified vendored files, and the calibration workflow (dataset names, removed annealing, Slurm resources and failure handling). Replace internal cluster paths with a generic shared root. Also dockerignore the VLM datasets so a local build cannot bake them into an image, and stop passing HF_TOKEN to the test job, which does not use it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The public repository stopped at the August snapshot of internal-pipette-scores and lacks every eval and scoring change made since.
Solution
vendor/ifbench-checkers), and the ifstruct bumpsCODEOWNERSHF_TOKENfrom the test jobTesting
uv run pytest: 1883 passed, 7 skippeduv run ruff check,uv run ruff format --check,uv lock --check