pass-at-k

Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with a 95% interval, train with GRPO/SFT/DPO, prove every gain on a held-out set. uv add whileai

Evaluate AI agents with Unix-style pipeline commands. Schema-driven adapters for any CLI agent, trajectory capture, pass@k metrics, and multi-run comparison.

A local harness for agent teams on Claude Code and Codex CLI: only what they prove counts as done. Waves and lanes split the work, reviewers from another model family judge it, councils keep the dissent that lost, typed decisions with Jev are measured before they are trusted, and approvals are bound to the bytes they approved. No model calls.

Codebase for Orthogonal Diverse Diffusion. We present a lightweight, training free method for improving sampling diversity and Pass@k in Diffusion Language Models.

Does coordinating several LLM attempts beat a single one? Local models, an exact oracle, repeat trials — measured on Advent of Code.

A reliability instrument for action-taking LLM agents: ground-truth, severity-priced damage measured across repeated runs

Offline-first Python framework for prompt-based program synthesis, test-based evaluation, and iterative repair

How many runs before your eval means anything? Reliability statistics for stochastic evals: audit miss rates, exact intervals, runs-needed.

Structured toolkit for evaluating and ranking AI-generated code — runs candidates against tests, scores them on a weighted rubric, and explains the verdict.

Marco de Production Readiness para agentes LLM de pre-venta: robustez (pass^k), seguridad ante prompt injection y cumplimiento de la Ley 21.719, en un veredicto go/no-go reproducible.

A tiny, dependency-free Python harness for scoring LLM answers against a gold set (exact / contains / numeric matching, CI-friendly).

Executable benchmark for LLM-generated HPC operational artifacts. Verifies SLURM job scripts by submission, execution and effective resource fit, catching silent errors that sbatch accepts.

Local-first reproducible LLM benchmark suite: 87 tasks, 16 scoring modes, multi-backend discovery, result diffs, and arena ELO artifacts.

Agent reliability harness measuring pass^k, not just pass@1 — with a degenerate-success audit that fails the benchmark if trivial baselines score.

Claude Code skill to test AI tasks, catch regressions, compare prompts and models, and measure pass@k. Built on affaan-m/ECC by @affaan-m.

Interactive calculator for AI agent reliability: why adding verifiers hits a ceiling and retries buy less than p^N suggests. Measured on ~131k runs across 4 benchmarks.

Evaluate LLMs the right way — confidence intervals, unbiased pass@k, significance testing, bias-controlled LLM-as-judge, contamination checks. A drop-in agent skill with a numpy stats core validated against ground truth.

ckl's personal benchmark for doc writing, infra code, and paper reading — one-click evaluation of the latest models via TUI coding CLIs with an interactive dashboard.

Deterministic reliability-floor metrics for long-horizon agents — pass^k, reliability decay curve, meltdown onset. Zero dependencies.

Math routing agent (FastAPI, Agno, Gemini, RAG, guardrails) with an LLM eval toolkit: SWE-bench-style patch harness, unbiased pass@k, rubric review with LLM-as-judge, and Cohen's kappa