Issues

• Model output can inject forged chat-role messages into the judge prompt (ChatML split, no sanitization)

  tatsu-lab/alpaca_eval#476 · Filter by author nikhilagrawalsf opened on Oct 17, 2026

• Failed judge calls are dropped from the win-rate denominator and nothing gates n_total against the instruction count, so win_rate rises as rows fail

  tatsu-lab/alpaca_eval#475 · Filter by author shaurya416 opened on Sep 23, 2026

• OpenAI-compat: openai_completions with openai_api_base /v1, not Hugging Face

  tatsu-lab/alpaca_eval#474 · Filter by author cursor opened on Aug 28, 2026

• Optional EvalPort interop for model_outputs/reference_outputs and pairwise preferences

  tatsu-lab/alpaca_eval#473 · Filter by author adhabnr-ux opened on Aug 22, 2026

• regex_parser extracts pairwise verdicts by match order over free text; candidates can seed the tokens it trusts

  tatsu-lab/alpaca_eval#472 · Filter by author AUTHENSOR opened on Aug 14, 2026

• AlpacaEval 2 leaderboard: discrete_win_rate for aligner-2b_gpt-4-turbo-2024-04-09 disagrees with its own n_wins/n_draws (only row of 223)

  tatsu-lab/alpaca_eval#471 · Filter by author ipezygj opened on Jul 30, 2026

• EvalScope integration: one-command evaluation support for AlpacaEval2.0

  tatsu-lab/alpaca_eval#470 · Filter by author Yunnglin opened on Jul 24, 2026

• Feature: Prompt compression to reduce evaluation token costs

  tatsu-lab/alpaca_eval#466 · Filter by author arjunkshah opened on Jun 29, 2026

• Unexpected keyword argument "leaderboard_mode" in print_leaderboard() when running alpaca_eval make_leaderboard

  tatsu-lab/alpaca_eval#460 · Filter by author takahashihiroshi opened on Jan 5, 2026

• dataset scripts no longer supported

  tatsu-lab/alpaca_eval#454 · Filter by author sunnyych opened on Aug 28, 2025
  1

• Cannot get length controlled win rate

  tatsu-lab/alpaca_eval#453 · Filter by author FocusLiwen opened on Jun 20, 2025
  1

• What models are used when GPT-4-1106-preview is inaccessible?

  tatsu-lab/alpaca_eval#452 · Filter by author fulQuan opened on Jun 10, 2025

• How to determine the possibility of another LLM as a judge model?

  tatsu-lab/alpaca_eval#450 · Filter by author AmberXu98 opened on Jun 6, 2025

• When fn_completions is set to huggingface_local_completions, the model will be reloaded for each chunk causing OOM error

  tatsu-lab/alpaca_eval#449 · Filter by author arushibear opened on Jun 3, 2025

• Why is the output “[\n {‘model’: ‘model_1’, ‘rank’: 1},\n {‘model’: ‘model_2’, ‘rank’: 2}\n]” and the final result is “preference”: 2.0, when these two do not correspond? What is the reason for this?

  tatsu-lab/alpaca_eval#447 · Filter by author FHhui opened on May 12, 2025

• "You must manually fix the score pair" Error While Using Custom Evaluator

  tatsu-lab/alpaca_eval#446 · Filter by author tommycwh opened on May 8, 2025
  1

• ValueError: cannot reindex on an axis with duplicate labels

  tatsu-lab/alpaca_eval#445 · Filter by author Liesy opened on May 1, 2025
  1

• when model output is like deepseek-r1 ,wondering if it is better to omit content inside <think>

  tatsu-lab/alpaca_eval#442 · Filter by author jijivski opened on Mar 22, 2025

• Allow specifying model revision

  tatsu-lab/alpaca_eval#441 · Filter by author tomtseng opened on Feb 8, 2025

• Can you realize deepseek as a judge?

  tatsu-lab/alpaca_eval#440 · Filter by author NuoJohnChen opened on Feb 8, 2025
  4

• openai.BadRequestError

  tatsu-lab/alpaca_eval#436 · Filter by author EnlightenedAI opened on Jan 24, 2025

• Question about best_of when use_beam_search=True

  tatsu-lab/alpaca_eval#435 · Filter by author ZeguanXiao opened on Jan 18, 2025

• alpaca eval config file

  tatsu-lab/alpaca_eval#433 · Filter by author zwRuan opened on Jan 4, 2025

• can't match the win rate posted on the leaderboard

  tatsu-lab/alpaca_eval#431 · Filter by author lindseyfeng opened on Jan 2, 2025

• The response was filtered due to the prompt triggering Azure OpenAI's content management policy

  tatsu-lab/alpaca_eval#430 · Filter by author stamina1121 opened on Jan 2, 2025
  3