fix(check_batch_cost): mark completed all-failed batches as processed

[devin-ai-integration]

TLDR

Problem this solves:

• Completed OpenAI batches with only failed rows never get marked processed
• These poison rows are re-polled forever and starve newer batches

How it solves it:

• Treat status=completed with output_file_id=None as terminal
• Mark it complete + batch_processed=True so polling stops

Relevant issues

Linear ticket

Resolves LIT-4826

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Root cause. In CheckBatchCost.check_batch_cost the loop had only two terminal branches:

if response.status == "completed" and response.output_file_id is not None:
    # track cost + mark batch_processed=True
elif response.status in ("failed", "expired", "cancelled"):
    # mark with that status + batch_processed=True

When every request in a batch fails, OpenAI reports status=completed with output_file_id=None (only an error_file_id). That matches neither branch, so batch_processed never flips to true. The row is re-polled every cycle, and because the poll query is take=MAX_OBJECTS_PER_POLL_CYCLE ordered created_at ASC, accumulating poison rows permanently occupy the oldest-N window and starve newer batches. The stale-cleanup pass does not rescue them either, since it excludes completed.

Fix. The completed-with-no-output_file_id case now falls into the terminal branch and is written back as complete + batch_processed=True with no cost tracking (no rows succeeded, so there is no usage to cost). Partially-failed batches still carry an output file and go through the existing cost-tracking branch unchanged.

Live proof (commit a7f416c4f1) against a source proxy on localhost:4000 (PROXY_BATCH_POLLING_INTERVAL=30) hitting real OpenAI with a managed batch whose every row is invalid (temperature out of range), so OpenAI completes it with only failed rows and no output file.

Upload the managed file and create the batch:

$ curl -s localhost:4000/v1/files -H "Authorization: Bearer sk-1234" \
    -F purpose=batch -F target_model_names=gpt-4o-mini \
    -F "file=@input.jsonl;type=application/jsonl"     # -> managed file id (base64)

$ curl -s localhost:4000/v1/batches -H "Authorization: Bearer sk-1234" \
    -H "Content-Type: application/json" \
    -d '{"input_file_id":"<managed_file_id>","endpoint":"/v1/chat/completions","completion_window":"24h"}'

Once OpenAI finished, the batch is completed with no output file and two failed rows:

$ curl -s localhost:4000/v1/batches/<managed_batch_id> -H "Authorization: Bearer sk-1234" \
    | jq '{status, output_file_id, error_file_id, request_counts}'
{
  "status": "completed",
  "output_file_id": null,
  "error_file_id": "bGl0ZWxsbV9wcm94eTph...",
  "request_counts": { "completed": 0, "failed": 2, "total": 2 }
}

With the fix, the poller retires the row instead of re-polling it forever:

$ psql -c "select status, batch_processed, (file_object->>'output_file_id') as output_file_id,
           (file_object->'request_counts') as counts
           from \"LiteLLM_ManagedObjectTable\" where file_purpose='batch' order by created_at desc limit 1;"
   status   | batch_processed | output_file_id |                  counts
------------+-----------------+----------------+-------------------------------------------
 completed  | t               |                | {"total": 2, "failed": 2, "completed": 0}

batch_processed=t on a completed batch with a null output_file_id is only reachable through the new branch. On the pre-fix source the row stays batch_processed=f and is re-selected every cycle.

A regression test pins this: tests/proxy_unit_tests/test_check_batch_cost.py::TestCheckBatchCost::test_completed_with_no_output_file_marks_job_processed drives a completed response with output_file_id=None through the real check_batch_cost loop and asserts the row is marked complete + batch_processed=True and that no output file is fetched. It fails on the pre-fix source and passes after.

Type

🐛 Bug Fix

Changes

• enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py: the terminal branch now also handles status=completed with output_file_id=None, writing status=complete + batch_processed=True.
• tests/proxy_unit_tests/test_check_batch_cost.py: added a regression test for the all-failed completed batch.

Link to Devin session: https://app.devin.ai/sessions/31841d4336cd405c8a5b57fb9611dc77

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Greptile Summary

This PR prevents completed all-failed batches from remaining in the polling queue.

• Treats provider responses with status="completed" and no output file as terminal.
• Persists the established managed-object status "complete" and sets batch_processed=True.
• Adds a regression test confirming the output file is not fetched and the batch is no longer eligible for polling.

Confidence Score: 5/5

The PR appears safe to merge, with the new terminal case aligned with existing persistence and polling conventions.

Completed batches without output files are now marked processed, while batches with output files retain the existing cost-tracking path and legacy-schema polling continues to exclude the persisted complete status.

Important Files Changed

Filename	Overview
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py	Extends terminal batch handling to completed responses without output files, consistently using the existing completed-to-complete status mapping.
tests/proxy_unit_tests/test_check_batch_cost.py	Adds focused regression coverage for persisting and retiring an all-failed completed batch without attempting output-file retrieval.

_{Reviews (1): Last reviewed commit: "fix(check_batch_cost): mark completed ba..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!