Problem
Large batch workloads can contain tens of millions of requests and run for days or weeks. Treating the full input as one execution unit makes failure recovery expensive: a worker, runtime, or control-plane interruption can require substantial work to be repeated, and users may need to split and resubmit datasets manually.
The current product limit and progress model are not suitable for a 20-million-request workload.
Expected outcome
A user can submit one large logical job. AIBrix divides and executes the work at a recoverable granularity, automatically retries unfinished work, and aggregates progress and results back into the original job.
Internal splitting and retry must be transparent to the user.
Acceptance criteria
- A single logical job supports a 20-million-request input without manual dataset splitting.
- The job can run across multiple days or weeks without depending on one long-lived worker or resource allocation.
- Worker, inference runtime, and control-plane restarts do not discard completed work.
- Recovery retries only unfinished or retryable work rather than restarting the complete input.
- Repeated failures of a small subset of requests do not prevent unrelated work from making progress.
- The final output has one committed result or terminal error for every valid input record.
- Aggregate progress remains available and accurate throughout recovery.
- Cancellation prevents new work from starting and safely concludes work already in flight.
- Control-plane and storage metadata remain practical at the 20-million-request scale and do not require one long-lived orchestration object per request.
- Failure-injection tests cover interruptions near the beginning, middle, and end of execution.
Out of scope
This issue does not prescribe a partition size, checkpoint format, queue implementation, or output storage format.
Problem
Large batch workloads can contain tens of millions of requests and run for days or weeks. Treating the full input as one execution unit makes failure recovery expensive: a worker, runtime, or control-plane interruption can require substantial work to be repeated, and users may need to split and resubmit datasets manually.
The current product limit and progress model are not suitable for a 20-million-request workload.
Expected outcome
A user can submit one large logical job. AIBrix divides and executes the work at a recoverable granularity, automatically retries unfinished work, and aggregates progress and results back into the original job.
Internal splitting and retry must be transparent to the user.
Acceptance criteria
Out of scope
This issue does not prescribe a partition size, checkpoint format, queue implementation, or output storage format.