[Feature] Model Scaling and Request Routing for Multi-LoRA Serving

[Suyi32]

🚀 Feature Description and Motivation

I would like to propose support for model scaling and request routing for Multi-LoRA serving in AIBrix.

This feature would enable AIBrix to scale LoRA adapters at runtime and route requests more effectively, improving flexibility and making LoRA serving more practical and efficient.

Use Case

This feature would make AIBrix more capable in Multi-LoRA serving scenarios where many LoRA adapters need to be served efficiently on shared infrastructure.

Proposed Solution

Task Breakdown

1. Add a Multi-LoRA serving example to the documentation and profile Multi-LoRA serving performance in AIBrix.

   • Measure performance tradeoffs, especially TTFT and TPOT latency degradation in Multi-LoRA serving scenarios.

2. Design and implement adapter placement/scaling and LoRA-aware request routing.

   • Currently, ModelAdapter.spec.replicas behaves as follows:

     • If replicas = 1, AIBrix skips scoring/routing and routes directly.
     • If replicas is omitted, the AIBrix gateway load-balances requests among pods.

   • To better support Multi-LoRA serving, I propose introducing two mechanisms:

     • Adapter placement policies: control where adapters are loaded based on base-model identity, max_lora_rank, max_loras, max_cpu_loras, and related constraints. Autoscale LoRA adapters when necessary.
     • LoRA-aware request routing: make routing decisions based on adapter state and pod load. Routing should prefer pods based on whether the requested adapter is already loaded on GPU and ready, the number of active requests, available capacity, memory pressure, and other relevant signals.