feat(framework): allow overriding the vLLM repository to target upstream vllm - #1029
feat(framework): allow overriding the vLLM repository to target upstream vllm#1029akii96 wants to merge 1 commit into
Conversation
Keep ROCm/vllm as the default while allowing operators to target an upstream vLLM checkout and PR stream. Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com>
2503264 to
7e194c3
Compare
|
@chaojhou not sure if we should open up to custom containers without some manual sanitization. I'm concerned aspects of the recommended container will be missing if a user adds in their own container/setup script and lead to secondary bugs. What do you think? The challenge I can see for egs. is that interfaces haven't been tested for different vLLM/SGLang versions and the right approach might be for us to plan how to enable all vLLM/SGLang versions |
|
Hi @akii96 ROCm/vllm has been deprecated (branches are used for local development AFAIK) and we use vllm-project/vllm for all practical usage. Can we just change the ROCm/vllm to vllm-project/vllm rather than adding new variables and code? |
|
@devalshahamd Yeah, we can! But after the discussion in meeting today I am just pondering now about what is expected of framework agent? For instance if I were to take a docker image with the latest upstream vllm image installed and then install hyperloom + the dependencies on it and run the optimize loop the framework agent then in most cases only rediscovers what is already merged and available in the installed vllm, is that correct? 😅 It might surface an env var or serving arg at best but then starts to overlap with what the hyperloom's grid sweeper does Did I get this correctly ? |
Summary
Allow operators to override Hyperloom’s vLLM repository while preserving the existing ROCm-fork default.
Motivation
Most vLLM development happens in
vllm-project/vllm, including new model support, quantization formats, scheduler changes, serving features, and general kernel integration. Deployments built from upstream vLLM should be able to search that larger and more current PR stream.Hyperloom currently searches
ROCm/vllmunconditionally. That fork may be valuable for AMD-specific fixes that may have not yet reached upstream, so this change preserves it as the default while allowing operators to select the repository that matches their installed runtime!Changes
HYPERLOOM_VLLM_REPO_URL.ROCm/vllmas the default.Testing