feat(vllm): add Prometheus /metrics-derived latency metrics (queue/prefill p50/p95)#274
Draft
atnair-amd wants to merge 1 commit into
Draft
feat(vllm): add Prometheus /metrics-derived latency metrics (queue/prefill p50/p95)#274atnair-amd wants to merge 1 commit into
atnair-amd wants to merge 1 commit into
Conversation
…efill p50/p95) Scrapes vLLM's own /metrics endpoint before/after each client run, diffs the queue-time and prefill-time histograms to isolate one cell's observations, and interpolates p50/p95 quantiles matching PromQL's histogram_quantile(). Gated as prom.* metrics via a parallel GATED_PROM_METRICS set (mirroring GATED_GPU_METRICS) to keep out of the locked client.* tiering partition.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/metricsPrometheus endpoint before/after each client run and diffs the queue-time/prefill-time histograms to isolate one sweep cell's observations from a reused server's cumulative counters.p50/p95quantiles matching PromQL'shistogram_quantile(), exposed as gated metrics:prom.queue_time_p50_ms,prom.queue_time_p95_ms,prom.prefill_time_p50_ms,prom.prefill_time_p95_ms.GATED_GPU_METRICSprecedent — a separateGATED_PROM_METRICSset, kept out ofvllm_parsing.py'sclient.*tiering machinery (that partition is asserted exactly by a locked invariant test).capture_gpu_metrics), avoiding the SSH-session race the GPU poller previously hit.Test plan
test_vllm_server_metrics.py(30 tests): bucket/sum/count parsing, before/after diffing, hand-computed interpolation math, end-to-end realistic scrape pairs, I/O-boundary scrape failure modes.test_vllm_config_loader.pywithGATED_PROM_METRICScoverage tests (missing-metric-raises / warns / full-set assertions).test_vllm_report_preset.py,test_vllm_job_server_reuse.py,test_vllm_job_ray_backend.pyall pass unchanged.run_all_unittests.py: 750 tests (34 new), same 2 failures + 3 errors as the unmodified base branch (pre-existing, in unrelatedtest_inferencing_config_loader.py/collection issues — confirmed via A/B stash comparison, not touched by this change).