Skip to content

[Bug] Fix external metric fetching for GPU optimizer autoscaling and replace dead scale-down annotation in samples #2616

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
varungup90 merged 5 commits into from
Aug 26, 2026
Merged

[Bug] Fix external metric fetching for GPU optimizer autoscaling and replace dead scale-down annotation in samples #2616

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5c91d92
Replace the unparsed kpa.autoscaling.aibrix.ai/scale-down-delay annot…
yaojiejia Aug 26, 2026
a9ef25c
Fetch external autoscaling metrics directly from the configured endpo…
yaojiejia Aug 26, 2026
93718d5
Merge branch 'main' into yaojiejia/fix-optimizer-kpa-autoscaling
varungup90 Aug 26, 2026
45a47ac
Stop the retry backoff timer on context cancellation instead of leavi…
yaojiejia Aug 26, 2026
0230459
Stop counting external metric fetch failures in the llm_engine_metric…
yaojiejia Aug 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion benchmarks/scenarios/autoscaling/deepseek-llm-7b-chat/kpa.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ metadata:
app.kubernetes.io/name: aibrix
app.kubernetes.io/managed-by: kustomize
annotations:
kpa.autoscaling.aibrix.ai/scale-down-delay: 3m
autoscaling.aibrix.ai/scale-down-cooldown-window: 3m
spec:
scalingStrategy: KPA
minReplicas: 1
Expand Down
4 changes: 2 additions & 2 deletions benchmarks/scenarios/autoscaling/deepseek-llm-7b-chat/optimizer-kpa.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,14 @@ metadata:
app.kubernetes.io/name: aibrix
app.kubernetes.io/managed-by: kustomize
annotations:
kpa.autoscaling.aibrix.ai/scale-down-delay: 0s
autoscaling.aibrix.ai/scale-down-cooldown-window: 0s
spec:
scalingStrategy: KPA
minReplicas: 1
maxReplicas: 8
metricsSources:
- endpoint: aibrix-gpu-optimizer.aibrix-system.svc.cluster.local:8080
metricSourceType: domain
metricSourceType: external
path: /metrics/default/deepseek-llm-7b-chat
protocolType: http
targetMetric: vllm:deployment_replicas
Expand Down
2 changes: 1 addition & 1 deletion config/samples/autoscaling_v1alpha1_mock_llama.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ metadata:
annotations:
autoscaling.aibrix.ai/max-scale-up-rate: "2"
autoscaling.aibrix.ai/max-scale-down-rate: "2"
kpa.autoscaling.aibrix.ai/scale-down-delay: "60s"
autoscaling.aibrix.ai/scale-down-cooldown-window: "60s"
namespace: aibrix-system
spec:
scaleTargetRef:
Expand Down
4 changes: 2 additions & 2 deletions development/app/config/heterogeneous/simulator_a40/patch_podautoscaler_a40.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@ kind: PodAutoscaler
metadata:
name: podautoscaler-simulator-llama2-7b-a40
annotations:
kpa.autoscaling.aibrix.ai/scale-down-delay: 0s
autoscaling.aibrix.ai/scale-down-cooldown-window: 0s
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: simulator-llama2-7b-a40
metricsSources:
- metricSourceType: domain
- metricSourceType: external
protocolType: http
endpoint: aibrix-gpu-optimizer.aibrix-system.svc.cluster.local:8080
path: /metrics/default/simulator-llama2-7b-a40
Expand Down
4 changes: 2 additions & 2 deletions development/app/config/simulator/patch_podautoscaler_a100.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@ kind: PodAutoscaler
metadata:
name: podautoscaler-simulator-llama2-7b-a100
annotations:
kpa.autoscaling.aibrix.ai/scale-down-delay: 0s
autoscaling.aibrix.ai/scale-down-cooldown-window: 0s
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: simulator-llama2-7b-a100
metricsSources:
- metricSourceType: domain
- metricSourceType: external
protocolType: http
endpoint: aibrix-gpu-optimizer.aibrix-system.svc.cluster.local:8080
path: /metrics/default/simulator-llama2-7b-a100
Expand Down
2 changes: 1 addition & 1 deletion development/app/config/templates/podautoscaler/podautoscaler_kpa.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ metadata:
app.kubernetes.io/name: aibrix
app.kubernetes.io/managed-by: kustomize
annotations:
kpa.autoscaling.aibrix.ai/scale-down-delay: 30s
autoscaling.aibrix.ai/scale-down-cooldown-window: 30s
namespace: default
spec:
scaleTargetRef:
Expand Down
2 changes: 1 addition & 1 deletion development/tutorials/distributed/fleet-autoscaling.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ metadata:
labels:
app.kubernetes.io/name: aibrix
annotations:
kpa.autoscaling.aibrix.ai/scale-down-delay: 1m
autoscaling.aibrix.ai/scale-down-cooldown-window: 1m
spec:
scalingStrategy: KPA
minReplicas: 1
Expand Down
14 changes: 11 additions & 3 deletions pkg/controller/podautoscaler/metrics/fetcher.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ package metrics
import (
"context"
"fmt"
"strings"

v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
Expand Down Expand Up @@ -272,9 +273,16 @@ func (f *ExternalMetricsFetcher) fetchFromGPUOptimizer(ctx context.Context, pod
"path", source.Path,
"metric", source.TargetMetric)

// Use the centralized engine fetcher for external HTTP calls
// This gives us a global value that we need to adapt to per-pod semantics
metricValue, err := f.engineFetcher.FetchTypedMetric(ctx, source.Endpoint, "external", "gpu-optimizer", source.TargetMetric)
protocol := source.ProtocolType
if protocol == "" {
protocol = autoscalingv1alpha1.HTTP
}
url := fmt.Sprintf("%s://%s/%s", protocol, source.Endpoint, strings.TrimLeft(source.Path, "/"))

// External metrics are not engine metrics: fetch the raw metric directly from the
// configured endpoint and path instead of resolving through the central registry.
// This gives us a global value that we need to adapt to per-pod semantics.
metricValue, err := f.engineFetcher.FetchRawMetric(ctx, url, source.Endpoint, source.TargetMetric)
if err != nil {
klog.Warningf("Failed to fetch metric %s from GPU-Optimizer %s: %v",
source.TargetMetric, source.Endpoint, err)
Expand Down
43 changes: 43 additions & 0 deletions pkg/metrics/engine_fetcher.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,49 @@ func (ef *EngineMetricsFetcher) FetchTypedMetric(ctx context.Context, endpoint,
metricName, identifier, ef.config.MaxRetries+1)
}

// FetchRawMetric fetches a metric by its raw Prometheus name from an explicit metrics URL,
// bypassing the central metric registry. External sources such as the GPU optimizer expose
// caller-defined metrics on caller-defined paths, so neither the registry's metric
// definitions nor its per-engine paths apply to them.
func (ef *EngineMetricsFetcher) FetchRawMetric(ctx context.Context, url, identifier, rawMetricName string) (MetricValue, error) {
for attempt := 0; attempt <= ef.config.MaxRetries; attempt++ {
if attempt > 0 {
delay := ef.calculateBackoffDelay(attempt)
klog.V(4).InfoS("Retrying raw metric fetch",
"attempt", attempt, "delay", delay, "identifier", identifier, "metric", rawMetricName)

select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(delay):
}

@gemini-code-assist gemini-code-assist Bot Aug 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using time.After in a select statement inside a loop can cause a temporary memory leak if the context is cancelled before the timer fires, as the underlying timer is not stopped and remains in memory until it expires. It is highly recommended to use time.NewTimer instead and ensure it is stopped when the select block exits.

			timer := time.NewTimer(delay)
			select {
			case <-ctx.Done():
				timer.Stop()
				return nil, ctx.Err()
			case <-timer.C:
			}

@googs1025 googs1025 Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plz check this to avoid memory leak

@yaojiejia yaojiejia Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plz check this to avoid memory leak

done

}

allMetrics, err := ef.fetchAllMetricsFromURL(ctx, url)
if err != nil {
klog.V(4).InfoS("Failed to fetch metrics from URL",
"attempt", attempt+1, "identifier", identifier, "url", url, "error", err)
continue
}

@gemini-code-assist gemini-code-assist Bot Aug 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If fetchAllMetricsFromURL fails because the context was cancelled or timed out, continuing the loop will log a misleading retry message and wait unnecessarily. Checking ctx.Err() immediately after the error allows the function to fail fast and return the context error without redundant logging or retries.

		allMetrics, err := ef.fetchAllMetricsFromURL(ctx, url)
		if err != nil {
			if ctx.Err() != nil {
				return nil, ctx.Err()
			}
			klog.V(4).InfoS("Failed to fetch metrics from URL",
				"attempt", attempt+1, "identifier", identifier, "url", url, "error", err)
			continue
		}


family, exists := allMetrics[rawMetricName]
if !exists || len(family.Metric) == 0 {
klog.V(4).InfoS("Raw metric not found in response",
"attempt", attempt+1, "identifier", identifier, "metric", rawMetricName)
continue
}

metricValue, err := GetCounterGaugeValue(family.Metric[0], family.GetType())
if err != nil {
return nil, fmt.Errorf("failed to parse raw metric %s from %s: %w", rawMetricName, identifier, err)
}
return metricValue, nil
}

return nil, fmt.Errorf("failed to fetch raw metric %s from %s after %d attempts",
rawMetricName, identifier, ef.config.MaxRetries+1)
}

// FetchAllTypedMetrics fetches all available typed metrics from an engine endpoint
func (ef *EngineMetricsFetcher) FetchAllTypedMetrics(ctx context.Context, endpoint, engineType, identifier string, requestedMetrics []string) (*EngineMetricsResult, error) {
result := &EngineMetricsResult{
Expand Down
49 changes: 49 additions & 0 deletions pkg/metrics/engine_fetcher_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -476,6 +476,55 @@ func TestEngineMetricsFetcher_RetryLogic(t *testing.T) {
})
}

func TestEngineMetricsFetcher_FetchRawMetric(t *testing.T) {
optimizerMetrics := `# HELP vllm:deployment_replicas Number of suggested replicas.
# TYPE vllm:deployment_replicas gauge
vllm:deployment_replicas{model_name="deepseek-r1-distill-llama-8b"} 3
`
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/metrics/default/deepseek-r1-distill-llama-8b" {
w.WriteHeader(404)
return
}
w.WriteHeader(200)
fmt.Fprint(w, optimizerMetrics)
}))
defer server.Close()

config := EngineMetricsFetcherConfig{
Timeout: 5 * time.Second,
MaxRetries: 0,
BaseDelay: 10 * time.Millisecond,
MaxDelay: 100 * time.Millisecond,
InsecureTLS: true,
}
fetcher := NewEngineMetricsFetcherWithConfig(config)
ctx := context.Background()

t.Run("FetchUnregisteredMetricFromExplicitPath", func(t *testing.T) {
url := server.URL + "/metrics/default/deepseek-r1-distill-llama-8b"
value, err := fetcher.FetchRawMetric(ctx, url, "gpu-optimizer", "vllm:deployment_replicas")

require.NoError(t, err)
assert.Equal(t, 3.0, value.GetSimpleValue())
})

t.Run("MetricMissingFromResponse", func(t *testing.T) {
url := server.URL + "/metrics/default/deepseek-r1-distill-llama-8b"
_, err := fetcher.FetchRawMetric(ctx, url, "gpu-optimizer", "vllm:missing_metric")

require.Error(t, err)
assert.Contains(t, err.Error(), "failed to fetch raw metric")
})

t.Run("WrongPathReturnsError", func(t *testing.T) {
url := server.URL + "/metrics"
_, err := fetcher.FetchRawMetric(ctx, url, "gpu-optimizer", "vllm:deployment_replicas")

require.Error(t, err)
})
}

func TestEngineMetricsFetcher_BackoffDelay(t *testing.T) {
config := DefaultEngineMetricsFetcherConfig()
fetcher := NewEngineMetricsFetcherWithConfig(config)
Expand Down
2 changes: 1 addition & 1 deletion samples/autoscaling/external-metrics-kpa.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ metadata:
app.kubernetes.io/name: aibrix
app.kubernetes.io/managed-by: kustomize
annotations:
kpa.autoscaling.aibrix.ai/scale-down-delay: 3m
autoscaling.aibrix.ai/scale-down-cooldown-window: 3m
spec:
scalingStrategy: KPA
minReplicas: 1
Expand Down
2 changes: 1 addition & 1 deletion samples/autoscaling/kpa.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ metadata:
app.kubernetes.io/name: aibrix
app.kubernetes.io/managed-by: kustomize
annotations:
kpa.autoscaling.aibrix.ai/scale-down-delay: 3m
autoscaling.aibrix.ai/scale-down-cooldown-window: 3m
spec:
scalingStrategy: KPA
minReplicas: 1
Expand Down
4 changes: 2 additions & 2 deletions samples/autoscaling/optimizer-kpa.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,14 @@ metadata:
app.kubernetes.io/name: aibrix
app.kubernetes.io/managed-by: kustomize
annotations:
kpa.autoscaling.aibrix.ai/scale-down-delay: 0s
autoscaling.aibrix.ai/scale-down-cooldown-window: 0s
spec:
scalingStrategy: KPA
minReplicas: 1
maxReplicas: 8
metricsSources:
- endpoint: aibrix-gpu-optimizer.aibrix-system.svc.cluster.local:8080
metricSourceType: domain
metricSourceType: external
path: /metrics/default/deepseek-r1-distill-llama-8b
protocolType: http
targetMetric: vllm:deployment_replicas
Expand Down
2 changes: 1 addition & 1 deletion samples/deepseek-r1/deepseek-r1-autoscaling.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ metadata:
labels:
app.kubernetes.io/name: aibrix
annotations:
kpa.autoscaling.aibrix.ai/scale-down-delay: 2m
autoscaling.aibrix.ai/scale-down-cooldown-window: 2m
spec:
scalingStrategy: KPA
minReplicas: 1
Expand Down
4 changes: 2 additions & 2 deletions samples/heterogeneous/deepseek-coder-7b-l20-podautoscaler.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,14 @@ metadata:
app.kubernetes.io/managed-by: kustomize
app.kubernetes.io/name: aibrix
annotations:
kpa.autoscaling.aibrix.ai/scale-down-delay: 0s
autoscaling.aibrix.ai/scale-down-cooldown-window: 0s
name: podautoscaler-deepseek-coder-7b-l20
namespace: default
spec:
maxReplicas: 10
metricsSources:
- endpoint: aibrix-gpu-optimizer.aibrix-system.svc.cluster.local:8080
metricSourceType: domain
metricSourceType: external
path: /metrics/default/deepseek-coder-7b-l20
protocolType: http
targetMetric: vllm:deployment_replicas
Expand Down
4 changes: 2 additions & 2 deletions samples/heterogeneous/deepseek-coder-7b-v100-podautoscaler.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,14 @@ metadata:
app.kubernetes.io/managed-by: kustomize
app.kubernetes.io/name: aibrix
annotations:
kpa.autoscaling.aibrix.ai/scale-down-delay: 0s
autoscaling.aibrix.ai/scale-down-cooldown-window: 0s
name: podautoscaler-deepseek-coder-7b-v100
namespace: default
spec:
maxReplicas: 10
metricsSources:
- endpoint: aibrix-gpu-optimizer.aibrix-system.svc.cluster.local:8080
metricSourceType: domain
metricSourceType: external
path: /metrics/default/deepseek-coder-7b-v100
protocolType: http
targetMetric: vllm:deployment_replicas
Expand Down
2 changes: 1 addition & 1 deletion samples/volcano-engine/autoscaler.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ metadata:
app.kubernetes.io/name: aibrix
app.kubernetes.io/managed-by: kustomize
annotations:
kpa.autoscaling.aibrix.ai/scale-down-delay: 5m
autoscaling.aibrix.ai/scale-down-cooldown-window: 5m
spec:
scalingStrategy: KPA
minReplicas: 1
Expand Down
Loading