proxyrunner support for contacting oauth server through https_proxy

@schonmann

This is similar to #6671 but for a use-case not enabling toolhive internal oauth proxy: the proxy runner failing to validate incoming oauth bearer token in the /initialize endpoint (following successfull DRC and Oauth token by the mcp client) due to not honoring the injected https_proxy/no_proxy environment variables

Here is my redacted configuration fragment

apiVersion: toolhive.stacklok.dev/v1beta1
kind: MCPServer
metadata:
  name: postgres-${binding_id}
spec:
...

  resourceOverrides:
    proxyDeployment:
      env:
      - name: https_proxy
        value: "http://myproxy:3128"
      - name: no_proxy
        value: "127.0.0.1,localhost,..."
      - name: TOOLHIVE_DEBUG
        value: "true"

  oidcConfigRef:
    name: postgres-mcp-oidc-config
    audience: postgres-mcp-...
    resourceUrl: https://postgres-mcp-redacted-host # ResourceURL is the public URL for OAuth protected resource metadata (RFC 9728). If not specified, defaults to the internal Kubernetes service URL.

apiVersion: toolhive.stacklok.dev/v1beta1
kind: MCPOIDCConfig
metadata:
  name: postgres-mcp-oidc-config
spec:
  type: inline
  inline:
    issuer: https://redacted-upstream-oidc-host/realms/service-manager-realm
    clientId: postgres-mcp-client
    jwksUrl: https://redacted-upstream-oidc-host/realms/service-manager-realm/protocol/openid-connect/certs

Observed symptom:

Fast (142ms) 401 response on mcp initialize request with response header:

www-authenticate Bearer realm="https://redacted-upstream-oidc-host/service-manager-realm", resource_metadata="https://postgres-mcp-redacted-host/.well-known/oauth-protected-resource/mcp", error="invalid_token", error_description="failed to parse token: token is unverifiable: error while executing keyfunc: failed to lookup JWKS: resource \"https://redacted-upstream-oidc-host/realms/service-manager-realm/protocol/openid-connect/certs\" is not ready"

The https_proxy and the no_proxy env variables are properly injected into proxy-runner pod (through support contributed in #983)

kubectl get   deployments.apps postgres-6a0b0b17-c3d7-433b-8c7c-8c7227db875b -o yaml | yq -o=p | grep -iE -A1 "https_proxy|no_proxy|image"
spec.template.spec.containers.0.env.0.name = https_proxy
spec.template.spec.containers.0.env.0.value = http://myproxy:3128
spec.template.spec.containers.0.env.1.name = no_proxy
spec.template.spec.containers.0.env.1.value = 127.0.0.1,localhost,...
--
spec.template.spec.containers.0.image = ghcr.io/stacklok/toolhive/proxyrunner:v0.51.2

However, the proxy runner still fails to reach the upstream OIDC server, unfortunately without friendly traces

{"time":"2026-10-02T09:45:21Z","level":"DEBUG","msg":"JWKS URL registered but first fetch not ready; fetching continues in background","jwks_url":"https://redacted-upstream-oidc-host/realms/service-manager-realm/protocol/openid-connect/certs","error":"failed to add resource: resource registered but not ready: context deadline exceeded"}

Associated code path

https://github.com/jwx-go/jwkfetch/blob/7e8d989175276661062d594a2bd372c0a3439dee/cache.go#L134-L151

	httpClient := c.httpClient
	if httpClient == nil {
		httpClient = DefaultHTTPClient()
	}
	resourceOptions = append(resourceOptions, httprc.WithHTTPClient(httpClient))


	r, err := httprc.NewResource[jwk.Set](u, &transformer{
		parseOptions: c.parseOptions,
		maxBodySize:  c.maxBodySize,
	}, resourceOptions...)
	if err != nil {
		return fmt.Errorf(`failed to create httprc.Resource: %w`, err)
	}
	if err := c.ctrl.Add(ctx, r, httprc.WithWaitReady(waitReady)); err != nil {
		return fmt.Errorf(`failed to add resource: %w`, err)
	}
	return nil
}

Same analysis over httpclient creation than the one from @schonmann in #6671

@gberche-orange

@gberche-orange Thanks for the detailed report. I reviewed #6670 with this case in mind: #6670 (review)

Incoming token validation uses the same HttpClientBuilder and injects its client into the JWKS cache (pkg/auth/token.go:671-684). So the current patch also addresses the missing proxy selection on this path, even without the embedded auth server.

It is not yet a confirmed complete fix for this deployment. #6670 remains under review, including its proxy/private-IP policy. If revised to explicit opt-in, incoming JWKS/discovery must be included alongside upstream OAuth/DCR.

Could you confirm, without sharing credentials or internal hostnames:

Does the proxy hostname resolve to a private address?
Does the proxy itself expect TLS for the configured https://myproxy:3128 URL?
Is spec.inline.jwksAllowPrivateIP configured, or left at its default?

The current default dial policy still rejects private proxy addresses. Enabling jwksAllowPrivateIP also permits private OIDC destinations, so it is not a narrowly scoped proxy permission or a general workaround. These details will help distinguish the proxy-selection fix from any remaining connectivity/TLS problem. The reporter's Kubernetes deployment has not yet been reproduced.

@Sanskarzz

Thanks @Sanskarzz for your prompt response, and inclusion of my use-case in your review at #6670 (review)

Here are my details about my use-case:

The JWKS endpoint (from keycloak) is not reachable directly due to network ACL, and use of the explicit HTTP proxy (HTTP CONNECT) is required to reach it.

Does the proxy hostname resolve to a private address?
Yes, the proxy hostname resolves to a IP private address within the documented range at
https://en.wikipedia.org/wiki/Private_network#Private_IPv4_addresses

Does the proxy itself expect TLS for the configured https://myproxy:3128 URL?
For this specific use-case, the proxy does not enforce use of TLS. I've fixed in the original report as https_proxy=http://myproxy:3128. In some other cases, HTTP proxy expects TLS, so support for this would also be useful in 2nd step (potentially requiring self signed ca certs injection as well)

Is spec.inline.jwksAllowPrivateIP configured, or left at its default?
It is left as default (false from https://docs.stacklok.com/toolhive/reference/crds/mcpoidcconfig#specinline)

The current default dial policy still rejects private proxy addresses. Enabling jwksAllowPrivateIP also permits private OIDC destinations

Thanks, I'll enable this field. Would it make sense to document in https://docs.stacklok.com/toolhive/reference/crds/mcpoidcconfig#specinline that this properyy also applies to tcp connection to http proxy configured from environment variable http_proxy? It isn't obvious to me from reading current doc

jwksAllowPrivateIP boolean JWKSAllowPrivateIP allows JWKS/OIDC endpoints on private IP addresses. Note: at runtime, if either JWKSAllowPrivateIP or ProtectedResourceAllowPrivateIP is true, private IPs are allowed for all OIDC HTTP requests (JWKS, discovery, introspection).default false

protectedResourceAllowPrivateIP boolean ProtectedResourceAllowPrivateIP allows protected resource endpoint on private IP addresses. Note: at runtime, if either ProtectedResourceAllowPrivateIP or JWKSAllowPrivateIP is true, private IPs are allowed for all OIDC HTTP requests (JWKS, discovery, introspection).default false

@gberche-orange

Thanks @gberche-orange, this clarifies the setup. https_proxy=http://myproxy:3128 is appropriate for accessing an HTTPS JWKS endpoint through a plain HTTP proxy using CONNECT; the JWKS connection remains HTTPS.

One clarification: enabling jwksAllowPrivateIP alone will not resolve this on v0.51.2, because that client still ignores the proxy environment variables. Proxy selection also needs the fix in #6670 or its revised implementation. With the current patch, the private-IP policy would then affect whether the proxy connection is permitted.

Your documentation suggestion makes sense. We should explain that interaction once the proxy policy is settled, including that jwksAllowPrivateIP permits private OIDC destinations too—it is broader than permission to reach only the proxy.

We should keep #6745 open until the supported configuration successfully fetches JWKS in your deployment.

@Sanskarzz

thanks @Sanskarzz for the clarification.

One additional user feedback/requirement related to operability: in order to be able to quickly diagnose that connection is rejected due to private ip filtering, user would expect a clear error log clarifying the rejected http proxy dial in.

Navigation Menu

Repository navigation

proxyrunner support for contacting oauth server through https_proxy #6745

Observed symptom:

Associated code path

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

proxyrunner support for contacting oauth server through https_proxy #6745

Description

Observed symptom:

Associated code path

Activity

Sanskarzz commented on Oct 4, 2026

gberche-orange commented on Oct 5, 2026

Sanskarzz commented on Oct 6, 2026

gberche-orange commented on Oct 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions