authserver: upstream IdP token requests time out after 10s, not 30s #6754

Bug description

The embedded auth server's HTTP client for the upstream identity provider keeps the 10s ResponseHeaderTimeout that networking.NewHttpClientBuilder sets by default (pkg/networking/http_client.go:319-325, applied in Build() at :427). newHTTPClientForHost (pkg/authserver/upstream/oauth2.go:1026-1031) builds that client for both the OIDC and the OAuth2 provider, and the builder has no setter for this timeout. A token endpoint that needs more than 10s to send its first byte therefore fails the code exchange and the refresh, although the client timeout (HttpTimeout) and the refresher's refreshTimeout (pkg/authserver/refresher.go:22) are both 30s.

The shared grant client leaves this timeout unset on purpose. pkg/oauthproto/grants.go:250-253: "some corporate IdP chains (Entra OBO, federated Okta) legitimately take >10s to produce the first byte. The outer Client.Timeout (defaultHTTPTimeout) is the authoritative budget for the whole exchange." grants_test.go:65-75 asserts it. The upstream provider client does the opposite.

Consequence: with a provider that revokes the presented refresh token when it processes the refresh, a refresh that ToolHive abandons at 10s and the provider still completes leaves the revoked token in storage (refresher.go:168-171 returns before writing), so the next refresh gets invalid_grant and the user has to sign in again.

Steps to reproduce

With Go 1.27 or later:

mkdir repro && cd repro
# save main.go below
go mod init repro
go get github.com/stacklok/toolhive@v0.51.4
go mod tidy
go run .

package main

import (
	"context"
	"fmt"
	"log/slog"
	"net/http"
	"net/http/httptest"
	"time"

	"github.com/stacklok/toolhive/pkg/authserver/upstream"
)

func main() {
	slog.SetDefault(slog.New(slog.DiscardHandler))

	// Token endpoint that needs 12 s before it sends response headers.
	idp := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		time.Sleep(12 * time.Second)
		w.Header().Set("Content-Type", "application/json")
		fmt.Fprint(w, `{"access_token":"at-2","token_type":"Bearer","refresh_token":"rt-2","expires_in":3600}`)
	}))
	defer idp.Close()

	p, err := upstream.NewOAuth2Provider(&upstream.OAuth2Config{
		CommonOAuthConfig: upstream.CommonOAuthConfig{
			ClientID:    "client",
			RedirectURI: "http://localhost:8080/oauth/callback",
		},
		AuthorizationEndpoint: idp.URL + "/authorize",
		TokenEndpoint:         idp.URL + "/token",
	})
	if err != nil {
		panic(err)
	}

	start := time.Now()
	tokens, err := p.RefreshTokens(context.Background(), "rt-1", "")
	if err != nil {
		fmt.Printf("refresh failed after %.1fs: %v\n", time.Since(start).Seconds(), err)
		return
	}
	fmt.Printf("refresh ok after %.1fs: refresh_token=%s\n", time.Since(start).Seconds(), tokens.RefreshToken)
}

Expected behavior

The refresh completes within the 30s client timeout: refresh ok after 12.0s: refresh_token=rt-2.

Actual behavior

refresh failed after 10.0s: request failed: Post "http://127.0.0.1:36321/token": net/http: timeout awaiting response headers

The OIDC provider's code exchange fails the same way. Over HTTPS with HTTP/2 the error ends in http2: timeout awaiting response headers.

Environment (if relevant)

• OS/version: Linux, Go 1.27.1
• ToolHive version: v0.51.4, also reproduced at main (326be93)

Additional context

#6194 shows the same 10s limit in a real deployment, cutting off the JWKS fetch on this client (net/http: timeout awaiting response headers); #6559 made the ID-token validation retry, the limit itself stayed.

I intend to fix this and have the change ready: add HttpClientBuilder.WithResponseHeaderTimeout (the 10s default stays for every other caller) and have newHTTPClientForHost set it to zero, so the 30s client timeout bounds each call on that client, as in grants.go. I'll open the PR referencing this issue.