PMM-15228 Enrich AuthServer with performance metrics

[maxkondr]

Ticket number: PMM-15228

Feature build: Percona-Lab/pmm-submodules#4484

This pull request adds detailed Prometheus metrics to the AuthServer component in order to monitor authentication request flows, cache usage, and request durations. It also ensures that metrics are accurately labeled, especially in error scenarios, and improves code clarity around request path handling.

Prometheus Metrics Integration:

• Added a new authMetrics struct to AuthServer that defines and registers Prometheus counters and histograms for tracking authentication requests, cache hits/misses, Grafana backend requests, cache size, and operation durations. (managed/services/grafana/auth_server.go)
• Implemented the prom.Collector interface for AuthServer, allowing it to be registered with Prometheus and report its metrics. (managed/services/grafana/auth_server.go)
• Registered the AuthServer as a Prometheus collector in the main application entrypoint. (managed/cmd/pmm-managed/main.go)

Metrics Collection in Auth Flow:

• Updated the authentication flow to increment the appropriate counters and observe durations for total request time, Grafana backend requests, database lookups, cache hits, and cache misses. (managed/services/grafana/auth_server.go) [1] [2] [3]
• Ensured that metrics for failed requests (e.g., bad requests) use cleaned and normalized route labels for consistency and security. (managed/services/grafana/auth_server.go)

Testing and Path Handling Improvements:

• Added tests to verify that metrics for bad requests use the cleaned route or fall back to the raw route if cleaning fails. (managed/services/grafana/auth_server_test.go)
• Centralized and clarified the use of cleaned request paths by ensuring path normalization occurs early and is consistently used throughout the authentication logic. (managed/services/grafana/auth_server.go) [1] [2]

These changes provide better observability into authentication operations and improve the maintainability and correctness of metrics reporting.

Summary by CodeRabbit

• New Features

  • Added expanded PMM Health dashboard coverage for service health, host resources, databases, storage, QAN ingestion, authentication, and runtime metrics.
  • Added dashboard sections, variables, thresholds, legends, and navigation links for easier monitoring and troubleshooting.

• Monitoring Improvements

  • Added detailed metrics for authentication requests, Grafana interactions, caching, and access-control operations.
  • Improved route handling in authentication monitoring for more accurate request reporting.

[codecov]

Codecov Report

❌ Patch coverage is 62.01550% with 49 lines in your changes missing coverage. Please review.
✅ Project coverage is 43.47%. Comparing base (31318c7) to head (024e2c2).
⚠️ Report is 103 commits behind head on main.

Files with missing lines	Patch %	Lines
managed/services/grafana/auth_server.go	62.50%	45 Missing and 3 partials ⚠️
managed/cmd/pmm-managed/main.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5670      +/-   ##
==========================================
- Coverage   43.59%   43.47%   -0.12%     
==========================================
  Files         415      433      +18     
  Lines       43134    35128    -8006     
  Branches        0      591     +591     
==========================================
- Hits        18804    15272    -3532     
+ Misses      22454    18365    -4089     
+ Partials     1876     1491     -385     

Flag	Coverage Δ
admin	34.96% <ø> (+0.17%)	⬆️
agent	?
managed	45.12% <62.01%> (+2.14%)	⬆️
unittests	41.29% <ø> (?)
vmproxy	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[coderabbitai]

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

💤 Files selected but had no reviewable changes (1)

• dashboards/dashboards/PMM Health/PMM_Health.json

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: fff68e55-3736-4fc9-ab00-aaa2caa5b0b4

📥 Commits

Reviewing files that changed from the base of the PR and between f753119 and 024e2c2.

📒 Files selected for processing (1)

• dashboards/dashboards/PMM Health/PMM_Health.json

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

Walkthrough

The PR regenerates the PMM Health Grafana dashboard and adds Prometheus instrumentation to Grafana authentication. Request paths are normalized before authorization checks, and authentication, cache, Grafana, and LBAC operations now expose metrics.

Changes

PMM Health dashboard

Layer / File(s)	Summary
Summary, node, and QAN monitoring dashboards/dashboards/PMM Health/PMM_Health.json	Updates service, host-resource, and QAN API monitoring panels and queries.
PMM ManageD and Grafana monitoring dashboards/dashboards/PMM Health/PMM_Health.json	Rebuilds PMM ManageD and Grafana sections for runtime, authentication, cache, process, and SQL metrics.
VictoriaMetrics, ClickHouse, and PostgreSQL monitoring dashboards/dashboards/PMM Health/PMM_Health.json	Expands datastore monitoring and updates dashboard metadata, variables, layouts, and visualization settings.

Grafana authentication observability

Layer / File(s)	Summary
Authentication metrics and collector registration managed/services/grafana/auth_server.go, managed/cmd/pmm-managed/main.go	Defines Prometheus metrics, implements collection, and registers AuthServer.
Request path normalization and response metrics managed/services/grafana/auth_server.go	Cleans original request paths and records authentication outcomes and durations.
Cache, Grafana, and LBAC operation metrics managed/services/grafana/auth_server.go, managed/services/grafana/auth_server_test.go	Records cache, Grafana, and LBAC metrics. Tests cover normalized and fallback bad-request routes.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant AuthServer
  participant LBAC
  participant Grafana
  participant Prometheus
  Client->>AuthServer: send authentication request
  AuthServer->>AuthServer: clean and validate request path
  AuthServer->>LBAC: resolve roles and filters
  LBAC-->>AuthServer: return authorization result
  AuthServer->>Grafana: authenticate authorized request
  Grafana-->>AuthServer: return response status
  AuthServer->>Prometheus: record request, cache, Grafana, and LBAC metrics
  AuthServer-->>Client: return authentication response

Loading

Suggested reviewers: ademidoff, nailya, jirictvrtka

Poem

A rabbit checks the panels bright,
Counts each metric through the night.
Clean paths guide the auth request,
Prometheus records each quest.
Grafana hops, the dashboard glows.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 44.44% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the ticket and the main change: adding performance metrics to AuthServer.
Description check	✅ Passed	The description includes the ticket, feature build, implementation details, testing changes, and related work information.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch PMM-15228-pmm-server-performance-metrics

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 8

🧹 Nitpick comments (1)

dashboards/dashboards/PMM Health/PMM_Health.json (1)

8089-8100: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Remove the hidden target with the invalid subquery.

This target contains two defects:

1. (irate(...[2m]))[$__interval] is not a valid subquery. A subquery requires a resolution after a colon, for example [$__interval:].
2. $peaks is not declared in the templating.list block at lines 9603-9725.

The target sets "hide": true, so the query never runs and no user sees the failure. The text remains as a broken artifact. Delete the target, or declare $peaks and correct the subquery syntax.

♻️ Proposed removal of the hidden target

-                        {
-                            "datasource": "Metrics",
-                            "editorMode": "code",
-                            "exemplar": false,
-                            "expr": "max_over_time( (irate(ClickHouseProfileEvents_ReadBackoff{instance=~\"$instance\"}[2m]))[$__interval] ) * $peaks",
-                            "hide": true,
-                            "interval": "",
-                            "legendFormat": "peaks - {{instance}}",
-                            "range": true,
-                            "refId": "A"
-                        },
                         {
                             "datasource": "Metrics",
                             "editorMode": "code",
                             "exemplar": false,
                             "expr": "rate(ClickHouseProfileEvents_ReadBackoff{job=\"clickhouse\"}[$__interval]) ",

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@dashboards/dashboards/PMM` Health/PMM_Health.json around lines 8089 - 8100,
Remove the hidden target identified by refId "A" containing the
ClickHouseProfileEvents_ReadBackoff max_over_time expression from the targets
array. Do not add a $peaks variable or alter the invalid subquery, since this
artifact should be deleted entirely.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@dashboards/dashboards/PMM` Health/PMM_Health.json:
- Line 3900: Update the description for the “Incoming Auth Requests Rate” panel
to describe interpreting the pmm_managed_auth_requests_total request-rate
metric, removing all references to type labels, latency spikes, and database or
Grafana duration breakdowns.
- Around line 2670-2690: Update dashboards/dashboards/PMM Health/PMM_Health.json
at lines 2670-2690 by replacing rate() with avg_over_time() for both process
memory gauge queries. At lines 6069-6089, apply the same replacement to both
Grafana memory series and change the virtual-memory range from [5m] to
$__interval. At lines 5578-5589, query the grafana_database_conn_open gauge
directly without rate().
- Around line 9428-9435: Update the “Fetched” target expression to filter on
service_name=~"$service_name" instead of instance=~"$node_name", matching the
sibling targets in the same panel while preserving the existing metric,
aggregation, and interval logic.
- Around line 4666-4676: Update the PromQL expression for the target in the
cache hit-rate panel so the numerator sums
irate(pmm_managed_auth_cache_total[$__rate_interval]) filtered with
status="hit", while retaining the unfiltered total in the denominator. Keep the
existing panel configuration and legend unchanged.
- Around line 6713-6722: Update the disk usage expression in the panel query
identified by refId A so vm_free_disk_space_bytes is aggregated with sum as
well, producing matching label sets for the denominator addition while
preserving the existing job and instance filters.
- Around line 2441-2481: Update the target expression for panel id 1081 to
calculate average failed-batch duration by dividing the rate of
qan_api2_data_ingestion_batch_save_seconds_sum by the corresponding rate of
qan_api2_data_ingestion_batch_save_seconds_count, preserving the existing error
filter and interval. Keep the seconds unit and “Batch save duration with Errors”
title aligned with this duration result.
- Around line 6428-6448: Update the Grafana statistic panels, including the
Organisations Count and Folders Count targets, to use the dashboard’s instance
variable in their Prometheus selectors instead of a fixed instance value, so
both non-HA and active/active HA generated node IDs return data. Preserve the
existing metric names and db_name="grafana" filtering.

In `@managed/services/grafana/auth_server.go`:
- Around line 267-273: Bound authentication metric labels to prevent arbitrary
client-controlled values from creating unbounded Prometheus series. In
managed/services/grafana/auth_server.go:267-273, keep only bounded method and
canonical-route dimensions for mAuthRequests; at 392-403, emit "invalid" instead
of the fallback path when original-request parsing fails; at 422-448, record the
resolved authorization-rule prefix or route template rather than req.URL.Path.
Update managed/services/grafana/auth_server_test.go:402-418 to assert the fixed
invalid-route label.

---

Nitpick comments:
In `@dashboards/dashboards/PMM` Health/PMM_Health.json:
- Around line 8089-8100: Remove the hidden target identified by refId "A"
containing the ClickHouseProfileEvents_ReadBackoff max_over_time expression from
the targets array. Do not add a $peaks variable or alter the invalid subquery,
since this artifact should be deleted entirely.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: aee931c4-6cdb-4bb4-8278-6ad04252c933

📥 Commits

Reviewing files that changed from the base of the PR and between a50c452 and f753119.

📒 Files selected for processing (4)

• dashboards/dashboards/PMM Health/PMM_Health.json
• managed/cmd/pmm-managed/main.go
• managed/services/grafana/auth_server.go
• managed/services/grafana/auth_server_test.go