cluster_link: export Schema Registry sync totals as prometheus metrics#31067
Open
nguyen-andrew wants to merge 3 commits into
Open
cluster_link: export Schema Registry sync totals as prometheus metrics#31067nguyen-andrew wants to merge 3 commits into
nguyen-andrew wants to merge 3 commits into
Conversation
pgellert
reviewed
Jul 10, 2026
| } | ||
|
|
||
| const auto sl_name = link_probe::shadow_link_name(link_name); | ||
| const auto& totals = status.totals_since_task_start; |
Contributor
There was a problem hiding this comment.
I think resolving the reference to totals_since_task_start this early might already be or could become problematic when mirroring_task::_status.totals_since_task_start is assigned a fresh value, which is currently done in mirroring_task::reset_sync_state().
Member
Author
There was a problem hiding this comment.
Good point, forgot about that
| { | ||
| sm::make_counter( | ||
| "schema_registry_subject_versions_changed", | ||
| [&totals] { return totals.subject_versions_changed; }, |
Contributor
There was a problem hiding this comment.
The mirroring_task has a reconcile stats field that is added onto the status counters on status reporting call. It might be simplest to either reuse mirroring_task::get_status_report() here too.
nguyen-andrew
force-pushed
the
sl-sr-prometheus
branch
from
9e5a5d5 to
bd9c5bb
Compare
Member
Author
|
Force push to address PR comments and add more tests. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR exposes Schema Registry shadow-linking totals_since_task_start counters as Prometheus counters (shadow_link_schema_registry_*) on both internal (/metrics) and public (/public_metrics) endpoints, labelled by shadow_link_name, and adds unit + ducktape coverage to validate behavior across leadership changes and task lifecycle.
Changes:
- Add a Schema Registry sync probe that registers
shadow_link_schema_registry_*counters on both metrics endpoints. - Extend Schema Registry shadow-linking tests (C++ and ducktape) to validate counter presence/values, pause/stop behavior, and leadership-tenure resets.
- Update build targets to include the new probe and metric-test utilities.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/rptest/tests/cluster_linking_schema_registry_sync_test.py | Adds ducktape assertions for the new Prometheus counters on both endpoints and leadership-movement behavior. |
| src/v/cluster_link/schema_registry_sync/tests/mirroring_task_test.cc | Adds unit tests validating metric export, pause/stop series lifetime, leadership tenure resets, and scrape-time behavior. |
| src/v/cluster_link/schema_registry_sync/tests/BUILD | Adds deps needed for new metric tests (metrics + test_utils metrics). |
| src/v/cluster_link/schema_registry_sync/probe.h | New probe interface for exporting Schema Registry sync totals as metrics. |
| src/v/cluster_link/schema_registry_sync/probe.cc | New probe implementation registering shadow_link_schema_registry_* counters. |
| src/v/cluster_link/schema_registry_sync/mirroring_task.h | Integrates the probe and adds helper for live status assembly. |
| src/v/cluster_link/schema_registry_sync/mirroring_task.cc | Registers/clears the probe on start/stop and reuses live-status helper in status reporting. |
| src/v/cluster_link/schema_registry_sync/BUILD | Adds probe sources/headers and metrics dependency to the library build. |
Comment on lines
+147
to
+149
| _probe.setup(get_link()->get_config()->name, [this] { | ||
| return get_live_sync_status().totals_since_task_start; | ||
| }); |
| #include "test_utils/metrics.h" | ||
| #include "test_utils/test.h" | ||
|
|
||
| #include <seastar/core/coroutine.hh> |
Comment on lines
+26
to
+30
| /// The owning task keeps the series registered while it is not stopped, | ||
| /// i.e. on the shard leading `_schemas/0`, including while paused (the | ||
| /// totals survive a pause). Stopping resets the totals, so a stale series | ||
| /// cannot linger after leadership moves away; a transfer may briefly | ||
| /// export the series on zero or two nodes. |
nguyen-andrew
force-pushed
the
sl-sr-prometheus
branch
from
bd9c5bb to
f0e4c08
Compare
Member
Author
|
Force push to rebase on latest dev. |
Collaborator
CI test resultstest results on build#86990
test results on build#87093
|
nguyen-andrew
force-pushed
the
sl-sr-prometheus
branch
2 times, most recently
from
fe63a2f to
6dd29c5
Compare
Expose the totals_since_task_start counters of the Schema Registry shadowing task as counters on both the internal and public metrics endpoints, labelled by shadow_link_name. The task drives the series off its state machine: they exist exactly while it surfaces Schema Registry status (any state but stopped), so a demoted leader stops exporting rather than double-counting into the aggregated sum (a transfer may briefly export on zero or two nodes), while pause/resume keeps the series continuous. stop() already resets the totals, so a fresh leader re-exports from zero.
Assert the probe's counters mirror totals_since_task_start on both the internal and public registries, labelled with the link name, and pin the series lifecycle: registered while active, kept (with values) across pause/resume, removed on stop so a non-leader cannot export misleading reset zeros.
Scrape the shadow_link_schema_registry_* counters in the existing SR sync tests: the e2e counter check now asserts both metrics endpoints agree with the admin API totals, the out-of-scope-reference test covers a nonzero errors counter, and the leadership-change test verifies the series follows _schemas/0 leadership (the old leader drops its series rather than exporting stale values). Disable the leader balancer for the suite: totals_since_task_start resets when _schemas/0 leadership moves, so a balancer-initiated move would strand the exact-value counter assertions. The leadership-change test moves leadership explicitly.
nguyen-andrew
force-pushed
the
sl-sr-prometheus
branch
from
6dd29c5 to
97d7209
Compare
Member
Author
|
Force push to address copilot comments. Force push to reword commit. Force push to rebase on latest dev and fix merge conflict in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Expose the
totals_since_task_startcounters of Schema Registryshadowing as
shadow_link_schema_registry_*counters on both theinternal and public prometheus endpoints, labelled by
shadow_link_name, with unit and ducktape coverage.Fixes https://redpandadata.atlassian.net/browse/CORE-16607
Backports Required
Release Notes
Features
shadow_link_schema_registry_*counters on the internal and publicmetrics endpoints, labelled by
shadow_link_name.