Add project routing telemetry infrastructure by quux00 · Pull Request #155997 · elastic/elasticsearch

quux00 · 2026-08-05T19:01:42Z

Adds the core data structures and recording hooks for CPS project routing usage telemetry in GET _cluster/stats:

ProjectRoutingUsageHolder: LongAdder-based counters for _search and ES|QL queries broken down by routing mode (alias_origin, alias_wildcard, custom_tags, named_expression) plus failure counters for recording failed CPS queries.
ProjectRoutingUsageSnapshot: Writeable point-in-time snapshot accumulated across nodes on the coordinator.
ClusterStatsResponse: emits separate top-level tags and project_routing blocks; project_routing.queries is the sum of search and esql totals computed at render time.
TransportSearchAction: records per-search telemetry gated on collectSearchTelemetry and hasLinkedProjects.
Stub infrastructure for Ticket 4: ClusterStatsTagsProvider, TagsConfigSnapshot, ActionPlugin extension point.

JSON field names follow the PM spec: queries,
queries_project_routing, alias_origin, alias_wildcard, custom_tags, named_expression, in_SET, failures.

Tests cover: wire serialization, add() accumulation, toXContent suppression rules, failure-method no-ops and counter semantics, and ClusterStatsResponse assembly.

elasticsearchmachine · 2026-08-05T19:02:08Z

Pinging @elastic/es-search-foundations (Team:Search Foundations)

Adds the core data structures and recording hooks for CPS project routing usage telemetry in GET _cluster/stats: - ProjectRoutingUsageHolder: LongAdder-based counters for _search and ES|QL queries broken down by routing mode (alias_origin, alias_wildcard, custom_tags, named_expression) plus failure counters for recording failed CPS queries. - ProjectRoutingUsageSnapshot: Writeable point-in-time snapshot accumulated across nodes on the coordinator. - ClusterStatsResponse: emits separate top-level `tags` and `project_routing` blocks; `project_routing.queries` is the sum of search and esql totals computed at render time. - TransportSearchAction: records per-search telemetry gated on collectSearchTelemetry and hasLinkedProjects. - Stub infrastructure for Ticket 4: ClusterStatsTagsProvider, TagsConfigSnapshot, ActionPlugin extension point. JSON field names follow the PM spec: `queries`, `queries_project_routing`, `alias_origin`, `alias_wildcard`, `custom_tags`, `named_expression`, `in_SET`, `failures`. Tests cover: wire serialization, add() accumulation, toXContent suppression rules, failure-method no-ops and counter semantics, and ClusterStatsResponse assembly.

smalyshev · 2026-08-06T16:42:56Z

+    int total,
+    int totalCustom,
+    List<String> names,
+    int namedRoutingExpressionsTotal,


Do we need the totals here? Couldn't we just calculate them from the List?

Good catch. Will be fixed in the next push.

smalyshev · 2026-08-06T16:49:06Z

+    public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
+        builder.field("total", total);
+        builder.field("total_custom", totalCustom);
+        builder.array("names", names.toArray(new String[0]));


I think XContentBuilder can handle lists? It has Iterable<?> support so I think it'd do the right thing?

👍 Will be changed in the next push.

smalyshev · 2026-08-06T16:53:03Z

        CCSTelemetrySnapshot ccsTelemetrySnapshot,
-        CCSTelemetrySnapshot esqlTelemetrySnapshot
+        CCSTelemetrySnapshot esqlTelemetrySnapshot,
+        ProjectRoutingUsageSnapshot projectRoutingUsageSnapshot


Do we need to ensure non-null here? I think we are going the other route with tags... I am not a huge fan of nulls, but here I am wondering if we have nothing to display (which would be the case for all non-CPS cases I imagine) why create the object at all?

See this comment: #155997 (comment)

smalyshev · 2026-08-06T16:53:48Z

        searchCcsMetrics.writeTo(out);
        esqlCcsMetrics.writeTo(out);
+        if (out.getTransportVersion().supports(PROJECT_ROUTING_USAGE_STATS)) {
+            projectRoutingUsageSnapshot.writeTo(out);


As per above, maybe optional object here would be better, for non-CPS cases?

See this comment: #155997 (comment)

smalyshev · 2026-08-06T16:54:57Z

    final CCSTelemetrySnapshot esqlMetrics;
+    final ProjectRoutingUsageSnapshot projectRoutingUsageSnapshot;
+    @Nullable
+    final TagsConfigSnapshot tagsConfig;


As noted above, here it's a bit inconsistent - one is nullable, the other is not. Is there a reason why? The seem to be both non-existant in non-CPS context, or am I missing something?

I agree this is a little confusing. It is true that ProjectRoutingUsage and TagsConfig will only be present in serverless and both absent (in the _cluster/stats output) for stateful. The reason for the different handling here (one is null, the other is never null but can be full of zeros) is that:

tagsConfig is nullable because it comes from an optional plugin registration that may not exist at all (no serverless plugin = null provider). Null here means "feature not wired up". It gets wired up only on the serverless side.

The projectRoutingUsageSnapshot comes from ProjectRoutingUsageHolder, which is a built-in holder that always exists in UsageService on the es-core side. It always has a valid snapshot to return.

Neither will "render" in _cluster/stats for stateful. For projectRoutingUsageSnapshot, it will not be included in the _cluster/stats output unlesstotalQueries > 0 . (see: https://github.com/elastic/elasticsearch/pull/155997/changes#diff-e4a6036f7e574278a739adbb118f2ab0031e4c1abbe92bac125bb732c8011993R187)

smalyshev · 2026-08-06T17:18:28Z

+        esqlQueriesTotal.increment();
+        if (setClauseUsed) esqlWithSet.increment();
+        if (info == null) return;
+        esqlWithProjectRouting.increment();


This part seems to be copy-paste from the same part of the function above, maybe should be one function somehow?

This will change with the change here.

smalyshev · 2026-08-06T17:21:06Z

+
+    // ES|QL endpoint
+    private final LongAdder esqlQueriesTotal = new LongAdder();
+    private final LongAdder esqlWithProjectRouting = new LongAdder();


It feels like search and ESQL counters both have common substructure:

record RoutingCounters(LongAdder projectRouting, LongAdder aliasOrigin, LongAdder aliasWildcard, LongAdder customTags, LongAdder namedExpression, LongAdder failures);

Good suggestion. Will be changed in the next push.

smalyshev · 2026-08-06T17:26:20Z

-    @Nullable List<ProjectRoutingInfo> linkedProjects // null when CPS is disabled
+    @Nullable List<ProjectRoutingInfo> linkedProjects, // null when CPS is disabled
+    @Nullable ProjectRoutingRequestInfo projectRoutingRequestInfo,
+    boolean hasLinkedProjects


Not sure if it's avoidable but this boolean does't feel right for me. It feels like we should already have this information without it, but I can't yet figure out how to get rid of it. Maybe will need to think more about it, for now I am just recording I don't like it :)

hasLinkedProjects is needed because linkedProjects reflects post-routing state and can't serve as the gate. Specifically, _alias:_origin (probably the most common routing expression) causes the resolver to set linkedProjects to empty (route to origin only), even when the project has configured linked projects; using linkedProjects.isEmpty() would silently suppress all counters for that case. hasLinkedProjects captures the pre-routing truth.

I added javadoc to cover the reasoning as well.

smalyshev · 2026-08-06T18:29:11Z

+ * @param usedAliasOrigin true when the expression was exactly {@code _alias:_origin}
+ */
+public record ProjectRoutingRequestInfo(
+    List<String> tagsUsedInRouting,


This as I understand is supposed to keep all tags, but in fact we're just interested in one question - were there any custom tags among them? So maybe it should be just boolean usedCustomTags instead? Or we're going to use more stuff from here in the future?

So maybe it should be just boolean usedCustomTags instead? Or we're going to use more stuff from here in the future?

Yes, right now it is just used to compute whether custom tags were used. So we don't need the list, a boolean would do, but the list does future-proof against new requirements, but perhaps that's unnecessary abstraction? I can make the change.

…ats/TransportClusterStatsAction.java Co-authored-by: Stanislav Malyshev <smalyshev@users.noreply.github.com>

smalyshev · 2026-08-06T22:15:54Z

        this.searchUsageHolder = usageService.getSearchUsageHolder();
        this.ccsUsageHolder = usageService.getCcsUsageHolder();
        this.esqlUsageHolder = usageService.getEsqlUsageHolder();
+        this.usageService = usageService;


Here's a bit of inconsistency - we store components of UsageService above and then also keep the whole thing. To be consistent, we probably want tagsProvider and getProjectRoutingUsageHolder individually? Or dispense with that pattern and keep only UsageService but now it's half-and-half.

Fixed. I've refactored to extract all five things we need (searchUsageHolder, ccsUsageHolder, esqlUsageHolder, projectRoutingUsageHolder, and tagsProvider) as final fields in the constructor, and dropped the usageService field entirely. This was also made cleaner by switching ClusterStatsTagsProvider to SPI registration using loadSingletonServiceProvider (comment), which means it's fully resolved before TransportClusterStatsAction is constructed — no more lazy read needed.

…uting-1

smalyshev

A couple of nitpicks, overall LGTM.

smalyshev · 2026-08-10T18:34:37Z

+        ProjectRoutingUsageSnapshot snapshot = new ProjectRoutingUsageSnapshot(5L, 0L, 0L, 0L, 0L, 0L, 0L, 3L, 0L, 0L, 0L, 0L, 0L, 0L, 0L);
+        ClusterStatsResponse response = buildResponse(snapshot, null);
+        String json = Strings.toString(response);
+        assertThat(json, containsString("\"project_routing\""));


String-matching JSON feels a bit fragile. Can't we use some existing XContent methods to do the same?

Good call out. Will be fixed in next push.

smalyshev · 2026-08-10T18:38:10Z

+ * A snapshot of the project's tag configuration for inclusion in {@code GET _cluster/stats}.
+ * Populated by the serverless cross-project module (Ticket 4). Emits the static config fields
+ * ({@code total}, {@code total_custom}, {@code names}, {@code named_routing_expressions}) inside
+ * the top-level {@code tags} object.


This comment is slightly inaccurate - tags name is not produced by this class but by the enveloping code, but the comment implies it builds the whole object.

Which raises an question - should we make it produce the whole object? I am fine either way but the comment needs to match what happens.

Code comment updated in next push.

…uting-1

smalyshev · 2026-08-10T19:43:36Z

+            1L  // esql: total=8, with_pr=4, alias_wildcard=2, in_SET=3, failures=1
+        );
+        String json = toJson(snap);
+        assertThat(json, not(containsString("\"search\"")));


May want to also fix JSON string matching here

quux00 requested a review from a team as a code owner August 5, 2026 19:01

quux00 added >non-issue Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch :Search Foundations/Search Catch all for Search Foundations v9.6.0 labels Aug 5, 2026

quux00 requested a review from smalyshev August 5, 2026 19:40

quux00 force-pushed the telemetry/project-routing-1 branch 2 times, most recently from ef4c9df to 79847fb Compare August 5, 2026 20:15

quux00 added 3 commits August 6, 2026 10:15

Additional test and some variable renames

af6076c

Fixed compilation error

61213a7

quux00 force-pushed the telemetry/project-routing-1 branch from 79847fb to 61213a7 Compare August 6, 2026 14:15

elasticsearchmachine added the serverless-linked Added by automation, don't add manually label Aug 6, 2026

smalyshev reviewed Aug 6, 2026

View reviewed changes

quux00 and others added 2 commits August 6, 2026 15:30

Update server/src/main/java/org/elasticsearch/action/admin/cluster/st…

d586afc

…ats/TransportClusterStatsAction.java Co-authored-by: Stanislav Malyshev <smalyshev@users.noreply.github.com>

PR changes

6e27857

smalyshev reviewed Aug 6, 2026

View reviewed changes

quux00 added 6 commits August 7, 2026 10:04

PR changes

8341614

Changed to loadSingletonServiceProvider for ClusterStatsTagsProvider

6c956e3

changes based on PR feedback

e61f04e

Merge remote-tracking branch 'elastic/main' into telemetry/project-ro…

2547bfa

…uting-1

Changes based on cursor-agent review of code

95263f7

Merge remote-tracking branch 'elastic/main' into telemetry/project-ro…

dec1b01

…uting-1

quux00 removed the serverless-linked Added by automation, don't add manually label Aug 7, 2026

smalyshev approved these changes Aug 10, 2026

View reviewed changes

quux00 added 2 commits August 10, 2026 15:24

PR feedback

aff7bdc

Merge remote-tracking branch 'elastic/main' into telemetry/project-ro…

4072e5b

…uting-1

quux00 enabled auto-merge (squash) August 10, 2026 19:26

smalyshev reviewed Aug 10, 2026

View reviewed changes

quux00 merged commit 86e30a6 into elastic:main Aug 10, 2026
38 checks passed

Uh oh!

Conversation

quux00 commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Aug 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quux00 Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smalyshev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

quux00 commented Aug 5, 2026 •

edited

Loading

quux00 Aug 7, 2026 •

edited

Loading