feat(nats): add JetStream data stream - #20980
Conversation
Add a new `jetstream` data stream that wraps the Metricbeat `nats/jetstream` metricset (available since Elastic Stack 9.1). The data stream collects stats, account, stream, and consumer metrics from the NATS `/jsz` monitoring endpoint, with optional name filters for accounts, streams, and consumers. - New data stream `packages/nats/data_stream/jetstream/` modeled after the existing `connection` data stream, with TSDB field mappings (metric_type, dimensions). - Package version bumped to 1.13.0; Kibana constraint raised to ^9.1.0 because the jetstream metricset is only in Agent 9.1+. - Docker test setup updated to NATS 2.10.27 with `-js` enabled. - Docs, changelog, and sample event updated. Closes elastic#10748 Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Add JetStream data stream to NATS integration
ReviewersBuildkite won't run for external contributors automatically; you need to add a comment:
NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details. |
There was a problem hiding this comment.
🟡 Changes recommended
The current changes introduce a shell portability failure in the dev docker run script and a likely TSDB index-template failure due to exceeding the default time-series dimension field limit.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds first-class JetStream monitoring to the nats integration by introducing a new jetstream metrics data stream backed by the NATS /jsz endpoint, along with supporting fields/docs and updated local system-test deployment assets.
Changes:
- Introduces a new
jetstreammetrics data stream (stats/account/stream/consumer categories) with templates, fields, and a sample event. - Extends existing
statsandconnectiondata streams with additional fields (server_name,kind,type) and updates package docs to mention JetStream. - Updates the
_devdocker deployment/test setup to run a modern NATS with JetStream enabled.
File summaries
| File | Description |
|---|---|
| packages/nats/manifest.yml | Bumps package version and updates metrics input description to include JetStream. |
| packages/nats/docs/README.md | Documents JetStream dataset, compatibility, and adds field references. |
| packages/nats/_dev/build/docs/README.md | Updates the docs template to include JetStream dataset documentation. |
| packages/nats/data_stream/stats/fields/fields.yml | Adds nats.stats.server_name field. |
| packages/nats/data_stream/connection/fields/fields.yml | Adds nats.connection.kind and nats.connection.type fields. |
| packages/nats/data_stream/jetstream/manifest.yml | Defines the new JetStream metrics data stream configuration and vars. |
| packages/nats/data_stream/jetstream/agent/stream/stream.yml.hbs | Adds agent stream template for configuring JetStream collection and filters. |
| packages/nats/data_stream/jetstream/fields/base-fields.yml | Adds base fields for the new JetStream dataset. |
| packages/nats/data_stream/jetstream/fields/ecs.yml | Declares ECS dimensions used by the JetStream TSDB mapping. |
| packages/nats/data_stream/jetstream/fields/package-fields.yml | Adds package-level NATS server fields used by JetStream. |
| packages/nats/data_stream/jetstream/fields/fields.yml | Adds JetStream-specific field definitions under nats.jetstream.*. |
| packages/nats/data_stream/jetstream/sample_event.json | Adds a sample event for documentation/examples. |
| packages/nats/data_stream/jetstream/_dev/test/system/test-default-config.yml | Adds default system test config for the new data stream. |
| packages/nats/changelog.yml | Adds changelog entry for JetStream data stream introduction. |
| packages/nats/_dev/deploy/variants.yml | Updates dev deployment to test against NATS 2.10.27. |
| packages/nats/_dev/deploy/docker/run.sh | Updates container runtime script to enable JetStream and set server/cluster names. |
| packages/nats/_dev/deploy/docker/Dockerfile | Updates NATS/Go/Alpine versions and bumps healthcheck retries. |
Review details
- Files reviewed: 17/17 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
🟡 Changes recommended
TSDS dimension definitions for JetStream entities can cause metric series collisions, and the traffic generator should fail fast after retry exhaustion to avoid flaky/no-signal system tests.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
packages/nats/data_stream/jetstream/manifest.yml:68
- The PR description/README mention the JetStream dataset requires Elastic Agent 9.1+, but the data stream description shown in Fleet doesn't surface that requirement. Consider including the requirement directly in this data stream description so users see it at configuration time.
title: NATS JetStream metrics
enabled: false
description: Collect JetStream metrics (stats, account, stream, consumer) from NATS servers
- Files reviewed: 18/18 changed files
- Comments generated: 5
- Review effort level: Lite
| fields: | ||
| - name: name | ||
| type: keyword | ||
| description: | | ||
| The name of the stream. |
| } | ||
|
|
||
| consumerCfg := jetstream.ConsumerConfig{ |
There was a problem hiding this comment.
🟡 Changes recommended
The PR introduces duration-like fields without format: duration and has version-requirement wording/enforcement mismatches (e.g., “JetStream 2.9+” and Agent minimum not reflected in package constraints).
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (6)
Previously missed (2) — in code that hasn't changed since the last review.
packages/nats/data_stream/jetstream/fields/fields.yml:89
- These values are described as a time interval, but the field lacks
format: duration, so Kibana will not treat it as a duration field. Adding the duration format will improve display/aggregation consistency with other NATS duration fields (e.g., nats.stats.uptime).
This issue also appears in the following locations of the same file:
- line 277
- line 435
- name: sync_interval
type: long
metric_type: gauge
description: |
The fsync/sync interval for page cache in the filestore.
packages/nats/data_stream/jetstream/manifest.yml:58
- This variable description says "Requires JetStream 2.9+", but JetStream is a feature of the NATS server. Referencing the NATS server version avoids ambiguity for users configuring the integration.
description: Enable collection of JetStream consumer metrics. Requires JetStream 2.9+.
packages/nats/data_stream/jetstream/fields/fields.yml:281
max_ageis a duration (nanoseconds) but is missingformat: duration, which makes it harder to visualize as time in Kibana. Consider marking it as a duration like other time-based NATS fields.
- name: max_age
type: long
metric_type: gauge
description: |
Maximum age of any message in the stream, expressed in nanoseconds.
packages/nats/data_stream/jetstream/fields/fields.yml:439
ack_waitis explicitly a duration (nanoseconds) but is missingformat: duration, which prevents duration-aware formatting in Kibana. Adding the duration format will make this field consistent with other duration fields in the package.
- name: ack_wait
type: long
metric_type: gauge
description: |
The duration (in nanoseconds) that the server will wait for an acknowledgment for any individual message once it has been delivered to a consumer. If an acknowledgment is not received in time, the message will be redelivered.
packages/nats/docs/README.md:776
- Same as above: this should reference the NATS server version rather than "JetStream 2.9+" to make the requirement unambiguous for users.
* `consumer` — Per-consumer metrics (delivered, ack floor, pending, config). Requires JetStream 2.9+.
packages/nats/_dev/build/docs/README.md:106
- Same wording issue here: use the NATS server version (e.g., "Requires NATS 2.9+") instead of "JetStream 2.9+" for clarity.
* `consumer` — Per-consumer metrics (delivered, ack floor, pending, config). Requires JetStream 2.9+.
- Files reviewed: 18/18 changed files
- Comments generated: 3
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
The stack compatibility constraints and a few correctness/documentation issues in the new JetStream additions should be addressed before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (4)
Previously missed (3) — in code that hasn't changed since the last review.
packages/nats/_dev/deploy/docker/jetstream-traffic.go:99
- The initial publish loop ignores errors from
js.Publish, which can make the traffic generator silently do nothing (and make system tests/debugging misleading) if JetStream isn’t ready yet.
for i := 0; i < 20; i++ {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
msg := fmt.Sprintf("order data %d", i)
_, _ = js.Publish(ctx, "orders.created", []byte(msg))
cancel()
packages/nats/_dev/deploy/docker/jetstream-traffic.go:113
- On publish failure in the main loop, the code
continues without logging. Even a simple log line helps diagnose why no traffic/consumer activity is being generated during system tests.
_, err := js.Publish(ctx, "orders.created", []byte(msg))
cancel()
if err != nil {
continue
}
packages/nats/data_stream/jetstream/fields/fields.yml:341
- The
ack_floorgroup description says AckFloor is the highest contiguous acknowledged sequence, butconsumer_seqis described as the lowest contiguous acknowledged sequence. This is internally inconsistent and will confuse users reading the exported fields.
This issue also appears on line 344 of the same file.
metric_type: gauge
description: |
The lowest contiguous consumer sequence number that has been acknowledged.
packages/nats/data_stream/jetstream/fields/fields.yml:346
- The
ack_floorgroup description says AckFloor is the highest contiguous acknowledged sequence, butstream_seqis described as the lowest contiguous acknowledged sequence. This should match the group description to avoid conflicting documentation.
metric_type: gauge
description: |
The lowest contiguous stream sequence number that has been acknowledged by the consumer.
- Files reviewed: 18/18 changed files
- Comments generated: 1
- Review effort level: Lite
…d add publish error logging
There was a problem hiding this comment.
🟡 Changes recommended
The changelog entry link and compatibility documentation need small but important fixes to match repository conventions and the updated package-wide version constraints.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
packages/nats/changelog.yml:6
- The new 1.13.0 changelog entry links to the issue, but this package’s changelog consistently links to the PR URL for each entry (see subsequent entries). Please change this link to the actual PR link before merging to keep release-note traceability consistent.
- version: "1.13.0"
changes:
- description: Add JetStream data stream for monitoring JetStream stats, accounts, streams, and consumers. Requires Elastic Agent 9.1+.
type: enhancement
link: https://github.com/elastic/integrations/issues/10748
- Files reviewed: 18/18 changed files
- Comments generated: 2
- Review effort level: Lite
There was a problem hiding this comment.
🔵 Needs a closer look
The JetStream data stream introduces an analyzed text field in a time_series metrics mapping and the name-filter vars are hidden from the Fleet UI, both of which should be addressed before merging.
Review details
Suppressed comments (4)
Previously missed (2) — in code that hasn't changed since the last review.
packages/nats/data_stream/jetstream/fields/fields.yml:248
nats.jetstream.stream.config.descriptionis defined astext, which introduces an analyzed field in a time_series metrics data stream. In the existing NATS metrics data streams, string fields are modeled askeyword; usingkeywordhere is more consistent and avoids unnecessary analysis/index overhead.
- name: description
type: text
description: |
The description of the stream.
packages/nats/data_stream/jetstream/manifest.yml:35
- The account name filter var is hidden from the Fleet UI (
show_user: false), which makes the documented name-filtering feature hard to discover/configure without editing advanced YAML.
This issue also appears in the following locations of the same file:
- line 44
- line 59
- name: jetstream_account_names
type: text
title: JetStream account names filter
multi: true
required: false
show_user: false
description: Filter account metrics by name. When empty, all accounts are reported.
packages/nats/data_stream/jetstream/manifest.yml:50
- The stream name filter var is hidden from the Fleet UI (
show_user: false), which makes stream name filtering difficult to configure via the standard integration settings.
- name: jetstream_stream_names
type: text
title: JetStream stream names filter
multi: true
required: false
show_user: false
description: Filter stream metrics by name. When empty, all streams are reported.
packages/nats/data_stream/jetstream/manifest.yml:65
- The consumer name filter var is hidden from the Fleet UI (
show_user: false), which makes consumer name filtering difficult to configure via the standard integration settings.
- name: jetstream_consumer_names
type: text
title: JetStream consumer names filter
multi: true
required: false
show_user: false
description: Filter consumer metrics by name. When empty, all consumers are reported.
- Files reviewed: 18/18 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🟢 Approval recommended
The changes are consistent across package metadata, docs, fields, and dev/system-test scaffolding, and no concrete defects were found in the updated/new files.
Review details
- Files reviewed: 18/18 changed files
- Comments generated: 0 new
- Review effort level: Lite
Proposed commit message
feat(nats): add JetStream data stream
WHAT:
jetstreamdata stream to the NATS package to collect metrics from the NATS/jszmonitoring endpoint.stats,account,stream, andconsumer, with optional name-filtering for accounts, streams, and consumers.nats.jetstream.*(covering cluster info, storage/memory quotas, message counters, stream states, and consumer ack/delivery floors).jetstream-traffic.go) to spin up streams and durable consumers and simulate active messaging during system tests.WHY:
Checklist
changelog.ymlfile.Author's Checklist
elastic-package check).fields.ymldefinitions match all metrics exported by NATS/jsz.stats,account,stream,consumer).How to test this PR locally
cd packages/natsjetstreamdata stream against the local test stack:elastic-package test system -d jetstreamnats.jetstream.stats.*,nats.jetstream.account.*,nats.jetstream.stream.*, andnats.jetstream.consumer.*.Related issues