Skip to content

[elasticsearch] Decode querylog NDJSON on the Agent - #20986

Open
consulthys wants to merge 1 commit into
mainfrom
consulthys/elasticsearch-querylog-ndjson-decode
Open

[elasticsearch] Decode querylog NDJSON on the Agent#20986
consulthys wants to merge 1 commit into
mainfrom
consulthys/elasticsearch-querylog-ndjson-decode

Conversation

@consulthys

@consulthys consulthys commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Proposed commit message

Query log files are already ECS JSON. Filebeat expands them with filestream + ndjson. The integration Agent stream was a plain logfile input, so parsing depended on pipeline-json.yml via the Fleet default_pipeline. That template (priority 200) loses to x-pack logs-elasticsearch.querylog@template (priority 240 after elastic/elasticsearch#158188). Events stayed as a JSON string in message and the dashboard was empty.

This expands NDJSON on the Agent (decode_json_fields + expand_keys), matching Filebeat. The ingest pipeline stays as a fallback for raw message lines (pipeline tests still cover that).

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

cd packages/elasticsearch
elastic-package test pipeline --data-streams querylog
elastic-package test system --data-streams querylog

On a 9.4+ cluster with elasticsearch.querylog.enabled: true and this package installed, documents in logs-elasticsearch.querylog-* should have elasticsearch.querylog.type populated even when logs-elasticsearch.querylog@template is the matching index template.

Related issues

The x-pack destination template wins over the Fleet ingest
pipeline, so querylog lines must be expanded on the shipper.

Co-authored-by: Cursor <cursoragent@cursor.com>
@consulthys
consulthys requested a review from a team as a code owner September 1, 2026 06:47
@consulthys consulthys added Integration:elasticsearch Elasticsearch Feature:Stack Monitoring Stack Monitoring Feature bugfix Pull request that fixes a bug issue Team:Stack Monitoring Stack Monitoring team [elastic/stack-monitoring] labels Sep 1, 2026
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!


The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

✅ All changelog entries have the correct PR link.

@adcleared

Copy link
Copy Markdown

Not really my place but raising the possibility that the integration pipeline might be more preferable? This would couple it to stack version and effectively make the existing pipeline for querylog in the integration (which does exist), meaningless?

@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

💚 Build Succeeded

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Pull request that fixes a bug issue Feature:Stack Monitoring Stack Monitoring Feature Integration:elasticsearch Elasticsearch Team:Stack Monitoring Stack Monitoring team [elastic/stack-monitoring]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Elasticsearch]: querylog Agent stream does not decode NDJSON, so events stay unparsed when the x-pack template wins

2 participants