Skip to content

[AWS] [SQS] Add detail dashboard - #20974

Open
mrg-elastic wants to merge 8 commits into
elastic:mainfrom
mrg-elastic:feat/aws-sqs-otel-detail-dashboard
Open

[AWS] [SQS] Add detail dashboard#20974
mrg-elastic wants to merge 8 commits into
elastic:mainfrom
mrg-elastic:feat/aws-sqs-otel-detail-dashboard

Conversation

@mrg-elastic

Copy link
Copy Markdown
Contributor

Proposed commit message

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

Each SQS document contains only one metric, so AND-ing two IS NOT NULL
conditions in the same WHERE made the query match nothing. Drop the
IS NOT NULL pre-filters and use COALESCE(metric, 0) in the EVAL so
NULL time-buckets evaluate cleanly to Healthy.
@mrg-elastic
mrg-elastic marked this pull request as ready for review August 31, 2026 15:14
@mrg-elastic
mrg-elastic requested a review from a team as a code owner August 31, 2026 15:14
@agithomas

Copy link
Copy Markdown
Contributor

Kindly include the updated dashboard screenshots
.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!


The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

✅ All changelog entries have the correct PR link.

@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

💚 Build Succeeded

History

@agithomas

Copy link
Copy Markdown
Contributor

For the initial info panels, see if you have to add AND stat == "Sum" , for improving the ESQL performance.

 Suggested ES|QL:
        SET unmapped_fields = "nullify";
        FROM metrics-aws.sqs.otel-*
        | WHERE QueueName IS NOT NULL AND QueueName != "" AND stat == "Sum"
        | INLINE STATS queue_last = MAX(@timestamp) BY QueueName, cloud.region
        | INLINE STATS latest_global = MAX(@timestamp)
        | WHERE DATE_DIFF("minutes", queue_last, latest_global) <= 5
        | STATS region = MAX(cloud.region)
        | KEEP region

@agithomas

Copy link
Copy Markdown
Contributor

So, in one account you can have an SQS queue named otel-test-healthy in us-east-1 and another otel-test-healthy in eu-west-1

A message queue's name must be unique within an AWS account and region. So, either add the drill-down to the SQS URL. Alternatively, add an AWS region so that if there is a duplicate, there is a chance of filtering the specific SQS instance

@agithomas

Copy link
Copy Markdown
Contributor

Primary metric: Oldest message age (s)

Suggested ES|QL:

SET unmapped_fields = "nullify";
FROM metrics-aws.sqs.otel-*
| WHERE QueueName IS NOT NULL AND QueueName != ""
  AND stat == "Maximum"
  AND `metrics.amazonaws.com/AWS/SQS/ApproximateAgeOfOldestMessage` IS NOT NULL
| INLINE STATS queue_last = MAX(@timestamp) BY QueueName, `cloud.region`
| INLINE STATS latest_global = MAX(@timestamp)
| WHERE DATE_DIFF("minutes", queue_last, latest_global) <= 5
| STATS v = MAX(`metrics.amazonaws.com/AWS/SQS/ApproximateAgeOfOldestMessage`) BY timestamp = BUCKET(@timestamp, 50, ?_tstart, ?_tend), QueueName, `cloud.region`
| SORT timestamp
| STATS sum_value = SUM(v) BY timestamp
| STATS avg_value = AVG(sum_value),
    last_value = LAST(sum_value, timestamp)
| KEEP avg_value, last_value

Replaced the final aggregation logic: instead of AVG(v)
and LAST(v) directly from bucketed queue maximums, the
query now computes SUM(v) BY timestamp (summing all
queues' max age per bucket), then AVG/LAST of those
sums. This aligns with the CloudWatch ground truth
command's SUM(SEARCH(...)) pattern, which aggregates the
metric across all queues. The original query averaged
individual queue maximums, severely underestimating the
total age when multiple queues had old messages.

@agithomas

Copy link
Copy Markdown
Contributor

Primary metric: Total queues

Suggested ESQL

        SET unmapped_fields = "nullify";
        TS metrics-aws.sqs.otel-*
        | WHERE QueueName IS NOT NULL AND QueueName != "" AND stat == "Sum"
        | STATS last_seen = MAX(@timestamp)
          BY QueueName, cloud.region
        | INLINE STATS latest = MAX(last_seen)
        | WHERE DATE_DIFF("minutes", last_seen, latest) <= 5
        | STATS total = COUNT(*)
        | KEEP total

Add the stat == "Sum" to avoid the risk of overcounting / improve performance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants