[aws.cloudtrail] Backport ingest pipeline improvements and bugfixes to backport-aws-6.x

[brijesh-elastic]

Proposed commit message

aws.cloudtrail: ingest pipeline improvements and fixes

Backport of #20403 to `backport-aws-6.x` so that users on the 8.x Kibana
stack (package 6.x, Kibana `^8.19.4 || ^9.2.1`) receive the CloudTrail
ingest pipeline improvements. This brings the `cloudtrail` data stream to
parity with `main`.

Because `backport-aws-6.x` did not have the intermediate CloudTrail change
from #20318, that change is included here as well (it is a prerequisite for
the #20403 pipeline and is part of the same CloudTrail quality effort).

Enhancements:
- Replace every `ignore_failure: true` with an `ignore_missing` flag, a
  condition guard, or an explicit `on_failure` handler reporting to
  `error.message`, so real faults surface instead of being silently swallowed.
- Add support for CloudTrail fields that were previously dropped.
- Populate the `cloud.provider`, `cloud.service.name` and `related.*` ECS fields.
- Stop storing the assumed-role ARN session name in `user.changes.*`. The session
  name is now added to `related.user` for cross-source correlation; when it is an
  email, both the full email and the local-part prefix are added. `user.name`
  continues to hold the IAM role name for detection rules. (#20318)

Bugfixes:
- Stop populating `file.hash.sha256` on digest records, where it held the hash of
  the previous digest file while `file.path` pointed at the current one. The value
  is still available as `aws.cloudtrail.digest.previous_hash_value` and in
  `related.hash`.
- Fix `DeleteObject` using `event.type: delete`, which is not a valid ECS value.
  It is now `deletion`.
- Fix `Publish`, `SendCommand` and `Converse` producing no `event.type`. They now
  use `info`, `start` and `info` respectively.
- Fix `retain: none` behaving like `retain: all`. It now drops both the keyword and
  the flattened copies as documented.
- Fix `retain: keyword` and `retain: minimal` discarding digest and Insights records.
  The setting governs API call payloads and no longer affects either record type.
- Fix `aws.cloudtrail.service_event_details` surviving `retain: flattened` and
  `retain: none`. It is now governed like the other payload fields.

Note

This is a backport of #20403 to backport-aws-6.x (the branch that keeps
Kibana 8.x support for the AWS package). The scope is limited to the
cloudtrail data stream and package-level files (changelog, manifest,
validation). No other data streams are modified.

What this includes vs. main

• The cloudtrail ingest pipeline, fields.yml, sample_event.json,
  data-stream manifest, and pipeline test fixtures are byte-identical to the
  merged main change (a3291a9 / [aws.cloudtrail] Ingest Pipeline Improvments and Bugfixes #20403).
• The assumed-role session-name change ([aws] cloudtrail: move assumed-role session name from user.changes.* to related.user #20318) is folded in because it did not
  exist on this branch and is a prerequisite for the [aws.cloudtrail] Ingest Pipeline Improvments and Bugfixes #20403 pipeline.
• Package version bumped to 6.22.0 (format_version kept at 3.4.0).
• SVR00009 added to validation.yml (pre-existing pipeline predates that check).

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

• Check out this branch and start a stack on a supported 8.x version, e.g. elastic-package stack up -d --version 8.19.4.
• From packages/aws, run: elastic-package test pipeline --data-streams cloudtrail -v

Related issues

• Requires [aws] Add mandatory tags to system test terraform (backport #20698) #20973
• Backport of [aws.cloudtrail] Ingest Pipeline Improvments and Bugfixes #20403
• Relates aws.cloudtrail: Ingest pipeline & ECS mapping improvements #20040
• Relates elastic/security-integrations#634

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[vera-review-bot]

Severity: 🟡 Medium confidence: high path: packages/aws/changelog.yml:10

Changelog links a different PR number

Details

The changelog entries added on lines 10, 15, 25 point at pull/20403 (line 10), pull/20318 (line 15), pull/20403 (line 25), but they were added in PR #​20972. They are likely leftover template placeholders or copies from another PR.

Recommendation:

Point each added changelog entry's link at this PR:

link: https://github.com/elastic/integrations/pull/20972

---

🤖 AI-Generated Review | Vera Review Bot - v0.3.0 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[vera-review-bot]

🟢 No issues across the latest commits e655c9b.

Review summary

Issues found across earlier commits ae4b364 — 1 medium

• 🟡 Changelog links a different PR number (link) (Unresolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.3.0 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[elastic-vault-github-plugin-prod]

✅ Package owners are in sync with main.

[infra-vault-gh-plugin-prod]

💔 Build Failed

• Buildkite Build
• Commit: e655c9b

Failed CI Steps

• Check integrations aws

History

• 💔 Build #48797 failed ae4b364

cc @brijesh-elastic