Skip to content

[Wiz] Set CDR transform destination indices to 3 primary shards - #21016

Open
moxarth-rathod wants to merge 1 commit into
elastic:mainfrom
moxarth-rathod:wiz-transform-shards-increase
Open

[Wiz] Set CDR transform destination indices to 3 primary shards#21016
moxarth-rathod wants to merge 1 commit into
elastic:mainfrom
moxarth-rathod:wiz-transform-shards-increase

Conversation

@moxarth-rathod

Copy link
Copy Markdown
Contributor

Proposed commit message

The latest_cdr_vulnerabilities and latest_cdr_misconfigurations Wiz transforms write to destination indices that are created with a single primary shard, because neither transform shipped a manifest.yml with a destination_index_template. At scale this trips Elasticsearch shard-size warnings.

This standardises both CDR "latest" transforms on number_of_shards: 3, a reasonable static middle-ground (~25 GB/shard at the reported 76 GB) that clears the warning for realistic index sizes without over-sharding small deployments.

Because number_of_shards is an immutable, create-time index setting, adding a destination_index_template and bumping fleet_transform_version alone is not enough: on upgrade Fleet reinstalls the transform but keeps the existing destination index, so an already-created 1-shard index would not be re-sharded. To make the new shard count take effect for existing customers, the destination index -vN suffix is bumped so a fresh index is created from the updated template, the transform repopulates it, and the move_on_creation: true alias moves onto the new index.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • Confirm the new -v3 / -v4 destination indices are created with 3 primary shards.
  • Confirm the security_solution-wiz.{vulnerability,misconfiguration}_latest aliases point at the new indices after install/upgrade.

How to test this PR locally

  1. Build/install the package: elastic-package check then elastic-package install (from packages/wiz).
  2. Verify the shard count on the new destination indices:
    • GET security_solution-wiz.vulnerability_latest-v3/_settingsindex.number_of_shards: 3
    • GET security_solution-wiz.misconfiguration_latest-v4/_settingsindex.number_of_shards: 3
  3. Verify the aliases resolve to the new indices:
    • GET security_solution-wiz.vulnerability_latest/_alias
    • GET security_solution-wiz.misconfiguration_latest/_alias
  4. Confirm both transforms start and populate the new destination indices.

Related issues

@moxarth-rathod moxarth-rathod self-assigned this Sep 2, 2026
@moxarth-rathod
moxarth-rathod requested review from a team as code owners September 2, 2026 11:11
@moxarth-rathod moxarth-rathod added enhancement New feature or request Integration:wiz Wiz Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels Sep 2, 2026
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!


The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Changelog link mismatch — expected https://github.com/elastic/integrations/pull/21016 in the following file(s):

  • packages/wiz/changelog.yml

Tip

If expected, add the changelog-link-check:skip label to skip this check. Or, if an issue link was intended, use .../issues/<n> instead.

View Buildkite build
@moxarth-rathod

@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Sep 2, 2026

Copy link
Copy Markdown

💔 Build Failed

Failed CI Steps

History

cc @moxarth-rathod

changes:
- description: Set the `latest_cdr_vulnerabilities` and `latest_cdr_misconfigurations` transform destination indices to 3 primary shards to avoid shard-size warnings at scale. The destination indices are re-created (`-v3` and `-v4` respectively).
type: enhancement
link: https://github.com/elastic/integrations/pull/1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟡 Medium confidence: high path: packages/wiz/changelog.yml:6

Changelog links a different PR number

Details

This changelog entry's link: points at pull/1, but it was added in PR #​21016. It is likely a leftover template placeholder or a copy from another PR.

Recommendation:

Point each added changelog entry's link at this PR:

link: https://github.com/elastic/integrations/pull/21016

🤖 AI-Generated Review | Vera Review Bot - v0.3.0 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@moxarth-rathod please fix this.

@vera-review-bot

Copy link
Copy Markdown

Review summary

Issues found across the latest commits 5b43cee — 1 medium
  • 🟡 Changelog links a different PR number (link) (Unresolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.3.0 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:wiz Wiz Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[wiz] Set transform destination indices to 3 primary shards to avoid shard-size warnings at scale

2 participants