[google_workspace] - Add optional chunk_duration to Reports API data streams by ShourieG · Pull Request #21049 · elastic/integrations

ShourieG · 2026-09-03T08:46:34Z

Type of change

Enhancement
Docs

Proposed commit message

google_workspace: add optional chunk_duration to Reports API data streams

The Reports API data streams advance their collection cursor only after
a full time window has been paginated. When a long collection is
interrupted, for example by an agent restart in an agentless deployment,
the input resumes from the start of the window and re-fetches the entire
backlog, which can stall progress on high-volume reports.

Add an optional, hidden chunk_duration setting to all thirteen Reports
API data streams. When set, each interval requests a window of at most
chunk_duration starting from the last completed position and advances
only after that window is collected, so an interrupted run re-fetches at
most one chunk. The window is clamped to now minus lag_time so it never
runs ahead of the lag margin. Leaving the setting empty, or giving a
non-positive value, keeps the previous unchunked behaviour, so existing
cursors are unaffected.

Bump the package to 3.9.0, document the collection window settings in
the README, and add system tests covering chunked collection for every
stream and the disabled fallback for the token stream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

…eams The Reports API data streams advance their collection cursor only after a full time window has been paginated. When a long collection is interrupted, for example by an agent restart in an agentless deployment, the input resumes from the start of the window and re-fetches the entire backlog, which can stall progress on high-volume reports. Add an optional, hidden chunk_duration setting to all thirteen Reports API data streams. When set, each interval requests a window of at most chunk_duration starting from the last completed position and advances only after that window is collected, so an interrupted run re-fetches at most one chunk. The window is clamped to now minus lag_time so it never runs ahead of the lag margin. Leaving the setting empty, or giving a non-positive value, keeps the previous unchunked behaviour, so existing cursors are unaffected. Bump the package to 3.9.0, document the collection window settings in the README, and add system tests covering chunked collection for every stream and the disabled fallback for the token stream. Updates elastic/sdh-beats#7536

github-actions · 2026-09-03T08:47:43Z

Elastic Docs Style Checker (Vale)

Summary: 14 suggestions found

💡 Suggestions (14): Optional style improvements. Apply when helpful.

File	Line	Rule	Message
packages/google_workspace/_dev/build/docs/README.md	93	Elastic.WordChoice	Consider using 'deactivates, deselects, hides, turns off, makes unavailable' instead of 'disables', unless the term is in the UI.
packages/google_workspace/data_stream/access_transparency/manifest.yml	31	Elastic.WordChoice	Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI.
packages/google_workspace/data_stream/admin/manifest.yml	31	Elastic.WordChoice	Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI.
packages/google_workspace/data_stream/context_aware_access/manifest.yml	31	Elastic.WordChoice	Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI.
packages/google_workspace/data_stream/device/manifest.yml	31	Elastic.WordChoice	Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI.
packages/google_workspace/data_stream/drive/manifest.yml	31	Elastic.WordChoice	Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI.
packages/google_workspace/data_stream/gcp/manifest.yml	31	Elastic.WordChoice	Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI.
packages/google_workspace/data_stream/group_enterprise/manifest.yml	31	Elastic.WordChoice	Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI.
packages/google_workspace/data_stream/groups/manifest.yml	31	Elastic.WordChoice	Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI.
packages/google_workspace/data_stream/login/manifest.yml	31	Elastic.WordChoice	Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI.
packages/google_workspace/data_stream/rules/manifest.yml	31	Elastic.WordChoice	Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI.
packages/google_workspace/data_stream/saml/manifest.yml	31	Elastic.WordChoice	Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI.
packages/google_workspace/data_stream/token/manifest.yml	31	Elastic.WordChoice	Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI.
packages/google_workspace/data_stream/user_accounts/manifest.yml	31	Elastic.WordChoice	Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI.

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

infra-vault-gh-plugin-prod · 2026-09-03T08:51:46Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elastic-vault-github-plugin-prod · 2026-09-03T08:54:59Z

✅ All changelog entries have the correct PR link.

Copilot

🟡 Changes recommended

The README documents Chunk Duration as being available in the UI “Advanced options” while the actual chunk_duration vars are hidden (show_user: false), which is likely to mislead users.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR enhances the google_workspace integration’s Reports API data streams by adding an optional chunk_duration configuration that limits each polling interval to a bounded time window, reducing re-fetching after interruptions (e.g., restarts) and improving forward progress on high-volume backlogs.

Changes:

Bump google_workspace package version to 3.9.0 and add a changelog entry for the enhancement.
Add hidden chunk_duration support to all 13 Reports API httpjson-based data streams, including cursor behavior updates for chunked windows.
Add system tests (and a shared mock endpoint) to validate chunked behavior across streams plus a disabled-chunk fallback test for token; document collection window behavior in the README.

File summaries

File	Description
packages/google_workspace/manifest.yml	Bumps the integration version to 3.9.0.
packages/google_workspace/changelog.yml	Adds a 3.9.0 enhancement entry for `chunk_duration`.
packages/google_workspace/docs/README.md	Documents collection window settings, including Chunk Duration.
packages/google_workspace/_dev/build/docs/README.md	Mirrors README doc updates for the build/docs source.
packages/google_workspace/_dev/deploy/docker/config.yml	Adds a shared mock endpoint for chunked-window system tests.
packages/google_workspace/data_stream/access_transparency/manifest.yml	Adds hidden `chunk_duration` var to the access_transparency stream.
packages/google_workspace/data_stream/access_transparency/agent/stream/httpjson.yml.hbs	Implements chunked endTime and cursor advancement logic.
packages/google_workspace/data_stream/access_transparency/_dev/test/system/test-chunked-config.yml	Adds system test config for chunked collection.
packages/google_workspace/data_stream/admin/manifest.yml	Adds hidden `chunk_duration` var to the admin stream.
packages/google_workspace/data_stream/admin/agent/stream/httpjson.yml.hbs	Implements chunked endTime and cursor advancement logic.
packages/google_workspace/data_stream/admin/_dev/test/system/test-chunked-config.yml	Adds system test config for chunked collection.
packages/google_workspace/data_stream/context_aware_access/manifest.yml	Adds hidden `chunk_duration` var to the context_aware_access stream.
packages/google_workspace/data_stream/context_aware_access/agent/stream/httpjson.yml.hbs	Implements chunked endTime and cursor advancement logic.
packages/google_workspace/data_stream/context_aware_access/_dev/test/system/test-chunked-config.yml	Adds system test config for chunked collection.
packages/google_workspace/data_stream/device/manifest.yml	Adds hidden `chunk_duration` var to the device stream.
packages/google_workspace/data_stream/device/agent/stream/httpjson.yml.hbs	Implements chunked endTime and cursor advancement logic.
packages/google_workspace/data_stream/device/_dev/test/system/test-chunked-config.yml	Adds system test config for chunked collection.
packages/google_workspace/data_stream/drive/manifest.yml	Adds hidden `chunk_duration` var to the drive stream.
packages/google_workspace/data_stream/drive/agent/stream/httpjson.yml.hbs	Implements chunked endTime and cursor advancement logic.
packages/google_workspace/data_stream/drive/_dev/test/system/test-chunked-config.yml	Adds system test config for chunked collection.
packages/google_workspace/data_stream/gcp/manifest.yml	Adds hidden `chunk_duration` var to the gcp stream.
packages/google_workspace/data_stream/gcp/agent/stream/httpjson.yml.hbs	Implements chunked endTime and cursor advancement logic.
packages/google_workspace/data_stream/gcp/_dev/test/system/test-chunked-config.yml	Adds system test config for chunked collection.
packages/google_workspace/data_stream/group_enterprise/manifest.yml	Adds hidden `chunk_duration` var to the group_enterprise stream.
packages/google_workspace/data_stream/group_enterprise/agent/stream/httpjson.yml.hbs	Implements chunked endTime and cursor advancement logic.
packages/google_workspace/data_stream/group_enterprise/_dev/test/system/test-chunked-config.yml	Adds system test config for chunked collection.
packages/google_workspace/data_stream/groups/manifest.yml	Adds hidden `chunk_duration` var to the groups stream.
packages/google_workspace/data_stream/groups/agent/stream/httpjson.yml.hbs	Implements chunked endTime and cursor advancement logic.
packages/google_workspace/data_stream/groups/_dev/test/system/test-chunked-config.yml	Adds system test config for chunked collection.
packages/google_workspace/data_stream/login/manifest.yml	Adds hidden `chunk_duration` var to the login stream.
packages/google_workspace/data_stream/login/agent/stream/httpjson.yml.hbs	Implements chunked endTime and cursor advancement logic.
packages/google_workspace/data_stream/login/_dev/test/system/test-chunked-config.yml	Adds system test config for chunked collection.
packages/google_workspace/data_stream/rules/manifest.yml	Adds hidden `chunk_duration` var to the rules stream.
packages/google_workspace/data_stream/rules/agent/stream/httpjson.yml.hbs	Implements chunked endTime and cursor advancement logic.
packages/google_workspace/data_stream/rules/_dev/test/system/test-chunked-config.yml	Adds system test config for chunked collection.
packages/google_workspace/data_stream/saml/manifest.yml	Adds hidden `chunk_duration` var to the saml stream.
packages/google_workspace/data_stream/saml/agent/stream/httpjson.yml.hbs	Implements chunked endTime and cursor advancement logic.
packages/google_workspace/data_stream/saml/_dev/test/system/test-chunked-config.yml	Adds system test config for chunked collection.
packages/google_workspace/data_stream/token/manifest.yml	Adds hidden `chunk_duration` var to the token stream.
packages/google_workspace/data_stream/token/agent/stream/httpjson.yml.hbs	Implements chunked endTime and cursor advancement logic.
packages/google_workspace/data_stream/token/_dev/test/system/test-chunked-config.yml	Adds system test config for chunked collection.
packages/google_workspace/data_stream/token/_dev/test/system/test-chunk-disabled-config.yml	Adds system test config verifying chunking is disabled for non-positive values.
packages/google_workspace/data_stream/user_accounts/manifest.yml	Adds hidden `chunk_duration` var to the user_accounts stream.
packages/google_workspace/data_stream/user_accounts/agent/stream/httpjson.yml.hbs	Implements chunked endTime and cursor advancement logic.
packages/google_workspace/data_stream/user_accounts/_dev/test/system/test-chunked-config.yml	Adds system test config for chunked collection.

Review details

Files reviewed: 45/45 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

vera-review-bot · 2026-09-03T09:02:31Z

🟢 No issues across the latest commits fe5533c.

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.3.0 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

elastic-vault-github-plugin-prod · 2026-09-03T09:53:57Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

infra-vault-gh-plugin-prod · 2026-09-03T09:54:00Z

💚 Build Succeeded

Buildkite Build
Commit: fe5533c

cc @ShourieG

mergify · 2026-09-03T11:55:14Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

elastic-vault-github-plugin-prod · 2026-09-03T13:53:38Z

Package google_workspace - 3.9.0 containing this change is available at https://epr.elastic.co/package/google_workspace/3.9.0/

ShourieG self-assigned this Sep 3, 2026

updated changelog

fe5533c

ShourieG marked this pull request as ready for review September 3, 2026 08:51

Copilot AI lite review requested due to automatic review settings September 3, 2026 08:51

ShourieG requested review from a team as code owners September 3, 2026 08:51

Copilot started reviewing on behalf of ShourieG September 3, 2026 08:52 View session

Copilot AI reviewed Sep 3, 2026

View reviewed changes

Comment thread packages/google_workspace/_dev/build/docs/README.md

Comment thread packages/google_workspace/docs/README.md

kcreddy approved these changes Sep 3, 2026

View reviewed changes

ShourieG merged commit fe6b994 into elastic:main Sep 3, 2026
11 checks passed

ShourieG deleted the bugfix/google_workspace branch September 4, 2026 04:41

Navigation Menu

Uh oh!

[google_workspace] - Add optional chunk_duration to Reports API data streams - #21049

[google_workspace] - Add optional chunk_duration to Reports API data streams#21049
ShourieG merged 2 commits into
elastic:mainfrom
ShourieG:bugfix/google_workspace

ShourieG commented Sep 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 3, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Sep 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

vera-review-bot Bot commented Sep 3, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Sep 3, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 3, 2026

Uh oh!

mergify Bot commented Sep 3, 2026

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ShourieG commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of change

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

github-actions Bot commented Sep 3, 2026

Elastic Docs Style Checker (Vale)

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 3, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Sep 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

vera-review-bot Bot commented Sep 3, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Sep 3, 2026

🚀 Benchmarks report

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 3, 2026

💚 Build Succeeded

Uh oh!

mergify Bot commented Sep 3, 2026

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ShourieG commented Sep 3, 2026 •

edited

Loading