[kolide] Add latest_device and latest_people transforms - #21005
Conversation
Add two latest transforms that maintain a current-state index of every Kolide device and person, suitable as a source for enrich policies. - latest_device: logs-kolide.device-* -> logs-kolide_latest.device, keyed on host.id. - latest_people: logs-kolide.people-* -> logs-kolide_latest.people, keyed on user.id. Both key on the stable Kolide identifier rather than a display name, so renaming a device or changing a person's email updates the existing document instead of creating a second entity. Display names such as host.hostname remain on the destination documents and stay available as enrich match fields. latest_device filters the source to event.kind: state so that sparse device webhook deliveries routed in from kolide.webhook cannot overwrite a full REST API inventory snapshot. Neither transform sets a retention policy. Both ingest pipelines assign a content fingerprint _id, so re-polling an unchanged record collides on that _id and the write is dropped rather than merged; the timestamps on an entity's newest source document therefore track the last time its content changed, not the last time it was seen. A time-based retention policy would evict entities that are still active in Kolide but have not changed within max_age. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ReviewersBuildkite won't run for external contributors automatically; you need to add a comment:
NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details. |
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
There are documentation consistency issues and the changelog entry’s PR link appears to be invalid/unresolvable, which should be corrected before merge.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds two latest Elasticsearch transforms to the Kolide integration to maintain “current-state” indices (one doc per device / person) suitable for enrich policies and entity lookups, and documents their usage.
Changes:
- Add
latest_deviceandlatest_peopletransforms with version-suffixed destination indices behindmove_on_creationaliases. - Define destination index templates and exported fields for both transforms.
- Update Kolide docs and bump package version to
0.3.0with a changelog entry.
File summaries
| File | Description |
|---|---|
| packages/kolide/manifest.yml | Bump package version for new feature release. |
| packages/kolide/elasticsearch/transform/latest_device/transform.yml | Define latest_device transform (source filter, latest config, alias). |
| packages/kolide/elasticsearch/transform/latest_device/manifest.yml | Destination index template for latest_device. |
| packages/kolide/elasticsearch/transform/latest_device/fields/base-fields.yml | Base ECS fields/constants for latest_device destination docs. |
| packages/kolide/elasticsearch/transform/latest_device/fields/beats.yml | Beats compatibility fields for latest_device destination docs. |
| packages/kolide/elasticsearch/transform/latest_device/fields/ecs.yml | ECS field declarations for latest_device destination docs. |
| packages/kolide/elasticsearch/transform/latest_device/fields/fields.yml | Kolide device vendor fields for latest_device destination docs. |
| packages/kolide/elasticsearch/transform/latest_device/fields/is-transform-source-false.yml | Mark destination docs as non-source for transform filtering. |
| packages/kolide/elasticsearch/transform/latest_people/transform.yml | Define latest_people transform (source filter, latest config, alias). |
| packages/kolide/elasticsearch/transform/latest_people/manifest.yml | Destination index template for latest_people. |
| packages/kolide/elasticsearch/transform/latest_people/fields/base-fields.yml | Base ECS fields/constants for latest_people destination docs. |
| packages/kolide/elasticsearch/transform/latest_people/fields/beats.yml | Beats compatibility fields for latest_people destination docs. |
| packages/kolide/elasticsearch/transform/latest_people/fields/ecs.yml | ECS field declarations for latest_people destination docs. |
| packages/kolide/elasticsearch/transform/latest_people/fields/fields.yml | Kolide people vendor fields for latest_people destination docs. |
| packages/kolide/elasticsearch/transform/latest_people/fields/is-transform-source-false.yml | Mark destination docs as non-source for transform filtering. |
| packages/kolide/docs/README.md | Document latest snapshot transforms and recommend them for enrich sources. |
| packages/kolide/_dev/build/docs/README.md | Mirror the docs README updates for the build-docs source. |
| packages/kolide/changelog.yml | Add changelog entry for the new transforms. |
Review details
- Files reviewed: 18/18 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| ### Host correlation for device checks | ||
|
|
||
| Check-run results identify the device only by its numeric Kolide device ID, which maps to `host.id`. The payload carries no hostname, so the integration does not set `host.name` on this data stream. You can correlate check runs with the `device`, `auth`, and `issues` data streams using the shared `host.id`. If you need `host.name` directly on check-run documents, you must enrich them at ingest time with an Elasticsearch [enrich policy](https://www.elastic.co/docs/manage-data/ingest/transform-enrich/data-enrichment) that maps `host.id` to `host.name` from the `device` data stream. This setup requires you to enable the `device` data stream and periodically refresh the enrich policy so new or renamed devices resolve correctly. | ||
| Check-run results identify the device only by its numeric Kolide device ID, which maps to `host.id`. The payload carries no hostname, so the integration does not set `host.name` on this data stream. You can correlate check runs with the `device`, `auth`, and `issues` data streams using the shared `host.id`. If you need `host.name` directly on check-run documents, you must enrich them at ingest time with an Elasticsearch [enrich policy](https://www.elastic.co/docs/manage-data/ingest/transform-enrich/data-enrichment) that maps `host.id` to `host.name` from the `device` data stream. This setup requires you to enable the `device` data stream and periodically refresh the enrich policy so new or renamed devices resolve correctly. The `logs-kolide_latest.device` index described in [Latest device and people snapshots](#latest-device-and-people-snapshots) is a convenient source for such a policy, since it already holds exactly one document per device. |
There was a problem hiding this comment.
Good catch — fixed in ae7aa94.
The paragraph named two different sources for the same policy: my appended sentence pointed at logs-kolide_latest.device, while the preceding sentence still described the policy as mapping host.id to host.name from the device data stream. Rewritten around a single source:
If you need
host.namedirectly on check-run documents, you must enrich them at ingest time with an Elasticsearch enrich policy that matches onhost.id. Uselogs-kolide_latest.deviceas the policy's source index: it is maintained by thelatest_devicetransform and holds exactly one document per device, so eachhost.idresolves to a single unambiguoushost.name. See Latest device and people snapshots for details. This setup requires you to enable thedevicedata stream, which is what feeds the transform, and to re-execute the enrich policy periodically so that new or renamed devices resolve correctly.
The device data stream is still required, but it is now stated as the transform's input rather than as the policy's source — which is also the more accurate framing, since an enrich policy over the raw data stream would see many documents per host.id and resolve ambiguously.
| - description: Add latest_device and latest_people transforms, which maintain a current-state index of every Kolide device and person. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/21000 |
| ### Host correlation for device checks | ||
|
|
||
| Check-run results identify the device only by its numeric Kolide device ID, which maps to `host.id`. The payload carries no hostname, so the integration does not set `host.name` on this data stream. You can correlate check runs with the `device`, `auth`, and `issues` data streams using the shared `host.id`. If you need `host.name` directly on check-run documents, you must enrich them at ingest time with an Elasticsearch [enrich policy](https://www.elastic.co/docs/manage-data/ingest/transform-enrich/data-enrichment) that maps `host.id` to `host.name` from the `device` data stream. This setup requires you to enable the `device` data stream and periodically refresh the enrich policy so new or renamed devices resolve correctly. | ||
| Check-run results identify the device only by its numeric Kolide device ID, which maps to `host.id`. The payload carries no hostname, so the integration does not set `host.name` on this data stream. You can correlate check runs with the `device`, `auth`, and `issues` data streams using the shared `host.id`. If you need `host.name` directly on check-run documents, you must enrich them at ingest time with an Elasticsearch [enrich policy](https://www.elastic.co/docs/manage-data/ingest/transform-enrich/data-enrichment) that maps `host.id` to `host.name` from the `device` data stream. This setup requires you to enable the `device` data stream and periodically refresh the enrich policy so new or renamed devices resolve correctly. The `logs-kolide_latest.device` index described in [Latest device and people snapshots](#latest-device-and-people-snapshots) is a convenient source for such a policy, since it already holds exactly one document per device. |
There was a problem hiding this comment.
Good catch — fixed in ae7aa94.
The paragraph named two different sources for the same policy: my appended sentence pointed at logs-kolide_latest.device, while the preceding sentence still described the policy as mapping host.id to host.name from the device data stream. Rewritten around a single source:
If you need
host.namedirectly on check-run documents, you must enrich them at ingest time with an Elasticsearch enrich policy that matches onhost.id. Uselogs-kolide_latest.deviceas the policy's source index: it is maintained by thelatest_devicetransform and holds exactly one document per device, so eachhost.idresolves to a single unambiguoushost.name. See Latest device and people snapshots for details. This setup requires you to enable thedevicedata stream, which is what feeds the transform, and to re-execute the enrich policy periodically so that new or renamed devices resolve correctly.
The device data stream is still required, but it is now stated as the transform's input rather than as the policy's source — which is also the more accurate framing, since an enrich policy over the raw data stream would see many documents per host.id and resolve ambiguously.
There was a problem hiding this comment.
🟢 Approval recommended
The transforms and docs changes look consistent with existing patterns, and the only flagged item is a minor convention improvement around labels.is_transform_source usage.
Review details
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
packages/kolide/elasticsearch/transform/latest_device/fields/is-transform-source-false.yml:4
labels.is_transform_sourceis introduced on the transform outputs as a constant"false", but the Kolide source data streams don’t currently set a corresponding"true"value. In other integrations that ship transforms, this label is usually set on both source (true) and destination (false) to make filtering unambiguous (e.g.,packages/island_browser/data_stream/device/fields/is-transform-source-true.yml). Consider addingis-transform-source-true.ymlunderpackages/kolide/data_stream/device/fields/(andpeople/fields/) to match the intended “distinguishes between source and output” semantics.
packages/kolide/elasticsearch/transform/latest_people/fields/is-transform-source-false.yml:4- Same as
latest_device: this transform output setslabels.is_transform_sourceto constant"false", but the Kolidepeoplesource data stream doesn’t currently set a corresponding"true". If the goal is easy filtering between source and transform output, consider addingis-transform-source-true.ymlto thepackages/kolide/data_stream/people/fields/source fields (and similarly fordevice).
- Files reviewed: 18/18 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🟢 Approval recommended
The transforms, templates, versioning, and documentation changes are consistent with existing “latest transform” patterns in this repo and align with the PR’s stated behavior.
Review details
- Files reviewed: 18/18 changed files
- Comments generated: 0 new
- Review effort level: Lite
The paragraph on enriching check-run documents recommended logs-kolide_latest.device as the enrich source while still describing the policy as mapping host.id to host.name from the `device` data stream, so it named two different sources for the same policy. Rewrite it around a single source: the policy matches on host.id against logs-kolide_latest.device, which holds exactly one document per device so each host.id resolves unambiguously. The `device` data stream is still required, now stated as the transform's input rather than as the policy's source. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
@vinit-chauhan I think this is what you meant from your previous comment on the other PR of mine, can you take a look? |
There was a problem hiding this comment.
🔵 Needs a closer look
The new transforms add labels.is_transform_source: "false" on outputs but Kolide sources don’t define the corresponding "true" field, which makes the “distinguish source vs output” label pattern incomplete and inconsistent with other transform-shipping packages.
Review details
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
packages/kolide/elasticsearch/transform/latest_device/fields/is-transform-source-false.yml:4
labels.is_transform_sourceis defined here as a constant "false" value, but the Kolide source data streams (e.g.kolide.device,kolide.people) do not appear to define the corresponding constant "true" field. In other integrations that ship transforms (e.g.island_browserandqualys_gav), the source dataset defineslabels.is_transform_source: "true"and the transform output defines"false", so users can reliably filter either side. Consider addingdata_stream/{device,people}/fields/is-transform-source-true.ymlso this field actually distinguishes source vs transform output across Kolide indices too.
packages/kolide/elasticsearch/transform/latest_people/fields/is-transform-source-false.yml:4- Same as
latest_device: this transform output defineslabels.is_transform_source: "false", but the Kolide source datasets do not definelabels.is_transform_source: "true". Adding the correspondingis-transform-source-true.ymlto thekolide.peopledata stream would make the label symmetric and consistent with other packages’ transform patterns.
- Files reviewed: 18/18 changed files
- Comments generated: 0 new
- Review effort level: Lite
The transforms' destination indices declare labels.is_transform_source as a constant "false", but the device and people data streams did not declare the corresponding "true", so the label could not actually be used to tell the two sides apart when querying across logs-kolide.* and logs-kolide_latest.*. Add is-transform-source-true.yml to both source data streams. This is the convention in 24 of the 26 packages that ship transforms, including island_browser, qualys_gav, and jamf_pro, whose transform layout this package already follows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Confirmed and fixed in a692785. I checked the claim against the repo before acting, and it holds: of the 26 packages that declare Added
Without the source side the label was inert — you could not use it to separate source from output when querying across
|
There was a problem hiding this comment.
🟢 Approval recommended
The new transforms and field additions follow existing repo patterns, validate cleanly against the Kolide ingestion behavior described, and the changelog/README updates align with the new assets.
Review details
- Files reviewed: 20/20 changed files
- Comments generated: 0 new
- Review effort level: Lite
|
@vera-review-bot review |
Elastic Docs Style Checker (Vale)Summary: 1 suggestion found 💡 Suggestions (1): Optional style improvements. Apply when helpful.
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
![vera-review-bot[bot]](https://avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
Review summaryIssues found across the latest commits a692785 — 1 medium
🤖 AI-Generated Review | Vera Review Bot - v0.3.0 | 📚 Knowledge base: integration-skills
|
Both latest transforms scan the whole source index pattern on their initial run, so without a _tier guard they can read documents out of searchable-snapshot storage. Matches the source-query convention used by the other latest transforms in the repo (wiz, crowdstrike, microsoft_defender_endpoint). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🟢 Approval recommended
The transforms, mappings/fields, labeling, versioning, and documentation changes are consistent with established patterns in the repo and appear internally coherent.
Review details
- Files reviewed: 20/20 changed files
- Comments generated: 0 new
- Review effort level: Lite
|
/test |
vinit-chauhan
left a comment
vinit-chauhan
left a comment
There was a problem hiding this comment.
Hey @hnguyen-coreweave - Thanks for the PR, the transforms looks good to me. 😄
|
✅ All changelog entries have the correct PR link. |
There was a problem hiding this comment.
🟢 Approval recommended
The transform configs, fields/mappings, docs, and versioning/changelog updates are internally consistent and follow established transform patterns used elsewhere in the repo.
Review details
- Files reviewed: 20/20 changed files
- Comments generated: 0 new
- Review effort level: Lite
🚀 Benchmarks reportTo see the full report comment with |
💚 Build Succeeded
History
|
|
Package kolide - 0.3.0 containing this change is available at https://epr.elastic.co/package/kolide/0.3.0/ |
Proposed commit message
Add two
latesttransforms to the Kolide package that maintain a current-state index of every device and person, intended as a source for enrich policies and entity lookups.latest_devicelogs-kolide.device-*logs-kolide_latest.devicehost.idlatest_peoplelogs-kolide.people-*logs-kolide_latest.peopleuser.idWHY: The
deviceandpeopledata streams retain one document per distinct state of each entity, becauseGET /devicesandGET /peopleexpose no modified-since cursor and are re-polled in full on every interval. Answering "what does this device look like right now" therefore requires a collapse or a top-hits aggregation on every query, and there is no stable index to point an enrich policy at. These transforms maintain that current-state view.WHAT: Both follow the pattern established by
qualys_gav,jamf_pro, andisland_browser:sort: "@timestamp",frequency: 30s, sync onevent.ingestedwith a 120s delay,deduce_mappings: false,unattended: true,start: true, and a version-suffixed destination index behind amove_on_creationalias. Three decisions differ from a straight copy and are worth review:1. Keyed on the stable identifier, not the display name.
host.idis the Kolide device ID anduser.idthe person ID; both survive renames and email changes, so an update replaces the existing document rather than creating a second entity. The mutable display names (host.hostname,host.name,user.email) remain on the destination documents and stay usable as enrich match fields — the transform key only determines document identity, not what a policy can match on.2.
latest_devicefilters the source toevent.kind: state.kolide.devicecarries two document shapes: full inventory snapshots polled from the REST API (event.kind: state), and sparse webhook deliveries routed in fromkolide.webhook(devices.created,devices.registered,devices.destroyed,device_trust.status_changed;event.kind: event), which populate onlyhost.id,host.name/host.hostname,user.*, andkolide.device.device_status/device_url. A latest transform replaces the whole destination document, so without this filter a webhook delivery arriving after a poll wins the bucket on@timestampand blanksserial,note,auth_state, and the rest until the next poll. Webhook events remain fully queryable inlogs-kolide.device-*.3. Neither transform sets a
retention_policy, deliberately. This departs from the packages above and the reasoning is specific to Kolide. Both ingest pipelines assign a content-fingerprint_id(device overevent.originalminuslast_seen_at, people minuslast_authenticated_at). These are logs data streams, so re-polling an unchanged record collides on that_idand the write is dropped rather than merged.@timestampandevent.ingestedon an entity's newest source document therefore track the last time its content changed, not the last time it was seen — and the customaryretention_policy: {field: event.ingested, max_age: 30d}would evict a device that is still active in Kolide but has not changed configuration within the window, silently emptying it out of the enrich source. Eachtransform.ymlcarries an inline comment recording this so it is not reintroduced later.The accepted trade-off for (3) is that devices and people deleted in Kolide remain in the destination indices until removed by other means.
Checklist
changelog.ymlfile.^8.19.0 || ^9.1.0; transforms use no features beyond that floor.)Version bumped 0.2.0 → 0.3.0 (minor: backward-compatible new feature).
.github/CODEOWNERSalready maps/packages/kolideto@elastic/integration-experience, so no change was needed there.Author's Checklist
host.idthe right entity key forlatest_device, rather thanhost.hostname? The reasoning is in point (1) above; the display names remain available as match fields either way.retention_policyacceptable? Point (3) explains why the usual 30d policy would be actively harmful here, but it does mean deleted entities persist.event.kind: statethe filter you would prefer for excluding webhook documents, versus something narrower such asinput.type: cel?How to test this PR locally
To exercise the transforms end to end against a running stack:
Then index two
kolide.devicesnapshot documents sharing ahost.idbut differing inhost.hostnameandkolide.device.note, plus one sparse document withevent.kind: eventcarrying the newest@timestamp. After the 120s sync delay,logs-kolide_latest.deviceshould hold exactly one document for thathost.id, showing the newer snapshot's values withserial/note/auth_stateintact and the webhook document'sdevice_statusabsent.I ran this locally against a 9.4.3 stack. Both transforms installed with the intended configuration and reached
state: startedwith zero index or search failures; two device snapshots plus one rename collapsed to a single destination document, the sparseevent.kind: eventdocument was correctly excluded despite having the newest@timestamp, and two person snapshots collapsed to one. All 26 existing pipeline tests pass.Related issues