[kolide] Add latest_device and latest_people transforms

packages/kolide/_dev/build/docs/README.md

@@ -49,7 +49,7 @@
 
 ### Host correlation for device checks
 
-Check-run results identify the device only by its numeric Kolide device ID, which maps to `host.id`. The payload carries no hostname, so the integration does not set `host.name` on this data stream. You can correlate check runs with the `device`, `auth`, and `issues` data streams using the shared `host.id`. If you need `host.name` directly on check-run documents, you must enrich them at ingest time with an Elasticsearch [enrich policy](https://www.elastic.co/docs/manage-data/ingest/transform-enrich/data-enrichment) that maps `host.id` to `host.name` from the `device` data stream. This setup requires you to enable the `device` data stream and periodically refresh the enrich policy so new or renamed devices resolve correctly.
+Check-run results identify the device only by its numeric Kolide device ID, which maps to `host.id`. The payload carries no hostname, so the integration does not set `host.name` on this data stream. You can correlate check runs with the `device`, `auth`, and `issues` data streams using the shared `host.id`. If you need `host.name` directly on check-run documents, you must enrich them at ingest time with an Elasticsearch [enrich policy](https://www.elastic.co/docs/manage-data/ingest/transform-enrich/data-enrichment) that matches on `host.id`. Use `logs-kolide_latest.device` as the policy's source index: it is maintained by the `latest_device` transform and holds exactly one document per device, so each `host.id` resolves to a single unambiguous `host.name`. See [Latest device and people snapshots](#latest-device-and-people-snapshots) for details. This setup requires you to enable the `device` data stream, which is what feeds the transform, and to re-execute the enrich policy periodically so that new or renamed devices resolve correctly.
 
 ### Document identity for requests
 
@@ -59,6 +59,25 @@
 
 The endpoint `GET /people` returns a full-table snapshot with no modified-since filter, so the integration re-fetches every person on every poll. To prevent unnecessary indexing, the integration deduplicates documents using a fingerprint of the entire raw record, excluding `last_authenticated_at`. This exclusion prevents a new document from being created every time an active person authenticates. A change to any other field (such as name, email, registered-device status, or SCIM usernames) produces a new document, while unchanged records are deduplicated across polls.
 
+### Latest device and people snapshots
+
+The `device` and `people` data streams keep a document per distinct state of each device and person, so querying them for "what does this device look like right now" means sorting and collapsing on every search. The integration ships two transforms that maintain that current-state view for you:
+
+| Transform | Source | Destination alias | Entity key |
+| --- | --- | --- | --- |
+| `latest_device` | `logs-kolide.device-*` | `logs-kolide_latest.device` | `host.id` |
+| `latest_people` | `logs-kolide.people-*` | `logs-kolide_latest.people` | `user.id` |
+
+Each destination index holds exactly one document per device or person, containing its most recently collected state. When a record changes, the transform replaces the previous document rather than adding to it. These indices are well suited as the source for an Elasticsearch [enrich policy](https://www.elastic.co/docs/manage-data/ingest/transform-enrich/data-enrichment): because the entity key only determines document identity, any field on the document can serve as the policy's match field. For example, you can enrich on `host.hostname` or `host.name` from `logs-kolide_latest.device`, or on `user.email` from `logs-kolide_latest.people`.
+
+Both transforms key on the stable Kolide identifier (`host.id` for devices, `user.id` for people) rather than on a display name, so renaming a device or changing a person's email updates the existing document instead of creating a second entity.
+
+The `latest_device` transform reads only the full inventory snapshots collected from the REST API (`event.kind: state`). Device webhook deliveries carry only a few fields, and because a transform replaces the whole destination document, including them would overwrite a complete snapshot with a sparse one. Webhook events remain fully queryable in `logs-kolide.device-*`.
+
+Both transforms exclude the cold and frozen data tiers from their source query so they never scan searchable-snapshot storage. Existing destination documents are unaffected, but a device or person whose newest source document has aged into those tiers is not repopulated if the transform is ever reinstalled and rebuilt from scratch.
+
+Neither transform applies a retention policy. An unchanged device or person is deduplicated at ingest, so the timestamps on its newest source document reflect the last time its content changed, not the last time it was seen. A time-based retention policy would therefore evict entities that are still active in Kolide but have not changed recently. The trade-off is that devices and people deleted in Kolide remain in the destination indices until you remove them.
+
 ### Supported use cases
 
 Integrating Kolide with Elastic provides a powerful solution for enhancing security posture and operational visibility. You can use this integration for the following use cases:

packages/kolide/changelog.yml

@@ -1,4 +1,12 @@
 # later versions go on top
+- version: "0.3.0"
+  changes:
+    - description: Add latest_device and latest_people transforms, which maintain a current-state index of every Kolide device and person.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/21005
+    - description: Add labels.is_transform_source to the device and people data streams, so transform source documents can be distinguished from transform output.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/21005
 - version: "0.2.1"
   changes:
     - description: Remove empty ILM policies section from the README.

packages/kolide/data_stream/device/fields/is-transform-source-true.yml

@@ -0,0 +1,4 @@
+- name: labels.is_transform_source
+  type: constant_keyword
+  description: Distinguishes between documents that are a source for a transform and documents that are an output of a transform, to facilitate easier filtering.
+  value: "true"

packages/kolide/data_stream/people/fields/is-transform-source-true.yml

@@ -0,0 +1,4 @@
+- name: labels.is_transform_source
+  type: constant_keyword
+  description: Distinguishes between documents that are a source for a transform and documents that are an output of a transform, to facilitate easier filtering.
+  value: "true"