Add initial_delay parameter to the transform _start API

[shahzad31]

Summary

Continuous transforms apply a single steady-state sync.time.delay to every checkpoint's upper time bound (now - delay). That delay has to be conservative enough to tolerate source ingest/refresh lag, so already-present or just-arrived data is not processed until it has aged past the full delay. For chained transforms (e.g. the Kibana SLO rollup → summary pair, both delay: 65s) this compounds into a visible ~60–120s lag before the first downstream document appears.

This PR adds an optional initial_delay parameter to the _start API:

POST _transform/my-transform/_start?initial_delay=0s

initial_delay is a one-time, reduced delay applied while the transform is still in its initial catch-up phase — i.e. until it has processed its first document — after which it reverts to the steady-state delay. Keying off "has processed data" rather than literally "checkpoint #1" matters for chained transforms: the source data often lands shortly after the downstream transform's first (empty) checkpoint, so a checkpoint-#1-only rule would revert to the steady-state delay before any data was ever seen.

Closes #150357

Why a _start parameter (not a sync config field)

Per review feedback (thanks @prwhelan), initial_delay is a one-time-use value that only matters on initial catch-up, so it is modelled on the existing _start?from= parameter rather than persisted into the transform config in the system index. It is a transient value carried on the start request and the persistent task params.

Implementation

• initial_delay is added to StartTransformAction.Request (REST param on _start) and carried into TransformTaskParams.
• The catch-up signal lives on TransformContext as a monotonic hasProcessedData flag, set once the indexer processes documents (TransformIndexer#finalizeCheckpoint, and seeded from persisted stats on restart so a restarting transform does not re-apply the override).
• TimeBasedCheckpointProvider selects initial_delay vs delay via that flag, honored by both createNextCheckpoint (the upper-bound calculation) and sourceHasChanged (the change-detection gate — these must agree or the gate would reject the just-landed backfill).
• Validated at _start: rejected for batch (non-continuous) transforms and when initial_delay > delay.

Backwards compatibility

• Wire format for StartTransformAction.Request and TransformTaskParams guarded by a new transport version (transform_start_initial_delay); older nodes simply do not see the field.
• The persisted transform config is unchanged — nothing new is written to the system index.

Scope

This bounds first-data latency by the transform's frequency rather than delay + frequency. Local end-to-end test against Kibana SLOs (summary frequency: 1m): first summary document dropped from ~122s to ~62s by starting with initial_delay=0s, with no orchestration changes. Reaching seconds-level latency additionally requires the downstream transform to be triggered promptly after the upstream writes (e.g. _schedule_now or a shorter frequency) and is intentionally out of scope.

Evidence (measured against a local Kibana)

Ran this change against a locally-built ES with a local Kibana that starts its SLO summary transform with initial_delay=0s (delay: 65s, frequency: 1m), then created an SLO and timed the first summary document:

Trial	rollup first SLI doc	first summary doc
baseline (no initial_delay)	~3s	~122s
initial_delay=0s (run 1)	2.8s	63.0s (at summary checkpoint #2)
initial_delay=0s (run 2)	2.4s	62.4s (at summary checkpoint #2)

That is a ~60s reduction in time-to-first SLO summary document, with no Kibana orchestration changes. The summary document landing at checkpoint #2 confirms the new behavior: checkpoint #1 was empty (the rollup had not written its SLI docs yet), so initial_delay stayed in effect for checkpoint #2 instead of reverting to the 65s steady-state delay.

Test plan

• StartTransformActionRequestTests / TransformTaskParamsTests — wire + XContent BWC for the new initial_delay field.
• TimeBasedCheckpointProviderTests — initial_delay kept until first non-empty checkpoint, switch to steady state once data processed, gate uses initial_delay during catch-up, clamping.
• RestStartTransformActionTests — initial_delay param parsing.
• TransformInitialDelayIT — end-to-end: a pre-present backfill is processed on the first checkpoint after _start?initial_delay=0s despite a long steady-state delay.
• transforms_start_stop.yml — _start with initial_delay, rejection for batch transforms and when initial_delay > delay.
• CI green.

Made with Cursor

[github-actions]

🔍 Preview links for changed docs

This PR only changes snippets or data files. No direct page links are available.
Browse the full preview

[github-actions]

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

• Check out the cumulative docs guidelines
• Reach out in the #docs Slack channel

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[prwhelan]

Please remove all of the benchmark files