Add per-process Spot to on-demand fallback hint for Google Batch - #7470
Open
fraser-combe wants to merge 1 commit into
Open
Add per-process Spot to on-demand fallback hint for Google Batch#7470fraser-combe wants to merge 1 commit into
fraser-combe wants to merge 1 commit into
Conversation
Add a `scheduling.spotAttempts` hint (a positive integer N) on the Google Batch executor: a process runs on Spot for its first N attempts and then falls back to on-demand (STANDARD) for later attempts. The attempt count is the greater of the task's execution attempt and submit attempt, so the fallback triggers both on a mid-run Spot reclaim (`task.attempt`) and on a failure to obtain a Spot VM (`task.submitAttempt`, together with `maxSubmitAwait`). This complements the existing `google.batch.maxSpotAttempts`, which sets the number of Batch-internal retries after a Spot reclaim but always re-requests a Spot VM, and applies to the whole run. The hint instead changes the provisioning model across Nextflow-level attempts, and is set per process. The docs call out the distinction. Provisioning is resolved once in resolveProvisioningModel() and used by both the instance policy and machine-type pricing; when the hint is unset the global `google.batch.spot` / `preemptible` behaviour is unchanged. Non-positive or non-integer values are rejected. Under an instance template the hint has no effect and a warning is logged. Unknown `google-batch/`-prefixed hints are rejected, per the hints ADR. Includes unit tests and docs. See nextflow-io#7396 Signed-off-by: Fraser Combe <fraser.combe@tempus.com>
✅ Deploy Preview for nextflow-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the gap that leaves Google Batch unable to express something both other
provisioning-aware executors already support: running a process on Spot and
automatically falling back to on-demand when Spot cannot be obtained or is reclaimed.
Motivation
Attempt-based Spot to on-demand fallback is expressible on AWS Batch today, and on the
Seqera executor as of #7412. On Google Batch it cannot be expressed at all:
queueclosure overtask.attempt, routing to a Spot vs on-demand compute environmentmachineRequirement.provisioning(spotFirst)plusmaxSpotAttempts, per process via hints; resolved server-side by the Platform schedulergoogle.batch.spot, a single global booleangoogle.batch.spotis a config option rather than a process directive, so it cannot be aclosure, cannot vary per process, and cannot vary per attempt. There is no workaround: Cloud
Batch has no queue abstraction to route through, and
google.batch.maxSpotAttemptsonlyretries on Spot (see below).
This matters most for the case that motivates the feature:
ZONE_RESOURCE_POOL_EXHAUSTEDwhile well under quota. A task that cannot obtain a Spot VM currently retries onto Spot
forever, or the run fails; there is no way to say "try Spot twice, then just pay for it."
The docs already recommend exactly this strategy without offering a mechanism. From
docs/guides/updating-spot-retries.mdx, Best Practices: "Consider partial usage of Spot:some workflows may mix on-demand instances for critical or long tasks and Spot Instances for
shorter, less critical tasks."
What this adds
A
scheduling.spotAttemptshint on the Google Batch executor, taking a positive integer N:attempts
1..Nrequest a Spot VM, attempts afterNrequest on-demand (STANDARD).The attempt count is
max(task.attempt, task.submitAttempt), so the fallback is driven byboth failure modes:
task.attemptcovers a mid-run Spot reclaimtask.submitAttemptcovers a failure to obtain a Spot VM, in combination withmaxSubmitAwaitThe second is the important one, and the reason a plain attempt counter is not enough: a
stockout is a submission-side failure, not an execution-side one.
Relationship to
google.batch.maxSpotAttemptsSimilar name, different mechanism. This is called out explicitly in the docs:
google.batch.maxSpotAttemptssets Cloud Batch's internal retry count after a Spotreclaim. Every retry requests Spot again. It is global to the run.
scheduling.spotAttemptschanges the provisioning model across Nextflow-level attempts,per process.
They compose: Batch-internal Spot retries first, then Nextflow-level escalation to on-demand.
Relationship to #7343
This composes with #7343 (@reece-maticebio) rather than duplicating it. That PR sets
provisioning statically per process and explicitly scoped
spotFirst-style fallback out;this adds the attempt-aware dimension. If #7343 lands first, this can be rebased to read
its resolved provisioning model as the starting rung instead of
google.batch.spot.Note that the Seqera executor resolves this in the Platform scheduler rather than in Nextflow, so there is no existing in-tree implementation of attempt-based provisioning escalation to build on.
One longer-term unification worth discussing: the shared Nextflow idiom here is a closure over
task.attempt, which is how AWS Batch users already do this viaqueue. Making #7343'sscheduling.provisioningModelaccept a dynamic value would subsume both, with an attemptcount as ergonomic shorthand. Out of scope here, happy to follow up.
Implementation notes
resolveProvisioningModel()and reused by both theinstance policy and machine-type pricing, so
PriceModelstays consistent with what isactually requested.
google.batch.spotandpreemptiblebehaviour is unchanged.spotandpreemptibleoptions behave there.google-batch/-prefixed hints are rejected, per the hints ADR.google-batch/-prefixed forms are accepted; prefixed wins.Tests
GoogleBatchTaskHandlerTest, 128 tests pass. New coverage: the attempt/submitAttemptescalation matrix, both hint forms, numeric and string values, invalid values, the unchanged
global-config fallback path, and prefixed-hint validation.
Note for reviewers
The
google-batch/hint validation scaffolding (HINT_PREFIX,KNOWN_HINTS,validateHints()) also appears in #7395. Theoverlap is about 30 lines and mechanical.
Closes #7396