Skip to content

Add per-process Spot to on-demand fallback hint for Google Batch - #7470

Open
fraser-combe wants to merge 1 commit into
nextflow-io:masterfrom
fraser-combe:feat/google-batch-spot-attempts
Open

Add per-process Spot to on-demand fallback hint for Google Batch#7470
fraser-combe wants to merge 1 commit into
nextflow-io:masterfrom
fraser-combe:feat/google-batch-spot-attempts

Conversation

@fraser-combe

Copy link
Copy Markdown

Closes the gap that leaves Google Batch unable to express something both other
provisioning-aware executors already support: running a process on Spot and
automatically falling back to on-demand when Spot cannot be obtained or is reclaimed.

Motivation

Attempt-based Spot to on-demand fallback is expressible on AWS Batch today, and on the
Seqera executor as of #7412. On Google Batch it cannot be expressed at all:

Executor Per-process Spot to on-demand? Mechanism
AWS Batch Yes, today dynamic queue closure over task.attempt, routing to a Spot vs on-demand compute environment
Seqera Yes machineRequirement.provisioning (spotFirst)plus maxSpotAttempts, per process via hints; resolved server-side by the Platform scheduler
Google Batch No google.batch.spot, a single global boolean

google.batch.spot is a config option rather than a process directive, so it cannot be a
closure, cannot vary per process, and cannot vary per attempt. There is no workaround: Cloud
Batch has no queue abstraction to route through, and google.batch.maxSpotAttempts only
retries on Spot (see below).

This matters most for the case that motivates the feature: ZONE_RESOURCE_POOL_EXHAUSTED
while well under quota. A task that cannot obtain a Spot VM currently retries onto Spot
forever, or the run fails; there is no way to say "try Spot twice, then just pay for it."

The docs already recommend exactly this strategy without offering a mechanism. From
docs/guides/updating-spot-retries.mdx, Best Practices: "Consider partial usage of Spot:
some workflows may mix on-demand instances for critical or long tasks and Spot Instances for
shorter, less critical tasks."

What this adds

A scheduling.spotAttempts hint on the Google Batch executor, taking a positive integer N:
attempts 1..N request a Spot VM, attempts after N request on-demand (STANDARD).

process EXAMPLE {
    errorStrategy 'retry'
    maxRetries 3
    hints 'scheduling.spotAttempts': '2'   // Spot for attempts 1 and 2, on-demand from attempt 3
}

The attempt count is max(task.attempt, task.submitAttempt), so the fallback is driven by
both failure modes:

  • task.attempt covers a mid-run Spot reclaim
  • task.submitAttempt covers a failure to obtain a Spot VM, in combination with maxSubmitAwait

The second is the important one, and the reason a plain attempt counter is not enough: a
stockout is a submission-side failure, not an execution-side one.

Relationship to google.batch.maxSpotAttempts

Similar name, different mechanism. This is called out explicitly in the docs:

  • google.batch.maxSpotAttempts sets Cloud Batch's internal retry count after a Spot
    reclaim. Every retry requests Spot again. It is global to the run.
  • scheduling.spotAttempts changes the provisioning model across Nextflow-level attempts,
    per process.

They compose: Batch-internal Spot retries first, then Nextflow-level escalation to on-demand.

Relationship to #7343

This composes with #7343 (@reece-maticebio) rather than duplicating it. That PR sets
provisioning statically per process and explicitly scoped spotFirst-style fallback out;
this adds the attempt-aware dimension. If #7343 lands first, this can be rebased to read
its resolved provisioning model as the starting rung instead of google.batch.spot.

Note that the Seqera executor resolves this in the Platform scheduler rather than in Nextflow, so there is no existing in-tree implementation of attempt-based provisioning escalation to build on.

One longer-term unification worth discussing: the shared Nextflow idiom here is a closure over
task.attempt, which is how AWS Batch users already do this via queue. Making #7343's
scheduling.provisioningModel accept a dynamic value would subsume both, with an attempt
count as ergonomic shorthand. Out of scope here, happy to follow up.

Implementation notes

  • Provisioning is resolved once in resolveProvisioningModel() and reused by both the
    instance policy and machine-type pricing, so PriceModel stays consistent with what is
    actually requested.
  • When the hint is unset, google.batch.spot and preemptible behaviour is unchanged.
  • Non-positive and non-integer values are rejected with a clear message.
  • Under an instance template the hint has no effect and warns, matching how the existing
    spot and preemptible options behave there.
  • Unknown google-batch/-prefixed hints are rejected, per the hints ADR.
  • Both the bare and google-batch/-prefixed forms are accepted; prefixed wins.

Tests

GoogleBatchTaskHandlerTest, 128 tests pass. New coverage: the attempt/submitAttempt
escalation matrix, both hint forms, numeric and string values, invalid values, the unchanged
global-config fallback path, and prefixed-hint validation.

Note for reviewers

The google-batch/ hint validation scaffolding (HINT_PREFIX, KNOWN_HINTS,
validateHints()) also appears in #7395. The
overlap is about 30 lines and mechanical.

Closes #7396

Add a `scheduling.spotAttempts` hint (a positive integer N) on the Google
Batch executor: a process runs on Spot for its first N attempts and then
falls back to on-demand (STANDARD) for later attempts. The attempt count
is the greater of the task's execution attempt and submit attempt, so the
fallback triggers both on a mid-run Spot reclaim (`task.attempt`) and on a
failure to obtain a Spot VM (`task.submitAttempt`, together with
`maxSubmitAwait`).

This complements the existing `google.batch.maxSpotAttempts`, which sets
the number of Batch-internal retries after a Spot reclaim but always
re-requests a Spot VM, and applies to the whole run. The hint instead
changes the provisioning model across Nextflow-level attempts, and is set
per process. The docs call out the distinction.

Provisioning is resolved once in resolveProvisioningModel() and used by
both the instance policy and machine-type pricing; when the hint is unset
the global `google.batch.spot` / `preemptible` behaviour is unchanged.
Non-positive or non-integer values are rejected. Under an instance
template the hint has no effect and a warning is logged. Unknown
`google-batch/`-prefixed hints are rejected, per the hints ADR.

Includes unit tests and docs.

See nextflow-io#7396

Signed-off-by: Fraser Combe <fraser.combe@tempus.com>
@fraser-combe
fraser-combe requested a review from a team as a code owner August 11, 2026 16:42
@netlify

netlify Bot commented Aug 11, 2026

Copy link
Copy Markdown

Deploy Preview for nextflow-docs ready!

Name Link
🔨 Latest commit bbcd6cc
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs/deploys/6a7b50f2523ae600085c17c3
😎 Deploy Preview https://deploy-preview-7470--nextflow-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: per-process Spot → on-demand fallback for Google Batch

1 participant