Skip to content

sched/backfill: keep hetjob preemption plans sticky #203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vwxyzjn wants to merge 6 commits into
base: slurm-25.05
Choose a base branch
from
+337 −14
Open

sched/backfill: keep hetjob preemption plans sticky #203

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
fb2751b
Add sticky hetjob preemption timeout
May 22, 2026
520fff0
Default sticky hetjob preemption timeout to 30 minutes
May 22, 2026
29617dd
Handle first-component hetjob preemption waits
May 27, 2026
26b295b
Pin hetjob starts to planned backfill nodes
May 29, 2026
5decdca
Preempt planned hetjob victims
May 29, 2026
0172697
Docs - document hetjob sticky preempt timeout
Jun 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions doc/html/sched_config.shtml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,10 @@ see the <a href="slurm.conf.html">slurm.conf(5)</a> man page.
<UL>
<LI><B>bf_continue</B> - If set, then continue backfill scheduling after
periodically releasing locks for other operations.</LI>
<LI><B>bf_hetjob_sticky_preempt_timeout=#</B> - Number of seconds the backfill
scheduler will keep retrying the same heterogeneous job start attempt after
preemption has started for one of its components. Default value is 1800 seconds.
Set to 0 to disable.</LI>
<LI><B>bf_interval=#</B> - Interval between backfill scheduling attempts.
Default value is 30 seconds.</LI>
<LI><B>bf_max_job_part=#</B> - Maximum number of jobs to initiate per partition
Expand Down
11 changes: 11 additions & 0 deletions doc/man/man5/slurm.conf.5
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3969,6 +3969,17 @@ option is disabled by default. If enabled and \fBbf_hetjob_prio=min\fR is not
set, then it would be automatically set.
.IP

.TP
\fBbf_hetjob_sticky_preempt_timeout=#\fR
Number of seconds the backfill scheduler will keep retrying the same
heterogeneous job start attempt after preemption has started for one of its
components. During this window, the scheduler waits for preempted jobs on the
planned nodes to finish cleanup instead of immediately rolling back the
heterogeneous job start and replanning against a different node set.
Set to 0 to disable this sticky preemption wait.
The default value is 1800 seconds.
.IP

.TP
\fBbf_hetjob_prio=[min|avg|max]\fR
At the beginning of each backfill scheduling cycle, a list of pending to be
Expand Down
Loading