sched/backfill: keep hetjob preemption plans sticky

[vwxyzjn]

This PR adds commit-like behavior for heterogeneous job starts when backfill uses QOS preemption. It is based on the production patch stack carried in vwxyzjn/slurm-containers#3.

Problem

When a heterogeneous job is scheduled by backfill and needs QOS preemption, the scheduler can lose the concrete plan between the planning and immediate start paths. In particular:

• backfill may produce a valid SchedNodeList for each hetjob component;
• _het_job_start_now() may retry against a broader or freshly computed node set;
• a component may return ESLURM_NODES_BUSY while preempted jobs are still cleaning up;
• if no runtime preemptee_job_list was produced yet, the start attempt can roll back and replan instead of committing to the planned victims.

That can repeatedly forecast a workable hetjob plan without actually following through on the preemption needed to make the plan available.

Changes

• Add bf_hetjob_sticky_preempt_timeout, defaulting to 1800 seconds.
• Preserve each hetjob component's planned node bitmap from backfill.
• Pin immediate hetjob start attempts to the planned component bitmap.
• Preserve planned QOS-preemptible victim job IDs overlapping the planned bitmap.
• If the pinned immediate start returns ESLURM_NODES_BUSY before runtime victims exist, revalidate and preempt the planned victims, then wait on the same sticky preemption path.
• Document the new scheduler parameter in slurm.conf(5) and sched_config.shtml.

Validation

• Cherry-picked cleanly onto current origin/slurm-25.05.
• Ran git diff --check origin/slurm-25.05..HEAD.

Full Slurm integration tests were not run locally.

[vwxyzjn]

This is a human’s note: basically we find het job preemption doesn’t work properly — normal qos jobs across partitions cannot reliably kill preemptible QoS jobs across partitions, despite available preemptible capacity.

We see degenerate behaviors — it preemptied some nodes in one partition, went on to preempt some nodes in another partition, realized the plan isn’t feasible and replans. A job would wait hours to tens of hours to start, and cause unnecessary preemptions.

The patch here is AI generated, but the testing is real. It is tried and tested in a production cluster, and it works very reliably and follow expectations.

[wickberg]

Thanks for the submission, we're looking into this a bit further internally still.

If you don't mind, we will need a Signed-off-by trailer on these commits to acknowledge the DCO [1] if this were to be merged upstream.

• Tim

[1] https://github.com/SchedMD/slurm/blob/master/CONTRIBUTING.md#developer-certificate-of-origin