slurmctld - Restore qos_ptr after multi-QOS job-queue build#208
Closed
Yash0270 wants to merge 1 commit into
Closed
slurmctld - Restore qos_ptr after multi-QOS job-queue build#208Yash0270 wants to merge 1 commit into
Yash0270 wants to merge 1 commit into
Conversation
_build_job_queue_for_qos() sets job_ptr->qos_ptr to each qos_list member while testing a pending multi-QOS job and never restores it, leaving the live pointer on the last (lowest priority) member after every scheduling cycle. The job_queue entries already captured their own qos_ptr, so reset the job's pointer to the highest-priority member (the pending default) instead. Both acct_policy and the priority/multifactor decay key usage accounting on job_ptr->qos_ptr, so a stale pointer can charge/uncharge a multi-QOS job's TRES against the wrong QOS member. Changelog: slurmctld - Fix multi-qos qos_ptr left pointing at the wrong QOS member after job queue build. Signed-off-by: Yash Gupta <guptayash0270@gmail.com>
Author
|
Closing as redundant. On review, the stale |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A job submitted with multiple QOS (
--qos=a,b) is evaluated against eachqos_listmember during scheduling:_build_job_queue_for_qos()(src/slurmctld/job_scheduler.c) setsjob_ptr->qos_ptrto each member to test runnability, but never restores it. After the loop,job_ptr->qos_ptris left pointing at the last-iterated (lowest-priority) member, on every scheduling cycle.Both per-user/per-account QOS TRES accounting (
acct_policy) and thepriority/multifactortres_run_secsdecay key usage onjob_ptr->qos_ptr. Leaving it on a stale member can cause a multi-QOS job's TRES (e.g.gres/gpu) to be charged/uncharged against the wrong QOS member.Fix
After the per-member test loop, reset
job_ptr->qos_ptrto the highest-priority member (the pending default), mirroring how pending multi-QOS jobs are resolved elsewhere (_set_highest_prio_qos_ptr()/_foreach_cache_update_job()). Thejob_queueentries already captured their ownqos_ptrat append time (job_queue_rec->qos_ptr), so dispatch is unaffected; only the job's transient live pointer is reset.Testing
Built
slurmctldfrom source and ran a multi-QOS workload on a two-node from-source cluster (low= per-user GRES cap,high= GrpTRES group cap). A multi-QOS job array distributed across both members stays correctly attributed per member across scheduling cycles andscontrol reconfigure(4 high / 4 low preserved), whereas without the fix the jobs collapse onto the highest-priority member.Target Release
26.05(bug fix).Related
Companion multi-QOS
qos_ptr-consistency fix for the state-reload path: #207. Same class of bug (a running multi-QOS job'sqos_ptrleft on the wrongqos_listmember, mis-charging accounting), different code path.