Skip to content

slurmctld - Restore qos_ptr after multi-QOS job-queue build#208

Closed
Yash0270 wants to merge 1 commit into
SchedMD:slurm-26.05from
Yash0270:fix/multiqos-buildqueue-qosptr
Closed

slurmctld - Restore qos_ptr after multi-QOS job-queue build#208
Yash0270 wants to merge 1 commit into
SchedMD:slurm-26.05from
Yash0270:fix/multiqos-buildqueue-qosptr

Conversation

@Yash0270

Copy link
Copy Markdown

Problem

A job submitted with multiple QOS (--qos=a,b) is evaluated against each qos_list member during scheduling: _build_job_queue_for_qos() (src/slurmctld/job_scheduler.c) sets job_ptr->qos_ptr to each member to test runnability, but never restores it. After the loop, job_ptr->qos_ptr is left pointing at the last-iterated (lowest-priority) member, on every scheduling cycle.

Both per-user/per-account QOS TRES accounting (acct_policy) and the priority/multifactor tres_run_secs decay key usage on job_ptr->qos_ptr. Leaving it on a stale member can cause a multi-QOS job's TRES (e.g. gres/gpu) to be charged/uncharged against the wrong QOS member.

Fix

After the per-member test loop, reset job_ptr->qos_ptr to the highest-priority member (the pending default), mirroring how pending multi-QOS jobs are resolved elsewhere (_set_highest_prio_qos_ptr() / _foreach_cache_update_job()). The job_queue entries already captured their own qos_ptr at append time (job_queue_rec->qos_ptr), so dispatch is unaffected; only the job's transient live pointer is reset.

Testing

Built slurmctld from source and ran a multi-QOS workload on a two-node from-source cluster (low = per-user GRES cap, high = GrpTRES group cap). A multi-QOS job array distributed across both members stays correctly attributed per member across scheduling cycles and scontrol reconfigure (4 high / 4 low preserved), whereas without the fix the jobs collapse onto the highest-priority member.

Target Release

26.05 (bug fix).

Related

Companion multi-QOS qos_ptr-consistency fix for the state-reload path: #207. Same class of bug (a running multi-QOS job's qos_ptr left on the wrong qos_list member, mis-charging accounting), different code path.

_build_job_queue_for_qos() sets job_ptr->qos_ptr to each qos_list member
while testing a pending multi-QOS job and never restores it, leaving the
live pointer on the last (lowest priority) member after every scheduling
cycle. The job_queue entries already captured their own qos_ptr, so reset
the job's pointer to the highest-priority member (the pending default)
instead. Both acct_policy and the priority/multifactor decay key usage
accounting on job_ptr->qos_ptr, so a stale pointer can charge/uncharge a
multi-QOS job's TRES against the wrong QOS member.

Changelog: slurmctld - Fix multi-qos qos_ptr left pointing at the wrong QOS member after job queue build.
Signed-off-by: Yash Gupta <guptayash0270@gmail.com>
@Yash0270

Copy link
Copy Markdown
Author

Closing as redundant. On review, the stale job_ptr->qos_ptr left after the queue-build loop is only ever parked on a pending job, and it is always NULLed or re-resolved before the QOS record it points to is freed: both the full assoc_mgr refresh (_foreach_cache_update_job) and incremental QOS removal (job_hold_by_qos_id -> job_fail_qos) do this under job_write_lock, which excludes the scheduler. So it prevents no use-after-free and no observable misaccounting. The genuine multi-QOS reload misattribution fix is tracked in #207.

@Yash0270 Yash0270 closed this Jun 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant