Fix multi-QOS per-user/account GRES usage recharged to wrong QOS on reload

[Yash0270]

Problem

A job can be submitted with more than one QOS (--qos=a,b). It runs under whichever member the scheduler picked, and its GRES/TRES usage is accounted to that member (saved in job_ptr->qos_id).

When the controller recovers job state from StateSaveLocation (a slurmctld restart or a backup takeover) it re-reads each running job and resets its QOS to the head of the QOS list instead of the member it actually started under, then recharges its per-user/per-account usage there. The list head is the first --qos member after sorting by priority; when the listed QOS share the same priority, the head is simply the first one listed.

So every running multi-QOS job a user has is re-attributed to that single head QOS, regardless of which member it really ran under. The head QOS's usage counter then grows past what is physically running, and new jobs are blocked with QOSMaxGRESPerUser. In production the head QOS's per-user counter showed ~976 GPUs in use while the user was actually running only ~300 (the two QOS had equal priority, so the head was just the first one in the job's --qos list).

Where it goes wrong: job_mgr_load_job_state() resolves the QOS via _get_qos_info(), which returns list_peek() (the list head) and overwrites job_ptr->qos_id with it; _restore_job_accounting() then recharges usage against that member.

The in-place scontrol reconfigure path is affected in Slurm 25.05 version but it got fixed in the future version, 25.11.

Fix

At state recovery, keep a running/suspended multi-QOS job on the member it was dispatched under (job_ptr->qos_id); only pending jobs fall back to the list head. This mirrors the pending-vs-running handling already in _foreach_cache_update_job(), and makes accounting reflect the QOS the job is physically running under.

That is the whole fix: one change in job_mgr_load_job_state(). The qos_ptr restore already present in _adjust_limit_usage() keeps acct_policy_add_job_submit() from leaving the job on the head member before the subsequent JOB_BEGIN, so nothing else is needed.

(Note for backports: on 25.05 that restore in _adjust_limit_usage() runs only for ACCT_POLICY_REM_SUBMIT, so the ADD_SUBMIT highest-QOS bump is left in place and this job_mgr.c change alone is not sufficient there; a 25.05 backport also needs _adjust_limit_usage() to skip _set_highest_prio_qos_ptr() for running jobs.)

Target Release

Targets 26.05.2 Slurm release. This is a minor patch to an existing bug.

Issue

Reported in SchedMD support ticket: https://support.schedmd.com/show_bug.cgi?id=23033

Testing

Regression testing: added an ATF test (testsuite/python/tests/test_124_2.py) with two cases:

• test_multiqos_gres_usage_survives_reconfigure guards the in-place cache-rebuild path.
• test_multiqos_gres_usage_survives_restart exercises the state-recovery path this fix targets; it fails on unpatched 26.05 and passes with the fix.

Functional testing: verified end-to-end on a from-source 26.05 cluster, comparing a plain-26.05 build against the patched build and triggering a slurmctld restart:

sbatch --qos=high     --gres=gpu:2 --wrap "sleep 900"   # fills high's per-user cap (MaxTRESPerUser=gres/gpu=2)
sbatch --qos=low,high --gres=gpu:2 --wrap "sleep 900"   # forced onto non-head 'low'
scontrol -o show assoc_mgr flags=qos     # per-user gres/gpu used
# restart slurmctld (recovers job state from StateSaveLocation)
scontrol -o show assoc_mgr flags=qos     # re-check

per-user gres/gpu	before fix	after fix
pre-restart	low=2 high=2	low=2 high=2
post-restart	low=0 high=4 (job re-attributed to the head)	low=2 high=2 (stays on low)

(For reference, scontrol reconfigure was also tested and is unaffected on 26.05, before and after the fix.)