sched: make GrpTRES post-select check preemption-aware#210
Draft
Yash0270 wants to merge 3 commits into
Draft
Conversation
When a pending job is blocked by a parent association's GrpTRES limit, the scheduler now retries the check with usage reduced by the TRES that preempting the identified victims would free. This enables cross-account QoS preemption within hierarchical account trees. The fix adds a preemptee_job_list parameter to acct_policy_job_runnable_post_select(). On the normal path (no preemptees, or GrpTRES passes), zero extra work is done. Only on GrpTRES failure with preemptees present, a single retry is attempted with adjusted usage — no function duplication, no extra allocations on the happy path. Bug: 23492
Yash0270
force-pushed
the
preempt-aware-grptres-check
branch
from
17ca9bb to
48b7086
Compare
ATF test (test_128_3.py) and Docker E2E test (test-atf-preempt.sh) for Bug 23492: QoS preemption across accounts sharing a parent GrpTRES. Tests cover: - Pure cross-account preemption (proj1 idle, parent full from siblings) - Multi-CPU preemption requiring multiple victims - Array job preemption across accounts - Reverse direction (low cannot preempt high) - Three-level hierarchy (org → pillar → projects) - Baseline (no contention) - Known edge case: leaf+parent both at limit (informational)
The preemption-aware GrpTRES post-select check (Bug 23492) credits the TRES that selected victims would free at each association level. But victim selection in slurm_find_preemptable_jobs() ordered equal-priority candidates arbitrarily, so when the preemptor's own leaf (or an intermediate) association was ALSO at its GrpTRES limit, the scheduler could pick a sibling-account victim that only relieved a shared ancestor. The leaf limit stayed violated and the job was left PENDING with AssocGrpCpuLimit, nondeterministically depending on candidate order. Break equal preempt-priority ties by association affinity: prefer the candidate closest (in the association tree) to the preemptor, so its own lower-priority jobs are preempted first, relieving the leaf and every ancestor together. This only reorders candidates the priority scheme already treats as equivalent, so it never overrides preempt-priority ordering. Add test_128_3.py::test_preemption_prefers_own_account for the leaf+parent-both-at-limit case, and a Dockerfile.base recipe for the slurm-preempt image the ATF Docker harness builds on. Bug: 23492
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NOT READY FOR REVIEW
When a pending job's QoS is configured to preempt running jobs (via PreemptType=preempt/qos), but the job is blocked by a parent association's GrpTRES limit, the scheduler now considers whether preempting the identified victims would free enough TRES to satisfy the association hierarchy limits.
Previously, acct_policy_job_runnable_post_select() would reject the job with ESLURM_ACCOUNTING_POLICY before preemption was executed, even when the preemptee_job_list contained jobs whose TRES would resolve the GrpTRES violation. This made cross-account QoS preemption within hierarchical account trees impossible.
The fix adds acct_policy_job_runnable_post_select_preemptees() which:
In select_nodes(), when the standard post_select check fails and a preemptee_job_list exists, the new preemption-aware function is called as a fallback before rejecting the job.
Bug: 23492