Skip to content

Add cause-aware batch job requeue limits#209

Open
Yash0270 wants to merge 1 commit into
SchedMD:masterfrom
Yash0270:feature/cause-aware-requeue
Open

Add cause-aware batch job requeue limits#209
Yash0270 wants to merge 1 commit into
SchedMD:masterfrom
Yash0270:feature/cause-aware-requeue

Conversation

@Yash0270

@Yash0270 Yash0270 commented Jun 29, 2026

Copy link
Copy Markdown

Summary

Add cause-aware requeue limits for batch jobs so node failures and preemption/operator requeues can be bounded independently from MaxBatchRequeue.

  • MaxNodeFailRequeue defaults to 5 and bounds node-failure requeues.
  • MaxPreemptRequeue defaults to 0 (unlimited) and bounds preemption/operator requeues.
  • MaxBatchRequeue keeps its existing launch/prolog failure behavior.
  • Jobs that exceed a configured cause limit are held with JobHeldMaxNodeFailRequeue or JobHeldMaxPreemptRequeue; scontrol release resets the counters.

This change was reviewed internally at Meta in Yash0270/slurm#4. We had originally planned to send a direct patch, but given the recent changes to CONTRIBUTING.md, we are opening it as a pull request here instead.

Target Release

Targets master / the next stable Slurm release because this adds slurm.conf parameters plus protocol/state-file guarded job fields.

Issue

Reported in SchedMD support ticket: https://support.schedmd.com/show_bug.cgi?id=24298

Implementation Notes

  • Adds per-job node-failure and preemption requeue counters.
  • Keeps the counters controller-internal and packs them only under the 26.11 protocol/state guard.
  • Updates config parsing/reporting, data parser coverage, Perl config bindings, reason-code docs, slurm.conf(5), and CHANGELOG/slurm-26.11.md.
  • Leaves generated build-system files untouched.

Memory Impact

Two uint32_t counters per job = 8 bytes per job (~0.74% of the ~1,074-byte job_record struct).

Cluster size Existing job_record memory Our addition
1,000 jobs ~1 MB 8 KB
10,000 jobs ~10 MB 78 KB
100,000 jobs ~102 MB 781 KB
1,000,000 jobs ~1 GB 7.6 MB

The counters are controller-internal: packed under the for_state guard (state-save files only, not in scontrol show job / squeue RPC responses), so there is no network/protocol overhead.

Testing

  • Whitespace/sanity: git diff --check origin/master...HEAD.
  • Added ATF coverage in testsuite/python/tests/test_105_11.py for node-failure holds, preemption/operator holds, user requeue and requeuehold exemptions, counter independence, scontrol release counter reset, and MaxPreemptRequeue=0 unlimited mode.
  • Functional testing: a Docker-based two-node e2e harness (Yash0270/slurm#5) exercised the same paths against a live cluster with real slurmctld / slurmd daemons; result: 9/9 PASS.

Contribution Notes

  • Single logical commit with Signed-off-by and Changelog: trailers per CONTRIBUTING.md.
  • No changes to generated build-system files (Makefile.in, configure).

@Yash0270 Yash0270 marked this pull request as ready for review June 29, 2026 17:44
@Yash0270 Yash0270 marked this pull request as draft July 2, 2026 04:10
Add MaxNodeFailRequeue and MaxPreemptRequeue to bound node-failure and
preemption/operator requeues independently from MaxBatchRequeue.

Each cause keeps its own per-job counter. Jobs that exceed the configured
limit are held with JobHeldMaxNodeFailRequeue or JobHeldMaxPreemptRequeue,
while a limit of 0 disables the hold. scontrol release resets the counters
so a released job gets a fresh allowance.

The new config fields and job counters are packed behind the 26.11
protocol/state version guard. The change also updates config reporting,
data parser coverage, Perl config bindings, reason-code docs, slurm.conf(5),
the changelog, and ATF tests for the new limits and exemption cases.

Changelog: Add MaxNodeFailRequeue and MaxPreemptRequeue to bound requeues by cause.
Signed-off-by: Yash Gupta <guptayash0270@gmail.com>
@Yash0270 Yash0270 force-pushed the feature/cause-aware-requeue branch from 4e85989 to d3d8e8b Compare July 2, 2026 19:38
@Yash0270 Yash0270 marked this pull request as ready for review July 2, 2026 20:14
@Yash0270

Yash0270 commented Jul 2, 2026

Copy link
Copy Markdown
Author

testsuite/python/tests/test_105_11.py merge conflict

It's due to new code landed in master. Once PR is reviewed, we can rename the test to the latest test number available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant