Make slurmctld periodic check interval configurable

[dkhokhlov]

Summary

Add SlurmctldParameters=periodic_check_interval=# to configure how often
slurmctld runs its periodic background check pass.

The default remains the existing PERIODIC_TIMEOUT value of 30 seconds. Sites
can lower the interval, for example to 2 seconds, to reduce delay before jobs
progress after powered-down nodes resume and register, such as when reducing
resume latency for suspended nodes.

Affects:

• slurm-smd-slurmctld

Details

The interval controls the background block that runs job time-limit handling,
reservation checks, and node timer checks. The same configured value is also
used for related job warning/time-limit timing checks so behavior stays
consistent.

Invalid values, 0, and INFINITE are rejected and fall back to the default.

Documentation for the new parameter was added to slurm.conf(5) and the power
save documentation.

A targeted ATF integration test was added to verify that lowering
periodic_check_interval allows a job waiting in CONFIGURING to advance
promptly after the node registers.

Validation

• git diff --check
• python -m py_compile testsuite/python/tests/test_141_1.py
• python -m pytest testsuite/python/tests/test_141_1.py -k periodic_check_interval --collect-only -q

Full execution of the new ATF test is expected to run in GitHub CI / isolated
test infrastructure, since Slurm ATF --auto-config manages the Slurm
environment and is not appropriate to run against a live local Slurm setup.

[dkhokhlov]

Submitted via the official channel per CONTRIBUTING.md (SchedMD does
not accept GitHub PRs). Closing here; tracking upstream at:

https://support.schedmd.com/show_bug.cgi?id=25294

[wickberg]

We do accept PRs now, although older versions of CONTRIBUTING.md still said we did not.

I've pushed a change to update the version on the 25.05/25.11 branches to help avoid confusion.

[dkhokhlov]

Debian downstream: bugs.debian.org/1138083

[mcmult]

Can you clarify what the effect you are trying to achieve here? This appears to hit a lot of different loop timers that aren't directly related to the issue in the summary and I want to make sure this is behaving as expected.

Thanks!

[dkhokhlov]

Can you clarify what the effect you are trying to achieve here? This appears to hit a lot of different loop timers that aren't directly related to the issue in the summary and I want to make sure this is behaving as expected.

Thanks!

The goal is to reduce the extra delay before jobs start after nodes in power save mode are resumed and register.

This does not change the underlying timeout values. It only changes how often slurmctld evaluates the existing PERIODIC_TIMEOUT polling block.

Today that poll is hardcoded at 30s and runs this existing group:

• job_time_limit()
• job_resv_check()
• check_node_timers()

The relevant path is in job_time_limit(): despite the name, it also rechecks CONFIGURING jobs and advances them when test_job_nodes_ready() succeeds:

test_job_nodes_ready() -> job_config_fini() -> launch_job()

So after ResumeProgram brings a node back from power save mode and slurmd registers, the job can still wait until the next hardcoded 30s PERIODIC_TIMEOUT poll before launching. Making that poll interval configurable lowers that average extra wait.