Skip to content

Add jitter to resource lock requeue delay to prevent starvation - #274

Merged
felix-kaestner merged 2 commits into
mainfrom
requeue
Apr 2, 2026
Merged

felix-kaestner merged 2 commits into
mainfrom
requeue

Conversation

@felix-kaestner

@felix-kaestner felix-kaestner commented Apr 2, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Add ±10% jitter to the RequeueAfter delay when a controller fails to acquire the per-device resource lock, preventing lock convoy starvation across all 25+ controllers

Problem

All controllers targeting the same device use a Lease-based resource lock to serialize
reconciliations. When the lock is already held, every controller requeues after exactly
time.Second. This creates a lock convoy: all blocked controllers wake up simultaneously,
compete for the lock in deterministic work-queue order, and the same controllers can
repeatedly win — starving others indefinitely.

Solution

Wrap the requeue delay with the existing Jitter() helper (±10%, range 900ms–1100ms).
This breaks the synchronized thundering herd by spreading wake-up times, giving each
controller a statistically fair chance to acquire the lock over time.

This is the same pattern used in Kubernetes leader election, client-go retry backoffs,
and other distributed systems to prevent convoy/starvation effects.

@felix-kaestner
felix-kaestner requested a review from a team April 2, 2026 07:59
@felix-kaestner
felix-kaestner enabled auto-merge (rebase) April 2, 2026 07:59
In 7fafc92
we adjusted the resource lock requeue dealy, however we forgot to also
adjust the default settings of the locker duration and renew interval
which we now adjust as well.
When multiple controllers targeting the same device fail to acquire the
per-device resource lock, they all requeue after exactly 1 second. This
creates a lock convoy: all controllers wake up simultaneously and
compete for the lock in a deterministic order, which can cause some
controllers to be perpetually starved.

By adding ±10% jitter (900ms–1100ms) to the requeue delay, controllers
wake up at slightly different times, breaking the synchronized
thundering herd and giving each controller a fair chance to acquire the
lock over time.
@felix-kaestner
felix-kaestner merged commit aefe3a6 into main Apr 2, 2026
10 checks passed
@felix-kaestner
felix-kaestner deleted the requeue branch April 2, 2026 08:08
@github-actions

github-actions Bot commented Apr 2, 2026

Copy link
Copy Markdown

Merging this branch will decrease overall coverage

Impacted Packages Coverage Δ 🤖
github.com/ironcore-dev/network-operator/cmd 0.00% (ø)
github.com/ironcore-dev/network-operator/internal/controller/cisco/nx 63.29% (ø)
github.com/ironcore-dev/network-operator/internal/controller/core 63.25% (-0.10%) 👎

Coverage by file

Changed files (no unit tests)

Changed File Coverage Δ Total Covered Missed 🤖
github.com/ironcore-dev/network-operator/cmd/main.go 0.00% (ø) 0 0 0
github.com/ironcore-dev/network-operator/internal/controller/cisco/nx/bordergateway_controller.go 52.14% (ø) 234 122 112
github.com/ironcore-dev/network-operator/internal/controller/cisco/nx/system_controller.go 64.04% (ø) 114 73 41
github.com/ironcore-dev/network-operator/internal/controller/cisco/nx/vpcdomain_controller.go 74.55% (ø) 224 167 57
github.com/ironcore-dev/network-operator/internal/controller/core/acl_controller.go 58.99% (ø) 139 82 57
github.com/ironcore-dev/network-operator/internal/controller/core/banner_controller.go 58.68% (ø) 167 98 69
github.com/ironcore-dev/network-operator/internal/controller/core/bgp_controller.go 58.99% (ø) 139 82 57
github.com/ironcore-dev/network-operator/internal/controller/core/bgp_peer_controller.go 65.34% (ø) 176 115 61
github.com/ironcore-dev/network-operator/internal/controller/core/certificate_controller.go 58.82% (ø) 153 90 63
github.com/ironcore-dev/network-operator/internal/controller/core/dns_controller.go 58.82% (ø) 136 80 56
github.com/ironcore-dev/network-operator/internal/controller/core/evpninstance_controller.go 67.96% (ø) 206 140 66
github.com/ironcore-dev/network-operator/internal/controller/core/interface_controller.go 74.28% (ø) 416 309 107
github.com/ironcore-dev/network-operator/internal/controller/core/isis_controller.go 60.78% (ø) 153 93 60
github.com/ironcore-dev/network-operator/internal/controller/core/lldp_controller.go 67.98% (ø) 203 138 65
github.com/ironcore-dev/network-operator/internal/controller/core/managementaccess_controller.go 57.35% (-1.47%) 136 78 (-2) 58 (+2) 👎
github.com/ironcore-dev/network-operator/internal/controller/core/ntp_controller.go 58.82% (ø) 136 80 56
github.com/ironcore-dev/network-operator/internal/controller/core/nve_controller.go 67.43% (-0.92%) 218 147 (-2) 71 (+2) 👎
github.com/ironcore-dev/network-operator/internal/controller/core/ospf_controller.go 59.34% (ø) 182 108 74
github.com/ironcore-dev/network-operator/internal/controller/core/pim_controller.go 60.78% (ø) 153 93 60
github.com/ironcore-dev/network-operator/internal/controller/core/prefixset_controller.go 62.59% (ø) 139 87 52
github.com/ironcore-dev/network-operator/internal/controller/core/routingpolicy_controller.go 66.14% (ø) 189 125 64
github.com/ironcore-dev/network-operator/internal/controller/core/snmp_controller.go 58.82% (ø) 136 80 56
github.com/ironcore-dev/network-operator/internal/controller/core/syslog_controller.go 58.99% (ø) 139 82 57
github.com/ironcore-dev/network-operator/internal/controller/core/user_controller.go 57.49% (ø) 167 96 71
github.com/ironcore-dev/network-operator/internal/controller/core/vlan_controller.go 61.74% (ø) 149 92 57
github.com/ironcore-dev/network-operator/internal/controller/core/vrf_controller.go 61.70% (ø) 141 87 54

Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code.

@hardikdr hardikdr added the area/switch-automation Automation processes for network switch management and operations. label Apr 3, 2026
@hardikdr hardikdr added this to Roadmap Apr 3, 2026
@github-project-automation github-project-automation Bot moved this to Done in Roadmap Apr 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/switch-automation Automation processes for network switch management and operations.

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants