Coordinator/place keepalive - #1884
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1884 +/- ##
========================================
- Coverage 46.0% 45.1% -0.9%
========================================
Files 180 180
Lines 14464 14764 +300
========================================
+ Hits 6654 6667 +13
- Misses 7810 8097 +287
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. |
e65832f to
34865ac
Compare
|
@Emantor updated to drop other commit |
Place acquisition is currently indefinite, which can lead to stale locks when users stop interacting with a place without releasing it. Add a lease-based locking mode that allows places to be acquired for a limited time and expire automatically unless extended. Leases are tied to the lifetime of a reservation. This prevents stale locks and reduces the need for manual cleanup. Signed-off-by: Asher Pemberton <asher.pemberton@arm.com> Reviewed-by: Asher Pemberton <asher.pemberton@arm.com> # gatekeeper Co-authored-by: Idan Saadon <idan.saadon@arm.com>
Send a dedicated lease-start request to exporters when a place is leased, including the coordinator default lease duration. This keeps lease acquisition separate from normal acquire requests. Signed-off-by: Asher Pemberton <asher.pemberton@arm.com> Reviewed-by: Asher Pemberton <asher.pemberton@arm.com> # gatekeeper Co-authored-by: Idan Saadon <idan.saadon@arm.com>
34865ac to
357ceca
Compare
|
note: have added a new commit remote/coordinator: add exporter notification for lease start |
Return FAILED_PRECONDITION when a leased place contains an orphaned acquired resource during lease extension. This keeps the existing lease alive until its current timeout instead of silently extending the reservation without notifying the exporter. Signed-off-by: Asher Pemberton <asher.pemberton@arm.com> Reviewed-by: Asher Pemberton <asher.pemberton@arm.com> # gatekeeper Co-authored-by: Idan Saadon <idan.saadon@arm.com>
|
note: have added another new commit for an edge case we found: remote/coordinator: fail lease extension for orphaned resourcesagain can squash post review |
|
While it's not really clear yet how we'd implement fencing (stopping access to resources) on the client on timeout (be it due to temporary network issues, suspend/resume on a laptop or something else), I'd prefer if we could extend the core acquire concept with timeouts. This would allow us to cleanly extend this to cover normal interactive lock/unlock as well. The new behavior could be opt-in via |
Background:
Labgrid currently supports place acquisition, but acquired places remain locked until they are released explicitly.
In practice, this can lead to stale locks, where a place stays occupied even after the user is no longer actively using it.
This reduces availability of shared lab resources and often requires manual intervention to recover locked places.
What problem this solves
This MR introduces lease-based place locking, allowing places to be acquired for a limited time and released automatically if the lease is not extended.
This prevents stale locks and keeps place usage self-cleaning, while also integrating naturally with the existing reservation and scheduler semantics.
Overview:
This change adds a new lease acquisition mode alongside the existing acquire mode:
be extended explicitly
Leased places are released automatically when the associated reservation expires, is cancelled, or when lease extension fails.
Lease timing and limits
Lease timing is not user-configurable via CLI. Instead, it is governed by coordinator configuration:
This design ensures consistent behavior across users and prevents misuse or overly long locks.
Lease extension failure behavior
If extending a lease fails (for example due to exporter/resource errors),
the coordinator will cancel the lease and release the place.
This avoids leaving the system in a partially extended or inconsistent state,
especially when a place consists of multiple resources.
From the user perspective, a failed extend means the lease is no longer valid
and the place must be leased again.
CLI examples
Create a reservation:
labgrid-client reserve board=imx8 # the LG_TOKEN will be printed upon successLease a place using the dedicated
leasecommand:labgrid-client -p +<LG_TOKEN> leaseExtend the reservation to keep the lease alive:
labgrid-client extend <LG_TOKEN> --keepalive # automatically renews lease until failure or max durationAcquire a place indefinitely (unchanged behaviour):
labgrid-client -p board-01 lockor
labgrid-client -p board-01 acquire