Metadata identity

[asher-pem-arm]

Description

This is the first part of the coordinator unary-RPC rework. It introduces metadata-based identity for clients and exporters, reducing the coordinator’s dependency on startup messages sent through the long-lived streams.

This PR:

• Adds client and server gRPC identity interceptors.
• Sends client/exporter identity and user-agent information as gRPC metadata.
• Initializes ClientStream and ExporterStream sessions from that metadata.
• Deprecates stream startup identity messages while retaining them as a fallback for older clients and exporters.
• Updates AcquirePlace, AllowPlace, and CreateReservation to infer caller identity from metadata or an existing stream session.
• No new coordinator RPCs are introduced in this part.

Motivation

Decouple ClientStream’s live-update role from identity and independent RPCs, allowing services and short-lived tools to perform bounded authenticated operations without maintaining stream state. This simplifies client implementations, reconnection handling, horizontal scaling, and per-operation authorization.

Compatibility note

Old clients and exporters remain compatible with the new coordinator through the startup-message fallback.
However, new clients and exporters stop sending startup identity messages and are therefore incompatible with coordinators predating this change. Those coordinators cannot establish the required stream session.

See also

Discussion on the gRPC unary rework: #1881
Follow-up RPC changes: #1919
The List RPC changes: #1933

[codecov]

Codecov Report

❌ Patch coverage is 47.50958% with 137 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.2%. Comparing base (fb48582) to head (1db182f).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...abgrid/remote/generated/labgrid_coordinator_pb2.py	0.8%	123 Missing ⚠️
labgrid/remote/coordinator.py	76.1%	10 Missing ⚠️
labgrid/remote/identity.py	91.4%	3 Missing ⚠️
labgrid/remote/grpc/interceptor/server.py	95.6%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #1918     +/-   ##
========================================
+ Coverage    61.0%   61.2%   +0.1%     
========================================
  Files         182     185      +3     
  Lines       14882   15020    +138     
========================================
+ Hits         9083    9195    +112     
- Misses       5799    5825     +26     

Flag	Coverage Δ
3.10	61.1% <47.5%> (+0.1%)	⬆️
3.11	61.2% <47.5%> (+0.1%)	⬆️
3.12	61.1% <47.5%> (+0.1%)	⬆️
3.13	61.1% <47.5%> (+0.1%)	⬆️
3.14	61.1% <47.5%> (+0.2%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[Emantor]

I took the liberty to reformatted for the PR template, this should make the description more readable.

[Emantor]

During the review of this PR yesterday it came up that we are missing a good justification for separate unary RPC instead of extending the ClientStream with filtering. The unary RPCs introduced in this PR duplicate the information on the stream and applications should be able to use the ClientStream for the same information.

[asher-pem-arm]

Hey @Emantor ,

Thanks for sharing the concern. The primary motivation for this came out of our work on the Labgrid fleeting plugin, and the more we thought about it the more sense it made.

At the moment ClientStream is doing two jobs: it is both a watch/update mechanism and part of the session/identity for independent RPCs. That coupling is what we are trying to remove. A service, CI worker, web backend, or short-lived tool should be able to make a bounded RPC with identity/auth metadata on that RPC, get a response, and finish.
As we found on our end with our fleeting plugin implementation, requiring those clients to maintain a bidirectional stream means they also need reconnect logic, caching, sync handling, readiness state, and failure handling to safely perform simple operations. On the Labgrid fleeting plugin we managed to remove ~1000 lines of code/tests when we moved to using the unary RPC implementation.

This also matters for production/distributed deployments. If every client has to establish and maintain a ClientStream session, then coordinator endpoint changes, load balancer migration, reconnects, and horizontally scaled services all need to manage stream/session recovery. In many cases those clients gain no benefit from a long-lived session. The intent here is to decouple those concerns: clients that need ClientStream for live updates can still use it.

A stream also makes auth harder as the coordinator has to continuously filter updates for each client and ensure no unauthorised state is ever sent. For unary RPCs, the auth decision is scoped to one operation and one response.

If you feel this is useful context I can add this to the discussion as needed. Thanks

[asher-pem-arm]

@Emantor as discussed I've updated this PR to only include the client stream decoupling

[Emantor]

Looking at the test output for this PR I see that the tests seem not to use the client identity:

pytest --cov-report=term-missing --cov=labgrid tests -k coordinator -s -vv
current_level: None
============================================================================================================ test session starts =============================================================================================================
platform linux -- Python 3.13.13, pytest-9.1.1, pluggy-1.6.0 -- /home/phoenix/work/ptx/labgrid/.venv/bin/python3
cachedir: .pytest_cache
benchmark: 5.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/phoenix/work/ptx/labgrid
configfile: pytest.ini (WARNING: ignoring pytest config in pyproject.toml!)
plugins: mock-3.15.1, isort-4.0.0, dependency-0.6.1, cov-7.1.0, benchmark-5.2.3, asyncio-1.4.0, labgrid-26.1.dev27
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 496 items / 478 deselected / 1 skipped / 18 selected

tests/test_coordinator.py::test_startup coordinator: INFO:root:loaded 0 place(s)
coordinator: INFO:root:Enabled channelz support
coordinator: INFO:root:Coordinator ready
coordinator: listening on [::]:20408
PASSED
------------------------------------------------------------------------------------------------------------- live log teardown --------------------------------------------------------------------------------------------------------------
INFO               root: stopping Coordinator pid=27317
coordinator: INFO:root:Starting graceful shutdown...
coordinator: INFO:root:Starting graceful shutdown...

tests/test_coordinator.py::test_coordinator_add_place coordinator: INFO:root:loaded 0 place(s)
coordinator: INFO:root:Enabled channelz support
coordinator: INFO:root:Coordinator ready
coordinator: listening on [::]:20408
coordinator: INFO:root:client connected: ipv4:127.0.0.1:42882
coordinator: WARNING:root:client ipv4:127.0.0.1:42882 did not provide identity metadata; using deprecated startup identity

But since this is with the client interceptor, shouldn't these use the new identity metadata?

Edit:Scratch that, the tests use a bespoke channel stub which does not add the metadata.

[Emantor]

The code looks good to me, all thats left in my opinion is to squash the last commit.
It was nice for reviewing, but it should be cleaner to not remove and re-add StartupDone support from a history point of view.

[jluebbe]

What is the meaning of

Signed-off-by: Asher Pemberton <asher.pemberton@arm.com>
Reviewed-by: Asher Pemberton <asher.pemberton@arm.com> # gatekeeper

? You can't add a Reviewed-by for code you've written yourself.

[Bastian-Krause]

No-op, drop that.

[Bastian-Krause]

Why do we need to pin to that exact version?

[Emantor]

This should be one logging.debug() line.

[Emantor]

This should be a single logging.debug line.

[Emantor]

Same as for ClientSession above, should be created in a single place.

[jluebbe]

The "why" of each change should also be explained in the commit messages.

Motivation

Decouple ClientStream’s live-update role from identity and independent RPCs, allowing services and short-lived tools to perform bounded authenticated operations without maintaining stream state. This simplifies client implementations, reconnection handling, horizontal scaling, and per-operation authorization.

I'm not fully convinced by the simplification arguments.

"client implementations" might be correct for other implementations, but labgrid's own client becomes more complex through the introduction of the Interceptors.

"reconnection handling" only applies to stateless clients, right? Anything using labgrid to access resources needs a ordered and loss-less stream of resource events to keep a consistent view of a Place. Supporting reconnection would mean buffering and replaying resource events upon reconnection, and I'm not sure if the complexity required for that is worth it.

"horizontal scaling" don't seem correct to me. As long as the state is held in the coordinator process (which I don't see us changing), there can only be a single instance as this ensures consistency.

"per-operation authorization" sounds right, as we no longer need to look up the ClientSession by peer.

So, I'd prefer a motivation description which make sense (primarily) from the perspective of using labgrid with the in-tree exporter and client.

[jluebbe]

I've been trying to understand why you chose to use a ContextVar instead of using the gRPC Server Side context object passed to each API method to inspect the invocation_metadata? It feels like the ContextVar adds a mechanism similar to what is already provided by the context parameter.

Later, on the authorization side, a function decorator could also access the ServicerContext parameter of the call to decide whether to allow/deny the call.

Without a ContextVar, we don't have easy access to the ClientIdentity, but as auth checks should be done early, I'm skeptical if that access is needed at all. Without the ContextVar, the complexity of the Interceptors wouldn't be needed either.

I'm not saying we must avoid ContextVar and Interceptors, but I want to understand why they are the right choice.