Grpc endpoints followup

README.md

@@ -56,6 +56,11 @@ foss42 APIs project has 3 parts:
 - [foss42/api](https://github.com/foss42/api): The FastAPI app which serves the APIs.
 - [foss42/foss42-core](https://github.com/foss42/foss42-core): The open source core python library which has the algorithms, the data and does all the heavy-lifting.
 
+## Repository structure
+
+- `src/` — the hosted Open Source APIs (FastAPI app), with `tests/` as their test suite.
+- `grpc/` and `mqtt/` — self-contained, **local protocol test rigs** for API Dash's gRPC and MQTT clients (they are not hosted). Each has its own `README.md`, `tests/` and `requirements-dev.txt`, and runs via Docker.
+
 ## How to Run Locally
 
 1. **Clone the repository:**

docs/grpc/README.md

@@ -67,7 +67,7 @@ Azure/production infra is wired up separately.
   `grpc/docker-compose.yml` -- the image (generates the Protobuf stubs at build
   time and mints a self-signed TLS cert on first start) and the one-command rig.
 - `docs/grpc/` -- this README plus per-scenario pages (reflection, unary, the
-  three streaming modes, metadata, errors, TLS).
+  three streaming modes, metadata, auth, errors, TLS).
 
 ## Run it
 
@@ -86,6 +86,30 @@ This starts one service:
 Stop it with `Ctrl-C`, or run detached with `-d` and stop via
 `docker compose -f grpc/docker-compose.yml down`.
 
+## Tests
+
+A pytest round-trip suite (`tests/grpc/test_grpc.py`) exercises every method of
+`apidash.test.TestService` against a running server -- the gRPC analogue of the
+MQTT rig's `tests/mqtt/test_mqtt.py`. It covers reflection, unary (`Echo`,
+`GetRandomUser`), all three streaming modes (`StreamTicks`, `SumNumbers`,
+`Chat`), request/response metadata (`EchoMetadata`), auth-via-metadata
+(`SecureEcho`) and error status codes (`RaiseError`).
+
+The Protobuf stubs are generated on the fly at test-collection time from
+`grpc/proto/apidash_test.proto` (via `grpc_tools.protoc`), so nothing generated
+is committed. Run it:
+
+```
+docker compose -f grpc/docker-compose.yml up --build -d   # start the server
+pip install -r requirements-dev.txt                       # grpcio + tools + reflection
+pytest tests/grpc/test_grpc.py
+```
+
+The whole module **skips gracefully** (it does not fail) when grpcio /
+grpcio-tools are missing or when no server is reachable at `localhost:9000`, so
+CI without a server stays green. Point it elsewhere with the `GRPC_HOST` /
+`GRPC_PORT` environment variables.
+
 ## Endpoints
 
 | Transport | Address | TLS | Notes |
@@ -131,15 +155,16 @@ that exercises it and the mock/random data it returns:
 | Server streaming | `apidash.test.TestService/StreamTicks` | A stream of `Tick`s with random values | [server_streaming](server_streaming.md) |
 | Client streaming | `apidash.test.TestService/SumNumbers` | `sum` / `count` / `average` of the numbers you send | [client_streaming](client_streaming.md) |
 | Bidirectional streaming | `apidash.test.TestService/Chat` | Each message echoed back, server-timestamped | [bidi](bidi.md) |
-| Metadata / auth-via-metadata | `apidash.test.TestService/EchoMetadata` | The request metadata, echoed back | [metadata](metadata.md) |
+| Metadata / auth-via-metadata | `apidash.test.TestService/EchoMetadata` | The request metadata echoed back, plus response initial + trailing metadata | [metadata](metadata.md) |
+| Auth (Bearer / API key) | `apidash.test.TestService/SecureEcho` | Echoes your message only with valid credentials; else `UNAUTHENTICATED` | [auth](auth.md) |
 | Errors / status codes | `apidash.test.TestService/RaiseError` | Fails with the gRPC status code you request | [errors](errors.md) |
 | TLS transport | any method on `localhost:9001` | Same methods over TLS | [tls](tls.md) |
 
 Full method list:
 
 | Service | Methods |
 | ----------- | ----------- |
-| `apidash.test.TestService` | `Echo`, `GetRandomUser`, `StreamTicks`, `SumNumbers`, `Chat`, `EchoMetadata`, `RaiseError` |
+| `apidash.test.TestService` | `Echo`, `GetRandomUser`, `StreamTicks`, `SumNumbers`, `Chat`, `EchoMetadata`, `SecureEcho`, `RaiseError` |
 | `grpc.reflection.v1alpha.ServerReflection` | `ServerReflectionInfo` (used by **Reflect**) |
 
 > **Reflection version note:** the server registers the standard **v1alpha**
@@ -155,7 +180,8 @@ Full method list:
 - [server_streaming](server_streaming.md) -- `StreamTicks` (one request -> stream)
 - [client_streaming](client_streaming.md) -- `SumNumbers` (stream -> one response)
 - [bidi](bidi.md) -- `Chat` (bidirectional streaming echo)
-- [metadata](metadata.md) -- `EchoMetadata` (custom headers + auth-via-metadata)
+- [metadata](metadata.md) -- `EchoMetadata` (custom request headers + response metadata)
+- [auth](auth.md) -- `SecureEcho` (auth-protected: Bearer token / API key)
 - [errors](errors.md) -- `RaiseError` (choose the gRPC status code)
 - [tls](tls.md) -- gRPC over TLS on `9001`
 

docs/grpc/auth.md

@@ -0,0 +1,109 @@
+---
+protocol: grpc
+title: gRPC Auth (SecureEcho)
+desc: SecureEcho is an auth-protected unary call -- it accepts the request only when you send a valid Bearer token or API key as metadata, and fails UNAUTHENTICATED otherwise, so you can test API Dash's Auth tab and API-key auth.
+path: grpc/auth
+---
+
+gRPC has no separate "auth" mechanism -- credentials travel as
+[metadata](metadata.md) (the gRPC equivalent of HTTP headers). A server enforces
+auth by reading that metadata and rejecting the call when it's missing or wrong.
+
+`apidash.test.TestService/SecureEcho` is built for testing exactly this. It
+behaves like [`Echo`](unary.md) -- echoes your `message` back with a server time
+and sequence number -- but **only if the call carries valid credentials**.
+Otherwise it fails with gRPC status **`16 UNAUTHENTICATED`**.
+
+## What it tests
+
+That API Dash can attach credentials to a gRPC call -- a **Bearer token** via
+the Auth tab (which becomes an `authorization` metadata entry) or an **API key**
+as a custom metadata header -- and that it surfaces the `UNAUTHENTICATED` status
+when they're absent or wrong.
+
+## Method
+
+```
+apidash.test.TestService/SecureEcho    # EchoRequest{message} -> EchoResponse{message, server_time, seq}
+```
+
+The request/response shapes are identical to `Echo`; the difference is the
+credential check on the incoming metadata. On success the echoed `message` is
+prefixed with `[authenticated]` so you can tell the two apart.
+
+## Accepted credentials
+
+The call is accepted if **EITHER** of these is present in the request metadata:
+
+| Metadata key | Value | How to send it in API Dash |
+| ----------- | ----------- | ----------- |
+| `authorization` | `Bearer test-token` | Auth tab -> Bearer token = `test-token` |
+| `x-api-key` | `test-apikey` | Add a metadata / header row `x-api-key` = `test-apikey` |
+
+Anything else -- no credentials, a wrong token, a wrong key -- is rejected.
+
+## Expected behavior
+
+| You send | Result |
+| ----------- | ----------- |
+| Metadata `authorization: Bearer test-token` | Success -- `message` echoed as `[authenticated] <your message>` |
+| Metadata `x-api-key: test-apikey` | Success -- same as above |
+| No credentials | Call fails with status `16 UNAUTHENTICATED` |
+| Wrong token / key (e.g. `Bearer nope`) | Call fails with status `16 UNAUTHENTICATED` |
+
+The failure message is:
+`missing or invalid credentials -- send 'authorization: Bearer test-token' or 'x-api-key: test-apikey'`.
+
+## Test it in API Dash
+
+1. **Reflect** against `localhost:9000` and pick
+   `apidash.test.TestService/SecureEcho`.
+2. Set the request to `{"message": "hello"}` and **Send** with no credentials.
+   - **Expected:** the call fails and API Dash shows status
+     **`16 UNAUTHENTICATED`** with the missing-credentials message.
+3. Open the **Auth** tab, choose **Bearer** and enter the token `test-token`
+   (API Dash puts `authorization: Bearer test-token` on the wire). **Send**.
+   - **Expected:** success -- the response echoes
+     `[authenticated] hello` with a `server_time` and `seq`.
+4. Alternatively, instead of the Auth tab, add a metadata row `x-api-key` =
+   `test-apikey` and **Send**.
+   - **Expected:** the same success response.
+
+## Sample Usage
+
+### grpcurl
+
+Pass credentials as metadata with `-H "key: value"`:
+
+```
+grpcurl -plaintext -d '{"message": "hi"}' \
+  -H "authorization: Bearer test-token" \
+  localhost:9000 apidash.test.TestService/SecureEcho
+# {
+#   "message": "[authenticated] hi",
+#   "server_time": "2026-08-24T...Z",
+#   "seq": 1
+# }
+```
+
+With an API key instead:
+
+```
+grpcurl -plaintext -d '{"message": "hi"}' \
+  -H "x-api-key: test-apikey" \
+  localhost:9000 apidash.test.TestService/SecureEcho
+# { "message": "[authenticated] hi", ... }
+```
+
+With no credentials it fails:
+
+```
+grpcurl -plaintext -d '{"message": "hi"}' \
+  localhost:9000 apidash.test.TestService/SecureEcho
+# ERROR:
+#   Code: Unauthenticated
+#   Message: missing or invalid credentials -- send 'authorization: Bearer test-token' or 'x-api-key: test-apikey'
+```
+
+For TLS use `localhost:9001` -- see [tls](tls.md). To just inspect which metadata
+reaches the server (without the auth check), use [`EchoMetadata`](metadata.md).

docs/grpc/metadata.md

@@ -1,7 +1,7 @@
 ---
 protocol: grpc
 title: gRPC Metadata & Auth
-desc: EchoMetadata reflects the request metadata (gRPC headers) back to you, so you can verify custom headers and auth-via-metadata (e.g. an authorization token) are sent.
+desc: EchoMetadata reflects the request metadata (gRPC headers) back to you AND sends response metadata (initial + trailing) back, so you can verify both custom request headers / auth-via-metadata and the response-metadata view.
 path: grpc/metadata
 ---
 
@@ -10,15 +10,26 @@ alongside every call. It's how clients send things like an `authorization`
 token, a request id, or an API key -- gRPC has no separate "auth" mechanism, so
 **auth is done via metadata**.
 
-`apidash.test.TestService/EchoMetadata` is built for testing this: it **returns
-the request metadata back to you** in the response, so you can confirm exactly
-which headers API Dash put on the wire.
+`apidash.test.TestService/EchoMetadata` is built for testing this in **both
+directions**:
+
+- **Request -> server:** it **returns the request metadata back to you** in the
+  response body, so you can confirm exactly which headers API Dash put on the
+  wire.
+- **Server -> response:** it also **sends response metadata back** -- initial
+  metadata (`x-server`, `x-echoed-count`) and trailing metadata (`x-trailer`) --
+  so API Dash's **response metadata / headers view** is testable too.
 
 ## What it tests
 
 That API Dash attaches custom metadata to a gRPC call (including an
 `authorization` header for auth-via-metadata) and that the values arrive at the
-server intact.
+server intact -- **and** that it surfaces the metadata the server sends back on
+the response (initial + trailing).
+
+> For an auth check that actually **rejects** calls without valid credentials
+> (rather than just echoing whatever you send), see [auth](auth.md)
+> (`SecureEcho`).
 
 ## Method
 
@@ -41,17 +52,37 @@ This makes it a quick way to prove an auth token or any custom header is actuall
 being sent -- if it comes back, the server received it. (Note gRPC lowercases
 metadata keys on the wire, so `Authorization` comes back as `authorization`.)
 
+## Response metadata (server -> client)
+
+Besides echoing the request metadata into the response **body**, the server
+attaches metadata to the **response** itself -- the mirror image of what you
+send. These are surfaced in API Dash's response metadata / headers view, not in
+the message body:
+
+| Metadata | When it arrives | Value |
+| ----------- | ----------- | ----------- |
+| `x-server` | initial metadata (response headers) | `apidash-grpc-test` |
+| `x-echoed-count` | initial metadata (response headers) | number of request metadata pairs the server received |
+| `x-trailer` | trailing metadata (response trailers) | `ok` |
+
+So a single `EchoMetadata` call exercises metadata in both directions:
+request -> server (echoed in the body) **and** server -> response (the initial +
+trailing metadata above).
+
 ## Test it in API Dash
 
 1. **Reflect** against `localhost:9000` and pick
    `apidash.test.TestService/EchoMetadata`.
 2. Add metadata rows to the request, e.g. `authorization` / `Bearer test-token`
    and `x-request-id` / `abc123`. Leave the request body empty.
 3. **Send**.
-   - **Expected:** the response `metadata` map contains
+   - **Expected (response body):** the response `metadata` map contains
      `authorization: Bearer test-token` and `x-request-id: abc123` (plus
      transport headers like `user-agent`). This is the pattern for testing
      **auth-via-metadata** against any real gRPC service.
+   - **Expected (response metadata / headers view):** `x-server:
+     apidash-grpc-test` and `x-echoed-count: <N>` in the initial metadata, and
+     `x-trailer: ok` in the trailing metadata.
 
 ## Sample Usage
 
@@ -74,6 +105,23 @@ grpcurl -plaintext \
 # }
 ```
 
+Add `-v` to also see the **response** metadata grpcurl received -- the server's
+initial and trailing metadata:
+
+```
+grpcurl -plaintext -v \
+  -H "authorization: Bearer test-token" \
+  localhost:9000 apidash.test.TestService/EchoMetadata
+# Response headers received:
+#   x-server: apidash-grpc-test
+#   x-echoed-count: 2
+# ...
+# Response trailers received:
+#   x-trailer: ok
+```
+
 In API Dash: add metadata rows to the request, call `EchoMetadata`, and confirm
-the same key/value pairs appear in the response. For TLS use `localhost:9001` --
-see [tls](tls.md).
+the same key/value pairs appear in the response body -- and that `x-server` /
+`x-echoed-count` / `x-trailer` appear in the response metadata view. For TLS use
+`localhost:9001` -- see [tls](tls.md). For an auth check that rejects invalid
+credentials, see [auth](auth.md).

grpc/README.md

@@ -0,0 +1,26 @@
+# gRPC test rig
+
+A self-contained, local gRPC test server (`apidash.test.TestService`) for
+exercising API Dash's gRPC client -- reflection, unary, all three streaming
+modes, metadata, auth-via-metadata and error codes. It is **not** part of the
+hosted Open Source APIs; it only runs locally via Docker.
+
+## Setup & run the server
+
+```bash
+docker compose -f grpc/docker-compose.yml up --build
+```
+
+Serves on `localhost:9000` (plaintext) and `localhost:9001` (TLS, self-signed).
+
+## Run the tests
+
+```bash
+pip install -r grpc/requirements-dev.txt
+pytest grpc/tests
+```
+
+The suite generates the Protobuf stubs on the fly and **skips gracefully** when
+grpcio/-tools are missing or no server is reachable on `localhost:9000`.
+
+See [`docs/grpc/`](../docs/grpc/) for the per-feature pages.

grpc/proto/apidash_test.proto

@@ -14,7 +14,8 @@
 //   StreamTicks    server stream  N random "ticks" (a price/sensor-style feed)
 //   SumNumbers     client stream  sum / count / average of the numbers you send
 //   Chat           bidi stream    echoes each message back, stamped server-side
-//   EchoMetadata   unary          reflects the call's gRPC metadata back to you
+//   EchoMetadata   unary          reflects request metadata + sends response metadata
+//   SecureEcho     unary          like Echo, but requires auth-via-metadata credentials
 //   RaiseError     unary          fails with the gRPC status code you request
 //
 // The server (server.py) has server reflection enabled, so API Dash's Reflect
@@ -45,10 +46,17 @@ service TestService {
   // echoed back with a server-applied timestamp.
   rpc Chat (stream ChatMessage) returns (stream ChatMessage);
 
-  // Unary: reflects the call's gRPC metadata (headers) back to you, for
-  // testing custom metadata and auth-via-metadata (e.g. an authorization token).
+  // Unary: reflects the call's gRPC metadata (headers) back to you, AND sends
+  // response metadata (initial + trailing) back, for testing custom metadata,
+  // auth-via-metadata (e.g. an authorization token) and response-header views.
   rpc EchoMetadata (Empty) returns (MetadataResponse);
 
+  // Unary: like Echo, but AUTH-PROTECTED via metadata. Accepts the call only if
+  // metadata `authorization` == "Bearer test-token" OR `x-api-key` ==
+  // "test-apikey"; otherwise fails UNAUTHENTICATED. Exercises the Auth tab
+  // (Bearer token -> authorization metadata) and API-key auth.
+  rpc SecureEcho (EchoRequest) returns (EchoResponse);
+
   // Unary: deliberately fails with the gRPC status code you request, for
   // testing how the client surfaces non-OK statuses. code = a gRPC status code.
   rpc RaiseError (ErrorRequest) returns (Empty);

grpc/requirements-dev.txt

@@ -0,0 +1,3 @@
+grpcio  # grpc/tests: local gRPC server round-trip tests (not needed by the prod app)
+grpcio-tools  # grpc/tests: generates the Protobuf stubs on the fly from grpc/proto (not needed by the prod app)
+grpcio-reflection  # grpc/tests: client-side reflection listing in the round-trip tests (not needed by the prod app)