Add OpenTelemetry Tracing for External Services

[genematx]

Where #1536 traces the request lifecycle (the FastAPI server span and internal phase spans), this PR adds spans for the external services a request touches, so a trace shows the full picture — the catalog, storage, cache, and outbound HTTP calls — and a Grafana service graph of Tiled's dependencies.

Stacked on #1536; that PR should merge first.

What's traced

Service	Instrumentation	Shows up as
PostgreSQL catalog & authentication	opentelemetry-instrumentation-asyncpg	SELECT/INSERT/… spans, db.system=postgresql, db.name=tiled_catalog / authn
SQL storage (tabular data, via ADBC)	opentelemetry-instrumentation-dbapi wrapping the ADBC connection, plus an explicit span around the bulk adbc_ingest write (which bypasses the DBAPI execute path)	db.system=postgresql, db.name=tiled_storage, adbc_ingest
Redis streaming cache	opentelemetry-instrumentation-redis	HSET/HGET/… spans, db.system=redis
Outbound HTTP (webhook delivery, OIDC auth, external policy servers)	opentelemetry-instrumentation-httpx	POST/GET client spans

These are client spans emitted by Tiled, so they keep service.name=tiled and are distinguished by db.system / the target host — the span correctly stays with its emitter rather than being re-labelled as another service.

Service graph

A peer.service attribute is set on each dependency so Grafana Tempo's Service Graph renders them as distinct nodes. Postgres is split by db.name; all webhook deliveries are grouped under a single webhooks node (identified by the X-Tiled-Event-ID header); other outbound calls are named by host. The example Collector / Tempo / Grafana config is wired up to generate and display it.

user → tiled → { tiled_catalog, tiled_storage, redis, webhooks, <oidc-host>, … }

Manual testing

Several integration tests are included in test_tracing.py, but it also informative to test the stack manually:

1. Start the example stack (Tiled + Postgres + Redis + Collector + Jaeger + Tempo + Grafana)

From the repo root (the two webhook env vars let the toy receiver in step 3 work; you can omit them if testing the webhook tracing is not in the scope):

TILED_WEBHOOKS_ALLOW_HTTP=true TILED_WEBHOOKS_ALLOW_PRIVATE_ADDRESSES=true \
  docker compose -f compose.dev.yml -f compose.monitoring.yml up --build

• Jaeger UI: http://localhost:16686 (select service tiled)
• Grafana → Explore → Tempo → Service Graph: http://localhost:3000

2. Exercise catalog (asyncpg), storage (ADBC), and streaming (Redis)

import numpy as np, pyarrow
from tiled.client import from_uri

c = from_uri("http://localhost:8000", api_key="secret")

# catalog (asyncpg): writes + reads
cont = c.create_container("demo")
arr = cont.write_array(np.arange(10), key="arr")
list(c)

# SQL storage (ADBC): appendable table -> adbc_ingest + DBAPI spans
z = c.create_container("demo_table", specs=["composite"])
tab = pyarrow.table({"x": [1, 2, 3], "y": [4.0, 5.0, 6.0]})
z.create_appendable_table(schema=tab.schema, key="t").append_partition(0, tab)

# Redis (streaming cache): subscribe, then push an update
sub = arr.subscribe(); sub.start_in_thread(start=0)
arr.patch(np.arange(10), offset=10, extend=True)

3. Exercise webhooks (httpx)

A minimal receiver (stdlib only), run on the host:

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        body = self.rfile.read(int(self.headers.get("Content-Length", 0)))
        print(self.headers.get("X-Tiled-Event-ID"), body)
        self.send_response(200); self.end_headers()

ThreadingHTTPServer(("0.0.0.0", 9000), H).serve_forever()

Register it and fire an event (the container reaches the host via host.docker.internal):

import httpx
httpx.post(
    "http://localhost:8000/api/v1/webhooks/target/",
    headers={"Authorization": "Apikey secret"},
    json={"url": "http://host.docker.internal:9000/hook"},
).raise_for_status()
c.create_container("fires_webhook")   # delivered via httpx -> a 'webhooks' client span

4. What you should see

• Jaeger (service tiled): request traces now contain child spans for SELECT/INSERT (db.system=postgresql), HSET/HGET (db.system=redis), adbc_ingest, and POST (webhook delivery). Startup / new-connection queries that run outside a request appear as their own short SELECT/select/show traces.
• Grafana → Tempo → Service Graph: tiled with edges to tiled_catalog, tiled_storage, redis, and webhooks.

The automated coverage lives in tests/test_tracing.py (extended with asyncpg / ADBC / Redis / httpx assertions); those tests run against live Postgres/Redis and skip when TILED_TEST_POSTGRESQL_URI / TILED_TEST_REDIS are unset.

Caveats

• Database tracing covers PostgreSQL (asyncpg + ADBC), not SQLite — the SQLAlchemy instrumentation that would cover SQLite does not yet support Tiled's SQLAlchemy version. SQLite / in-memory setups still get the request and phase spans from Add Basic OpenTelemetry Tracing #1536.

---

Follow up on: PR #1536
Related Issue: #1000

Checklist

• Add a Changelog entry
• Add the ticket number which this PR closes to the comment section

[danielballan]

Looks good. Other than my comments already in #1536, I just have one question here.

[danielballan]

I feel confused by the name adbc_current_catalog because I don't except to see "ADBC" and "catalog" together. ADBC is for tabular storage, not catalog. It seems possible (or likely!) that I'm missing something though.

[genematx]

Great question, I was a bit confused too at first. adbc_current_catalog is an internal attribute that tells OpenTelemetry which name to use to mark the connection (in this case, the database name, which would resolve to tiled_storage).
I've added a comment to make it clear (and while doing so, fixed a small bug too).

[genematx]

Added tests for tracing with DuckDB as well.