refactor(llmobs): abstract llmobs from llama_index integration code by quinna-h · Pull Request #20860 · DataDog/dd-trace-py

quinna-h · 2026-10-07T14:59:47Z

Description

Stacked on PR #20760 which introduced the LlmEvents span-lifecycle events and first LLMObs subscriber.

Changes:

Decouple llama-index instrumentation from LLMObs by moving integration handling into shared lifecycle subscribers.

Contrib creates LlmRequestEvent instances and owns APM model/provider tags without importing LLMObs.
Three shared subscribers select the integration using the event’s component. Anthropic and llama-index register their integration classes through ddtrace.integration_hooks, including when patched manually.
Each request retains its integration on its execution context, preserving finalization across unpatch/repatch.
Streaming handlers dispatch completion through the context without holding an integration directly.

Testing

Existing llama_index and llmobs suites are the behavioral contract and pass unchanged.

Risks

Additional Notes

base_stream_handler.py has no LLM Observability dependencies, but it lived under ddtrace/llmobs/_integrations, so every contrib that traces streamed responses (and AI Guard) imported from the LLMObs product. Move it to ddtrace/contrib/internal/stream_handler.py and update all importers. The unit tests are renamed to tests/llmobs/test_stream_handler.py, and the new path is added to the llmobs suitespec component so changes to it keep triggering the LLM integration suites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Applies the same inversion the Anthropic integration received to LlamaIndex, so the contrib no longer imports LLMObs. The contrib previously constructed a LlamaIndexIntegration in patch(), stashed it on llama_index.core._datadog_integration, and hung it off every LlmRequestEvent, which meant ddtrace/contrib/internal/llama_index imported ddtrace.llmobs. Now the contrib dispatches the shared LlmEvents span lifecycle and LLMObs attaches its own subscribers on the llama_index.patch core event, mirroring ddtrace/llmobs/_contrib/anthropic. - patch(): drop the integration instance and _get_integration(); take integration_config/service from config.llama_index; own the llama_index.request.model and llama_index.request.provider APM tags via event.tags so they are set whether or not LLMObs is present; dispatch llama_index.patch / llama_index.unpatch. - _streaming.py: drop the integration parameter; the stream handlers only need the context to dispatch the deferred ended event. - LlamaIndexIntegration._set_base_span_tags is now a no-op: the contrib owns the APM tags and LlamaIndex has no per-request base_url to record. - The LLMObs subscribers register off the patch event, with the already-patched fallback keyed on llama_index.core rather than the top-level package. The dependency direction detector drops from 183 to 181 violations (both contrib.internal.llama_index -> llmobs._integrations edges removed, no new violations, no uncovered modules). Testing: llmobs::llama_index (61 passed) and llmobs::anthropic (108 passed) on py3.14; llmobs::llmobs matches the branch baseline exactly (1563 passed, same 11 pre-existing RC/sampling failures with and without this change). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

cit-pr-commenter-54b7da · 2026-10-07T15:03:25Z

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

.agents/skills/llmobs-integrations/SKILL.md                             @DataDog/ml-observability
.agents/skills/llmobs-integrations/references/failure-modes.md          @DataDog/ml-observability
.agents/skills/llmobs-integrations/references/implementation-guide.md   @DataDog/ml-observability
ddtrace/contrib/internal/llama_index/_constants.py                      @DataDog/python-guild @DataDog/apm-idm-python
ddtrace/contrib/internal/llama_index/_streaming.py                      @DataDog/python-guild @DataDog/apm-idm-python
ddtrace/contrib/internal/llama_index/patch.py                           @DataDog/python-guild @DataDog/apm-idm-python
ddtrace/llmobs/_contrib/_subscribers.py                                 @DataDog/ml-observability
ddtrace/llmobs/_contrib/anthropic/__init__.py                           @DataDog/ml-observability
ddtrace/llmobs/_contrib/llama_index/__init__.py                         @DataDog/ml-observability
ddtrace/llmobs/_integrations/llama_index.py                             @DataDog/ml-observability
pyproject.toml                                                          @DataDog/python-guild
tests/contrib/llama_index/test_llama_index_llmobs.py                    @DataDog/python-guild @DataDog/apm-idm-python
tests/contrib/llama_index/test_llama_index_patch.py                     @DataDog/python-guild @DataDog/apm-idm-python

cit-pr-commenter-54b7da · 2026-10-07T15:03:38Z

Circular import analysis

⚠️ Existing circular imports

There are 1 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector

cit-pr-commenter-54b7da · 2026-10-07T15:03:42Z

Dependency direction analysis

📈 Existing violations got worse

1 pre-existing violation(s) increased in severity (e.g. their target became more depended-on, or got pulled into an import cycle), though the edge itself isn't new:

ddtrace.contrib._events.llm -×-> ddtrace.llmobs._integrations.base  (contrib -> product:llmobs, score=24, +1 vs base)

⚠️ Existing dependency direction violations

There are 114 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 114 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=121)
ddtrace.llmobs._integrations.google_adk -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)
ddtrace.llmobs._integrations.anthropic -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)
ddtrace.llmobs._integrations.bedrock_agents -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)
ddtrace.aiguard._api_client -×-> ddtrace.trace  (product:aiguard -> product:tracing, score=119)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:
uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

✅ Dependency direction violations removed

2 violation(s) have been removed by this PR.

ddtrace.contrib.internal.llama_index._streaming -×-> ddtrace.llmobs._integrations  (contrib -> product:llmobs, score=15)
ddtrace.contrib.internal.llama_index.patch -×-> ddtrace.llmobs._integrations  (contrib -> product:llmobs, score=15)

chatgpt-codex-connector · 2026-10-07T18:50:18Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-28T22:04:20.057968Z	`fdb4c04`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

emmettbutler

Pretty much matches what we discussed. Nice work.

fix in-flight requests after unpatch()

dubloom

I'll make an experiment but I still doubt a bit the usefullned of the mlobs specific span lifecycle events. Like maybe we can leverage the ones that were pre-existing

Yun-Kim

small nits but otherwise lgtm

wconti27

PR definitely moving in the right direction. Sorry for all the repeated "can we centralize" but id love to strip all those small things someone has to do when writing an integration, and only require things that are truly 1 off per integration

quinna-h · 2026-10-29T14:03:55Z

Deferring remaining comments to follow up #21333 for broader subscriber refactor (shared lifecycle handling, centralized product registration and patch notifications, and removal of per-integration LLMObs subscriber helpers).

emmettbutler and others added 8 commits October 2, 2026 10:46

abstract llmobs from anthropic contrib using the subscriber pattern

72cb2ae

rm cassettes

b2f95ee

merge whoopsie

d991e24

import cycle

b288318

fix test isolation problem exposed by refactor

8416503

refactor out subscribers

f831724

quinna-h marked this pull request as ready for review October 7, 2026 18:50

quinna-h requested review from a team as code owners October 7, 2026 18:50

quinna-h requested review from Yun-Kim and dubloom and removed request for a team October 7, 2026 18:50

quinna-h requested a review from emmettbutler October 7, 2026 18:51

emmettbutler approved these changes Oct 8, 2026

View reviewed changes

wconti27 reviewed Oct 8, 2026

View reviewed changes

Comment thread ddtrace/llmobs/_contrib/llama_index/subscribers.py Resolved

emmettbutler and others added 7 commits October 9, 2026 05:46

Merge branch 'main' into emmett.butler/llmobs-contrib

6a1dcdc

Merge branch 'main' into emmett.butler/llmobs-contrib

28a077b

typing

1397a45

update rules about direct patch() calls

5398902

fix in-flight requests after unpatch()

Merge branch 'main' into emmett.butler/llmobs-contrib

2842d96

Merge branch 'main' into emmett.butler/llmobs-contrib

26e1d6d

Merge branch 'main' into emmett.butler/llmobs-contrib

bd86331

fix(llmobs): preserve llama-index stream finalization after unpatch

4c7c4ab

gh-worker-ownership-write-b05516 Bot removed the request for review from a team October 26, 2026 17:25

refactor(llmobs): share lifecycle subscribers across integrations

90733a6

quinna-h force-pushed the quinna.halim/llmobs-contrib-llama-index branch from 0d51d5d to 90733a6 Compare October 26, 2026 17:32

emmettbutler added the changelog/no-changelog A changelog entry is not required for this PR. label Oct 26, 2026

quinna-h requested a review from wconti27 October 26, 2026 18:51

wconti27 reviewed Oct 26, 2026

View reviewed changes

Comment thread ddtrace/llmobs/_contrib/_subscribers.py Resolved

wconti27 reviewed Oct 26, 2026

View reviewed changes

Comment thread ddtrace/llmobs/_contrib/_subscribers.py Resolved

wconti27 reviewed Oct 26, 2026

View reviewed changes

Comment thread ddtrace/contrib/internal/llama_index/patch.py Resolved

wconti27 reviewed Oct 26, 2026

View reviewed changes

Comment thread ddtrace/contrib/internal/llama_index/patch.py Resolved

wconti27 reviewed Oct 26, 2026

View reviewed changes

Comment thread ddtrace/llmobs/_integrations/llama_index.py Resolved

taegyunkim removed their request for review October 26, 2026 19:57

dubloom reviewed Oct 27, 2026

View reviewed changes

Yun-Kim approved these changes Oct 27, 2026

View reviewed changes

Comment thread ddtrace/contrib/internal/llama_index/patch.py Resolved

Comment thread ddtrace/contrib/internal/llama_index/patch.py Resolved

quinna-h added 5 commits October 28, 2026 11:12

refactor(llama_index): share model and provider tag constants

962391a

chore: merge main and resolve skill migration conflicts

64d9811

docs(llama_index): correct provider tag comment

3be7f26

refactor(llama_index): reuse component for integration hooks

d4bd69d

refactor(llama_index): derive tag names from component

37c4f15