Repository navigation
feat(llmobs): report Nova Sonic text token counts - #20819
Conversation
…o codex/novasonic-token-breakdown
Codeowners resolved asResolved from the full PR diff against |
Dependency direction analysis
|
Circular import analysis
|
🎉 All green!🧪 All tests passed 🔗 Commit SHA: 2ff069b | Docs | View more details | Give us feedback! |
ab9d7b2 to
1919a75
Compare
Move the Python 3.12 and 3.13 locks from the retired .riot directory to tests/requirements_locks without changing dependency pins. This matches the suitespec lock paths and existing CODEOWNERS rule. Validation: scripts/test-requirements check validated all 2208 concrete locks; scripts/lint checks passed.
Snapshot caller LLMObs identity and metadata before the SDK invocation can outlive its caller. Scope turn emission to that snapshot and restore the receiver context on every exit, using the turn root for each direct child. Preserve application tags in propagated contexts and remove the obsolete post-creation parent-ID repair from the integration base. Keep late audio on an interrupted response while user speech is unfinished, and ignore late END_TURN events that would extend its interruption boundary. Respect configured event-size limits when allocating shared audio payload budget. Finalize input, output, and duplex context-entry failures without replacing SDK exceptions or cancellation. Validation: 99 Nova tests pass on each of Python 3.12 and 3.13; 92 shared propagation/agent-attribution tests pass. Full scripts/lint checks pass. Import analysis finds no new cycles or violating dependency edges versus the prior PR head; all existing tests and recorded-session replays pass.
Only populate aggregate token metrics after a valid usage counter arrives. Preserve explicit zeros, derive totals only from two known directions, and reset presence with each turn without changing cumulative deduplication. Track observed audio per turn and direction independently of byte retention. Keep empty error/tool-only content empty, use fallback markers only for observed audio that cannot be attached, and preserve real inline audio. Add usage, close/error/cancellation, audio-presence and restart regressions. The restart tests confirm that distinct LLMObs provider objects share the same ContextVar and preserve caller identity/application/sampling context. Validation: 127 Nova Sonic tests passed on each of Python 3.12 and 3.13; full scripts/lint checks passed.
![datadog-prod-us1-6[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
A turn with no usage event bypasses gap tracking, causing the next reported cumulative increase—including audio and text tokens—to be attributed entirely to a later turn.
🤖 Bits Code Review · Commit 548c0e4
XG-xin
left a comment
XG-xin
left a comment
There was a problem hiding this comment.
Just the question from bot's review, otherwise looks good to me!
brettlangdon
left a comment
brettlangdon
left a comment
There was a problem hiding this comment.
should you also update CODEOWNERS for ddtrace/contrib/internal/aws_sdk_bedrock_runtime and tests/contrib/aws_sdk_bedrock_runtime to be for MLObs?
|
The missing-usage-event finding is confirmed in the existing aggregate/audio accounting and is tracked in #21344. This PR keeps that correction separate from the named text counters. |
There was a problem hiding this comment.
More details
The text-token accounting preserves explicit zeros and omits conflicting text without discarding independently valid audio. Static inspection supports the handling of aggregate gaps; runtime behavior was not exercised.
🤖 Bits Code Review · Commit 0d15be4
…o codex/novasonic-token-breakdown
9c072b2 to
2ff069b
Compare
|
/code blockers |
|
View all feedbacks in Devflow UI.
Checking merge blockers for #20819...
No merge blockers detected. |
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
Description
Preserves Nova Sonic 2 cumulative input/output text counters as per-turn
input_text_tokensandoutput_text_tokensalongside the existing audio counts. Each modality uses an independent baseline. Reported zeros are retained, and missing or invalid counter fields remain absent.Validates combined audio/text deltas against the inclusive aggregate. Conflicting text is omitted while independently valid audio is retained. Recovery from observed aggregate-counter gaps spanning turns suppresses ambiguous aggregate/modality counts while reestablishing their baselines.
Updates Nova integration/test ownership to MLObs and documents the named-token rules under
.agents/skills, preserving.claude/skillsas the shared symlink. Includes currentmainand retains its native UUID implementation, registration updates, and supported Python matrix.Testing
scripts/run-tests -r --venv 25344eb73ed4: 215 passed, Python 3.12.15 and aws-sdk-bedrock-runtime 0.11.0, after rebuilding against the merged main.scripts/lint checks: passed, including mypy, security, and structural checks.@DataDog/ml-observability; diff whitespace checks pass.Risks
Optional text counts remain absent when attribution is incomplete or inconsistent. Inclusive totals and independently valid audio retain their existing semantics.
An entire turn without any
usageEventcan still have its unknown usage included in the next turn's cumulative delta. This pre-existing aggregate/audio accounting case is tracked separately in #21344; this PR does not claim to fix it.Additional Notes
The initial Nova Sonic integration from #20524 is already merged into
main. Its existing release note remains unchanged; the new Feature note covers the additional named text-token metrics. The PR is based onmainand no longer requires that prerequisite to merge first.