Skip to content

fix(tracing): keep W3C tracestate dd= member within 256 chars - #20820

Open
julinvictus wants to merge 6 commits into
DataDog:mainfrom
julinvictus:fix/w3c-tracestate-dd-member-length
Open

julinvictus wants to merge 6 commits into
DataDog:mainfrom
julinvictus:fix/w3c-tracestate-dd-member-length

Conversation

@julinvictus

@julinvictus julinvictus commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Description

w3c_get_dd_list_member keeps the dd= list-member of the W3C tracestate header under 256 characters, but it only counted the tag text. Three things that end up in the header were not counted:

  • the ; separators between tags (1 char each)
  • the p:<16 hex>; parent id field, prepended at injection time by w3c_tracestate_add_p / the native build_tracestate (19 chars)
  • the dd= prefix (3 chars)

So the injected member could exceed the limit by roughly 30 characters. This now shows up in practice because LLM Observability propagates more _dd.p.llmobs_* tags (llmobs_sid, llmobs_pagent_span_id, llmobs_pagent_name). With a typical set of those tags, the injected dd= member was 285 characters. Proxies that enforce the W3C limit drop the whole tracestate header, which breaks LLM Observability trace linking across services.

The fix:

  • Reserves space for dd= and p:<16 hex>; up front (_W3C_DD_LIST_MEMBER_RESERVED_LEN).
  • Counts the required fields (s:, o:, t.dm:, t.usr.id:) as they are written, separators included.
  • Counts 1 extra char for the ; before each optional _dd.p.* tag.

As before, tags that don't fit are dropped whole and never truncated.

Before (285 chars):

dd=p:10a3f888ec177dd0;s:1;o:rum;t.llmobs_parent_id:10707708093473052057;t.llmobs_trace_id:141883241939442010658227520463581072062;t.llmobs_ml_app:datasciencetiger;t.llmobs_sid:11a0eec0-ad0a-4f4d-8ae8-7f358ed1b153;t.llmobs_sr:1;t.llmobs_sd:1;t.llmobs_pagent_span_id:14260131734498464268

After (240 chars, t.llmobs_pagent_span_id dropped because it no longer fits):

dd=p:61d200da4b9294a3;s:1;o:rum;t.llmobs_parent_id:10707708093473052057;t.llmobs_trace_id:141883241939442010658227520463581072062;t.llmobs_ml_app:datasciencetiger;t.llmobs_sid:11a0eec0-ad0a-4f4d-8ae8-7f358ed1b153;t.llmobs_sr:1;t.llmobs_sd:1

Testing

  • Reproduced the overflow on v4.15.2 by injecting headers from an active span with the LLM Observability tags shown above (285 chars). With this patch applied to the same version, the member is 240 chars. The diff applies cleanly to v4.15.2.
  • Ran the existing w3c_get_dd_list_member / tracecontext / inject tests from tests/tracer/test_utils.py and tests/tracer/test_propagation.py with and without the patch. The patch introduced no new failures.

Risks

  • p: space is reserved even when no p: field is added (no active Datadog span and no known last parent id). In that case up to 19 fewer characters are used for propagated tags than strictly allowed.
  • When the budget is exceeded, the tags that come last in the trace's meta order are dropped. That ordering behavior is unchanged, but more tags can now be dropped than before, which is required to respect the limit. Today, with the LLM Observability tags, llmobs_pagent_span_id is the one most likely to be dropped.

Additional Notes

Opened from a fork, so some required CI can't run here and will need a maintainer to mirror the branch.

🤖 Generated with Claude Code

@julinvictus
julinvictus requested a review from a team as a code owner October 5, 2026 22:33
@julinvictus
julinvictus requested review from emmettbutler and removed request for a team October 5, 2026 22:33
@datadog-prod-us1-5

This comment has been minimized.

@julinvictus
julinvictus requested a review from a team as a code owner October 5, 2026 22:44
@julinvictus
julinvictus requested review from a team as code owners October 6, 2026 16:25
@julinvictus
julinvictus requested review from florentinl and rachelyangdog and removed request for a team October 6, 2026 16:25

@Yun-Kim Yun-Kim left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small nits but LGTM otherwise!

Comment thread ddtrace/internal/utils/http.py Outdated
Comment thread ddtrace/internal/utils/http.py Outdated
Comment thread ddtrace/internal/utils/http.py
Comment thread releasenotes/notes/fix-w3c-tracestate-dd-member-length-3f1c9a7e5d2b4c60.yaml Outdated
@julinvictus

Copy link
Copy Markdown
Collaborator Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-10-07 01:31:15 UTC ℹ️ Start processing command /merge


2026-10-07 01:31:18 UTC ❌ MergeQueue

The base branch of this pull request is currently frozen (see incident #61179).

If you need to change something else, your pull request can still be merged by commenting /merge -p incident.

@julinvictus julinvictus self-assigned this Oct 7, 2026

@brettlangdon brettlangdon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit about brevity in the release note, otherwise lgtm

Comment thread releasenotes/notes/fix-w3c-tracestate-dd-member-length-3f1c9a7e5d2b4c60.yaml Outdated
Comment thread ddtrace/internal/utils/http.py Outdated
julinvictus and others added 2 commits October 7, 2026 16:05
@julinvictus

Copy link
Copy Markdown
Collaborator Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-10-07 23:54:24 UTC ℹ️ Start processing command /merge


2026-10-07 23:54:27 UTC ❌ MergeQueue

The base branch of this pull request is currently frozen (see incident #61179).

If you need to change something else, your pull request can still be merged by commenting /merge -p incident.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants