[mabdinur]

Description

Reduce automatic startup overhead by keeping inactive LLMObs, debugging, and symbol-upload implementations unloaded. Preserve remote activation, public SDK exports, default code-origin tags, and required early AppSec/debugging hooks. No new configuration is required.

Ordinary requests, unrelated remote-config updates, fork, and shutdown do not initialize inactive implementations. Snapshot collection loads only for an active debug session.

Testing

CI startup benchmarks, comparing base b2fc819 with a77a6ae5: Python 3.12, Linux x86_64, 60 samples per configuration. Times include process shutdown.

Scenario	Base	PR	Reduction
import ddtrace	465.4 ms	463.7 ms	Essentially unchanged
import ddtrace.auto	836.3 ms	690.6 ms	145.8 ms / 17.4%
ddtrace-run	1204.4 ms	1059.6 ms	144.8 ms / 12.0%
Auto + Flask + 10 requests	1058.7 ms	928.1 ms	130.7 ms / 12.3%
Launcher + Flask + 10 requests	1407.5 ms	1296.2 ms	111.3 ms / 7.9%

All 19 configurations completed. The request-inclusive results retain 111–131 ms of savings, supporting reduced work across the process lifetime. Pyprof and module inventories show 60 fewer ddtrace modules loaded by auto-instrumentation.

Targeted Python 3.12 tests passed for LLMObs, debugging, AppSec, tracing, Flask, symbol uploads, remote configuration, and fork/shutdown. New tests cover cold activation, config removal, manual activation, concurrent snapshot initialization, and activation/shutdown races. Formatting, lint, typing, suite coverage, and benchmark ownership checks passed; no new import cycles or dependency violations.

Risks

Changes product initialization order. Activation, shutdown, and fork behavior need coverage across supported Python/platform combinations.

Additional Notes

Independent draft PR. Startup benchmarks passed on the current head; the full CI pipeline is still running. The local 19-configuration comparison also completed, but variation in unchanged controls makes its timings inconclusive; the table uses CI measurements.

[cit-pr-commenter-54b7da]

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 104 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 104 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=122)
ddtrace.llmobs._integrations.pydantic_ai -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=120)
ddtrace.llmobs._integrations.vertexai -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=120)
ddtrace.llmobs._integrations.llama_index -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=120)
ddtrace.llmobs._integrations.base -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=120)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

[cit-pr-commenter-54b7da]

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

.agents/skills/apm-integrations/references/design-decisions.md          @DataDog/python-guild @DataDog/apm-idm-python
.gitlab/benchmarks/slos/apm-core-python.yml                             @DataDog/apm-core-python
benchmarks/startup/config.yaml                                          @DataDog/apm-core-python
benchmarks/startup/scenario.py                                          @DataDog/apm-core-python
ddtrace/appsec/_asm_request_context.py                                  @DataDog/asm-python
ddtrace/appsec/_utils.py                                                @DataDog/asm-python
ddtrace/contrib/internal/trace_utils_base.py                            @DataDog/python-guild @DataDog/apm-idm-python
ddtrace/debugging/__init__.py                                           @DataDog/debugger-python
ddtrace/debugging/_origin/_span_snapshot.py                             @DataDog/debugger-python
ddtrace/debugging/_origin/span.py                                       @DataDog/debugger-python
ddtrace/debugging/_products/code_origin/span.py                         @DataDog/debugger-python
ddtrace/debugging/_products/dynamic_instrumentation.py                  @DataDog/debugger-python
ddtrace/debugging/_products/exception_replay.py                         @DataDog/debugger-python
ddtrace/debugging/_products/live_debugger.py                            @DataDog/debugger-python
ddtrace/ext/git.py                                                      @DataDog/ci-app-libraries @DataDog/apm-core-python
ddtrace/internal/README.md                                              @DataDog/apm-core-python
ddtrace/internal/_unpatched.py                                          @DataDog/python-guild
ddtrace/internal/products.py                                            @DataDog/apm-core-python
ddtrace/internal/symbol_db/remoteconfig.py                              @DataDog/debugger-python
ddtrace/internal/utils/http.py                                          @DataDog/apm-core-python
ddtrace/llmobs/__init__.py                                              @DataDog/ml-observability
ddtrace/llmobs/_evaluators/__init__.py                                  @DataDog/ml-observability
ddtrace/llmobs/_product.py                                              @DataDog/ml-observability
releasenotes/notes/reduce-disabled-product-startup-overhead-59624438644093e9.yaml  @DataDog/apm-python
tests/contrib/flask/test_startup.py                                     @DataDog/python-guild @DataDog/apm-idm-python
tests/debugging/origin/test_span.py                                     @DataDog/debugger-python
tests/debugging/test_sampling.py                                        @DataDog/debugger-python
tests/internal/test_products.py                                         @DataDog/apm-core-python
tests/llmobs/test_llmobs_rc.py                                          @DataDog/ml-observability

[datadog-datadog-us1-prod]

 

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a77a6ae | Docs | View more details | Give us feedback!}

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-11-03 14:30:47

Comparing candidate commit a77a6ae in PR branch codex/startup-disabled-products with baseline commit b2fc819 in branch main.

📊 Benchmarking dashboard

Found 11 performance improvements and 0 performance regressions! Performance is the same for 600 metrics, 10 unstable metrics, 1 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:startup-ddtrace_run

• 🟩 execution_time [-146.296ms; -143.294ms] or [-12.147%; -11.898%]

scenario:startup-ddtrace_run_appsec

• 🟩 execution_time [-138.083ms; -134.469ms] or [-11.163%; -10.871%]

scenario:startup-ddtrace_run_flask_requests

• 🟩 execution_time [-117.279ms; -105.357ms] or [-8.332%; -7.485%]

scenario:startup-ddtrace_run_profiling

• 🟩 execution_time [-146.243ms; -141.713ms] or [-11.706%; -11.344%]

scenario:startup-ddtrace_run_runtime_metrics

• 🟩 execution_time [-136.225ms; -131.635ms] or [-11.432%; -11.047%]

scenario:startup-ddtrace_run_telemetry_disabled

• 🟩 execution_time [-141.280ms; -137.999ms] or [-12.316%; -12.030%]

scenario:startup-ddtrace_run_telemetry_enabled

• 🟩 execution_time [-139.790ms; -136.404ms] or [-11.695%; -11.411%]

scenario:startup-import_ddtrace_auto

• 🟩 execution_time [-147.200ms; -144.310ms] or [-17.601%; -17.255%]

scenario:startup-import_ddtrace_auto_django

• 🟩 execution_time [-160.021ms; -156.610ms] or [-16.785%; -16.427%]

scenario:startup-import_ddtrace_auto_flask

• 🟩 execution_time [-139.297ms; -135.959ms] or [-13.025%; -12.713%]

scenario:startup-import_ddtrace_auto_flask_requests

• 🟩 execution_time [-134.163ms; -127.169ms] or [-12.672%; -12.011%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

• unstable execution_time [-712.250ns; +737.308ns] or [-6.619%; +6.852%]

scenario:coreapiscenario-core_dispatch_1_listener

• unstable execution_time [-31.634ns; +39.328ns] or [-4.602%; +5.721%]

scenario:coreapiscenario-core_dispatch_50_listeners

• unstable execution_time [-1943.749ns; +1860.147ns] or [-9.743%; +9.324%]

scenario:coreapiscenario-core_dispatch_exception_listeners

• unstable execution_time [-1977.947ns; +1697.872ns] or [-10.227%; +8.779%]

scenario:coreapiscenario-core_dispatch_listeners

• unstable execution_time [-386.727ns; +368.474ns] or [-8.861%; +8.442%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

• unstable execution_time [-225.943ns; +223.316ns] or [-8.367%; +8.270%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

• unstable execution_time [-91.428ns; +86.561ns] or [-6.451%; +6.108%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

• unstable execution_time [-4962.591ns; +4597.417ns] or [-9.889%; +9.161%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

• unstable execution_time [-995.459ns; +970.308ns] or [-9.351%; +9.115%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

• unstable execution_time [-61.485µs; +52.120µs] or [-10.466%; +8.872%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

Known flaky benchmarks without significant changes:

• scenario:packagespackageforrootmodulemapping-cache_off