Skip to content

Rewrite kernel profiler for correct tree nesting and GPU timing - #3694

Draft
Alexandr-Solovev wants to merge 1 commit into
uxlfoundation:mainfrom
Alexandr-Solovev:dev/asolovev_profiler_ai
Draft

Rewrite kernel profiler for correct tree nesting and GPU timing#3694
Alexandr-Solovev wants to merge 1 commit into
uxlfoundation:mainfrom
Alexandr-Solovev:dev/asolovev_profiler_ai

Conversation

@Alexandr-Solovev

Copy link
Copy Markdown
Contributor

Rebuild the profiler tree from per-thread stacks tracked in the singleton instead of a global level counter. Regular tasks now record a parent index at push time (the caller thread's innermost open task, with fallback to the innermost regular task across threads for worker-thread threading tasks); level derives from parent + 1. This fixes the case where a task opened on one thread and closed on another (or threading tasks under a regular parent) produced garbled nesting.

Merge phase now keys sibling collapse on (parent_idx, name) via an unordered_map, so it is O(n) and correctly matches only true siblings. Levels-and-names on their own were ambiguous: two tasks at the same depth under different parents could collapse into one.

Cache ONEDAL_VERBOSE once per process in a function-local static instead of a namespace-scope volatile int. The previous form gave every TU that included the header its own copy, so daal_verbose_val observed different values across .so boundaries.

Move the once-per-process suppression sets for the threading logger and tracer messages onto the profiler singleton so their lifetimes are bounded to the profiler, not the process; hoist unique-name checks under the existing mutex.

On the oneAPI side, replace the start_task(name, queue) pattern with a queue_sync_guard RAII wrapper. The guard issues wait_and_throw() before start_task() and again on scope exit so recorded durations reflect device-side execution rather than host enqueue. The pre-region wait happens before the timer, so already-in-flight work on the queue is not charged to the region. When the profiler is disabled the guard is a nullptr no-op and no waits are issued.

Uniquify the local variable name via LINE so multiple profiled scopes in the same block coexist; drop the ternary-of-mixed-types pattern and gate each START_TASK* macro on the enable predicate directly. SERVICE task variants gate their logger side-effects on is_service_debug_enabled (ONEDAL_VERBOSE=5) rather than is_logger_enabled (=1/4), matching the pre-refactor behavior — otherwise hot paths like table2ndarray would flood the output under LOGGER mode.

Validated: make onedal_dpc -j112 clean; CPU + GPU cov_dense_batch and kmeans_lloyd_dense_batch under ONEDAL_VERBOSE=3 produce well-formed analyzer trees with correct parent-child nesting and durations.

Description


Checklist:

Completeness and readability

  • I have commented my code, particularly in hard-to-understand areas.
  • I have updated the documentation to reflect the changes or created a separate PR with updates and provided its number in the description, if necessary.
  • Git commit message contains an appropriate signed-off-by string (see CONTRIBUTING.md for details).
  • I have resolved any merge conflicts that might occur with the base branch.

Testing

  • I have run it locally and tested the changes extensively.
  • All CI jobs are green or I have provided justification why they aren't.
  • I have extended testing suite if new functionality was introduced in this PR.

Performance

  • I have measured performance for affected algorithms using scikit-learn_bench and provided at least a summary table with measured data, if performance change is expected.
  • I have provided justification why performance and/or quality metrics have changed or why changes are not expected.
  • I have extended the benchmarking suite and provided a corresponding scikit-learn_bench PR if new measurable functionality was introduced in this PR.

Rebuild the profiler tree from per-thread stacks tracked in the singleton
instead of a global level counter. Regular tasks now record a parent index
at push time (the caller thread's innermost open task, with fallback to
the innermost regular task across threads for worker-thread threading
tasks); level derives from parent + 1. This fixes the case where a task
opened on one thread and closed on another (or threading tasks under a
regular parent) produced garbled nesting.

Merge phase now keys sibling collapse on (parent_idx, name) via an
unordered_map, so it is O(n) and correctly matches only true siblings.
Levels-and-names on their own were ambiguous: two tasks at the same depth
under different parents could collapse into one.

Cache ONEDAL_VERBOSE once per process in a function-local static instead
of a namespace-scope `volatile int`. The previous form gave every TU that
included the header its own copy, so `daal_verbose_val` observed different
values across .so boundaries.

Move the once-per-process suppression sets for the threading logger and
tracer messages onto the profiler singleton so their lifetimes are bounded
to the profiler, not the process; hoist unique-name checks under the
existing mutex.

On the oneAPI side, replace the `start_task(name, queue)` pattern with a
`queue_sync_guard` RAII wrapper. The guard issues `wait_and_throw()` before
`start_task()` and again on scope exit so recorded durations reflect
device-side execution rather than host enqueue. The pre-region wait
happens before the timer, so already-in-flight work on the queue is not
charged to the region. When the profiler is disabled the guard is a
nullptr no-op and no waits are issued.

Uniquify the local variable name via __LINE__ so multiple profiled scopes
in the same block coexist; drop the ternary-of-mixed-types pattern and
gate each START_TASK* macro on the enable predicate directly. SERVICE
task variants gate their logger side-effects on `is_service_debug_enabled`
(ONEDAL_VERBOSE=5) rather than `is_logger_enabled` (=1/4), matching the
pre-refactor behavior — otherwise hot paths like `table2ndarray` would
flood the output under LOGGER mode.

Validated: make onedal_dpc -j112 clean; CPU + GPU cov_dense_batch and
kmeans_lloyd_dense_batch under ONEDAL_VERBOSE=3 produce well-formed
analyzer trees with correct parent-child nesting and durations.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
}

// Reads ONEDAL_VERBOSE once per process; the C++17 function-local static gives thread-safe
// single initialization and a single storage instance across all TUs (fixes the previous

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could explain what a TU is.


#include <chrono>
#include <thread>
#include <cerrno>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are macros from this header used somewhere?


~queue_sync_guard() {
if (queue_) {
#if (!defined(DAAL_NOTHROW_EXCEPTIONS))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Throwing on a destructor is undefined behavior. It should be catched regardless of this macro.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants