Add data-loading bottleneck diagnostic skill

[rostan-t]

Category:

Other (e.g. Documentation, Tests, Configuration)

Description:

Create a skill allowing agent to detect and localize data loading bottlenecks in PyTorch training.

The workflow roughly contains two important phases:

• Detection of whether or not there's a data loading bottleneck using DALI's LoaderEvaluator
• Bottleneck localization with profiling

The complete workflow is more complex and looks as follows:

1. Identify the workload, environment and if it's at all possible to run.
2. Run (bounded) training, measure loader wait. If LoaderEvaluator is or can be made importable , continue to 3. Else, go to 4.
3. Replay the training with cached batches using LoaderEvaluator's replay mode. Establish if there's a measured bottleneck. If we this steps provdes that there's no bottleneck, go to 5. Otherwise, continue to 4. to localize.
4. Run profiling to localize the bottleneck if we come from 3., or to detect it if coming from 2.
5. Report the results.

This was tested on multiple models, with different environment requirements, including simple ResNet50, OpenCLIP, LeWorldModel, and Hugging Face timm.

In terms of the model used by the agent, I found that GPT 5.6 Terra or equivalent is a sound minimum requirement to run this reliably, although it can sometimes fail to accurately follow all instructions, like the shape of the report.

I unfortunately couldn't commit evaluations as the skill CI doesn't support GPU environments.

Additional information:

Affected modules and functionalities:

Skill created.

Key points relevant for the review:

• Is the skill's workflow sound?
• Can the skill miss an existing bottlenecks or detect one where there isn't?
• Is the skill too defensive on things like requiring equivalent work for replay?

Note that if you run locally, depending on the workload, it may take time to complete.

Tests:

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Skill CI doesn't support GPU workers yet

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other (skill)
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: DALI-4796

[greptile-apps]

Greptile Summary

The PR adds an agent skill for diagnosing PyTorch data-loading bottlenecks through preflight checks, bounded Real/Replay comparisons, and Nsight Systems profiling.

• Adds workflow instructions, profiling and DALI replay references, and a structured report template.
• Adds environment/evaluation definitions for input-bound, compute-bound, and out-of-scope workloads.
• Adds scripts for collecting environment readiness and summarizing Nsight trace data.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
skills/data-loading-bottleneck/SKILL.md	Defines the end-to-end diagnostic routing, equivalence requirements, verdict thresholds, and reporting contract.
skills/data-loading-bottleneck/references/profiling.md	Documents production-path NVTX instrumentation, Nsight capture validation, and evidence requirements for localization.
skills/data-loading-bottleneck/references/pytorch-dali.md	Documents bounded LoaderEvaluator construction and Real/Replay work-equivalence checks.
skills/data-loading-bottleneck/scripts/collect_preflight.py	Collects workload, environment, GPU, dependency, filesystem, and Git readiness into a preflight artifact.
skills/data-loading-bottleneck/scripts/summarize_nsys.py	Exports and summarizes the selected NVTX window, CUDA activity, worker coverage, and loader-wait overlap.
skills/data-loading-bottleneck/evals/evals.json	Defines representative input-bound, compute-bound, and inference scenarios for evaluating skill behavior.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Preflight workload and environment] --> B{Workload runnable?}
    B -- No --> I[Report INCONCLUSIVE]
    B -- Yes --> C[Run bounded Real workload]
    C --> D{LoaderEvaluator available?}
    D -- Yes --> E[Run equivalent Replay]
    E --> F{Replay speedup}
    F -- <= 1.10x --> N[Report NOT DETECTED]
    F -- > 1.10x --> P[Profile production path]
    D -- No --> P
    P --> G{Validated input-path delay?}
    G -- Yes --> H[Report DETECTED or POTENTIAL and localize]
    G -- No --> I

Loading

_{Reviews (3): Last reviewed commit: "Add eval suite for data-loading bottlene..." | Re-trigger Greptile}

[JanuszL]

/nvskills-ci

[JanuszL]

/nvskills-ci

[mdabek-nvidia]

Is this useful only for only pyTorch based training?

[rostan-t]

Partly, yes. The LoaderEvaluator tool assumes PyTorch's DataLoader. Most of the guidance was written with PyTorch in mind although a lot of it is library-agnostic.

I'd need to test with other frameworks to identify possible gaps and I'd rather leave it for a future version.

[mdabek-nvidia]

Is the intent to update version number manually? Can it re-use DALI versioning ?

[rostan-t]

Since skills are distributed independently from DALI (consumed by NVIDIA/skills), I wouldn't be sure what version to use here. Should it be 2.3.0 or the current version in VERSION 2.4.0dev? Do we open a PR to update the skill versions each time we update DALI?

[mdabek-nvidia]

The skill assumes that the platform that we run the preflight is configured so that it can run training.
Would it be possible to extend it to create virtual environment and execute pip install -f requiremnets.txt if such file exist?

[rostan-t]

I think it's generally safe to assume that somebody interested in evaluating if training performance is bottlenecked by data loading has a working environment. We want to use this environment with the exact pinned library versions they are using. Those might not be the same as a potential requirements.txt file or it might not tell the full picture because it's not uncommon for projects to have other files like requirements.dev.txt.

Additionally, supporting requirements.txt only would be incomplete, pyproject.toml is also often used (with e.g. uv or poetry), some projects have a conda configuration instead, others rely on Docker containers, etc.

[mdabek-nvidia]

Wouldn't it be better to create a venv?

[rostan-t]

We want to reuse the user's env if it exist and not add a new library that they didn't request in it. pip install --target allows installing a library in a custom location for temporary use and the user doesn't have to even know about it.

[mdabek-nvidia]

Would it be possible to add a graph (run a python script), that would show how much improvement we get for specific batch sizes and number or workers?

[rostan-t]

Initially, the skill had instructions to explore solutions for detected issues (like changing the number of workers). It ended up increasing the (already long) time needed to run the skill because the agent could also need to handle OOMs created by increasing the number of workers, etc.
I preferred keeping it focused on identification and attribution, with suggestions that users can try (or ask the agent to try) if they want to.

[mdabek-nvidia]

I guess this could be a skill, but I understand that we need 100% reproducibility, correct?

[rostan-t]

The reason for having a script here is that I found that less capable models could have a hard time properly summarizing, even when given proper instructions. This would also consume a lot of unnecessary tokens, especially when agents would loop over failures such as trying why there SQL queries with joins over four tables didn't work the first time.