Skip to content

feat: add end-to-end ML workflow tutorial notebook and docs - #728

Open
prakharsingh-74 wants to merge 30 commits into
kubeflow:mainfrom
prakharsingh-74:E2E-workflow-tutorial
Open

feat: add end-to-end ML workflow tutorial notebook and docs#728
prakharsingh-74 wants to merge 30 commits into
kubeflow:mainfrom
prakharsingh-74:E2E-workflow-tutorial

Conversation

@prakharsingh-74

Copy link
Copy Markdown
Contributor

fixes: #258

Description

This PR introduces a unified end-to-end ML workflow tutorial to demonstrate how the various clients in the Kubeflow SDK (OptimizerClient, TrainerClient, and ModelRegistryClient) integrate.

Why is this needed?

Currently, each SDK client is documented in isolation. Providing a cohesive workflow tutorial helps users understand the intended developer experience of utilizing the SDK as a unified ML platform.

Changes

  1. Tutorial Notebook: Added examples/end-to-end-tutorial.ipynb which walks through:
    • Setting up hyperparameter optimization (Katib) via OptimizerClient.
    • Executing final training (Trainer) with the optimal parameters via TrainerClient.
    • Registering the trained model version/artifacts via ModelRegistryClient (includes a mock fallback for robust E2E test execution).
  2. Sphinx Docs: Created a dedicated tutorial page docs/source/getting-started/end-to-end-workflow.rst, linked it from the getting-started index cards, and registered it in the main toctree index.
  3. E2E Workflow Configuration: Updated .github/workflows/test-e2e.yaml to install the hub dependency extra and execute the new notebook as part of E2E verification.

Verification

  • Ran uv run ruff check and uv run ruff format to ensure styling guidelines are met.
  • Ran uv run ty check kubeflow/hub to verify type safety.
  • Ran uv run sphinx-build -b html docs/source docs/_build/html to verify the documentation builds successfully without new warnings or compilation issues.

Signed-off-by: prakharsingh-74 <prakharsingh7014@gmail.com>
@review-notebook-app

Copy link
Copy Markdown

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@google-oss-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kramaranya for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@prakharsingh-74

Copy link
Copy Markdown
Contributor Author

@andreyvelich Can you please review this PR?
No updates on the previous PR so i have taken this issue

@prakharsingh-74 prakharsingh-74 changed the title Add end-to-end ML workflow tutorial notebook and docs docs: add end-to-end ML workflow tutorial notebook and docs Aug 6, 2026
@prakharsingh-74 prakharsingh-74 changed the title docs: add end-to-end ML workflow tutorial notebook and docs feat: add end-to-end ML workflow tutorial notebook and docs Aug 6, 2026

@andreyvelich andreyvelich left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great, thank you for working on it @prakharsingh-74!
It might be also good to add example with PipelinesClient() that runs
Spark Data Processing -> PyTorch Training -> HP Tuning -> Model Registration.
Do we prefer to add another Notebook for it?

cc @MStokluska @kramaranya @Fiona-Waters @tariq-hasan

/ok-to-test

Signed-off-by: prakharsingh-74 <prakharsingh7014@gmail.com>
Signed-off-by: prakharsingh-74 <prakharsingh7014@gmail.com>
@google-oss-prow google-oss-prow Bot added size/XXL and removed size/XL labels Aug 8, 2026
Signed-off-by: prakharsingh-74 <prakharsingh7014@gmail.com>
Signed-off-by: prakharsingh-74 <prakharsingh7014@gmail.com>
@prakharsingh-74
prakharsingh-74 force-pushed the E2E-workflow-tutorial branch 2 times, most recently from 14c2e88 to 7c20ec0 Compare August 8, 2026 08:44
Signed-off-by: prakharsingh-74 <prakharsingh7014@gmail.com>
Signed-off-by: prakharsingh-74 <prakharsingh7014@gmail.com>
Signed-off-by: prakharsingh-74 <prakharsingh7014@gmail.com>
…aining functions

Signed-off-by: prakharsingh-74 <prakharsingh7014@gmail.com>
Signed-off-by: prakharsingh-74 <prakharsingh7014@gmail.com>
Signed-off-by: prakharsingh-74 <prakharsingh7014@gmail.com>
@prakharsingh-74

Copy link
Copy Markdown
Contributor Author

@andreyvelich sir, can you guide me for resolving the CI fails which are occuring on adding of KFP control plane?

@andreyvelich

Copy link
Copy Markdown
Member

@HumairAK @kramaranya @MStokluska @mprahl @alyssacgoins would you be able to help @prakharsingh-74 correctly install KFP control plane in our CI?

@alyssacgoins

Copy link
Copy Markdown

Hey @prakharsingh-74 you’ll want to use manifests/kustomize/env/platform-agnostic instead of manifests/kustomize/env/dev, and you can also bump the version up to 2.17.0 from of 2.2.0

Comment thread .github/workflows/test-e2e.yaml Outdated
@prakharsingh-74

Copy link
Copy Markdown
Contributor Author

Thanks for the mentorship, @alyssacgoins!

Ready for the review and merge, @andreyvelich.

Also, if there are any more issues available, feel free to assign or point me to them. I’ll be happy to take them up and solve them.

@alyssacgoins

Copy link
Copy Markdown

/lgtm

@google-oss-prow google-oss-prow Bot removed the lgtm label Aug 14, 2026
Signed-off-by: prakharsingh-74 <prakharsingh7014@gmail.com>
@prakharsingh-74

Copy link
Copy Markdown
Contributor Author

@andreyvelich Sir, PR is ready for the final look

@alyssacgoins

Copy link
Copy Markdown

/lgtm

@google-oss-prow google-oss-prow Bot added the lgtm label Aug 18, 2026
Comment thread examples/pipelines-end-to-end-tutorial.ipynb Outdated
@google-oss-prow google-oss-prow Bot removed the lgtm label Aug 25, 2026
@google-oss-prow

Copy link
Copy Markdown
Contributor

New changes are detected. LGTM label has been removed.

Signed-off-by: prakharsingh-74 <prakharsingh7014@gmail.com>
Signed-off-by: prakharsingh-74 <prakharsingh7014@gmail.com>
Signed-off-by: prakharsingh-74 <prakharsingh7014@gmail.com>
Comment thread docs/source/getting-started/pipelines-end-to-end-workflow.rst Outdated
Signed-off-by: prakharsingh-74 <prakharsingh7014@gmail.com>
@prakharsingh-74

Copy link
Copy Markdown
Contributor Author

@alyssacgoins can you please check why CI are getting failed again?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add End-to-End Workflow Tutorial

3 participants