How to use SlicedClassificationMetrics

[asteroth]

Hope it's the right place to ask the question like that but I couldn't find any reference example anywhere on the net.
I want to use the sliced metrics and for the life of mine I can't figure out how are they supposed to be used.

kfp.version >>>> '2.15.2'

Running in a pipeline

I expected the sliced metrics to use the same syntax as the normal metrics:

@dsl.component(base_image=DEFAULT_IMAGE, packages_to_install=["numpy"])
def evaluate(model_in: dsl.Input[dsl.Model]
         , slicer: dsl.Output[dsl.SlicedClassificationMetrics]):
    import numpy as np
    # simulate doing something with a model:
    with open(model_in.path, "rb") as fin:
        model = np.load(fin)
    model[1, 1] += 1

    # simulate per batch metric:
    for x in range(100):
        randf = np.random.random()
        slicer.log_roc_data_point("potato", threshold=1/(x+1), tpr=randf, fpr=np.cos(randf))
        slicer.log_roc_data_point("strawberry", threshold=1/(x+1), tpr=randf, fpr=np.sin(randf))

the pipeline fails with an attribute error.

I have a feeling I need to define the SlicedClassificationMetrics somewhere else first and make it both an input and an output to pass it from there but I just can't figure out how is it supposed to work exactly.

Trying to just use a dummy locally:

I looked at the test code in the pipelines wiki and it appears this should work:

import kfp
metrics = kfp.dsl.SlicedClassificationMetrics()
metrics.log_roc_data_point("slice1", threshold=0.12, tpr=27.5, fpr=86.32)
#  autocomplete suggests only metrics.log_roc_reading exists, but it also gives me attribute error

but it gives me AttributeError: 'SlicedClassificationMetrics' object has no attribute '_sliced_metrics' indicating the constructor does not build a complete object even though the arguments are all flagged as optional.

metrics._sliced_metrics = {}  # add an empty dict just to see what happens
# neither works:
metrics.log_roc_reading("slice1", threshold=1.0, tpr=26.5, fpr=88.4)
metrics.log_roc_data_point("potato", threshold=1.0, tpr=26.5, fpr=88.0)

[farfromexact]

The immediate problem is that SlicedClassificationMetrics has a different API from ClassificationMetrics. log_roc_data_point() is the method for the non-sliced artifact; a sliced artifact uses log_roc_reading() and includes the slice name.

Your component should look like this:

@dsl.component(base_image=DEFAULT_IMAGE, packages_to_install=["numpy"])
def evaluate(
    model_in: dsl.Input[dsl.Model],
    slicer: dsl.Output[dsl.SlicedClassificationMetrics],
):
    import numpy as np

    with open(model_in.path, "rb") as fin:
        model = np.load(fin)

    for x in range(100):
        threshold = 1 / (x + 1)
        randf = float(np.random.random())
        slicer.log_roc_reading(
            "potato", threshold=threshold, tpr=randf, fpr=float(np.cos(randf))
        )
        slicer.log_roc_reading(
            "strawberry", threshold=threshold, tpr=randf, fpr=float(np.sin(randf))
        )

The output artifact is injected by the component runtime. Do not instantiate SlicedClassificationMetrics() in a local script and do not set _sliced_metrics yourself; that is an internal implementation detail. If you test locally, the constructor may not be fully initialized because it is normally used as an injected output artifact.

The current SDK reference documents log_roc_reading(slice, threshold, tpr, fpr) here: https://kubeflow-pipelines.readthedocs.io/en/latest/source/dsl.html. Also make sure the rates are in the expected numeric range rather than percentages. log_roc_data_point() is the reason for the attribute error in the pipeline example.

[asteroth]

Thanks for reply - I tried running it in the pipelines on a proper deployment and get an error in the component log:

AttributeError: 'SlicedClassificationMetrics' object has no attribute '_sliced_metrics'

I made a minimal self contained example:

@dsl.component(base_image=DEFAULT_IMAGE)
def evaltest(model_in: str
         , slicer: dsl.Output[dsl.SlicedClassificationMetrics]):
    import numpy as np
    # simulate doing something with a model:
    print(f"Model dummy: {model_in}")

    # simulate per batch metric:
    for x in range(100):
        threshold = 1 / (x + 1)
        randf = float(np.random.random())
        slicer.log_roc_reading(
            "potato", threshold=threshold, tpr=randf, fpr=float(np.cos(randf))
        )
        slicer.log_roc_reading(
            "strawberry", threshold=threshold, tpr=randf, fpr=float(np.sin(randf))
        )


from kfpipe.components import evaltest
@dsl.pipeline(name="input", description="figure out sliced metric")
def test01():
    t1 = evaltest(model_in="dummy")
    t1.set_caching_options(False)
    

kfp_client_manager = kfp_client.KFPClientManager(<connection params>)
client = kfp_client_manager.create_kfp_client()
client.create_run_from_pipeline_func(test01
                                     , experiment_name=os.environ.get("KFP_EXPERIMENT", "default")
                                     , namespace="project-1"
                                     )

Full error trace:

  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/lib/python3.12/site-packages/kfp/dsl/executor_main.py", line 109, in <module>
    executor_main()
  File "/usr/local/lib/python3.12/site-packages/kfp/dsl/executor_main.py", line 101, in executor_main
    output_file = executor.execute()
                  ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kfp/dsl/executor.py", line 453, in execute
    result = self.func(**func_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.w0b7sjqbDL/ephemeral_component.py", line 19, in evaltest
    slicer.log_roc_reading(
  File "/usr/local/lib/python3.12/site-packages/kfp/dsl/types/artifact_types.py", line 434, in log_roc_reading
    self._upsert_classification_metrics_for_slice(slice)
  File "/usr/local/lib/python3.12/site-packages/kfp/dsl/types/artifact_types.py", line 408, in _upsert_classification_metrics_for_slice
    if slice not in self._sliced_metrics:
                    ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'SlicedClassificationMetrics' object has no attribute '_sliced_metrics'
Model dummy: dummy
I0803 08:34:17.928133      31 launcher_v2.go:193] publish success.
I0803 08:34:17.980905      31 client.go:750] Attempting to update DAG state
F0803 08:34:18.006463      31 main.go:58] failed to execute component: exit status 1
time="2026-08-03T08:34:18.200Z" level=info msg="sub-process exited" argo=true error="<nil>"

Do I need to explicitly add something to the component container?