From acfd969ce5cbb33ce83a8a385a090c4b19a0bc4c Mon Sep 17 00:00:00 2001 From: Jim Bennett Date: Wed, 22 Jul 2026 18:33:19 -0700 Subject: [PATCH] docs: migrate quickstart-evals notebook to current phoenix-evals API Update the evaluations quickstart off the deprecated v7 stack (ArizeExportClient/export_model_to_df, OpenAIModel, llm_classify, log_evaluations_sync) onto the current API: - export spans via ArizeClient().spans.export_to_df - build the judge with phoenix.evals LLM + create_classifier - run with evaluate_dataframe - flatten with to_annotation_dataframe and log via client.spans.update_evaluations Keeps the notebook in sync with the AX docs evaluations quickstart. Co-Authored-By: Claude Opus 4.8 (1M context) --- python/llm/evaluation/quickstart-evals.ipynb | 156 ++----------------- 1 file changed, 12 insertions(+), 144 deletions(-) diff --git a/python/llm/evaluation/quickstart-evals.ipynb b/python/llm/evaluation/quickstart-evals.ipynb index d237db7..d7b7356 100644 --- a/python/llm/evaluation/quickstart-evals.ipynb +++ b/python/llm/evaluation/quickstart-evals.ipynb @@ -5,35 +5,7 @@ "metadata": { "id": "SLdlB9yTCDxG" }, - "source": [ - "
\n", - "

\n", - " \"arize\n", - "
\n", - " Docs\n", - " |\n", - " GitHub\n", - " |\n", - " Slack Community\n", - "

\n", - "
\n", - "\n", - "

Evaluations Quickstart

\n", - "\n", - "## Overview\n", - "Evaluations are essential to understanding how well your model is performing in real-world scenarios, allowing you to identify strengths, weaknesses, and areas of improvement.\n", - "\n", - "Offline evaluations are run as code and then sent back to Arize using `log_evaluations_sync`.\n", - "\n", - "This guide assumes you have traces in Arize and are looking to run an evaluation to measure your application performance.\n", - "\n", - "To add evaluations you can set up online evaluations as a task to run automatically, or you can follow the steps below to generate evaluations and log them to Arize:\n", - "\n", - "1. Install the Arize SDK\n", - "2. Import your spans in code\n", - "3. Run a custom evaluator using Phoenix\n", - "4. Log evaluations back to Arize" - ] + "source": "
\n

\n \"arize\n
\n Docs\n |\n GitHub\n |\n Slack Community\n

\n
\n\n

Evaluations Quickstart

\n\n## Overview\nEvaluations are essential to understanding how well your model is performing in real-world scenarios, allowing you to identify strengths, weaknesses, and areas of improvement.\n\nOffline evaluations are run as code and then sent back to Arize using `client.spans.update_evaluations`.\n\nThis guide assumes you have traces in Arize and are looking to run an evaluation to measure your application performance.\n\nTo add evaluations you can set up online evaluations as a task to run automatically, or you can follow the steps below to generate evaluations and log them to Arize:\n\n1. Install the Arize SDK\n2. Import your spans in code\n3. Run a custom evaluator using Phoenix Evals\n4. Log evaluations back to Arize" }, { "cell_type": "markdown", @@ -49,11 +21,7 @@ "execution_count": null, "metadata": {}, "outputs": [], - "source": [ - "!pip install -q arize arize-phoenix-evals\n", - "\n", - "!pip install -q openai pandas nest_asyncio" - ] + "source": "!pip install -q 'arize>=8.0.0' arize-phoenix-evals\n\n!pip install -q openai pandas" }, { "cell_type": "markdown", @@ -101,64 +69,21 @@ "execution_count": null, "metadata": {}, "outputs": [], - "source": [ - "# import statements required for getting your spans\n", - "from datetime import datetime, timedelta\n", - "from arize.exporter import ArizeExportClient\n", - "from arize.utils.types import Environments\n", - "\n", - "start_time = datetime.now() - timedelta(days=14) # 14 days ago\n", - "end_time = datetime.now() # Today\n", - "\n", - "# Exporting your dataset into a dataframe\n", - "client = ArizeExportClient(api_key=API_KEY)\n", - "primary_df = client.export_model_to_df(\n", - " space_id=os.environ[\"SPACE_ID\"],\n", - " model_id=\"tracing-haiku-tutorial\", # change this to the name of your project\n", - " environment=Environments.TRACING,\n", - " start_time=start_time,\n", - " end_time=end_time,\n", - ")" - ] + "source": "# import statements required for getting your spans\nfrom datetime import datetime, timedelta, timezone\nfrom arize import ArizeClient\n\nstart_time = datetime.now(timezone.utc) - timedelta(days=14) # 14 days ago\nend_time = datetime.now(timezone.utc) # Today\n\n# Exporting your spans into a dataframe\nclient = ArizeClient(api_key=API_KEY)\nprimary_df = client.spans.export_to_df(\n space_id=SPACE_ID,\n project_name=\"tracing-haiku-tutorial\", # change this to the name of your project\n start_time=start_time,\n end_time=end_time,\n)" }, { "cell_type": "markdown", "metadata": { "id": "jd7GFiuoCDxM" }, - "source": [ - "## Run a custom evaluator using Phoenix\n", - "\n", - "Create a prompt template for the LLM to judge the quality of your responses. You can utilize any of the Arize Evaluator Templates or you can create your own. Below is an example which judges the positivity or negativity of the LLM output." - ] + "source": "## Run a custom evaluator using Phoenix Evals\n\nCreate a classifier for the LLM to judge the quality of your responses. You can utilize any of the Arize Evaluator Templates or you can create your own. Below is an example which judges the positivity or negativity of the LLM output. `create_classifier` relies on the judge's tool-calling / structured-output support, so use a non-reasoning model such as GPT-4.1." }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], - "source": [ - "import os\n", - "from phoenix.evals import OpenAIModel, llm_classify\n", - "\n", - "eval_model = OpenAIModel(\n", - " model=\"gpt-4o\", temperature=0, api_key=os.environ[\"OPENAI_API_KEY\"]\n", - ")\n", - "\n", - "MY_CUSTOM_TEMPLATE = \"\"\"\n", - " You are evaluating the positivity or negativity of the responses to questions.\n", - " [BEGIN DATA]\n", - " ************\n", - " [Question]: {input}\n", - " ************\n", - " [Response]: {output}\n", - " [END DATA]\n", - "\n", - "\n", - " Please focus on the tone of the response.\n", - " Your answer must be single word, either \"positive\" or \"negative\"\n", - " \"\"\"" - ] + "source": "from phoenix.evals import LLM, create_classifier\n\njudge = LLM(provider=\"openai\", model=\"gpt-4.1\")\n\nMY_CUSTOM_TEMPLATE = \"\"\"\n You are evaluating the positivity or negativity of the responses to questions.\n [BEGIN DATA]\n ************\n [Question]: {input}\n ************\n [Response]: {output}\n [END DATA]\n\n\n Please focus on the tone of the response.\n Your answer must be single word, either \"positive\" or \"negative\"\n \"\"\"\n\ntone_eval = create_classifier(\n name=\"tone_eval\",\n prompt_template=MY_CUSTOM_TEMPLATE,\n llm=judge,\n choices={\"positive\": 1.0, \"negative\": 0.0},\n direction=\"maximize\",\n)" }, { "cell_type": "markdown", @@ -204,28 +129,14 @@ "metadata": { "id": "-GyksRt_CDxN" }, - "source": [ - "Use the `llm_classify` function to run the evaluation using your custom template. You will be using the dataframe from the traces you generated above. We also add `nest_asyncio` to run the evaluations concurrently (if you are running multiple evaluations)." - ] + "source": "Use the `evaluate_dataframe` function to run the evaluation using your classifier. You will be using the dataframe from the traces you generated above. It runs the judge calls concurrently and returns the dataframe with a `tone_eval_score` column holding the label, score, and explanation." }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], - "source": [ - "import nest_asyncio\n", - "\n", - "nest_asyncio.apply()\n", - "\n", - "evals_df = llm_classify(\n", - " dataframe=primary_df,\n", - " template=MY_CUSTOM_TEMPLATE,\n", - " model=eval_model,\n", - " rails=[\"positive\", \"negative\"],\n", - " provide_explanation=True,\n", - ")" - ] + "source": "from phoenix.evals import evaluate_dataframe\n\nresults_df = evaluate_dataframe(dataframe=primary_df, evaluators=[tone_eval])" }, { "cell_type": "markdown", @@ -241,71 +152,28 @@ "metadata": { "id": "bfeISZDNCDxN" }, - "source": [ - "# Log evaluations back to Arize\n", - "Use the `log_evaluations_sync` function as part of our Python SDK to attach evaluations you've run to traces. The code below assumes that you have already completed an evaluation run, and you have the `evals_dataframe` object. It also assumes you have a `traces_dataframe` object to get the `span_id` that you need to attach the evals.\n", - "\n", - "The `evals_dataframe` requires four columns, which should be auto-generated for you based on the evaluation you ran using Phoenix. The `` must be alphanumeric and cannot have hyphens or spaces.\n", - "- `eval..label`\n", - "- `eval..score`\n", - "- `eval..explanation`\n", - "- `context.span_id`\n", - "\n", - "An example evaluation data dictionary would look like:\n", - "```python\n", - "evaluation_data = {\n", - " 'context.span_id': ['74bdfb83-a40e-4351-9f41-19349e272ae9'], # Use your span_id\n", - " 'eval.myeval.label': ['accuracy'], # Example label name\n", - " 'eval.myeval.score': [0.95], # Example label value\n", - " 'eval.myeval.explanation': [\"some explanation\"]\n", - "}\n", - "evaluation_df = pd.DataFrame(evaluation_data)\n", - "```" - ] + "source": "# Log evaluations back to Arize\nUse the `update_evaluations` method on the Arize SDK client to attach the evaluations you've run to traces. It requires four columns, and the `` must be alphanumeric and cannot have hyphens or spaces.\n- `eval..label`\n- `eval..score`\n- `eval..explanation`\n- `context.span_id`" }, { "cell_type": "markdown", "metadata": { "id": "pUWLwKDOCDxN" }, - "source": [ - "Here is sample code to log the evaluations back to Arize. The API reference can be found here." - ] + "source": "`to_annotation_dataframe` flattens the nested `tone_eval_score` column into `label`, `score`, and `explanation` while preserving `context.span_id` (the join key `export_to_df` already provides). Rename those to the reserved `eval..*` columns and upload with `client.spans.update_evaluations`. The API reference can be found here." }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], - "source": [ - "evals_df[\"eval.tone_eval.label\"] = evals_df[\"label\"]\n", - "evals_df[\"eval.tone_eval.explanation\"] = evals_df[\"explanation\"]\n", - "evals_df.head()" - ] + "source": "from phoenix.evals.utils import to_annotation_dataframe\n\nannotations = to_annotation_dataframe(dataframe=results_df)\n\nevals_df = annotations.rename(\n columns={\n \"label\": \"eval.tone_eval.label\",\n \"score\": \"eval.tone_eval.score\",\n \"explanation\": \"eval.tone_eval.explanation\",\n }\n)[\n [\n \"context.span_id\",\n \"eval.tone_eval.label\",\n \"eval.tone_eval.score\",\n \"eval.tone_eval.explanation\",\n ]\n]\nevals_df.head()" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], - "source": [ - "import os\n", - "from arize.pandas.logger import Client\n", - "\n", - "ARIZE_API_KEY = os.environ.get(\"ARIZE_API_KEY\")\n", - "SPACE_ID = os.environ.get(\"SPACE_ID\")\n", - "\n", - "# Initialize Arize client to log evaluations\n", - "arize_client = Client(\n", - " space_id=SPACE_ID, api_key=ARIZE_API_KEY\n", - ")\n", - "\n", - "# Set the evals_df to have the correct span ID to log it to Arize\n", - "evals_df[\"context.span_id\"] = primary_df[\"context.span_id\"]\n", - "\n", - "# send the eval to Arize\n", - "arize_client.log_evaluations_sync(evals_df, \"tracing-haiku-tutorial\")" - ] + "source": "# send the evals to Arize\nclient.spans.update_evaluations(\n space_id=SPACE_ID,\n project_name=\"tracing-haiku-tutorial\", # your project name\n dataframe=evals_df,\n)" } ], "metadata": { @@ -315,4 +183,4 @@ }, "nbformat": 4, "nbformat_minor": 0 -} +} \ No newline at end of file