Skip to content

docs: migrate quickstart-evals notebook to current phoenix-evals API #75

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jimbobbennett wants to merge 1 commit into from
Closed

docs: migrate quickstart-evals notebook to current phoenix-evals API #75

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
156 changes: 12 additions & 144 deletions python/llm/evaluation/quickstart-evals.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,35 +5,7 @@
"metadata": {
"id": "SLdlB9yTCDxG"
},
"source": [
"<center>\n",
" <p style=\"text-align:center\">\n",
" <img alt=\"arize logo\" src=\"https://storage.googleapis.com/arize-assets/arize-logo-white.jpg\" width=\"300\"/>\n",
" <br>\n",
" <a href=\"https://docs.arize.com/arize/\">Docs</a>\n",
" |\n",
" <a href=\"https://github.com/Arize-ai/client_python\">GitHub</a>\n",
" |\n",
" <a href=\"https://arize-ai.slack.com/join/shared_invite/zt-11t1vbu4x-xkBIHmOREQnYnYDH1GDfCg\">Slack Community</a>\n",
" </p>\n",
"</center>\n",
"\n",
"<center><h1>Evaluations Quickstart </h1></center>\n",
"\n",
"## Overview\n",
"Evaluations are essential to understanding how well your model is performing in real-world scenarios, allowing you to identify strengths, weaknesses, and areas of improvement.\n",
"\n",
"Offline evaluations are run as code and then sent back to Arize using `log_evaluations_sync`.\n",
"\n",
"This guide assumes you have traces in Arize and are looking to run an evaluation to measure your application performance.\n",
"\n",
"To add evaluations you can set up online evaluations as a task to run automatically, or you can follow the steps below to generate evaluations and log them to Arize:\n",
"\n",
"1. Install the Arize SDK\n",
"2. Import your spans in code\n",
"3. Run a custom evaluator using Phoenix\n",
"4. Log evaluations back to Arize"
]
"source": "<center>\n <p style=\"text-align:center\">\n <img alt=\"arize logo\" src=\"https://storage.googleapis.com/arize-assets/arize-logo-white.jpg\" width=\"300\"/>\n <br>\n <a href=\"https://docs.arize.com/arize/\">Docs</a>\n |\n <a href=\"https://github.com/Arize-ai/client_python\">GitHub</a>\n |\n <a href=\"https://arize-ai.slack.com/join/shared_invite/zt-11t1vbu4x-xkBIHmOREQnYnYDH1GDfCg\">Slack Community</a>\n </p>\n</center>\n\n<center><h1>Evaluations Quickstart </h1></center>\n\n## Overview\nEvaluations are essential to understanding how well your model is performing in real-world scenarios, allowing you to identify strengths, weaknesses, and areas of improvement.\n\nOffline evaluations are run as code and then sent back to Arize using `client.spans.update_evaluations`.\n\nThis guide assumes you have traces in Arize and are looking to run an evaluation to measure your application performance.\n\nTo add evaluations you can set up online evaluations as a task to run automatically, or you can follow the steps below to generate evaluations and log them to Arize:\n\n1. Install the Arize SDK\n2. Import your spans in code\n3. Run a custom evaluator using Phoenix Evals\n4. Log evaluations back to Arize"
},
{
"cell_type": "markdown",
Expand All @@ -49,11 +21,7 @@
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install -q arize arize-phoenix-evals\n",
"\n",
"!pip install -q openai pandas nest_asyncio"
]
"source": "!pip install -q 'arize>=8.0.0' arize-phoenix-evals\n\n!pip install -q openai pandas"
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -101,64 +69,21 @@
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# import statements required for getting your spans\n",
"from datetime import datetime, timedelta\n",
"from arize.exporter import ArizeExportClient\n",
"from arize.utils.types import Environments\n",
"\n",
"start_time = datetime.now() - timedelta(days=14) # 14 days ago\n",
"end_time = datetime.now() # Today\n",
"\n",
"# Exporting your dataset into a dataframe\n",
"client = ArizeExportClient(api_key=API_KEY)\n",
"primary_df = client.export_model_to_df(\n",
" space_id=os.environ[\"SPACE_ID\"],\n",
" model_id=\"tracing-haiku-tutorial\", # change this to the name of your project\n",
" environment=Environments.TRACING,\n",
" start_time=start_time,\n",
" end_time=end_time,\n",
")"
]
"source": "# import statements required for getting your spans\nfrom datetime import datetime, timedelta, timezone\nfrom arize import ArizeClient\n\nstart_time = datetime.now(timezone.utc) - timedelta(days=14) # 14 days ago\nend_time = datetime.now(timezone.utc) # Today\n\n# Exporting your spans into a dataframe\nclient = ArizeClient(api_key=API_KEY)\nprimary_df = client.spans.export_to_df(\n space_id=SPACE_ID,\n project_name=\"tracing-haiku-tutorial\", # change this to the name of your project\n start_time=start_time,\n end_time=end_time,\n)"
},
{
"cell_type": "markdown",
"metadata": {
"id": "jd7GFiuoCDxM"
},
"source": [
"## Run a custom evaluator using Phoenix\n",
"\n",
"Create a prompt template for the LLM to judge the quality of your responses. You can utilize any of the Arize Evaluator Templates or you can create your own. Below is an example which judges the positivity or negativity of the LLM output."
]
"source": "## Run a custom evaluator using Phoenix Evals\n\nCreate a classifier for the LLM to judge the quality of your responses. You can utilize any of the Arize Evaluator Templates or you can create your own. Below is an example which judges the positivity or negativity of the LLM output. `create_classifier` relies on the judge's tool-calling / structured-output support, so use a non-reasoning model such as GPT-4.1."
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"from phoenix.evals import OpenAIModel, llm_classify\n",
"\n",
"eval_model = OpenAIModel(\n",
" model=\"gpt-4o\", temperature=0, api_key=os.environ[\"OPENAI_API_KEY\"]\n",
")\n",
"\n",
"MY_CUSTOM_TEMPLATE = \"\"\"\n",
" You are evaluating the positivity or negativity of the responses to questions.\n",
" [BEGIN DATA]\n",
" ************\n",
" [Question]: {input}\n",
" ************\n",
" [Response]: {output}\n",
" [END DATA]\n",
"\n",
"\n",
" Please focus on the tone of the response.\n",
" Your answer must be single word, either \"positive\" or \"negative\"\n",
" \"\"\""
]
"source": "from phoenix.evals import LLM, create_classifier\n\njudge = LLM(provider=\"openai\", model=\"gpt-4.1\")\n\nMY_CUSTOM_TEMPLATE = \"\"\"\n You are evaluating the positivity or negativity of the responses to questions.\n [BEGIN DATA]\n ************\n [Question]: {input}\n ************\n [Response]: {output}\n [END DATA]\n\n\n Please focus on the tone of the response.\n Your answer must be single word, either \"positive\" or \"negative\"\n \"\"\"\n\ntone_eval = create_classifier(\n name=\"tone_eval\",\n prompt_template=MY_CUSTOM_TEMPLATE,\n llm=judge,\n choices={\"positive\": 1.0, \"negative\": 0.0},\n direction=\"maximize\",\n)"
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -204,28 +129,14 @@
"metadata": {
"id": "-GyksRt_CDxN"
},
"source": [
"Use the `llm_classify` function to run the evaluation using your custom template. You will be using the dataframe from the traces you generated above. We also add `nest_asyncio` to run the evaluations concurrently (if you are running multiple evaluations)."
]
"source": "Use the `evaluate_dataframe` function to run the evaluation using your classifier. You will be using the dataframe from the traces you generated above. It runs the judge calls concurrently and returns the dataframe with a `tone_eval_score` column holding the label, score, and explanation."
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import nest_asyncio\n",
"\n",
"nest_asyncio.apply()\n",
"\n",
"evals_df = llm_classify(\n",
" dataframe=primary_df,\n",
" template=MY_CUSTOM_TEMPLATE,\n",
" model=eval_model,\n",
" rails=[\"positive\", \"negative\"],\n",
" provide_explanation=True,\n",
")"
]
"source": "from phoenix.evals import evaluate_dataframe\n\nresults_df = evaluate_dataframe(dataframe=primary_df, evaluators=[tone_eval])"
},
{
"cell_type": "markdown",
Expand All @@ -241,71 +152,28 @@
"metadata": {
"id": "bfeISZDNCDxN"
},
"source": [
"# Log evaluations back to Arize\n",
"Use the `log_evaluations_sync` function as part of our Python SDK to attach evaluations you've run to traces. The code below assumes that you have already completed an evaluation run, and you have the `evals_dataframe` object. It also assumes you have a `traces_dataframe` object to get the `span_id` that you need to attach the evals.\n",
"\n",
"The `evals_dataframe` requires four columns, which should be auto-generated for you based on the evaluation you ran using Phoenix. The `<eval_name>` must be alphanumeric and cannot have hyphens or spaces.\n",
"- `eval.<eval_name>.label`\n",
"- `eval.<eval_name>.score`\n",
"- `eval.<eval_name>.explanation`\n",
"- `context.span_id`\n",
"\n",
"An example evaluation data dictionary would look like:\n",
"```python\n",
"evaluation_data = {\n",
" 'context.span_id': ['74bdfb83-a40e-4351-9f41-19349e272ae9'], # Use your span_id\n",
" 'eval.myeval.label': ['accuracy'], # Example label name\n",
" 'eval.myeval.score': [0.95], # Example label value\n",
" 'eval.myeval.explanation': [\"some explanation\"]\n",
"}\n",
"evaluation_df = pd.DataFrame(evaluation_data)\n",
"```"
]
"source": "# Log evaluations back to Arize\nUse the `update_evaluations` method on the Arize SDK client to attach the evaluations you've run to traces. It requires four columns, and the `<eval_name>` must be alphanumeric and cannot have hyphens or spaces.\n- `eval.<eval_name>.label`\n- `eval.<eval_name>.score`\n- `eval.<eval_name>.explanation`\n- `context.span_id`"
},
{
"cell_type": "markdown",
"metadata": {
"id": "pUWLwKDOCDxN"
},
"source": [
"Here is sample code to log the evaluations back to Arize. The API reference can be found <a href=\"https://arize-client-python.readthedocs.io/en/latest/llm-api/logger.html#arize.pandas.logger.Client.log_evaluations_sync\">here</a>."
]
"source": "`to_annotation_dataframe` flattens the nested `tone_eval_score` column into `label`, `score`, and `explanation` while preserving `context.span_id` (the join key `export_to_df` already provides). Rename those to the reserved `eval.<eval_name>.*` columns and upload with `client.spans.update_evaluations`. The API reference can be found <a href=\"https://arize.com/docs/ax/api-reference\">here</a>."
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"evals_df[\"eval.tone_eval.label\"] = evals_df[\"label\"]\n",
"evals_df[\"eval.tone_eval.explanation\"] = evals_df[\"explanation\"]\n",
"evals_df.head()"
]
"source": "from phoenix.evals.utils import to_annotation_dataframe\n\nannotations = to_annotation_dataframe(dataframe=results_df)\n\nevals_df = annotations.rename(\n columns={\n \"label\": \"eval.tone_eval.label\",\n \"score\": \"eval.tone_eval.score\",\n \"explanation\": \"eval.tone_eval.explanation\",\n }\n)[\n [\n \"context.span_id\",\n \"eval.tone_eval.label\",\n \"eval.tone_eval.score\",\n \"eval.tone_eval.explanation\",\n ]\n]\nevals_df.head()"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"from arize.pandas.logger import Client\n",
"\n",
"ARIZE_API_KEY = os.environ.get(\"ARIZE_API_KEY\")\n",
"SPACE_ID = os.environ.get(\"SPACE_ID\")\n",
"\n",
"# Initialize Arize client to log evaluations\n",
"arize_client = Client(\n",
" space_id=SPACE_ID, api_key=ARIZE_API_KEY\n",
")\n",
"\n",
"# Set the evals_df to have the correct span ID to log it to Arize\n",
"evals_df[\"context.span_id\"] = primary_df[\"context.span_id\"]\n",
"\n",
"# send the eval to Arize\n",
"arize_client.log_evaluations_sync(evals_df, \"tracing-haiku-tutorial\")"
]
"source": "# send the evals to Arize\nclient.spans.update_evaluations(\n space_id=SPACE_ID,\n project_name=\"tracing-haiku-tutorial\", # your project name\n dataframe=evals_df,\n)"
}
],
"metadata": {
Expand All @@ -315,4 +183,4 @@
},
"nbformat": 4,
"nbformat_minor": 0
}
}