Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,7 @@ experiment_results = await px_client.experiments.run_experiment(
dataset=dataset,
task=task,
experiment_name="concise-template",
experiment_description="explicitly instuct the llm to be concise",
experiment_description="explicitly instruct the llm to be concise",
experiment_metadata={"vendor": "openai", "model": gpt_4o},
evaluators=EVALUATORS,
)
Expand Down
2 changes: 1 addition & 1 deletion docs/phoenix/cookbook/evaluation/evaluate-an-agent.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -561,7 +561,7 @@ You should now see eval labels in Phoenix.

## Function Calling Evals using Ground Truth

The above example works, however if you have ground truth labled data, you can use that data to get an even more accurate measure of your router's performance by running an experiments.
The above example works, however if you have ground truth labeled data, you can use that data to get an even more accurate measure of your router's performance by running an experiments.

Experiments also follow a standard step-by-step process in Phoenix:

Expand Down
2 changes: 1 addition & 1 deletion docs/phoenix/cookbook/evaluation/evaluate-rag.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ colab.research.google.com

In this tutorial we will look into building a RAG pipeline and evaluating it with Phoenix Evals.

It has the the following sections:
It has the following sections:

1. Understanding Retrieval Augmented Generation (RAG).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -662,7 +662,7 @@ main().catch((err) => {

Individual bad traces can be anecdotal, edge cases. Clusters help you see what to prioritize. This clustering method comes from qualitative research:

1. **Open coding.** In qualitative research, "coding" means annotating data with labels, not programing. Pull a sample of traces and write free-form notes on anything wrong with each one. Don't pre-define categories before looking at the traces, or you'll miss the failure modes unique to your application.
1. **Open coding.** In qualitative research, "coding" means annotating data with labels, not programming. Pull a sample of traces and write free-form notes on anything wrong with each one. Don't pre-define categories before looking at the traces, or you'll miss the failure modes unique to your application.
2. **Axial coding.** Group the notes around shared themes (the "axes" the method is named for) producing a failure taxonomy: distinct, named categories. You can do this yourself like grouping socks after doing the laundry, but an LLM is often faster and just as good.
3. **Count.** Tally failures per category. This step creates a prioritized roadmap. In [one of Hamel Husain's client engagements](https://hamel.dev/blog/posts/field-guide/#bottom-up-vs.-top-down-analysis), three categories accounted for over 60% of all problems.
4. **Iterate to saturation.** Keep sampling until new traces stop revealing new categories. Rule of thumb: review at least 100 traces to start; once ~20 consecutive traces turn up nothing new, you're saturated.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ response = openai.ChatCompletion.create(

You can use phoenix spans and traces to inspect the invocation parameters of the function to

1. verify the inputs to the model in form of the the user message
1. verify the inputs to the model in form of the user message

2. verify your request to Open AI

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: "Concepts: Datasets"

## Datasets

Datasets are integral to evaluation and experimentation. They are collections of examples that provide the `inputs` and, optionally, expected `reference` outputs for assessing your application. Each example within a dataset represents a single data point, consisting of an `inputs` dictionary, an optional `output` dictionary, and an optional `metadata` dictionary. The `optional` output dictionary often contains the the expected LLM application output for the given input.
Datasets are integral to evaluation and experimentation. They are collections of examples that provide the `inputs` and, optionally, expected `reference` outputs for assessing your application. Each example within a dataset represents a single data point, consisting of an `inputs` dictionary, an optional `output` dictionary, and an optional `metadata` dictionary. The `optional` output dictionary often contains the expected LLM application output for the given input.

Datasets allow you to collect data from production, staging, evaluations, and even manually. The examples collected are then used to run experiments and evaluations to track improvements.

Expand Down
2 changes: 1 addition & 1 deletion docs/phoenix/integrations/evaluation-integrations/uqlm.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,7 @@ We will walk through the key steps in the documentation below. Check out the ful
uqlm_df
```

That's it! Congratulations, you have sucessfully run the Uncertainty Quantification for Language Models eval. Take it a step further by following the steps below.
That's it! Congratulations, you have successfully run the Uncertainty Quantification for Language Models eval. Take it a step further by following the steps below.

7. Generate-and-score with your LLM client

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ With the Phoenix Client SDK you can:

* [Pull prompts](/docs/phoenix/prompt-engineering/how-to-prompts/using-a-prompt) templates by name, version, or tag

* [Format prompt](/docs/phoenix/prompt-engineering/how-to-prompts/using-a-prompt#using-a-prompt) templates with runtime variables and use them in your code. Native support for OpenAI, Anthropic, Gemini, Vercel AI SDK, and more. No propriatry client necessary.
* [Format prompt](/docs/phoenix/prompt-engineering/how-to-prompts/using-a-prompt#using-a-prompt) templates with runtime variables and use them in your code. Native support for OpenAI, Anthropic, Gemini, Vercel AI SDK, and more. No proprietary client necessary.

* Support for [tool calling](/docs/phoenix/prompt-engineering/concepts-prompts/prompts-concepts#tools) and [response formats](/docs/phoenix/prompt-engineering/concepts-prompts/prompts-concepts#response-format). Execute tools defined within the prompt. Phoenix prompts encompasses more than just the text and messages.

Expand Down
2 changes: 1 addition & 1 deletion docs/phoenix/resources/phoenix-to-arize-ax-migration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ results/ # Overview of import and export jobs

The formatting of some span attributes may not be compatible between Arize AX and Phoenix. We tried to cover as many cases as possible, but there may be some missing ones. If you encounter errors:

1. Check the `results/` folder to see what errors have occured
1. Check the `results/` folder to see what errors have occurred
2. Fix them in the project's `traces.json` file
3. Re-import the data to Arize AX

Expand Down
2 changes: 1 addition & 1 deletion docs/phoenix/self-hosting/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ The following environment variables will control how your phoenix server runs.

* **PHOENIX\_ALLOW\_EXTERNAL\_RESOURCES:** Controls whether external resources (such as Google Fonts) are loaded in the web interface. Defaults to `true`. Set to `false` in air-gapped environments to prevent external requests that can cause UI loading delays. Available since version 11.15.0.

* **PHOENIX\_SQL\_DATABASE\_URL:** The SQL database URL to use when logging traces and evals. if you plan on using SQLite, it's advised to to use a persistent volume and simply point the `PHOENIX_WORKING_DIR` to that volume. If URL is not specified, by default Phoenix starts with a file-based SQLite database in a temporary folder, the location of which will be shown at startup. Phoenix also supports PostgresSQL as shown below:
* **PHOENIX\_SQL\_DATABASE\_URL:** The SQL database URL to use when logging traces and evals. if you plan on using SQLite, it's advised to use a persistent volume and simply point the `PHOENIX_WORKING_DIR` to that volume. If URL is not specified, by default Phoenix starts with a file-based SQLite database in a temporary folder, the location of which will be shown at startup. Phoenix also supports PostgreSQL as shown below:

* PostgreSQL, e.g. `postgresql://@host/dbname?user=user&password=password` or `postgresql://user:password@host/dbname`

Expand Down
2 changes: 1 addition & 1 deletion docs/phoenix/self-hosting/features/authentication.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ The following environment variables are optional but recommended:

<table data-header-hidden data-full-width="false"><thead><tr><th width="300">Variable</th><th>Description</th></tr></thead><tbody><tr><td><strong>PHOENIX_USE_SECURE_COOKIES</strong></td><td>If set to <strong>True</strong>, access and refresh tokens will be stored in secure cookies. Defaults to <strong>False</strong>.</td></tr><tr><td><strong>PHOENIX_CSRF_TRUSTED_ORIGINS</strong></td><td>A comma-separated list of origins allowed to bypass Cross-Site Request Forgery (CSRF) protection. This setting is recommended when configuring OAuth2 clients or sending password reset emails. If this variable is left unspecified or contains no origins, CSRF protection will not be enabled. In such cases, when a request includes <code>origin</code> or <code>referer</code> headers, those values will not be validated.</td></tr></tbody></table>

Deploy Phoenix with the above environment variables set. You will know that you have setup authentication correctly if the UI navigates to to a login screen.
Deploy Phoenix with the above environment variables set. You will know that you have setup authentication correctly if the UI navigates to a login screen.

By default Phoenix will create an admin user account. To get started:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ Below is an example of logging conversations:
current_span.set_attribute(SpanAttributes.SESSION_ID, session_id)
current_span.set_attribute(SpanAttributes.INPUT_VALUE, messages[-1].get('content'))

# Propagate the session_id down to spans crated by the OpenAI instrumentation
# Propagate the session_id down to spans created by the OpenAI instrumentation
# This is not strictly necessary, but it helps to correlate the spans to the same session
with using_session(session_id):
response = client.chat.completions.create(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ Manual annotation gives you ground truth, but it doesn't scale. We can review ma

Sometimes, our users are already telling you what's working. Every thumbs up, thumbs down, "this wasn't helpful" click, or escalation to a human agent is feedback. Let's store that feedback in Phoenix, so that we can attach it to our traces!

Let's simulate a thumbs up/thumbs down feature, and then store that as annotations to our traces in Phoenix. This will give us metrics on how satified our users are.
Let's simulate a thumbs up/thumbs down feature, and then store that as annotations to our traces in Phoenix. This will give us metrics on how satisfied our users are.

## Get the Span ID from Running Code

Expand Down
Loading