Skip to content

docs: migrate cookbook notebooks to current phoenix-evals API - #76

Closed
jimbobbennett wants to merge 5 commits into
mainfrom
docs/cookbooks-current-evals-api
Closed

docs: migrate cookbook notebooks to current phoenix-evals API#76
jimbobbennett wants to merge 5 commits into
mainfrom
docs/cookbooks-current-evals-api

Conversation

@jimbobbennett

@jimbobbennett jimbobbennett commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

What

Migrates the eval engine in four cookbook notebooks off the deprecated phoenix-evals interface onto the current API, keeping them in sync with the paired Arize AX docs cookbooks (migrated in a corresponding docs PR).

Notebook Change
python/llm/agents/agents-cookbook.ipynb OpenAIModel/llm_classify → LLM.generate_text + create_classifier/evaluate_dataframe; drop nest_asyncio
python/llm/agents/openai-agents-cookbook.ipynb experiment evaluator → async create_classifier + async_evaluate; drop nest_asyncio
python/llm/agents/couchbase_langgraph_agentic_rag.ipynb prebuilt DocumentRelevanceEvaluator/CorrectnessEvaluator from phoenix.evals.metrics; also fixes a pre-existing undefined-variable bug in run_evaluators
python/llm/experiments/text2sql-experiment.ipynb eval engine → create_classifier; experiment flow → v8 client.datasets.create / client.experiments.run (the v7 arize.experimental API is dropped in arize>=8)

python/cookbooks/phoenix_evals_examples/ragas_retrieval_evals_tutorial.ipynb was checked and left unchanged — it is already on the current API.

Why

phoenix-evals 3.x removed OpenAIModel/GeminiModel/llm_classify/llm_generate (the classic interface moved to phoenix.evals.legacy). These notebooks are linked directly from AX docs cookbooks; this keeps notebook and doc consistent.

Testing

The eval/generation API transforms are verified against installed arize 8.35.0 / arize-phoenix-evals 3.1.0, and the text2sql v8 experiment flow mirrors a doc snippet that was run end-to-end against a live AX space. The heavy framework paths (couchbase cluster, openai-agents, langgraph, DuckDB dataset) were not executed end-to-end; all four notebooks pass nbformat validation with zero residual deprecated tokens.

🤖 Generated with Claude Code

@jimbobbennett @claude
docs: migrate cookbook notebooks to current phoenix-evals API
66fbcf4 
Migrate the eval engine in four cookbook notebooks off the deprecated
phoenix-evals interface (OpenAIModel / llm_classify / prebuilt rails
templates) onto the current LLM + create_classifier / evaluate_dataframe
/ metrics API. text2sql additionally moves its experiment flow to the v8
arize SDK (client.datasets.create / client.experiments.run), matching the
paired AX docs page.

- agents-cookbook.ipynb: LLM.generate_text + create_classifier/evaluate_dataframe
- openai-agents-cookbook.ipynb: async create_classifier evaluator; drop nest_asyncio
- couchbase_langgraph_agentic_rag.ipynb: prebuilt DocumentRelevance/Correctness evaluators
- text2sql-experiment.ipynb: v8 datasets/experiments + async is_sql classifier

Keeps notebooks in sync with the migrated AX cookbook docs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@review-notebook-app

review-notebook-app Bot commented Jul 23, 2026

Copy link
Copy Markdown

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@github-actions

github-actions Bot commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown

🤖 Model version check

⚠️ 2 pre-existing outdated model(s) in files this PR touches — not blocking

These are on unchanged lines, so the check still passes — but since you're already editing these files, consider updating them too.

Location Found Suggested Why
python/llm/agents/agents-cookbook.ipynb:333 gpt-4o-mini gpt-5.4-mini mini tier → latest mini
python/llm/experiments/text2sql-experiment.ipynb:180 gpt-4o gpt-5.5 full tier → latest full

⚠ 5 item(s) to review (not blocking)

Prose mentions, specialised variants (*-codex, *-chat-latest), or GPT-5/o-series code changes (max_tokens → max_completion_tokens, drop temperature).

Location Found Suggested Why
python/llm/agents/couchbase_langgraph_agentic_rag.ipynb:375 temperature — GPT-5/o-series: remove temperature (unsupported on reasoning models)
python/llm/agents/couchbase_langgraph_agentic_rag.ipynb:423 temperature — GPT-5/o-series: remove temperature (unsupported on reasoning models)
python/llm/agents/couchbase_langgraph_agentic_rag.ipynb:446 temperature — GPT-5/o-series: remove temperature (unsupported on reasoning models)
python/llm/agents/couchbase_langgraph_agentic_rag.ipynb:470 temperature — GPT-5/o-series: remove temperature (unsupported on reasoning models)
python/llm/agents/couchbase_langgraph_agentic_rag.ipynb:658 max_tokens — GPT-5/o-series: rename max_tokens → max_completion_tokens

See the check-models skill. Policy date: 2026-06-18. Add check-models:ignore to a line to skip it.

ℹ️ Platform-wrapped IDs (Bedrock [region.]anthropic.claude-…, Databricks databricks-claude-…, OpenRouter/LiteLLM provider/model) are flagged on their embedded model name — bump the version but keep the platform's ID format (e.g. Bedrock 4.x needs a us./eu./apac. inference-profile prefix). See the skill's Platform-specific IDs section.

jimbobbennett and others added 3 commits July 22, 2026 19:27
@jimbobbennett @claude
fix: bump generation model gpt-4o -> gpt-5.4-mini (check-models)
31f797e 
The migrated LLM.generate_text generation model introduced gpt-4o on a
changed line, which the check-models gate flags. Use the current-tier
gpt-5.4-mini to match the eval docs' app-model convention.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jimbobbennett @claude
fix: migrate notebook experiments to v8 API + fix runtime bugs
e0f2806 
- Migrate v7 experiments/datasets API → v8 (datasets.create, experiments.run,
  experiments.create precomputed via list_examples for server example IDs) in
  agents-cookbook, openai-agents-cookbook, couchbase; text2sql already v8.
- Remove max_tokens from gpt-5.4-mini generate_text calls (GPT-5 rejects max_tokens).
- Fix client name collision: Arize client renamed to `arize_client` (text2sql,
  agents-cookbook) so it no longer clobbers the OpenAI `client`.
- text2sql no_error/has_results now return EvaluationResult (bare floats leave
  reserved eval columns null → Flight rejects).
- text2sql verified live end-to-end (both experiments, n_errors=0). couchbase v8
  path faithful but not live-verified (needs a Couchbase cluster).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jimbobbennett @claude
fix: make agent cookbooks run end-to-end (verified live)
834f423 
Found by actually executing the notebooks against a live AX space:

- openai-agents-cookbook: restore nest_asyncio (import + apply + pip). The async
  experiment task fails with "asyncio.run() cannot be called from a running event
  loop" in a notebook without it; dropping it during migration was wrong.
- agents-cookbook: attach server example IDs by a unique row_key instead of joining
  on the question text. Questions repeat, so the content join exploded rows and left
  eval columns null -> experiments.create 400. Also assign eval labels before the join.

Both now run clean (n_errors=0; experiments.create/experiments.run succeed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jimbobbennett

jimbobbennett commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Pairs with docs PR Arize-ai/docs#767 (deprecated-API / v7→v8 upgrade).

@jimbobbennett

jimbobbennett commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

couchbase update: verified the Couchbase integration + migrated arize logging against a local Couchbase (podman couchbase/server:enterprise-7.6.2): ingested 99 chunks into a vector index (all-MiniLM-L12-v2, 384-dim scoped FTS index), confirmed similarity_search returns hits, and confirmed the v8 client.experiments.create precomputed-logging path works with couchbase-derived data (clean example_id join). Not run: the full langgraph + agentc agent graph, which needs a pinned langchain<0.3 (uses langchain_core.pydantic_v1) + an agentc catalog — orthogonal to this API migration.

@jimbobbennett @claude
fix: make couchbase agentic-RAG notebook run end-to-end (drop agentc)
9ef6fdb 
Migrate off Couchbase Agent Catalog so the notebook is self-contained and runs
on current langchain, and fix two pre-existing bugs found by running it against
a local Couchbase (podman couchbase/server:enterprise-7.6.2):

- Drop agentc: inline the retriever tool (create_retriever_tool over the vector
  store) and the relevance-grading + RAG prompts (previously fetched from Agent
  Catalog, whose source files aren't in the repo).
- Modernize to current langchain: langchain_core.pydantic_v1 -> pydantic,
  CouchbaseVectorStore -> CouchbaseSearchVectorStore,
  create_retriever_tool from langchain_core.tools.retriever, inline rlm/rag-prompt.
- Update models to gpt-4.1 (tool-calling + structured output).
- Fix reset_vector_store: the document-load block was indented under `if results:`,
  so a fresh store never ingested (empty retrieval -> grader always "not relevant"
  -> graph never reaches generate). Dedented + added a short FTS catch-up wait.
- Import ChatOpenAI before the graph's first run.

Verified end-to-end: docs ingested, similarity search returns hits, agent graph
runs (relevant/rewrite decisions), and 4 k/chunk experiments log to Arize (0 errors).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jimbobbennett

jimbobbennett commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

couchbase update (resolved): the couchbase cookbook + notebook now run end-to-end against a local Couchbase (podman couchbase/server:enterprise-7.6.2). Migrated off Couchbase Agent Catalog (inlined the retriever tool + grading/RAG prompts — the catalog source files were never in the repo, so it was not runnable as shipped), modernized to current langchain (pydantic v2, CouchbaseSearchVectorStore), and fixed a pre-existing reset_vector_store indentation bug that left a fresh store empty. Verified: doc ingestion, vector search, the LangGraph agent graph (relevant/rewrite decisions), and 4 k/chunk experiments logged to Arize — 0 errors. Agent Catalog is no longer required.

@jimbobbennett
jimbobbennett marked this pull request as draft July 24, 2026 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jimbobbennett