docs(skills): weekly audit — 2026-08-26

[mikeldking]

Phoenix Skills Audit — 2026-08-19 → 2026-08-26

Audited against: origin/main at d002a45899014c9c9644aeaee59fa6140bca6305
Commits analyzed: 97 (2 produced skill edits, 8 read in depth, the rest skipped)

Local main was already level with origin/main (git rev-list --count main..origin/main → 0).

Edits applied

phoenix-evals

• references/evaluators-pre-built.md — added a "Retrieval quality: document
  relevance vs. retrieval relevance" section for the new
  RetrievalRelevanceEvaluator / createRetrievalRelevanceEvaluator shipped in
  7efa7b21f (feat(evals): add retrieval relevance evaluator, feat(evals): add retrieval relevance evaluator #14766).
  Grounded in:

  • packages/phoenix-evals/src/phoenix/evals/metrics/retrieval_relevance.py:13
    (class), :74 (RetrievalRelevanceInputSchema — input, context),
    :80 (__init__(self, llm: LLM, **kwargs: Any))
  • packages/phoenix-evals/src/phoenix/evals/metrics/__init__.py:10,27 (export)
  • packages/phoenix-evals/src/phoenix/evals/__generated__/classification_evaluator_configs/_retrieval_relevance_classification_evaluator_config.py:7,9,16
    (name="retrieval_relevance", optimization_direction="maximize",
    choices={"relevant": 1.0, "irrelevant": 0.0})
  • js/packages/phoenix-evals/src/llm/createRetrievalRelevanceEvaluator.ts:6,20,65
    (args interface, RetrievalRelevanceEvaluationRecord, factory) and
    js/packages/phoenix-evals/src/llm/index.ts:10 +
    js/packages/phoenix-evals/src/index.ts (root re-export, so the
    @arizeai/phoenix-evals import in the snippet resolves)

  The section documents the input / context field conventions verbatim from
  the docstrings (user request, not a reformulated query; all retrieved items
  joined), the maximize direction, and the two rubric rules most likely to
  surprise a reader — relevance is not correctness, and a failed retrieval is
  irrelevant.

• references/evaluators-pre-built.md — updated the opening capability list
  from "RAG quality (faithfulness, correctness, document relevance)" to include
  retrieval relevance, so the enumeration matches metrics/__init__.py.

• references/evaluators-rag.md — added a pointer in the "Retrieval Evaluation
  (IR Metrics)" section: IR metrics need query-document relevance labels, so
  when you don't have them, reach for the LLM judge instead. Distinguishes
  per-document (DocumentRelevanceEvaluator, for precision@k / recall@k) from
  holistic per-step (RetrievalRelevanceEvaluator). Same commit 7efa7b21f.

• references/experiments-harbor-python.md — new file for
  phoenix.client.harbor, added in f7c6cb3be (feat(client): record Harbor
  jobs in Phoenix, feat(client): record Harbor jobs in Phoenix #15547). Maps to phoenix-evals rather than the client's
  file path because the feature it exposes is datasets + experiments.
  Grounded in:

  • packages/phoenix-client/src/phoenix/client/harbor/__init__.py:1-15
    (exported symbols: PhoenixJobPlugin, HarborPluginError,
    DEFAULT_EXPERIMENT_NAME_TEMPLATE, EXPERIMENT_NAME_TEMPLATE_FIELDS)
  • packages/phoenix-client/src/phoenix/client/harbor/_plugin.py:46,49-93
    (constructor kwargs and their documented defaults; trace_mode accepts only
    "none" and raises ValueError otherwise)
  • packages/phoenix-client/src/phoenix/client/harbor/_naming.py:18-29
    (default template "{job.name} · {agent.name} · {agent.model}" and the six
    EXPERIMENT_NAME_TEMPLATE_FIELDS entries, quoted verbatim), :52
    (validate_experiment_naming — mutual exclusivity), :103
    (validate_experiment_name_for_plan — exact name rejected for multi-slice jobs)
  • packages/phoenix-client/src/phoenix/client/harbor/_adapter.py:123,257-280
    (the HarborPluginError messages reproduced in the "Failure modes" section)
  • packages/phoenix-client/src/phoenix/client/harbor/_recorder.py:66,111,228
    (one versioned dataset per job, one experiment per agent config, one run per
    terminal trial) and _plugin.py:246 ("Not recorded yet: scores.")
  • packages/phoenix-client/pyproject.toml:47-49 (harbor extra,
    harbor>=0.21.0; python_version >= '3.12') and :56 (entry point name
    arize-phoenix, so the Harbor flag is --plugin arize-phoenix)

  The endpoint default is documented as get_base_url() resolution
  (PHOENIX_ENDPOINT, then PHOENIX_COLLECTOR_ENDPOINT) rather than the
  plugin docstring's bare "PHOENIX_COLLECTOR_ENDPOINT", because
  _plugin.py:80 calls get_base_url(), whose key order is
  packages/phoenix-client/src/phoenix/client/utils/config.py:57,67-74.

• SKILL.md — added the Quick Reference row
  Record a Harbor job as an experiment → experiments-harbor-python, so the new
  file is not an orphan.

phoenix-tracing

No edits. No commit in the window touched packages/phoenix-otel/,
js/packages/phoenix-otel/src/, OpenInference semantic conventions, or span
attributes. The only js/packages/phoenix-otel change was a package.json
dependency bump (050a10e1e, fa2c7c59c).

phoenix-cli

No edits. Verified against the tree rather than inferred:

• js/packages/phoenix-cli/src/commands/ is unchanged in the window — no new
  subcommand, flag, or JSON output field. The only CLI source change was
  js/packages/phoenix-cli/src/pxi/types.ts in 391d0ddaa.
• The released CLI change in the window (7cfc1aa53, v1.16.1) is a PXI model
  recommendation list edit (gemini-3.7-flash in, gemini-3.5-flash out,
  claude-opus-5 added). The skill documents no PXI model list, so there is
  nothing to update.
• The two new REST endpoints below are not wrapped by any px command — see
  "Out-of-scope findings".

Phase-2 tagged candidates and their disposition

Commit	Tag	Disposition
7efa7b21f retrieval relevance evaluator	evals	Edited — pre-built + RAG references
f7c6cb3be record Harbor jobs in Phoenix	evals	Edited — new experiments-harbor-python.md
b28cf94fc upgrade to anthropic SDK v1	evals	Dropped. In packages/phoenix-evals/pyproject.toml the change is anthropic>0.18.0 → anthropic>=1,<2 in the dev and test optional-dependency groups only (verified in the diff); production deps are untouched, and the skill pins no versions. The user-facing half of this commit is phoenix-client's Anthropic prompt helper moving temperature/top_p into extra_body — prompts, which none of the three skills document. Noted below.
8ffd93899 PATCH /projects/{id}/retention	cli	Dropped from the skills. No px command wraps it (grep retention js/packages/phoenix-cli/src/ → no hits) and no Python/TS client resource method exists (grep retention packages/phoenix-client/src/phoenix/client/resources/ → no hits); only generated OpenAPI types landed. Noted below.
59c247a19 GET /model_providers	cli	Dropped from the skills. Same reason — REST-only, no px wrapper. The CLI already reads the equivalent data through the existing modelProviders GraphQL field in js/packages/phoenix-cli/src/pxi/preflight.ts:37, which is unchanged. Noted below.
391d0ddaa PXI GraphQL mutations by default	cli	Dropped. The CLI-side change is a type in src/pxi/types.ts; the operator-facing knob is the PHOENIX_AGENTS_DISABLE_GRAPHQL_MUTATIONS env var, which is server-side and governs the PXI agent. The phoenix-cli skill documents trace/span/dataset debugging and px api graphql, not PXI session behavior. Noted below.
ce13c6f30 PXI browser-action meta-tools	cli	Dropped. Server-side PXI tool surface (src/phoenix/server/api/routers/agents.py); no CLI command or JSON shape changed.
bcac8f624 session filter NUL message	cli	Dropped. src/phoenix/trace/dsl/filter.py change only rewords a SyntaxError message on Python ≥3.11 so it stops quoting the tokenizer. The filter-expression grammar the skill documents is unchanged.

Skipped commits

Release/bookkeeping: a015c6f69, 1915396ea, 49cf5a722, c4a1068df,
08922f3d0, f8561dc16, 9478f954b, c619fef00, 540f1a544, 722a495b3,
7cfc1aa53 (CHANGELOG/version only — content triaged above), b44ffda67
(regenerated OpenAPI artifacts for the endpoints already listed).

Dependency bumps: 050a10e1e, fa2c7c59c, 5e2becf1d, 55184f4c2,
198945957, 25e7c5cd6, 48f18e98d (sqlean).

Cost/pricing manifests: b0c1e60c5, 4580f348a, ad9b3a112, 5150c4d8e.

Frontend / UI only: 410a45bb4, 9fbdebb38, 2518e2b2a, ef9f238c0,
e0a11a1c2, 2fe40f9f6, d76cdf197, 11bca548a, 056afc3ad, 31afcc737,
0457b2541, 7a6583190, 8c143b8b1.

Internal server / agents refactors: f08a46cf6, 31fafc5fe, e2cf8be0a,
c08877f47, d38df33be (server-side evaluator prompt validation error text),
7a5bc5578, f12126cc2, 7b77585f5, db334e6da.

Docs and CI: d002a4589, 8cdad3c75, 729ff110d, 1a537fcd9, 506ebdca6,
5ece96933, 840fa1e0b, c121df660, 3a42bc101, 54c57cd8f, 198344c4e,
a38430963, e66c1fee8, 0b2321e1f, 81bd0695f, 230b7ed5d, and the
Arize-AX / onboarding docs series (97429c9c3 … bd13ea95c).

Out-of-scope findings

Flagged for their owners; no files outside the three target skills were touched.

1. PATCH /v1/projects/{project_identifier}/retention (8ffd93899) assigns
   an existing trace retention policy to a project, or resets it to the default
   with policy_id: null (src/phoenix/server/api/routers/v1/projects.py:335-400).
   It has no Python client resource method, no TypeScript client helper, and no
   px command — only generated types. Worth wrapping in
   client.projects and/or px project, at which point it belongs in
   phoenix-tracing/references/projects-python.md and phoenix-cli/SKILL.md.

2. GET /v1/model_providers (59c247a19) lists built-in provider families,
   narrowed by PHOENIX_ALLOWED_PROVIDERS
   (src/phoenix/server/api/routers/v1/model_providers.py:38-66). Also
   unwrapped by any client or CLI command. Its sibling
   GET /v1/custom_model_providers is referenced in the endpoint description.

3. Anthropic prompt-helper shape change (b28cf94fc):
   phoenix.client.helpers.sdk.anthropic.messages now carries temperature /
   top_p inside extra_body on AnthropicMessageModelKwargs, since
   anthropic 1.x dropped them as keyword arguments of messages.create().
   from_anthropic still accepts the old top-level shape. This is a
   user-visible change to prompt.format() output for Anthropic prompts, but
   prompts are not documented by any of the three external-facing skills.

4. PHOENIX_AGENTS_DISABLE_GRAPHQL_MUTATIONS (391d0ddaa) lets operators
   disable PXI's GraphQL mutations server-side regardless of client settings.
   This is a deployment knob for the PXI agent — likely phoenix-server
   territory, not one of the three skills.

5. Python-only feature: the Harbor plugin has no TypeScript mirror
   (grep -rli harbor js/packages/ → no hits). If a TS Harbor integration is
   planned, experiments-harbor-python.md will need a -typescript.md sibling.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
arize-phoenix	🟢 Ready	View Preview	Aug 26, 2026, 6:05 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.