docs(evaluate): document rubric_based_multi_turn_trajectory_quality_v1 and clarify Rubric.type / criterion.rubrics by zettaittenani · Pull Request #1853 · google/adk-docs

zettaittenani · 2026-06-22T10:51:37Z

What

Closes a documentation gap around rubric-based evaluation criteria in docs/evaluate/criteria.md. Three small additions:

New section: rubric_based_multi_turn_trajectory_quality_v1. The metric is registered in metric_evaluator_registry.py and exists in upstream ADK 2.x, but the explanatory section that the other two rubric-based metrics have was missing. Also adds the corresponding row to the criteria table at the top of the page.

Rubric.type notes added to each rubric-based metric section. Each RubricBasedEvaluator subclass uses a RUBRIC_TYPE ClassVar that acts as a filter — rubrics carried on EvalCase.rubrics are only picked up when their type field matches the evaluator's expected value:

Metric	Required `Rubric.type`
`rubric_based_final_response_quality_v1`	`FINAL_RESPONSE_QUALITY`
`rubric_based_tool_use_quality_v1`	`TOOL_USE_QUALITY`
`rubric_based_multi_turn_trajectory_quality_v1`	`TRAJECTORY_QUALITY`
The existing examples also now include the `type` field explicitly so users have a complete template.

criterion.rubrics requirement clarified. RubricBasedEvaluator.__init__ asserts criterion.rubrics is non-empty (rubric_based_evaluator.py:332), and EvalCase.rubrics is additive on top of the criterion-level list (create_effective_rubrics_list). The "Notes On Rubrics" block in each rubric-based metric section now states both.

Why

Without these notes, three failure modes happen quickly in practice (see #1852 for full context):

Users see rubric_based_multi_turn_trajectory_quality_v1 in the criteria table but no example of how to build an EvalConfig entry for it.
Users put rubrics on EvalCase.rubrics with type unset (or mismatched) and the rubrics are silently dropped.
Users put all rubrics on EvalCase.rubrics and none on the criterion, and hit AssertionError: Rubrics are required. at evaluator init.

All three are observable in RubricBasedEvaluator source but aren't surfaced in the user-facing docs.

Changes

Only docs/evaluate/criteria.md is modified. No code changes.

Testing

mkdocs serve locally; verified the new section renders, the criteria table at the top is consistent with the sections below, and existing internal anchors still resolve.

CLA

Individual CLA signed; cla/google check is passing on this PR.

Notes

Opening as draft per the discussion in docs(evaluate): missing section for rubric_based_multi_turn_trajectory_quality_v1 + Rubric.type / criterion.rubrics requirements #1852 — happy to adjust scope / shape based on maintainer feedback before flipping to ready-for-review.

…1 and clarify Rubric.type / criterion.rubrics Closes three gaps in docs/evaluate/criteria.md: 1. Adds a section for `rubric_based_multi_turn_trajectory_quality_v1`. The metric is registered in metric_evaluator_registry.py and implemented in rubric_based_multi_turn_trajectory_evaluator.py upstream, but the explanatory section that its two sibling rubric-based metrics have was missing. Also adds the corresponding row to the criteria table. 2. Adds a "Notes On Rubrics" subsection under each of the three rubric-based metric sections, stating the required Rubric.type value (FINAL_RESPONSE_QUALITY / TOOL_USE_QUALITY / TRAJECTORY_QUALITY) and the criterion-vs-EvalCase rubric relationship (criterion.rubrics required and non-empty; EvalCase.rubrics is additive, filtered by type). 3. Adds the explicit `type` field to existing JSON examples in the two existing rubric-based sections so users have a complete template. Refs google#1852.

netlify · 2026-06-22T10:51:41Z

✅ Deploy Preview for adk-docs-preview ready!

Name	Link
🔨 Latest commit	`23cf409`
🔍 Latest deploy log	https://app.netlify.com/projects/adk-docs-preview/deploys/6a5661dc5ef56600088c130a
😎 Deploy Preview	https://deploy-preview-1853--adk-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

koverholt

Thanks for the PR and additions / fixes! The new section, table row, and the non-empty/additive notes are accurate and verified against adk-python.

Requesting changes on one substantive point, flagged inline: the Rubric.type filter only applies to EvalCase.rubrics, not to criterion-level rubrics in EvalConfig. The "Notes On Rubrics" bullets state it as a blanket rule, and the examples add type to criterion-level rubrics, which together imply criterion rubrics are dropped on a type mismatch, which they aren't. See the inline suggestions on both the bullets and the example type fields across all three sections. Everything else looks good.

(Cross-referencing #1833 as the original adk-bot issue where these were originally flagged.)

koverholt · 2026-07-14T17:06:31Z

+
+- Rubrics on `EvalConfig.criteria["rubric_based_final_response_quality_v1"].rubrics` **must be non-empty** — `RubricBasedEvaluator` asserts this at init time.
+- Rubrics on `EvalCase.rubrics` are *additive* on top of the criterion-level list, not a replacement. The effective rubric set passed to the judge is the union of both.
+- Only rubrics whose `type` field is `"FINAL_RESPONSE_QUALITY"` are picked up by this criterion. Rubrics with other `type` values (e.g. `"TRAJECTORY_QUALITY"`, `"TOOL_USE_QUALITY"`) are ignored.


This is inaccurate for criterion-level rubrics. In create_effective_rubrics_list, only rubrics from EvalCase.rubrics are filtered by type; the criterion-level rubrics in EvalConfig are added unconditionally regardless of type. As written, a reader will think their EvalConfig rubrics get dropped unless type matches.

Suggested change

- Only rubrics whose `type` field is `"FINAL_RESPONSE_QUALITY"` are picked up by this criterion. Rubrics with other `type` values (e.g. `"TRAJECTORY_QUALITY"`, `"TOOL_USE_QUALITY"`) are ignored.

- Rubrics supplied per-case via `EvalCase.rubrics` are filtered by `type`: only those whose `type` is `"FINAL_RESPONSE_QUALITY"` are merged in. Criterion-level rubrics in `EvalConfig` are **not** filtered by `type`.

koverholt · 2026-07-14T17:06:42Z

+
+- Rubrics on `EvalConfig.criteria["rubric_based_tool_use_quality_v1"].rubrics` **must be non-empty** — `RubricBasedEvaluator` asserts this at init time.
+- Rubrics on `EvalCase.rubrics` are *additive* on top of the criterion-level list, not a replacement. The effective rubric set passed to the judge is the union of both.
+- Only rubrics whose `type` field is `"TOOL_USE_QUALITY"` are picked up by this criterion. Rubrics with other `type` values (e.g. `"FINAL_RESPONSE_QUALITY"`, `"TRAJECTORY_QUALITY"`) are ignored.


Same scoping issue: only EvalCase.rubrics are filtered by type in create_effective_rubrics_list. Criterion-level rubrics are added unconditionally.

Suggested change

- Only rubrics whose `type` field is `"TOOL_USE_QUALITY"` are picked up by this criterion. Rubrics with other `type` values (e.g. `"FINAL_RESPONSE_QUALITY"`, `"TRAJECTORY_QUALITY"`) are ignored.

- Rubrics supplied per-case via `EvalCase.rubrics` are filtered by `type`: only those whose `type` is `"TOOL_USE_QUALITY"` are merged in. Criterion-level rubrics in `EvalConfig` are **not** filtered by `type`.

koverholt · 2026-07-14T17:06:56Z

+
+- Rubrics on `EvalConfig.criteria["rubric_based_multi_turn_trajectory_quality_v1"].rubrics` **must be non-empty** — `RubricBasedEvaluator` asserts this at init time.
+- Rubrics on `EvalCase.rubrics` are *additive* on top of the criterion-level list, not a replacement. The effective rubric set passed to the judge is the union of both.
+- Only rubrics whose `type` field is `"TRAJECTORY_QUALITY"` are picked up by this criterion. Rubrics with other `type` values (e.g. `"FINAL_RESPONSE_QUALITY"`, `"TOOL_USE_QUALITY"`) are ignored.


Same scoping issue: only EvalCase.rubrics are filtered by type. Criterion-level rubrics are added unconditionally.

Suggested change

- Only rubrics whose `type` field is `"TRAJECTORY_QUALITY"` are picked up by this criterion. Rubrics with other `type` values (e.g. `"FINAL_RESPONSE_QUALITY"`, `"TOOL_USE_QUALITY"`) are ignored.

- Rubrics supplied per-case via `EvalCase.rubrics` are filtered by `type`: only those whose `type` is `"TRAJECTORY_QUALITY"` are merged in. Criterion-level rubrics in `EvalConfig` are **not** filtered by `type`.

koverholt · 2026-07-14T17:07:24Z

            "text_property": "The agent's response is direct and to the point."
-          }
+          },
+          "type": "FINAL_RESPONSE_QUALITY"


type on a criterion-level rubric has no effect (it's only used to filter EvalCase.rubrics). Adding it here implies criterion rubrics are type-filtered, which they aren't. Recommend removing these two type lines, or moving the demonstration into a separate short EvalCase.rubrics snippet where type actually matters.

koverholt · 2026-07-14T17:07:36Z

            "text_property": "The agent calls the GeoCoding tool before calling the GetWeather tool."
-          }
+          },
+          "type": "TOOL_USE_QUALITY"


Same as the final-response example: remove type from these criterion-level rubrics, or demonstrate it in an EvalCase.rubrics example instead, since type is inert at the criterion level.

koverholt · 2026-07-14T17:08:04Z

+          "rubric_content": {
+            "text_property": "The agent asks about individual factors (age, prior conditions, current medication) before giving any individualized advice."
+          },
+          "type": "TRAJECTORY_QUALITY"


Same as above: type on criterion-level rubrics is inert (only EvalCase.rubrics are type-filtered). Remove or relocate to an EvalCase.rubrics example.

koverholt · 2026-07-14T17:16:48Z

+          "rubric_content": {
+            "text_property": "When the user discloses risk-relevant information late in the conversation, the agent revisits and corrects earlier advice rather than leaving it standing."
+          },
+          "type": "TRAJECTORY_QUALITY"


Same as line 461: type here is inert (criterion-level rubric). Remove both, or move to an EvalCase.rubrics example.

zettaittenani marked this pull request as ready for review July 14, 2026 16:18

Merge branch 'main' into docs/evaluate-rubric-clarifications

23cf409

zettaittenani mentioned this pull request Jul 14, 2026

Found docs updates needed from ADK python release v2.0.0 to v2.2.0 #1833

Open

koverholt requested changes Jul 14, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

docs(evaluate): document rubric_based_multi_turn_trajectory_quality_v1 and clarify Rubric.type / criterion.rubrics#1853

docs(evaluate): document rubric_based_multi_turn_trajectory_quality_v1 and clarify Rubric.type / criterion.rubrics#1853
zettaittenani wants to merge 2 commits into
google:mainfrom
zettaittenani:docs/evaluate-rubric-clarifications

zettaittenani commented Jun 22, 2026 •

edited

Loading

Uh oh!

netlify Bot commented Jun 22, 2026 •

edited

Loading

Uh oh!

koverholt left a comment

Uh oh!

koverholt Jul 14, 2026

Uh oh!

koverholt Jul 14, 2026

Uh oh!

koverholt Jul 14, 2026

Uh oh!

koverholt Jul 14, 2026

Uh oh!

koverholt Jul 14, 2026

Uh oh!

koverholt Jul 14, 2026

Uh oh!

koverholt Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	- Only rubrics whose `type` field is `"FINAL_RESPONSE_QUALITY"` are picked up by this criterion. Rubrics with other `type` values (e.g. `"TRAJECTORY_QUALITY"`, `"TOOL_USE_QUALITY"`) are ignored.
	- Rubrics supplied per-case via `EvalCase.rubrics` are filtered by `type`: only those whose `type` is `"FINAL_RESPONSE_QUALITY"` are merged in. Criterion-level rubrics in `EvalConfig` are not filtered by `type`.

	- Only rubrics whose `type` field is `"TOOL_USE_QUALITY"` are picked up by this criterion. Rubrics with other `type` values (e.g. `"FINAL_RESPONSE_QUALITY"`, `"TRAJECTORY_QUALITY"`) are ignored.
	- Rubrics supplied per-case via `EvalCase.rubrics` are filtered by `type`: only those whose `type` is `"TOOL_USE_QUALITY"` are merged in. Criterion-level rubrics in `EvalConfig` are not filtered by `type`.

Uh oh!

Conversation

zettaittenani commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Changes

Testing

CLA

Notes

Uh oh!

netlify Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for adk-docs-preview ready!

Uh oh!

koverholt left a comment

Choose a reason for hiding this comment

Uh oh!

koverholt Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

koverholt Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

koverholt Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

koverholt Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

koverholt Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

koverholt Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

koverholt Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zettaittenani commented Jun 22, 2026 •

edited

Loading

netlify Bot commented Jun 22, 2026 •

edited

Loading