feat(datagen): add the corpus generation tooling - #15698
Open
anticorrelator wants to merge 95 commits into
Open
Conversation
Adds scripts/datagen (mock OpenAI-compatible provider plus two PEP 723 scenario scripts using real OpenInference instrumenters) and two recorded corpora (openai_chat_sessions, langchain_agent_rag) as OTLP protobuf-JSON lines with manifests under src/phoenix/datagen/corpora. Claude-Session: https://claude.ai/code/session_01YF3zGrMPmFKZhQUjowsCJi
Adds src/phoenix/datagen (corpus loader for local paths and URLs, session-aware replayer with ID/timestamp rewriting, contamination-mixture anomaly injection with ground-truth manifests, OTLP/HTTP export) and wires a lazily imported 'phoenix datagen' subcommand with env-var/flag config. Claude-Session: https://claude.ai/code/session_01YF3zGrMPmFKZhQUjowsCJi
Spans are grouped by recorded trace_id across all corpus lines, so corpora batched one-request-per-span replay as intact traces; manifest validation now counts distinct trace ids and total spans. Claude-Session: https://claude.ai/code/session_01YF3zGrMPmFKZhQUjowsCJi
Bundles the recorded corpora in the wheel, restores parent-child end containment after latency redraws, rebases span-event timestamps, honors PHOENIX_CLIENT_HEADERS, preserves dangling recorded parents, and pins the numeric, export, and corpus-fidelity contracts in tests. Claude-Session: https://claude.ai/code/session_01YF3zGrMPmFKZhQUjowsCJi
Adds --project/PHOENIX_PROJECT_NAME with a datagen-<corpus> default emitted as the openinference.project.name resource attribute, derives emitted IDs from the seed plus a per-run nonce so same-seed reruns stay comparable without colliding, and retries OTLP export with capped exponential backoff instead of exiting on transport errors. Claude-Session: https://claude.ai/code/session_01YF3zGrMPmFKZhQUjowsCJi
Adds an opt-in docker-compose datagen profile, a disabled-by-default Helm datagen deployment, a kustomize datagen overlay, a commented-out Render worker, and a self-hosting docs page covering local, Compose, Helm, Kustomize, Render, Railway, and Cloud Run flows. Claude-Session: https://claude.ai/code/session_01YF3zGrMPmFKZhQUjowsCJi
…stribution Adds the v2 fragment-bank schema and loader, a resumable USD-capped generation control plane with OpenAI Batch support, deterministic fake tools, self-play and scripted recording lanes, six archetype recorders, MinHash dedup and quality gating with atomic bank packaging, an archetype-safe replay session composer with lognormal session/gap knobs, checksum-verified bank fetch/pull with lazy resolution, a datagen-assets release workflow (--latest=false), a wheel starter-assets size gate, and checksum-pinned container asset baking. Claude-Session: https://claude.ai/code/session_01YF3zGrMPmFKZhQUjowsCJi
Every recorder script's PEP 723 block now resolves under the repository's three-day `exclude-newer` window, and each recorder was run end-to-end, keyless, against the in-repo mock provider to confirm it emits its archetype's span kinds with session.id attached. - tool_agent, graph_multi_agent: langchain-core 1.5.6, langchain-openai 1.5.1, openinference-instrumentation-langchain 0.1.70. The previous langchain-openai/openai pins were mutually unsatisfiable, and the 2024-era LangChain instrumenter predates get_attributes_from_context. - langchain_agent_rag: llama-index-core 0.14.23, the newest release outside the freshness window. - guardrailed_app: guardrails-ai 0.5.0. Every published OpenInference Guardrails instrumenter supports only `>=0.4.5,<0.5.1`, so the 0.6.7 pin left the instrumentor disabled and no GUARDRAIL span was recorded. - Each recorder that imports `openinference.instrumentation` directly now declares it, and openai is pinned at one version across the directory. The mock provider now synthesizes tool-call arguments from the caller's own declared tool schema instead of a fixed delivery-estimate shape, so the tool-agent registry validates them, and it serves a server-sent-event stream when a request asks for one, which the plain-chat recorder needs.
…d model backends Versioned application-profile contract with canonical run snapshots; matrix v2 draws every conversational field inside one profile with reproducible per-field streams, 10% targeted-seed cells, and Beta(2,8) seed intensities; structured OpenAI and codex-exec backends with provider-aware attempt accounting (priced vs subscription); scripted and self-play lanes consume profile draws. Claude-Session: https://claude.ai/code/session_01Jrru1FDRB5uKGFGq6Rwxst
…ents Profile seeds gain authored subtle/moderate/strong effect variants with a closed vocabulary (corpus edits, tool-result overlays, simulator traits); a pure materializer maps (cell_id, seed_id, intensity) to one variant and projects a seed-metadata-free environment into both lanes; fake tools overlay successful results after native handling; targeted cells add only a natural route; transcripts reject internal seed language. Claude-Session: https://claude.ai/code/session_01Jrru1FDRB5uKGFGq6Rwxst
Immutable judging-input and judgment sidecars keyed by cell and fragment; recorded seed-engagement signals (tool overlay applications, perturbed- document serving, trait seeds by construction) determine ambient proximity; deterministic routing judges every trap-proximate fragment plus a 5% stratified baseline; versioned three-label rubric (survived/degraded/failed) through the frontier ModelBackend binding; outcomes project into schema-v2 quality metadata and never gate acceptance. Claude-Session: https://claude.ai/code/session_01Jrru1FDRB5uKGFGq6Rwxst
Add --rate-schedule (flat|business-hours), --timezone, --backfill, and --error-rate to phoenix datagen. The replayer paces a virtual timeline that follows weekday/weekend hourly tiers, can start in the past and catch up at exporter throughput, and can mark recorded LLM/TOOL spans as failed with an exception event and ancestor status propagation. Anomaly manifest rows gain an additive kind and timing fields. Flat defaults leave emitted requests unchanged. Claude-Session: https://claude.ai/code/session_01EvQfhu4vtHASNDPReHB5t9
Runs can allocate one provider or tool fault per matrix cell (--fault-fraction, --fault-modes) with base-bank lineage (--base-scenario-name, --base-archive-sha256). The mock provider injects a selected fault once so the real SDK and instrumenters record retry and exception topology; tool exceptions cross the agent loop as error tool messages. Every fault fragment is judged through a dedicated route. bank.py gains package and merge commands that union a validated supplement into its base archive, rebuilding aggregates and recording per-input instrumenter provenance. README documents the supplemental procedure and the prepare-only publication handoff. Claude-Session: https://claude.ai/code/session_01EvQfhu4vtHASNDPReHB5t9
Chat fixtures carry authored opening phrasings; live recording picks one per run so the session list does not repeat identical first messages. Conditioned runs keep their materialized opening. Claude-Session: https://claude.ai/code/session_014gvDFFS2FTKCnjCnQpcdng
The datagen unit tests import the recorder modules under scripts/, but the unit-test job's sparse checkout dropped that directory when the DSL compile checks moved to their own job, so collection failed with ModuleNotFoundError. Re-add it. Also pin the live chat test to the authored opening so the new per-run phrasing choice cannot make its assertions flaky. Claude-Session: https://claude.ai/code/session_014gvDFFS2FTKCnjCnQpcdng
The test passed every coding fixture to the scripted recorder, which raises for the new live-only tasks. Select only the fixtures that carry a deterministic scripted episode. Claude-Session: https://claude.ai/code/session_014gvDFFS2FTKCnjCnQpcdng
The recorder and corpus-pipeline tests import scripts.datagen.*, which forced the Unit Tests CI job to check out all of scripts/ and forced a mypy override for the un-followable script modules. That put dev tooling on the wrong side of the product test boundary. - Move the ten scripts-importing test files to scripts/datagen/tests/ with their own conftest and fragment_bank fixture copy - Restore the Unit Tests sparse-checkout to main's list - Add a path-filtered Datagen Tooling Tests job that runs uv run pytest scripts/datagen/tests - Drop the scripts.datagen.* mypy override Claude-Session: https://claude.ai/code/session_014gvDFFS2FTKCnjCnQpcdng
Split per review feedback: the generation tooling (scripts/datagen) and the deployment integration + docs (compose, kustomize, helm, render, self-hosting page) move to follow-up PRs. This PR keeps the feature that stands alone: the phoenix.datagen runtime (fetcher, loader, composer, replayer, exporter), the datagen server subcommand, and their unit tests. Reverts the langchain test pins and mypy carve-outs the generation tests had pushed into the shared dev environment. Claude-Session: https://claude.ai/code/session_014gvDFFS2FTKCnjCnQpcdng
Recorder scripts, condition perturbations, corpus packer, and publish staging for the datagen corpus, as PEP 723 scripts run with uv run --script — their dependencies stay out of the Phoenix development environment. Tests cover the dependency-free pipeline pieces (conditions, packer, fetcher roundtrip) and run in a dedicated path-filtered CI job; recorder behavior is exercised by generating a corpus rather than by unit tests. Claude-Session: https://claude.ai/code/session_014gvDFFS2FTKCnjCnQpcdng
dosubot
Bot
added
the
size:XXL
anticorrelator
changed the title
The unit-test tox env installs with -U and resolves pydantic-ai-slim fresh, so it picked up 2.34.0, whose source no longer matches the vendored data-stream protocol types in src/phoenix/db/types. test_data_stream_protocol_compatibility now fails on every PR (verified: 2.33.0 passes, 2.34.0 fails the request-type and schema parity tests). Cap the unit env at 2.33 parity until the vendored types are re-synced. Claude-Session: https://claude.ai/code/session_014gvDFFS2FTKCnjCnQpcdng
A span carrying only prompt and total counts (the embedding-span shape) fell through to the independent total jitter, so the emitted total could drop below the jittered prompt count. Recompute the total from the jittered components whenever either one is present, leaving the independent jitter for spans that carry only a total. Claude-Session: https://claude.ai/code/session_014gvDFFS2FTKCnjCnQpcdng
Main upgraded pydantic-ai-slim to 2.34 and re-synced the vendored data-stream types (#15714), so the stopgap cap this branch carried is no longer needed. Claude-Session: https://claude.ai/code/session_014gvDFFS2FTKCnjCnQpcdng
Main removed phoenix.utilities.re with the legacy client cleanup (#15709), which deleted parse_env_headers from the server package. Parse the W3C Baggage-style header string locally instead of reaching into the client package. Claude-Session: https://claude.ai/code/session_014gvDFFS2FTKCnjCnQpcdng
Review feedback: hide the datagen subcommand from the top-level help (help=SUPPRESS, matching db) while documenting it for direct --help invocation, add the no-compatibility disclaimer to the package docstring, and move the package to phoenix.experimental.datagen so the wheel's import surface carries the same signal. Claude-Session: https://claude.ai/code/session_014gvDFFS2FTKCnjCnQpcdng
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #15614 (the datagen replayer runtime, now merged); targets
main.Adds the corpus generation tooling that produces what the replayer consumes:
scripts/datagen/recorder scripts (PEP 723, run viauv run --script) for six application archetypes, with offline scripted providers and live-model recordingorganic_conditions.json) and the condition materializerscripts/datagen/README.mdRecorder dependencies (langchain, llama-index, guardrails, instrumenters) stay entirely inside the scripts' PEP 723 metadata — nothing is added to the Phoenix development environment or the unit-test requirements.
Tests cover only the dependency-free pipeline pieces (conditions, packer, fetcher roundtrip) and run in a dedicated path-filtered
Datagen Tooling TestsCI job, separate from the product unit suite. Recorder behavior is verified by generating a corpus, not by unit tests.https://claude.ai/code/session_014gvDFFS2FTKCnjCnQpcdng