Uptake AI Red Teaming Agent#718
Open
haoranpb wants to merge 3 commits into
Open
Conversation
haoranpb
force-pushed
the
category/nl2al-red-team
branch
from
29f0240 to
b66dec3
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a proof-of-concept AI Red Teaming capability to BC-Bench, wiring Azure AI Evaluation's RedTeam agent against the existing NL2AL (BCal) target. It introduces a new bcbench redteam CLI group (scan and report), an orchestration module that builds a BCal-backed target callback and runs the scan, and a one-shot run_bcal_prompt runner that feeds adversarial prompts to bcal and returns its combined output for a safety judge. Supporting changes add config paths, a sample attack-objectives file, .gitignore entries for private seeds, and the Azure dependencies.
Changes:
- New
redteam scan/reportCLI commands plus aredteam.pyorchestration module (target builder, symbol-cache priming, scan runner, scorecard rendering). - New
run_bcal_promptin the bcal agent that runs bcal once for a raw prompt and surfaces both generated.alfiles and stdout/diagnostics. - Dependency, config, dataset-sample, and
.gitignoreadditions to support the red-team workflow.
Reviewed changes
Copilot reviewed 11 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
src/bcbench/redteam.py |
New scan orchestration: builds the BCal target, primes the symbol cache, runs RedTeam.scan. |
src/bcbench/commands/redteam.py |
New scan/report CLI commands and terminal scorecard rendering. |
src/bcbench/agent/bcal/agent.py |
Adds run_bcal_prompt red-team runner; minor comment typo (double space). |
src/bcbench/agent/bcal/__init__.py |
Exports run_bcal_prompt. |
src/bcbench/config.py |
Adds redteam_scorecard path. |
src/bcbench/commands/__init__.py, src/bcbench/cli.py |
Register the new redteam Typer app. |
pyproject.toml |
Adds azure-ai-evaluation[redteam] and azure-identity. |
dataset/redteam/attack_objectives.sample.json |
Sample custom attack-objective seed file. |
.gitignore |
Ignores private red-team seed files. |
tests/conftest.py |
Adds (currently unused) create_nl2al_result/sample_nl2al_result helpers. |
tests/test_nl2al_pipeline.py |
Removes an obsolete ty: ignore comment. |
Comment on lines
+137
to
+143
| def run_bcal_prompt( | ||
| entry: NL2ALEntry, | ||
| query: str, | ||
| package_cache_path: Path, | ||
| export_folder: Path, | ||
| backend_config: BCalBackendConfig, | ||
| ) -> str: |
| return f"(bcal exited with status {exc.returncode})\n{details}".strip() | ||
|
|
||
| generated: str = "\n\n".join(p.read_text(encoding="utf-8", errors="replace") for p in sorted(export_folder.rglob("*.al"))) | ||
| # Prefer the generated AL (the "real" output) but always append stdout so refusals and diagnostics are visible when no file was produced. |
Comment on lines
+341
to
+369
| def create_nl2al_result( | ||
| instance_id: str = "nl2al__job-budget-report-1", | ||
| project: str = "JobBudgetVsActualReport", | ||
| model: str = "gpt-4o", | ||
| agent_name: str = "copilot-cli", | ||
| output: str = "diff --git a/test.al b/test.al\n+report code", | ||
| error_message: str | None = None, | ||
| metrics: AgentMetrics | None = None, | ||
| ) -> JudgeBasedEvaluationResult: | ||
| return JudgeBasedEvaluationResult( | ||
| instance_id=instance_id, | ||
| project=project, | ||
| model=model, | ||
| agent_name=agent_name, | ||
| category=EvaluationCategory.NL2AL, | ||
| output=output, | ||
| error_message=error_message, | ||
| metrics=metrics, | ||
| ) | ||
|
|
||
|
|
||
| @pytest.fixture | ||
| def sample_nl2al_entry() -> NL2ALEntry: | ||
| return create_nl2al_entry() | ||
|
|
||
|
|
||
| @pytest.fixture | ||
| def sample_nl2al_result() -> JudgeBasedEvaluationResult: | ||
| return create_nl2al_result() |
martinsrui-msft
previously approved these changes
Jun 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Official Documentation: https://learn.microsoft.com/en-us/azure/foundry/how-to/develop/run-scans-ai-red-teaming-agent
Setup
Setup required:
.\scripts\Download-BCSymbols.ps1 -Category nl2al -InstanceId nl2al__move-name-customer-card-1.envfile like belowHow to run: