From 31b102b6e814a9073542ddc4db6ab6b681672bee Mon Sep 17 00:00:00 2001 From: bcbench Date: Mon, 6 Jul 2026 09:44:37 +0200 Subject: [PATCH 1/3] docs: design spec and implementation plan for marketplace & local plugins experiment toggle Adds a design spec and a task-by-task implementation plan for a new experiment toggle that installs agent plugins (a marketplace pinned to a git commit, or a local marketplace) into the running CLI via its own 'plugin marketplace add' + 'plugin install' commands, records them in ExperimentConfiguration, and removes them after the run. Docs only; no code changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...026-07-06-marketplace-and-local-plugins.md | 973 ++++++++++++++++++ ...ace-and-local-plugins-experiment-design.md | 241 +++++ 2 files changed, 1214 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-06-marketplace-and-local-plugins.md create mode 100644 docs/superpowers/specs/2026-07-03-marketplace-and-local-plugins-experiment-design.md diff --git a/docs/superpowers/plans/2026-07-06-marketplace-and-local-plugins.md b/docs/superpowers/plans/2026-07-06-marketplace-and-local-plugins.md new file mode 100644 index 000000000..42f67751a --- /dev/null +++ b/docs/superpowers/plans/2026-07-06-marketplace-and-local-plugins.md @@ -0,0 +1,973 @@ +# Marketplace & Local Plugins Experiment Toggle — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add a config-driven experiment toggle that installs agent plugins (marketplace-pinned-to-commit or local) into the running CLI via its own `plugin marketplace add` + `plugin install` commands, records them in results, and cleanly removes them afterward. + +**Architecture:** A new `operations/plugin_operations.py` reads a `plugins` list from `agent/shared/config.yaml`, clones each marketplace at its pinned commit into `/.bcbench/plugins/`, installs the requested plugins by name via the CLI commands (user scope, both Copilot and Claude), and returns records folded into `ExperimentConfiguration`. A symmetric teardown (`uninstall` + `marketplace remove`) runs in a `finally` in each agent runner. `.bcbench` content is removed by `git clean` and an inline clean-before. Config injection (`extraKnownMarketplaces`) is NOT used — it is trust-dialog-gated and ignored in headless mode. + +**Tech Stack:** Python 3.13, Pydantic v2, Typer, `subprocess` for `git` + the agent CLIs, pytest with `unittest.mock`, `uv` for running commands. + +## Global Constraints + +- Python `>=3.13`; strong typing / type hints required (ruff `ANN`). +- Run everything through `uv` (e.g. `uv run pytest ...`, `uv run ruff ...`). +- Use the module logger (`get_logger(__name__)`); never `print` in `src/bcbench`. +- Line length 200; ruff rules per `pyproject.toml`. `subprocess.run` MUST pass an explicit `check=` (ruff `PLW1510`). +- No new domain model types for recording — `ExperimentConfiguration.plugins` is `list[str] | None`, mirroring `mcp_servers`. +- Marketplace/local source content must be a **marketplace root** (a directory containing `.claude-plugin/marketplace.json` or `.github/plugin/marketplace.json`). +- Do NOT modify the AL-LSP plugin mechanism (`build_al_lsp_plugin`, `--al-lsp`, `--plugin-dir`). +- Commit after every task (frequent commits). + +--- + +### Task 1: Add `plugins` field to `ExperimentConfiguration` + +**Files:** +- Modify: `src/bcbench/types.py` (class `ExperimentConfiguration`, ~lines 74-104) +- Test: `tests/test_experiment_configuration.py` + +**Interfaces:** +- Produces: `ExperimentConfiguration(plugins: list[str] | None = None)`; `is_empty()` returns True only when `plugins is None` along with the existing fields. + +- [ ] **Step 1: Write the failing tests** + +Add these methods to class `TestExperimentConfiguration` in `tests/test_experiment_configuration.py`: + +```python + def test_default_plugins_is_none(self): + config = ExperimentConfiguration() + + assert config.plugins is None + assert config.is_empty() + + def test_with_plugins(self): + plugins = ["frontend-web-dev@a1b2c3d4", "my-local-plugin@local"] + config = ExperimentConfiguration(plugins=plugins) + + assert config.plugins == plugins + assert not config.is_empty() + + def test_empty_plugins_list_is_not_empty_config(self): + config = ExperimentConfiguration(plugins=[]) + + assert config.plugins == [] + assert not config.is_empty() +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `uv run pytest tests/test_experiment_configuration.py -v` +Expected: FAIL — `TypeError: unexpected keyword argument 'plugins'` (and `test_default_plugins_is_none` fails on `config.plugins`). + +- [ ] **Step 3: Add the field and update `is_empty`** + +In `src/bcbench/types.py`, inside `class ExperimentConfiguration`, add the field after `custom_agent`: + +```python + # Custom agent name used in experiment (if any) + custom_agent: str | None = None + + # Plugins installed for this experiment: "@" (marketplace) or "@local" + plugins: list[str] | None = None +``` + +Then extend `is_empty`: + +```python + def is_empty(self) -> bool: + """Check if this configuration has all default/empty values. + + An empty configuration means no special experiment settings were used. + This is useful for comparing with None (no experiment) vs default experiment. + """ + return self.mcp_servers is None and self.al_lsp_enabled is False and self.custom_instructions is False and self.skills_enabled is False and self.custom_agent is None and self.plugins is None +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `uv run pytest tests/test_experiment_configuration.py -v` +Expected: PASS (all tests, including the 3 new ones). + +- [ ] **Step 5: Commit** + +```bash +git add src/bcbench/types.py tests/test_experiment_configuration.py +git commit -m "feat: record installed plugins in ExperimentConfiguration" +``` + +--- + +### Task 2: Add `clone_at_commit` git helper + +**Files:** +- Modify: `src/bcbench/operations/git_operations.py` +- Modify: `src/bcbench/operations/__init__.py` (export) +- Test: `tests/test_git_operations.py` + +**Interfaces:** +- Produces: `clone_at_commit(repo: str, commit: str, dest: Path) -> None` — clones `repo` (a `owner/repo` slug or a full git URL) at `commit` into `dest` (created if missing), shallowly. + +- [ ] **Step 1: Write the failing test** + +Add to `tests/test_git_operations.py` (create the file if it does not exist, with the import block shown): + +```python +from pathlib import Path +from unittest.mock import call, patch + +from bcbench.operations.git_operations import clone_at_commit + + +@patch("bcbench.operations.git_operations.subprocess.run") +def test_clone_at_commit_owner_repo_builds_https_url(mock_run, tmp_path): + dest = tmp_path / "clone" + + clone_at_commit("github/awesome-copilot", "a" * 40, dest) + + assert dest.is_dir() + commands = [c.args[0] for c in mock_run.call_args_list] + assert ["git", "init", "-q"] in commands + assert ["git", "remote", "add", "origin", "https://github.com/github/awesome-copilot.git"] in commands + assert ["git", "fetch", "--depth", "1", "origin", "a" * 40] in commands + assert ["git", "checkout", "-q", "FETCH_HEAD"] in commands + + +@patch("bcbench.operations.git_operations.subprocess.run") +def test_clone_at_commit_full_url_used_verbatim(mock_run, tmp_path): + clone_at_commit("https://gitlab.com/o/r.git", "b" * 40, tmp_path / "c") + + remote_calls = [c.args[0] for c in mock_run.call_args_list if c.args[0][:3] == ["git", "remote", "add"]] + assert remote_calls == [["git", "remote", "add", "origin", "https://gitlab.com/o/r.git"]] +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run pytest tests/test_git_operations.py -k clone_at_commit -v` +Expected: FAIL — `ImportError: cannot import name 'clone_at_commit'`. + +- [ ] **Step 3: Implement `clone_at_commit`** + +Append to `src/bcbench/operations/git_operations.py`: + +```python +def clone_at_commit(repo: str, commit: str, dest: Path) -> None: + """Shallow-clone `repo` at a specific `commit` into `dest`. + + Args: + repo: A GitHub `owner/repo` slug or a full git URL (`https://...` or `...git`). + commit: The 40-char commit SHA to check out (GitHub allows fetching a SHA directly). + dest: Target directory (created if missing). + """ + url = repo if ("://" in repo or repo.endswith(".git")) else f"https://github.com/{repo}.git" + logger.info(f"Cloning {url} @ {commit} into {dest}") + dest.mkdir(parents=True, exist_ok=True) + subprocess.run(["git", "init", "-q"], cwd=dest, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, check=True) + subprocess.run(["git", "remote", "add", "origin", url], cwd=dest, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, check=True) + subprocess.run(["git", "fetch", "--depth", "1", "origin", commit], cwd=dest, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, check=True) + subprocess.run(["git", "checkout", "-q", "FETCH_HEAD"], cwd=dest, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, check=True) + logger.info(f"Cloned {url} @ {commit}") +``` + +In `src/bcbench/operations/__init__.py`, add `clone_at_commit` to the `git_operations` import block and to `__all__` (keep both alphabetized): + +```python +from bcbench.operations.git_operations import ( + apply_patch, + checkout_commit, + clean_project_paths, + clean_repo, + clone_at_commit, + commit_changes, + stage_and_get_diff, +) +``` + +```python + "checkout_commit", + "clean_project_paths", + "clean_repo", + "clone_at_commit", + "commit_changes", +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `uv run pytest tests/test_git_operations.py -k clone_at_commit -v` +Expected: PASS (both tests). + +- [ ] **Step 5: Commit** + +```bash +git add src/bcbench/operations/git_operations.py src/bcbench/operations/__init__.py tests/test_git_operations.py +git commit -m "feat: add clone_at_commit git helper for pinned marketplace clones" +``` + +--- + +### Task 3: `plugin_operations.py` — materialize + marketplace-name helpers + +**Files:** +- Create: `src/bcbench/operations/plugin_operations.py` +- Test: `tests/test_plugin_operations.py` + +**Interfaces:** +- Produces: + - `class InstalledPlugin(NamedTuple)` with `plugin: str`, `marketplace: str`, `record: str`. + - `_read_marketplace_name(marketplace_dir: Path) -> str` — reads `name` from `.claude-plugin/marketplace.json` or `.github/plugin/marketplace.json`. + - `_materialize(entry_cfg: dict, entry: BaseDatasetEntry, plugins_root: Path) -> tuple[Path, str]` — clones (marketplace) or copies (local) into `plugins_root`, returns `(marketplace_dir, record_suffix)` where `record_suffix` is the commit (marketplace) or the literal `"local"`. + +- [ ] **Step 1: Write the failing tests** + +Create `tests/test_plugin_operations.py`: + +```python +import json +from pathlib import Path + +import pytest + +from bcbench.exceptions import AgentError +from bcbench.operations import plugin_operations as po +from tests.conftest import create_dataset_entry + + +def _make_marketplace(root: Path, manifest_rel: str = ".claude-plugin/marketplace.json", name: str = "probe-mp") -> Path: + """Create a minimal marketplace directory and return its root.""" + manifest = root / manifest_rel + manifest.parent.mkdir(parents=True, exist_ok=True) + manifest.write_text(json.dumps({"name": name, "plugins": [{"name": "probe-plugin", "source": "./probe-plugin"}]}), encoding="utf-8") + plugin_json = root / "probe-plugin" / ".claude-plugin" / "plugin.json" + plugin_json.parent.mkdir(parents=True, exist_ok=True) + plugin_json.write_text(json.dumps({"name": "probe-plugin", "version": "0.0.1"}), encoding="utf-8") + return root + + +def test_read_marketplace_name_claude_plugin_path(tmp_path): + _make_marketplace(tmp_path, ".claude-plugin/marketplace.json", name="my-mp") + assert po._read_marketplace_name(tmp_path) == "my-mp" + + +def test_read_marketplace_name_github_plugin_path(tmp_path): + _make_marketplace(tmp_path, ".github/plugin/marketplace.json", name="gh-mp") + assert po._read_marketplace_name(tmp_path) == "gh-mp" + + +def test_read_marketplace_name_missing_raises(tmp_path): + with pytest.raises(AgentError, match="marketplace.json"): + po._read_marketplace_name(tmp_path) + + +def test_materialize_marketplace_clones_at_commit(tmp_path, monkeypatch): + plugins_root = tmp_path / "plugins_root" + plugins_root.mkdir() + + def fake_clone(repo: str, commit: str, dest: Path) -> None: + _make_marketplace(dest, name="cloned-mp") + + monkeypatch.setattr(po, "clone_at_commit", fake_clone) + entry = create_dataset_entry() + entry_cfg = {"source": "marketplace", "repo": "github/awesome-copilot", "commit": "a" * 40, "plugins": ["probe-plugin"]} + + marketplace_dir, record_suffix = po._materialize(entry_cfg, entry, plugins_root) + + assert (marketplace_dir / ".claude-plugin" / "marketplace.json").is_file() + assert record_suffix == "a" * 40 + + +def test_materialize_local_copies_from_instructions(tmp_path, monkeypatch): + plugins_root = tmp_path / "plugins_root" + plugins_root.mkdir() + source_root = tmp_path / "instructions" + _make_marketplace(source_root / "plugins" / "my-mp", name="local-mp") + + monkeypatch.setattr(po, "_get_source_instructions_path", lambda repo: source_root) + entry = create_dataset_entry() + entry_cfg = {"source": "local", "path": "plugins/my-mp", "plugins": ["probe-plugin"]} + + marketplace_dir, record_suffix = po._materialize(entry_cfg, entry, plugins_root) + + assert (marketplace_dir / ".claude-plugin" / "marketplace.json").is_file() + assert record_suffix == "local" + + +def test_materialize_local_missing_raises(tmp_path, monkeypatch): + monkeypatch.setattr(po, "_get_source_instructions_path", lambda repo: tmp_path / "nope") + entry = create_dataset_entry() + entry_cfg = {"source": "local", "path": "plugins/absent", "plugins": ["x"]} + + with pytest.raises(AgentError, match="Local plugin marketplace not found"): + po._materialize(entry_cfg, entry, tmp_path / "plugins_root") + + +def test_materialize_unknown_source_raises(tmp_path): + entry = create_dataset_entry() + with pytest.raises(AgentError, match="Unknown plugin source"): + po._materialize({"source": "bogus", "plugins": []}, entry, tmp_path) +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `uv run pytest tests/test_plugin_operations.py -v` +Expected: FAIL — `ModuleNotFoundError: No module named 'bcbench.operations.plugin_operations'`. + +- [ ] **Step 3: Create `plugin_operations.py` with the helpers** + +Create `src/bcbench/operations/plugin_operations.py`: + +```python +"""Install agent plugins (marketplace or local) declared in config, via the CLI's plugin commands. + +Config injection (`extraKnownMarketplaces` / `enabledPlugins`) is trust-dialog-gated and ignored in +headless mode, so we drive the CLI's real `plugin marketplace add` + `plugin install` commands. +Marketplace content is cloned at its pinned commit into `/.bcbench/plugins/` (repo-local, +cleaned by `git clean` and an inline clean-before). +""" + +from __future__ import annotations + +import json +import subprocess +from pathlib import Path +from shutil import copytree, rmtree +from typing import NamedTuple + +from bcbench.dataset import BaseDatasetEntry +from bcbench.exceptions import AgentError +from bcbench.logger import get_logger +from bcbench.operations.git_operations import clone_at_commit +from bcbench.operations.instruction_operations import _get_source_instructions_path +from bcbench.types import AgentType + +logger = get_logger(__name__) + +# NOTE: do NOT import from `bcbench.agent.*` here. `bcbench.agent.__init__` imports the runners, +# which import `bcbench.operations` — importing agent from operations would create a cycle. We +# therefore inline the `.bcbench/plugins` cleanup instead of reusing `remove_agent_plugin`. +_BCBENCH_ROOT = ".bcbench" +_PLUGINS_FOLDER = "plugins" # under /.bcbench/plugins/ +_MARKETPLACE_MANIFESTS = (".claude-plugin/marketplace.json", ".github/plugin/marketplace.json") + + +class InstalledPlugin(NamedTuple): + plugin: str + marketplace: str + record: str # "@" (marketplace) or "@local" + + +def _slug(text: str) -> str: + return "".join(c if c.isalnum() else "-" for c in text).strip("-") + + +def _read_marketplace_name(marketplace_dir: Path) -> str: + for rel in _MARKETPLACE_MANIFESTS: + manifest = marketplace_dir / rel + if manifest.is_file(): + name = json.loads(manifest.read_text(encoding="utf-8")).get("name") + if name: + return name + raise AgentError(f"No marketplace.json with a 'name' found under {marketplace_dir}") + + +def _materialize(entry_cfg: dict, entry: BaseDatasetEntry, plugins_root: Path) -> tuple[Path, str]: + """Clone (marketplace) or copy (local) the marketplace into plugins_root. + + Returns: + (marketplace_dir, record_suffix) where record_suffix is the pinned commit or "local". + """ + source = entry_cfg["source"] + match source: + case "marketplace": + repo: str = entry_cfg["repo"] + commit: str = entry_cfg["commit"] + dest = plugins_root / _slug(repo) + clone_at_commit(repo, commit, dest) + return dest, commit + case "local": + rel_path: str = entry_cfg["path"] + src = _get_source_instructions_path(entry.repo) / rel_path + if not src.is_dir(): + raise AgentError(f"Local plugin marketplace not found: {src}") + dest = plugins_root / _slug(rel_path) + if dest.exists(): + rmtree(dest) + copytree(src, dest) + return dest, "local" + case _: + raise AgentError(f"Unknown plugin source: {source!r}") +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `uv run pytest tests/test_plugin_operations.py -v` +Expected: PASS (7 tests). + +- [ ] **Step 5: Commit** + +```bash +git add src/bcbench/operations/plugin_operations.py tests/test_plugin_operations.py +git commit -m "feat: add plugin materialize + marketplace-name helpers" +``` + +--- + +### Task 4: `plugin_operations.py` — setup + teardown commands + +**Files:** +- Modify: `src/bcbench/operations/plugin_operations.py` +- Modify: `src/bcbench/operations/__init__.py` (exports) +- Test: `tests/test_plugin_operations.py` + +**Interfaces:** +- Consumes: `InstalledPlugin`, `_materialize`, `_read_marketplace_name` (Task 3); `AgentType`. +- Produces: + - `setup_plugins_from_config(agent_config: dict, entry: BaseDatasetEntry, repo_path: Path, agent_type: AgentType, cli_cmd: str) -> list[InstalledPlugin]` + - `teardown_plugins(cli_cmd: str, agent_type: AgentType, installed: list[InstalledPlugin]) -> None` + +- [ ] **Step 1: Write the failing tests** + +Append to `tests/test_plugin_operations.py`: + +```python +from unittest.mock import call + +from bcbench.types import AgentType + + +class _Recorder: + def __init__(self): + self.calls: list[list[str]] = [] + + def __call__(self, args, **kwargs): + self.calls.append(args) + + class _R: + returncode = 0 + + return _R() + + +def _marketplace_entry_cfg(): + return {"source": "marketplace", "repo": "github/awesome-copilot", "commit": "a" * 40, "plugins": ["probe-plugin"]} + + +def test_setup_no_plugins_returns_empty(tmp_path): + installed = po.setup_plugins_from_config({}, create_dataset_entry(), tmp_path, AgentType.COPILOT, "copilot") + assert installed == [] + + +def test_setup_disabled_entry_skipped(tmp_path, monkeypatch): + monkeypatch.setattr(po.subprocess, "run", _Recorder()) + cfg = {"plugins": [{**_marketplace_entry_cfg(), "enabled": False}]} + installed = po.setup_plugins_from_config(cfg, create_dataset_entry(), tmp_path, AgentType.COPILOT, "copilot") + assert installed == [] + + +def test_setup_installs_and_records(tmp_path, monkeypatch): + def fake_clone(repo, commit, dest): + _make_marketplace(dest, name="awesome-copilot") + + rec = _Recorder() + monkeypatch.setattr(po, "clone_at_commit", fake_clone) + monkeypatch.setattr(po.subprocess, "run", rec) + + cfg = {"plugins": [_marketplace_entry_cfg()]} + installed = po.setup_plugins_from_config(cfg, create_dataset_entry(), tmp_path, AgentType.COPILOT, "copilot") + + assert installed == [po.InstalledPlugin(plugin="probe-plugin", marketplace="awesome-copilot", record="probe-plugin@" + "a" * 40)] + # marketplace add + install were issued (self-heal remove/uninstall may precede them) + assert ["copilot", "plugin", "marketplace", "add", str(tmp_path / ".bcbench" / "plugins" / "github-awesome-copilot")] in rec.calls + assert ["copilot", "plugin", "install", "probe-plugin@awesome-copilot"] in rec.calls + + +def test_setup_failure_tears_down_partial_and_raises(tmp_path, monkeypatch): + def fake_clone(repo, commit, dest): + _make_marketplace(dest, name="awesome-copilot") + + monkeypatch.setattr(po, "clone_at_commit", fake_clone) + + def flaky_run(args, **kwargs): + class _R: + returncode = 0 + + if args[:4] == ["copilot", "plugin", "marketplace", "add"]: + raise po.subprocess.CalledProcessError(1, args) + return _R() + + monkeypatch.setattr(po.subprocess, "run", flaky_run) + cfg = {"plugins": [_marketplace_entry_cfg()]} + + with pytest.raises(AgentError, match="Plugin setup failed"): + po.setup_plugins_from_config(cfg, create_dataset_entry(), tmp_path, AgentType.COPILOT, "copilot") + + +def test_teardown_copilot_uninstalls_by_name(monkeypatch): + rec = _Recorder() + monkeypatch.setattr(po.subprocess, "run", rec) + installed = [po.InstalledPlugin("probe-plugin", "awesome-copilot", "probe-plugin@abc")] + + po.teardown_plugins("copilot", AgentType.COPILOT, installed) + + assert ["copilot", "plugin", "uninstall", "probe-plugin"] in rec.calls + assert ["copilot", "plugin", "marketplace", "remove", "awesome-copilot"] in rec.calls + + +def test_teardown_claude_uninstalls_by_qualified_name(monkeypatch): + rec = _Recorder() + monkeypatch.setattr(po.subprocess, "run", rec) + installed = [po.InstalledPlugin("probe-plugin", "awesome-copilot", "probe-plugin@abc")] + + po.teardown_plugins("claude", AgentType.CLAUDE, installed) + + assert ["claude", "plugin", "uninstall", "probe-plugin@awesome-copilot"] in rec.calls + assert ["claude", "plugin", "marketplace", "remove", "awesome-copilot"] in rec.calls + + +def test_teardown_empty_is_noop(monkeypatch): + rec = _Recorder() + monkeypatch.setattr(po.subprocess, "run", rec) + po.teardown_plugins("copilot", AgentType.COPILOT, []) + assert rec.calls == [] +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `uv run pytest tests/test_plugin_operations.py -k "setup or teardown" -v` +Expected: FAIL — `AttributeError: module 'bcbench.operations.plugin_operations' has no attribute 'setup_plugins_from_config'`. + +- [ ] **Step 3: Implement setup + teardown** + +Append to `src/bcbench/operations/plugin_operations.py`: + +```python +def _run_plugin_cmd(cli_cmd: str, args: list[str], *, check: bool) -> None: + subprocess.run([cli_cmd, "plugin", *args], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, check=check) + + +def _uninstall_args(agent_type: AgentType, plugin: str, marketplace: str) -> list[str]: + # Copilot uninstalls by plugin name; Claude by "@" (both verified live). + match agent_type: + case AgentType.COPILOT: + return ["uninstall", plugin] + case AgentType.CLAUDE: + return ["uninstall", f"{plugin}@{marketplace}"] + case _: + raise AgentError(f"Unsupported agent type for plugin teardown: {agent_type}") + + +def setup_plugins_from_config(agent_config: dict, entry: BaseDatasetEntry, repo_path: Path, agent_type: AgentType, cli_cmd: str) -> list[InstalledPlugin]: + """Install every enabled plugin entry into the CLI, before the agent launches. + + Returns the installed plugins (for recording and teardown). Raises AgentError on failure + (after tearing down any partial installs). Skips silently when no entry is enabled. + """ + entries = [e for e in (agent_config.get("plugins") or []) if e.get("enabled", True)] + if not entries: + return [] + + plugins_root = repo_path / _BCBENCH_ROOT / _PLUGINS_FOLDER + if plugins_root.exists(): + rmtree(plugins_root) # clean-before: wipe stale clones + plugins_root.mkdir(parents=True, exist_ok=True) + + installed: list[InstalledPlugin] = [] + try: + for entry_cfg in entries: + marketplace_dir, record_suffix = _materialize(entry_cfg, entry, plugins_root) + marketplace_name = _read_marketplace_name(marketplace_dir) + + # self-heal: drop any stale registration left by a crashed prior run (best-effort) + for plugin in entry_cfg["plugins"]: + _run_plugin_cmd(cli_cmd, _uninstall_args(agent_type, plugin, marketplace_name), check=False) + _run_plugin_cmd(cli_cmd, ["marketplace", "remove", marketplace_name], check=False) + + _run_plugin_cmd(cli_cmd, ["marketplace", "add", str(marketplace_dir)], check=True) + for plugin in entry_cfg["plugins"]: + _run_plugin_cmd(cli_cmd, ["install", f"{plugin}@{marketplace_name}"], check=True) + installed.append(InstalledPlugin(plugin=plugin, marketplace=marketplace_name, record=f"{plugin}@{record_suffix}")) + logger.info(f"Installed plugin {plugin}@{marketplace_name}") + except (subprocess.CalledProcessError, OSError) as e: + teardown_plugins(cli_cmd, agent_type, installed) + raise AgentError(f"Plugin setup failed: {e}") from e + + return installed + + +def teardown_plugins(cli_cmd: str, agent_type: AgentType, installed: list[InstalledPlugin]) -> None: + """Uninstall plugins and remove their marketplaces (best-effort; never raises).""" + if not installed: + return + for p in installed: + _run_plugin_cmd(cli_cmd, _uninstall_args(agent_type, p.plugin, p.marketplace), check=False) + for marketplace in dict.fromkeys(p.marketplace for p in installed): + _run_plugin_cmd(cli_cmd, ["marketplace", "remove", marketplace], check=False) + logger.info(f"Tore down plugins: {[p.record for p in installed]}") +``` + +In `src/bcbench/operations/__init__.py`, add the imports and `__all__` entries (keep alphabetized within the block): + +```python +from bcbench.operations.plugin_operations import setup_plugins_from_config, teardown_plugins +``` + +```python + "setup_hooks", + "setup_instructions_from_config", + "setup_plugins_from_config", + "setup_repo_prebuild", + "stage_and_get_diff", + "teardown_plugins", +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `uv run pytest tests/test_plugin_operations.py -v` +Expected: PASS (all tests). + +- [ ] **Step 5: Commit** + +```bash +git add src/bcbench/operations/plugin_operations.py src/bcbench/operations/__init__.py tests/test_plugin_operations.py +git commit -m "feat: install/teardown plugins via CLI commands (user scope, both agents)" +``` + +--- + +### Task 5: Wire plugin setup/teardown into the Copilot runner + +**Files:** +- Modify: `src/bcbench/agent/copilot/agent.py` +- Test: `tests/test_plugin_operations.py` + +**Interfaces:** +- Consumes: `setup_plugins_from_config`, `teardown_plugins` (Task 4); `ExperimentConfiguration.plugins` (Task 1). + +- [ ] **Step 1: Write the failing test** + +Append to `tests/test_plugin_operations.py`: + +```python +from unittest.mock import MagicMock, patch + +from bcbench.dataset import BaseDatasetEntry + + +@patch("bcbench.agent.copilot.agent.teardown_plugins") +@patch("bcbench.agent.copilot.agent.setup_plugins_from_config") +@patch("bcbench.agent.copilot.agent.parse_tool_usage_from_hooks", return_value=None) +@patch("bcbench.agent.copilot.agent.parse_metrics", return_value=None) +@patch("bcbench.agent.copilot.agent.setup_hooks") +@patch("bcbench.agent.copilot.agent.setup_custom_agent", return_value=None) +@patch("bcbench.agent.copilot.agent.setup_agent_skills", return_value=False) +@patch("bcbench.agent.copilot.agent.setup_instructions_from_config", return_value=False) +@patch("bcbench.agent.copilot.agent.build_al_lsp_plugin", return_value=None) +@patch("bcbench.agent.copilot.agent.build_mcp_config", return_value=(None, None)) +@patch("bcbench.agent.copilot.agent.build_prompt", return_value="do the task") +@patch("bcbench.agent.copilot.agent.shutil.which", return_value="copilot") +@patch("bcbench.agent.copilot.agent.subprocess.run") +def test_copilot_runner_records_plugins_and_tears_down( + mock_run, _which, _prompt, _mcp, _lsp, _instr, _skills, _agent, _hooks, _pm, _tu, mock_setup, mock_teardown, tmp_path +): + from bcbench.agent.copilot.agent import run_copilot_agent + from bcbench.types import EvaluationCategory + + mock_run.return_value = MagicMock(stderr=b"") + mock_setup.return_value = [po.InstalledPlugin("frontend-web-dev", "awesome-copilot", "frontend-web-dev@a1b2c3d4")] + entry = MagicMock(spec=BaseDatasetEntry) + entry.instance_id = "microsoftInternal__NAV-1" + + _metrics, config = run_copilot_agent(entry=entry, model="m", category=EvaluationCategory.BUG_FIX, repo_path=tmp_path, output_dir=tmp_path) + + assert config.plugins == ["frontend-web-dev@a1b2c3d4"] + mock_setup.assert_called_once() + mock_teardown.assert_called_once() +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run pytest tests/test_plugin_operations.py -k copilot_runner -v` +Expected: FAIL — `AttributeError`/`ImportError` for `setup_plugins_from_config` in `bcbench.agent.copilot.agent` (not imported/wired yet), or `config.plugins` is `None`. + +- [ ] **Step 3: Wire the runner** + +In `src/bcbench/agent/copilot/agent.py`, update the operations import line to include the new functions: + +```python +from bcbench.operations import setup_agent_skills, setup_custom_agent, setup_hooks, setup_instructions_from_config, setup_plugins_from_config, teardown_plugins +``` + +Resolve the CLI binary **before** the plugin setup and reuse it for the launch. Replace the existing block that computes `config`/finds `copilot_cmd` so it reads: + +```python + tool_log_path: Path = setup_hooks(repo_path, AgentType.COPILOT, output_dir) + + # Prefer copilot.exe over copilot.bat/copilot.cmd shims on Windows: the .bat shim invokes PowerShell, + # which re-parses arguments and corrupts prompts containing double quotes (e.g. JSON examples). + copilot_cmd = shutil.which("copilot.exe") or shutil.which("copilot.cmd") or shutil.which("copilot") + if not copilot_cmd: + raise AgentError("Copilot CLI not found in PATH. Please ensure it is installed and available.") + + installed_plugins = setup_plugins_from_config(copilot_config, entry, repo_path, AgentType.COPILOT, copilot_cmd) + + config = ExperimentConfiguration( + mcp_servers=mcp_server_names, + al_lsp_enabled=lsp_plugin_dir is not None, + custom_instructions=instructions_enabled, + skills_enabled=skills_enabled, + custom_agent=custom_agent, + plugins=[p.record for p in installed_plugins] or None, + ) + + logger.info(f"Executing Copilot CLI in directory: {repo_path}") + logger.debug(f"Using prompt:\n{prompt}") +``` + +Then delete the now-duplicate `copilot_cmd = shutil.which(...)` / `if not copilot_cmd: raise AgentError(...)` block that currently sits just before the `try:` (it has moved up). + +Wrap the agent subprocess in a `finally` that tears the plugins down. Change the `try:` that runs the agent so it ends with: + +```python + return metrics, config + except subprocess.TimeoutExpired: + logger.exception(f"Copilot CLI timed out after {_config.timeout.agent_execution} seconds") + metrics = AgentMetrics(execution_time=_config.timeout.agent_execution) + raise AgentTimeoutError("Copilot CLI timed out", metrics=metrics, config=config) from None + except subprocess.CalledProcessError as e: + logger.exception(f"Copilot CLI execution failed with error {e.stderr}") + raise AgentError(f"Copilot CLI execution failed: {e}") from None + except Exception: + logger.exception("Unexpected error running Copilot CLI") + raise + finally: + teardown_plugins(copilot_cmd, AgentType.COPILOT, installed_plugins) +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `uv run pytest tests/test_plugin_operations.py -k copilot_runner -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/bcbench/agent/copilot/agent.py tests/test_plugin_operations.py +git commit -m "feat: install plugins in the Copilot runner and tear down after the run" +``` + +--- + +### Task 6: Wire plugin setup/teardown into the Claude runner + +**Files:** +- Modify: `src/bcbench/agent/claude/agent.py` +- Test: `tests/test_plugin_operations.py` + +**Interfaces:** +- Consumes: `setup_plugins_from_config`, `teardown_plugins`; `ExperimentConfiguration.plugins`. + +- [ ] **Step 1: Write the failing test** + +Append to `tests/test_plugin_operations.py`: + +```python +@patch("bcbench.agent.claude.agent.teardown_plugins") +@patch("bcbench.agent.claude.agent.setup_plugins_from_config") +@patch("bcbench.agent.claude.agent.parse_tool_usage_from_hooks", return_value=None) +@patch("bcbench.agent.claude.agent.parse_metrics", return_value=None) +@patch("bcbench.agent.claude.agent.setup_hooks") +@patch("bcbench.agent.claude.agent.setup_custom_agent", return_value=None) +@patch("bcbench.agent.claude.agent.setup_agent_skills", return_value=False) +@patch("bcbench.agent.claude.agent.setup_instructions_from_config", return_value=False) +@patch("bcbench.agent.claude.agent.build_al_lsp_plugin", return_value=None) +@patch("bcbench.agent.claude.agent.build_mcp_config", return_value=(None, None)) +@patch("bcbench.agent.claude.agent.build_prompt", return_value="do the task") +@patch("bcbench.agent.claude.agent.shutil.which", return_value="claude") +@patch("bcbench.agent.claude.agent.subprocess.run") +def test_claude_runner_records_plugins_and_tears_down( + mock_run, _which, _prompt, _mcp, _lsp, _instr, _skills, _agent, _hooks, _pm, _tu, mock_setup, mock_teardown, tmp_path +): + from bcbench.agent.claude.agent import run_claude_code + from bcbench.types import EvaluationCategory + + mock_run.return_value = MagicMock(stdout=b'{"result": "ok"}') + mock_setup.return_value = [po.InstalledPlugin("frontend-web-dev", "awesome-copilot", "frontend-web-dev@a1b2c3d4")] + entry = MagicMock(spec=BaseDatasetEntry) + entry.instance_id = "microsoftInternal__NAV-1" + + _metrics, config = run_claude_code(entry=entry, model="m", category=EvaluationCategory.BUG_FIX, repo_path=tmp_path, output_dir=tmp_path) + + assert config.plugins == ["frontend-web-dev@a1b2c3d4"] + mock_setup.assert_called_once() + mock_teardown.assert_called_once() +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run pytest tests/test_plugin_operations.py -k claude_runner -v` +Expected: FAIL — `setup_plugins_from_config` not wired in `bcbench.agent.claude.agent`, or `config.plugins is None`. + +- [ ] **Step 3: Wire the runner** + +In `src/bcbench/agent/claude/agent.py`, update the operations import: + +```python +from bcbench.operations import setup_agent_skills, setup_custom_agent, setup_hooks, setup_instructions_from_config, setup_plugins_from_config, teardown_plugins +``` + +Resolve the CLI binary before plugin setup and reuse it. Replace the block that builds `config` and finds `claude_cmd` so it reads: + +```python + tool_log_path: Path = setup_hooks(repo_path, AgentType.CLAUDE, output_dir) + + claude_cmd = shutil.which("claude") + if not claude_cmd: + raise AgentError("Claude Code not found in PATH. Please ensure it is installed and available.") + + installed_plugins = setup_plugins_from_config(claude_config, entry, repo_path, AgentType.CLAUDE, claude_cmd) + + config = ExperimentConfiguration( + mcp_servers=mcp_server_names, + al_lsp_enabled=lsp_plugin_dir is not None, + custom_instructions=instructions_enabled, + skills_enabled=skills_enabled, + custom_agent=custom_agent, + plugins=[p.record for p in installed_plugins] or None, + ) + + logger.info(f"Executing Claude Code in directory: {repo_path}") + logger.debug(f"Using prompt:\n{prompt}") +``` + +Then delete the now-duplicate `claude_cmd = shutil.which("claude")` / `if not claude_cmd: raise AgentError(...)` block that currently sits just before the `try:`. + +Add a `finally` to the agent-run `try` block so it ends with: + +```python + return metrics, config + except subprocess.TimeoutExpired: + logger.exception(f"Claude Code timed out after {_config.timeout.agent_execution} seconds") + metrics = AgentMetrics(execution_time=_config.timeout.agent_execution) + raise AgentTimeoutError("Claude Code timed out", metrics=metrics, config=config) from None + except subprocess.CalledProcessError as e: + logger.exception(f"Claude Code execution failed with error {e.stderr}") + raise AgentError(f"Claude Code execution failed: {e.stderr}") from e + except Exception: + logger.exception("Unexpected error running Claude Code") + raise + finally: + teardown_plugins(claude_cmd, AgentType.CLAUDE, installed_plugins) +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `uv run pytest tests/test_plugin_operations.py -k claude_runner -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/bcbench/agent/claude/agent.py tests/test_plugin_operations.py +git commit -m "feat: install plugins in the Claude runner and tear down after the run" +``` + +--- + +### Task 7: Config schema, docs, mock scenario, and full verification + +**Files:** +- Modify: `src/bcbench/agent/shared/config.yaml` +- Modify: `EXPERIMENT.md` +- Modify: `src/bcbench/commands/evaluate.py` (`MockEvaluationPipeline` scenarios) +- Test: full suite + lint + +**Interfaces:** +- Consumes: everything above. No new public interface. + +- [ ] **Step 1: Add the `plugins` block to `config.yaml`** + +In `src/bcbench/agent/shared/config.yaml`, add this block after the `agents:` block (before `mcp:`): + +```yaml +# controls installing agent plugins (skills/agents/mcp/hooks bundles) into the CLI for the run. +# each enabled entry is installed via the CLI's `plugin marketplace add` + `plugin install` +# (user scope, both Copilot and Claude), then uninstalled after the run. Marketplace content is +# cloned at the pinned commit into /.bcbench/plugins/ (removed by the existing cleanup). +# NOTE: config injection (extraKnownMarketplaces/enabledPlugins) is trust-dialog-gated and +# ignored in headless mode, so we drive the real CLI commands instead. +# source: "marketplace" (repo + commit) or "local" (path under instructions/-/, +# which must be a marketplace root containing .claude-plugin/marketplace.json). +plugins: [] + # - source: marketplace + # enabled: true + # repo: "github/awesome-copilot" + # commit: "<40-char-commit-sha>" + # plugins: ["frontend-web-dev"] + # - source: local + # enabled: false + # path: "plugins/my-local-plugin" + # plugins: ["my-local-plugin"] +``` + +- [ ] **Step 2: Document the toggle in `EXPERIMENT.md`** + +In `EXPERIMENT.md`, add a row to the config table (after the `mcp.servers` row): + +```markdown +| `plugins` | _(empty)_ | List of agent plugins to install for the run (marketplace pinned to a commit, or local). Each enabled entry is installed via the CLI's `plugin marketplace add` + `plugin install` and removed afterward. | +``` + +And add this subsection after the "Custom instructions / skills / custom agents" subsection: + +```markdown +### Marketplace & local plugins + +`plugins` is a list; each entry is toggled by its own `enabled` (default `true`): + +- `source: marketplace` — `repo` (`owner/repo` or git URL) + `commit` (pinned for reproducibility) + `plugins` (names to install). +- `source: local` — `path` (relative to `instructions/-/`, pointing at a marketplace root with `.claude-plugin/marketplace.json`) + `plugins`. + +At runtime the marketplace is cloned at its commit into `/.bcbench/plugins/`, installed with the CLI's own commands (user scope), and uninstalled after the run. Installed plugins are recorded in the result's `ExperimentConfiguration.plugins` as `"@"` / `"@local"`. +``` + +- [ ] **Step 3: Add a plugins scenario to the mock pipeline** + +In `src/bcbench/commands/evaluate.py`, in `MockEvaluationPipeline.run_agent`, extend the `experiment_config_scenarios` list to include a plugins example (add one element): + +```python + experiment_config_scenarios: list[ExperimentConfiguration | None] = [ + ExperimentConfiguration(mcp_servers=["magic-mcp"], custom_instructions=True, custom_agent="custom-agent-v1"), + ExperimentConfiguration(mcp_servers=["magic-mcp"]), + ExperimentConfiguration(custom_instructions=True), + None, + ExperimentConfiguration(), + ExperimentConfiguration(custom_agent="custom-agent-v1"), + ExperimentConfiguration(plugins=["frontend-web-dev@a1b2c3d4"]), + ] +``` + +- [ ] **Step 4: Run the full test suite** + +Run: `uv run pytest -q` +Expected: PASS. If `tests/test_result_serialization.py` (or another snapshot test) fails because it compares a full `ExperimentConfiguration` dump and now sees the new `plugins` key, update the expected value in that test to include `"plugins": null` (or the recorded list) — do not remove the field. + +- [ ] **Step 5: Lint / format** + +Run: `uv run ruff check src/bcbench/operations/plugin_operations.py src/bcbench/operations/git_operations.py src/bcbench/agent/copilot/agent.py src/bcbench/agent/claude/agent.py src/bcbench/types.py tests/test_plugin_operations.py tests/test_git_operations.py` +Then: `uv run ruff format .` +Expected: no errors; formatter reports files unchanged or reformats cleanly. + +- [ ] **Step 6: Commit** + +```bash +git add src/bcbench/agent/shared/config.yaml EXPERIMENT.md src/bcbench/commands/evaluate.py tests/ +git commit -m "docs+config: document plugins toggle, add config block and mock scenario" +``` + +--- + +## Notes for the implementer + +- **Do not** attempt to load plugins via `extraKnownMarketplaces`/`enabledPlugins` injection — verified (and confirmed by gist `alexey-pelykh/566a4e5160b305db703d543312a1e686`) to be ignored in headless mode. +- The full lifecycle (`marketplace add` → `install` → `uninstall` → `marketplace remove`) was verified live on both CLIs against a local marketplace: non-interactive, offline, exit 0, baseline restored. Copilot uninstalls by ``; Claude by `@` — `_uninstall_args` encodes this. +- `git fetch --depth 1 origin ` works on GitHub (arbitrary SHA fetch is allowed). If a target host disallows it, the clone will fail loudly and surface as `AgentError` — acceptable (fails the entry like other setup failures). +- Setup ordering: `setup_plugins_from_config` runs after `setup_hooks` and other `setup_*` — it does not touch `.github`/`.claude`, so instruction/skill setup is unaffected. diff --git a/docs/superpowers/specs/2026-07-03-marketplace-and-local-plugins-experiment-design.md b/docs/superpowers/specs/2026-07-03-marketplace-and-local-plugins-experiment-design.md new file mode 100644 index 000000000..d95ffd6c4 --- /dev/null +++ b/docs/superpowers/specs/2026-07-03-marketplace-and-local-plugins-experiment-design.md @@ -0,0 +1,241 @@ +# Design: Marketplace & Local Plugins as an Experiment Toggle + +- **Date:** 2026-07-03 +- **Status:** Approved (design); pending implementation plan +- **Category impact:** None (agent-setup only; applies to all categories) + +## 1. Summary + +Add a config-driven experiment toggle that installs agent **plugins** — from a +marketplace pinned to a git commit, or from a local folder — into the running +CLI, so their skills are native during a run. This lets us measure whether a +plugin improves benchmark performance, alongside the existing +instruction/skill/agent/MCP toggles. + +Activation uses each CLI's **real plugin commands** (`plugin marketplace add` + +`plugin install`), run as ordinary subprocess calls in the setup phase **before** +launching the agent. This is deliberate: declarative `extraKnownMarketplaces` / +`enabledPlugins` config is processed only via the interactive trust dialog and is +**ignored in headless mode** — which is exactly how BC-Bench invokes both CLIs +(`copilot -p`, `claude --print`) — so it cannot be used (see §2.3). To pin a +commit we clone the marketplace ourselves into `/.bcbench/` and add it as a +**local** marketplace. Multiple plugins can be enabled at once. + +Both CLIs install at **user scope** (project scope is likewise ignored headless), +so both get the same **small, targeted** post-run teardown (`plugin uninstall` + +`marketplace remove`, verified reversible). Cloned content lives under `.bcbench` +(removed by the existing PR #651 cleanup). Installed plugins are recorded in +`ExperimentConfiguration`. + +## 2. Background + +### 2.1 Experiment model +Experiment levers live in `agent/shared/config.yaml` and are applied by +`operations/*` setup functions inside the agent runners +(`agent/copilot/agent.py`, `agent/claude/agent.py`) **before** the CLI subprocess +launches; each returns a value folded into `ExperimentConfiguration`, persisted +with the result. Between-run isolation comes from `clean_repo` (`git reset --hard` ++ `git clean -fd`) plus each setup function cleaning before it writes. + +### 2.2 Existing `.bcbench` plumbing to reuse (PR #651, commit `7eab2f9`, Haoran) +`agent/shared/plugin.py` provides `_PLUGIN_ROOT = Path(".bcbench")`, +`write_agent_plugin(...)` (creates dirs; `.bcbench` is created at run time, not +committed), and `remove_agent_plugin(repo_path, folder)` (clean-before). This +design reuses that runtime-created, repo-local root to hold cloned marketplace +content. + +### 2.3 Why the commands (config injection does not work headless) +Verified empirically **and** corroborated by an independent investigation +(gist `alexey-pelykh/566a4e5160b305db703d543312a1e686`): + +- Declarative `extraKnownMarketplaces` + `enabledPlugins` in project/repo settings + (`.claude/settings.json`, `.github/copilot/settings.json`) is processed **only + during the interactive trust-dialog handler**. In headless mode (`-p` / + `--print`) the trust dialog is skipped, so the config is **never processed**. + - My tests: repo-local `.github/copilot/settings.json` made a marketplace show + in `plugin marketplace list`, but a real `copilot -p` session did **not** + auto-install it (the plugin's skill never loaded; `installed-plugins/` + unchanged before/after). + - Plugin commands read from separate user-level storage + (`~/.claude/plugins/known_marketplaces.json` / `installed_plugins.json`; + `~/.copilot/config.json` `installedPlugins` + `settings.json`), which is where + `plugin install` writes. +- **Project scope is also ignored headless**, so `--scope project` does not help. + Both CLIs must install at **user scope**. +- Hand-writing those user-storage files is brittle (undocumented schema); the + documented, robust path is the CLI commands. **Decision: use the commands.** + +### 2.4 CLI facts (verified live on this machine) +Copilot CLI `1.0.69-0`, Claude Code `2.1.161`. + +- Full Copilot lifecycle against a **local** marketplace (a dir with + `.claude-plugin/marketplace.json` + a plugin with `.claude-plugin/plugin.json`), + non-interactive, offline, exit 0 each: + - `copilot plugin marketplace add ` → "Marketplace added successfully." + - `copilot plugin install @` → "Plugin installed successfully. Installed 1 skill." + - `copilot plugin uninstall ` / `plugin marketplace remove ` → success; baseline fully restored. +- Marketplace source accepts a **local path** (the marketplace ROOT — the dir + containing `marketplace.json`); the CLI records a local source as + `{"source":"directory","path":…}`. The plugin is selected `@` + where `` is the `name` inside `marketplace.json`. Copilot reads the + catalog from `.github/plugin/marketplace.json` or `.claude-plugin/marketplace.json`. +- Neither CLI has a `--ref`/`--commit` flag → we pin by cloning ourselves. +- Claude exposes the same command set (`plugin marketplace add|remove`, + `plugin install|uninstall`); use default **user** scope (see §2.3). +- Agent launch, model, MCP, permissions, auth are unaffected: the agent command + line is unchanged; the plugin is simply installed into the CLI beforehand. + +## 3. Goals / Non-goals + +### Guiding principle +Keep it simple and reuse existing architecture. Use the CLIs' documented commands +rather than reverse-engineering config or fighting the headless trust gate. Keep +cloned content repo-local under `.bcbench`. The teardown is minimal, symmetric +with setup, uses documented commands, and is scoped to exactly what we installed. + +### Goals +- Config-driven, per-entry enable/disable, multiple plugins at once. +- Marketplace plugins pinned to a git commit (reproducible); local plugins too. +- Plugins installed by name via the CLI's real commands, on Copilot and Claude. +- No config-home redirection; residue removed by a targeted post-run cleanup. +- Installed plugins recorded in `ExperimentConfiguration`. + +### Non-goals +- Changing/migrating the AL-LSP plugin (left as-is). +- Selecting which plugin agent *drives* the run (existing `--agent` covers it). +- Relying on `extraKnownMarketplaces`/`enabledPlugins` injection (ignored headless). +- Redirecting `COPILOT_HOME` / `CLAUDE_CONFIG_DIR`; hand-writing CLI storage files. +- Dataset, category, or scoring changes. + +## 4. Design + +### 4.1 Flow (mirrors the existing setup_* functions) +New `operations/plugin_operations.py`: + +```python +class InstalledPlugin(NamedTuple): + plugin: str # "frontend-web-dev" + marketplace: str # "awesome-copilot" + record: str # "frontend-web-dev@a1b2c3d4" (for ExperimentConfiguration) + +def setup_plugins_from_config( + agent_config: dict, entry: BaseDatasetEntry, repo_path: Path, + agent_type: AgentType, cli_cmd: str, +) -> list[InstalledPlugin]: ... + +def teardown_plugins(cli_cmd: str, installed: list[InstalledPlugin]) -> None: ... +``` + +`setup_plugins_from_config` clones/copies enabled plugins under `.bcbench`, runs +`marketplace add` + `install` (user scope), and returns what it installed (for the +result record and teardown). The agent launch command is unchanged. + +### 4.2 Materialize content into `.bcbench/` +- **Clean before / self-heal:** `remove_agent_plugin` for stale folders, and + defensively `uninstall` / `marketplace remove` our names (in case a prior run + crashed before teardown). +- **marketplace** (`repo` + `commit`): shallow-clone into + `/.bcbench/plugins/` and `git checkout ` (small + reused git helper). Read the marketplace `name` from its `marketplace.json`. +- **local** (`path`): copy into `/.bcbench/plugins/`. `path` resolves + under `src/bcbench/agent/shared/instructions//`. +- Failures raise `AgentError`. + +### 4.3 Install via the CLI commands (user scope, both CLIs) +For each enabled entry, using the resolved CLI binary: +- ` plugin marketplace add /.bcbench/plugins/` +- ` plugin install @` for each requested plugin +- Verify with ` plugin list`; failures raise `AgentError`. + +### 4.4 Teardown (symmetric, both CLIs) +In a `finally` around the agent run, for each installed plugin: +- ` plugin uninstall ` then ` plugin marketplace remove ` + (verified reversible; best-effort, logged, never masks the run outcome). +- `.bcbench` cloned content is removed by `remove_agent_plugin` / `git clean -fd`. +- The developer's real global config is returned to baseline. + +### 4.5 Config schema (`config.yaml`) +`plugins` is a flat list; each entry toggles independently. + +```yaml +plugins: + - source: marketplace + enabled: true # default true when omitted + repo: "github/awesome-copilot" # owner/repo or git URL + commit: "a1b2c3d4..." # pinned commit (required for marketplace) + plugins: ["frontend-web-dev"] # plugin name(s) to install + - source: local + enabled: false # kept in file, parked + path: "plugins/my-local-plugin" # under instructions// + plugins: ["my-local-plugin"] +``` + +Only `enabled: true` entries are installed. `enabled` defaults to `true`. + +### 4.6 Wiring into the runners +```python +installed = setup_plugins_from_config(config, entry, repo_path, agent_type, cli_cmd) +try: + result = subprocess.run(cmd_args, cwd=str(repo_path), ...) # unchanged agent launch + ... +finally: + teardown_plugins(cli_cmd, installed) + +experiment_config = ExperimentConfiguration( + ..., plugins=[p.record for p in installed] or None, +) +``` + +### 4.7 Recording (`types.py`) +Mirror the existing `mcp_servers: list[str]` — no new field type. + +```python +class ExperimentConfiguration(BaseModel): + ... + plugins: list[str] | None = None # e.g. ["frontend-web-dev@a1b2c3d4", "my-local-plugin@local"] + + def is_empty(self) -> bool: + return ( ... and self.plugins is None ) +``` +`"@"` (marketplace) / `"@local"` (local) keeps runs auditable. + +### 4.8 AL-LSP +Untouched (`build_al_lsp_plugin`, `--al-lsp`, its `--plugin-dir`). Only the new +plugins feature uses the commands; the `local` source could express AL-LSP later. + +## 5. Testing +Mirror `test_mcp_config.py` / `test_custom_instructions.py` (subprocess + git mocked): +- `tests/test_plugin_operations.py`: per-entry `enabled` (default true, disabled + skipped); marketplace clone/checkout; local copy into `.bcbench`; marketplace-name + read from `marketplace.json`; command construction for `marketplace add` / + `install` (user scope, both CLIs); `InstalledPlugin` records; `teardown_plugins` + builds correct `uninstall` / `marketplace remove`; clean-before self-heal; + `AgentError` on failure. +- Extend `test_experiment_configuration.py`: new `plugins` field + `is_empty`. +- Update mock `ExperimentConfiguration` scenarios + (`commands/evaluate.py` `MockEvaluationPipeline`, result-serialization tests). + +## 6. Documentation +- `EXPERIMENT.md`: add a `plugins` row + short schema/pinning note. +- `config.yaml`: document the `plugins` block in comments. + +## 7. Open questions / verification (small, non-blocking) +1. Confirm the Claude lifecycle (`marketplace add` + `install` + `uninstall` + + `marketplace remove`, user scope) end-to-end, mirroring the verified Copilot + lifecycle. High confidence; verify during implementation. +2. Ordering vs `setup_instructions_from_config`: run plugin setup after it. Low + risk (global config, not repo files), but keep the order explicit. +3. Migrate AL-LSP to a `local` plugin entry (deferred). +4. Optional agent-selection knob in a `plugins` entry (deferred; `--agent` covers it). + +## 8. References (verified) +- Independent investigation: gist `alexey-pelykh/566a4e5160b305db703d543312a1e686` + — `extraKnownMarketplaces`/`enabledPlugins` are trust-dialog-gated and ignored in + headless mode; recommends explicit `plugin marketplace add` / `install` commands. +- Live probes: full Copilot plugin lifecycle against a local marketplace + (non-interactive, offline, exit 0, "Installed 1 skill", baseline restored); a real + `copilot -p` session did **not** auto-install from repo-local settings. +- PR #651 / commit `7eab2f9` (Haoran) — `agent/shared/plugin.py`. +- `agent/copilot/agent.py`, `agent/claude/agent.py` — runner + `shutil.which` CLI + resolution + subprocess pattern to reuse. +- `types.py` — `ExperimentConfiguration`, `AgentType`. From 469ac69f06cf088c7bf669d95c7f2a36d4b64d12 Mon Sep 17 00:00:00 2001 From: bcbench Date: Mon, 6 Jul 2026 10:40:07 +0200 Subject: [PATCH 2/3] docs: per-entry isolated CLI home for concurrency safety (PR #733 review) Addresses review: execution-based categories run entries as a parallel matrix (max-parallel 64) on a shared self-hosted runner, and the CLI plugin store is user-scope/global, so concurrent installs/teardowns would race. Fix: install into a fresh per-entry config home (COPILOT_HOME / CLAUDE_CONFIG_DIR under .bcbench), applied to both the plugin commands and the agent launch. This removes teardown entirely (isolated home discarded with the checkout) and moots the teardown_plugins signature comment. Verified live on both CLIs (fresh home: install + skill load + headless session, real config untouched). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...026-07-06-marketplace-and-local-plugins.md | 343 ++++++++---------- ...ace-and-local-plugins-experiment-design.md | 174 +++++---- 2 files changed, 269 insertions(+), 248 deletions(-) diff --git a/docs/superpowers/plans/2026-07-06-marketplace-and-local-plugins.md b/docs/superpowers/plans/2026-07-06-marketplace-and-local-plugins.md index 42f67751a..4056dae0e 100644 --- a/docs/superpowers/plans/2026-07-06-marketplace-and-local-plugins.md +++ b/docs/superpowers/plans/2026-07-06-marketplace-and-local-plugins.md @@ -2,9 +2,9 @@ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. -**Goal:** Add a config-driven experiment toggle that installs agent plugins (marketplace-pinned-to-commit or local) into the running CLI via its own `plugin marketplace add` + `plugin install` commands, records them in results, and cleanly removes them afterward. +**Goal:** Add a config-driven experiment toggle that installs agent plugins (marketplace-pinned-to-commit or local) into the running CLI via its own `plugin marketplace add` + `plugin install` commands, records them in results, and keeps concurrent entries isolated. -**Architecture:** A new `operations/plugin_operations.py` reads a `plugins` list from `agent/shared/config.yaml`, clones each marketplace at its pinned commit into `/.bcbench/plugins/`, installs the requested plugins by name via the CLI commands (user scope, both Copilot and Claude), and returns records folded into `ExperimentConfiguration`. A symmetric teardown (`uninstall` + `marketplace remove`) runs in a `finally` in each agent runner. `.bcbench` content is removed by `git clean` and an inline clean-before. Config injection (`extraKnownMarketplaces`) is NOT used — it is trust-dialog-gated and ignored in headless mode. +**Architecture:** A new `operations/plugin_operations.py` reads a `plugins` list from `agent/shared/config.yaml`, clones each marketplace at its pinned commit into `/.bcbench/plugins/`, and installs the requested plugins by name via the CLI commands into a **fresh per-entry config home** (`COPILOT_HOME` / `CLAUDE_CONFIG_DIR` under `/.bcbench/`). It returns `(plugin_records, env_overrides)`: the records fold into `ExperimentConfiguration`, and `env_overrides` is merged into the agent-launch environment so the agent runs against the same isolated home. The per-entry home means concurrent matrix entries never share the user-scope plugin store — so there is **no teardown** (the home is discarded with the checkout; a clean-before removes any stale one). Config injection (`extraKnownMarketplaces`) is NOT used — it is trust-dialog-gated and ignored in headless mode. **Tech Stack:** Python 3.13, Pydantic v2, Typer, `subprocess` for `git` + the agent CLIs, pytest with `unittest.mock`, `uv` for running commands. @@ -16,6 +16,7 @@ - Line length 200; ruff rules per `pyproject.toml`. `subprocess.run` MUST pass an explicit `check=` (ruff `PLW1510`). - No new domain model types for recording — `ExperimentConfiguration.plugins` is `list[str] | None`, mirroring `mcp_servers`. - Marketplace/local source content must be a **marketplace root** (a directory containing `.claude-plugin/marketplace.json` or `.github/plugin/marketplace.json`). +- `operations/plugin_operations.py` must NOT import from `bcbench.agent.*` (would create a cycle: `bcbench.agent.__init__` imports the runners, which import `bcbench.operations`). - Do NOT modify the AL-LSP plugin mechanism (`build_al_lsp_plugin`, `--al-lsp`, `--plugin-dir`). - Commit after every task (frequent commits). @@ -114,7 +115,7 @@ Add to `tests/test_git_operations.py` (create the file if it does not exist, wit ```python from pathlib import Path -from unittest.mock import call, patch +from unittest.mock import patch from bcbench.operations.git_operations import clone_at_commit @@ -213,7 +214,6 @@ git commit -m "feat: add clone_at_commit git helper for pinned marketplace clone **Interfaces:** - Produces: - - `class InstalledPlugin(NamedTuple)` with `plugin: str`, `marketplace: str`, `record: str`. - `_read_marketplace_name(marketplace_dir: Path) -> str` — reads `name` from `.claude-plugin/marketplace.json` or `.github/plugin/marketplace.json`. - `_materialize(entry_cfg: dict, entry: BaseDatasetEntry, plugins_root: Path) -> tuple[Path, str]` — clones (marketplace) or copies (local) into `plugins_root`, returns `(marketplace_dir, record_suffix)` where `record_suffix` is the commit (marketplace) or the literal `"local"`. @@ -320,17 +320,22 @@ Create `src/bcbench/operations/plugin_operations.py`: Config injection (`extraKnownMarketplaces` / `enabledPlugins`) is trust-dialog-gated and ignored in headless mode, so we drive the CLI's real `plugin marketplace add` + `plugin install` commands. -Marketplace content is cloned at its pinned commit into `/.bcbench/plugins/` (repo-local, -cleaned by `git clean` and an inline clean-before). +Because the plugin store is user-scope/global and execution-based categories run entries as a +parallel matrix on a shared self-hosted runner, each entry installs into a fresh per-entry config +home (`COPILOT_HOME` / `CLAUDE_CONFIG_DIR`) under `/.bcbench/`, which the runner also applies +to the agent launch. Marketplace content is cloned at its pinned commit into `/.bcbench/plugins/`. + +NOTE: do NOT import from `bcbench.agent.*` here — `bcbench.agent.__init__` imports the runners, +which import `bcbench.operations`; importing agent from operations would create a cycle. """ from __future__ import annotations import json +import os import subprocess from pathlib import Path from shutil import copytree, rmtree -from typing import NamedTuple from bcbench.dataset import BaseDatasetEntry from bcbench.exceptions import AgentError @@ -341,20 +346,11 @@ from bcbench.types import AgentType logger = get_logger(__name__) -# NOTE: do NOT import from `bcbench.agent.*` here. `bcbench.agent.__init__` imports the runners, -# which import `bcbench.operations` — importing agent from operations would create a cycle. We -# therefore inline the `.bcbench/plugins` cleanup instead of reusing `remove_agent_plugin`. _BCBENCH_ROOT = ".bcbench" _PLUGINS_FOLDER = "plugins" # under /.bcbench/plugins/ _MARKETPLACE_MANIFESTS = (".claude-plugin/marketplace.json", ".github/plugin/marketplace.json") -class InstalledPlugin(NamedTuple): - plugin: str - marketplace: str - record: str # "@" (marketplace) or "@local" - - def _slug(text: str) -> str: return "".join(c if c.isalnum() else "-" for c in text).strip("-") @@ -397,6 +393,8 @@ def _materialize(entry_cfg: dict, entry: BaseDatasetEntry, plugins_root: Path) - raise AgentError(f"Unknown plugin source: {source!r}") ``` +(`os` and `subprocess` are used by the setup function added in Task 4.) + - [ ] **Step 4: Run tests to verify they pass** Run: `uv run pytest tests/test_plugin_operations.py -v` @@ -411,35 +409,32 @@ git commit -m "feat: add plugin materialize + marketplace-name helpers" --- -### Task 4: `plugin_operations.py` — setup + teardown commands +### Task 4: `plugin_operations.py` — install into a per-entry isolated home **Files:** - Modify: `src/bcbench/operations/plugin_operations.py` -- Modify: `src/bcbench/operations/__init__.py` (exports) +- Modify: `src/bcbench/operations/__init__.py` (export) - Test: `tests/test_plugin_operations.py` **Interfaces:** -- Consumes: `InstalledPlugin`, `_materialize`, `_read_marketplace_name` (Task 3); `AgentType`. +- Consumes: `_materialize`, `_read_marketplace_name` (Task 3); `AgentType`. - Produces: - - `setup_plugins_from_config(agent_config: dict, entry: BaseDatasetEntry, repo_path: Path, agent_type: AgentType, cli_cmd: str) -> list[InstalledPlugin]` - - `teardown_plugins(cli_cmd: str, agent_type: AgentType, installed: list[InstalledPlugin]) -> None` + - `setup_plugins_from_config(agent_config: dict, entry: BaseDatasetEntry, repo_path: Path, agent_type: AgentType, cli_cmd: str) -> tuple[list[str], dict[str, str]]` — returns `(plugin_records, env_overrides)`; `env_overrides` is `{"COPILOT_HOME": }` / `{"CLAUDE_CONFIG_DIR": }` (or `{}` when nothing is enabled) that the runner must merge into the agent-launch env. - [ ] **Step 1: Write the failing tests** Append to `tests/test_plugin_operations.py`: ```python -from unittest.mock import call - from bcbench.types import AgentType class _Recorder: def __init__(self): - self.calls: list[list[str]] = [] + self.calls: list[tuple[list[str], dict | None]] = [] def __call__(self, args, **kwargs): - self.calls.append(args) + self.calls.append((args, kwargs.get("env"))) class _R: returncode = 0 @@ -452,18 +447,20 @@ def _marketplace_entry_cfg(): def test_setup_no_plugins_returns_empty(tmp_path): - installed = po.setup_plugins_from_config({}, create_dataset_entry(), tmp_path, AgentType.COPILOT, "copilot") - assert installed == [] + records, env = po.setup_plugins_from_config({}, create_dataset_entry(), tmp_path, AgentType.COPILOT, "copilot") + assert records == [] + assert env == {} def test_setup_disabled_entry_skipped(tmp_path, monkeypatch): monkeypatch.setattr(po.subprocess, "run", _Recorder()) cfg = {"plugins": [{**_marketplace_entry_cfg(), "enabled": False}]} - installed = po.setup_plugins_from_config(cfg, create_dataset_entry(), tmp_path, AgentType.COPILOT, "copilot") - assert installed == [] + records, env = po.setup_plugins_from_config(cfg, create_dataset_entry(), tmp_path, AgentType.COPILOT, "copilot") + assert records == [] + assert env == {} -def test_setup_installs_and_records(tmp_path, monkeypatch): +def test_setup_installs_into_isolated_home_and_records(tmp_path, monkeypatch): def fake_clone(repo, commit, dest): _make_marketplace(dest, name="awesome-copilot") @@ -472,142 +469,120 @@ def test_setup_installs_and_records(tmp_path, monkeypatch): monkeypatch.setattr(po.subprocess, "run", rec) cfg = {"plugins": [_marketplace_entry_cfg()]} - installed = po.setup_plugins_from_config(cfg, create_dataset_entry(), tmp_path, AgentType.COPILOT, "copilot") + records, env = po.setup_plugins_from_config(cfg, create_dataset_entry(), tmp_path, AgentType.COPILOT, "copilot") - assert installed == [po.InstalledPlugin(plugin="probe-plugin", marketplace="awesome-copilot", record="probe-plugin@" + "a" * 40)] - # marketplace add + install were issued (self-heal remove/uninstall may precede them) - assert ["copilot", "plugin", "marketplace", "add", str(tmp_path / ".bcbench" / "plugins" / "github-awesome-copilot")] in rec.calls - assert ["copilot", "plugin", "install", "probe-plugin@awesome-copilot"] in rec.calls + home = str(tmp_path / ".bcbench" / "copilot-home") + assert records == ["probe-plugin@" + "a" * 40] + assert env == {"COPILOT_HOME": home} + add_call = next(c for c in rec.calls if c[0][:4] == ["copilot", "plugin", "marketplace", "add"]) + install_call = next(c for c in rec.calls if c[0][:3] == ["copilot", "plugin", "install"]) + assert add_call[0][4] == str(tmp_path / ".bcbench" / "plugins" / "github-awesome-copilot") + assert install_call[0] == ["copilot", "plugin", "install", "probe-plugin@awesome-copilot"] + # commands ran with the isolated home in their env + assert add_call[1]["COPILOT_HOME"] == home + assert install_call[1]["COPILOT_HOME"] == home -def test_setup_failure_tears_down_partial_and_raises(tmp_path, monkeypatch): + +def test_setup_claude_uses_claude_config_dir(tmp_path, monkeypatch): def fake_clone(repo, commit, dest): _make_marketplace(dest, name="awesome-copilot") monkeypatch.setattr(po, "clone_at_commit", fake_clone) - - def flaky_run(args, **kwargs): - class _R: - returncode = 0 - - if args[:4] == ["copilot", "plugin", "marketplace", "add"]: - raise po.subprocess.CalledProcessError(1, args) - return _R() - - monkeypatch.setattr(po.subprocess, "run", flaky_run) + monkeypatch.setattr(po.subprocess, "run", _Recorder()) cfg = {"plugins": [_marketplace_entry_cfg()]} - with pytest.raises(AgentError, match="Plugin setup failed"): - po.setup_plugins_from_config(cfg, create_dataset_entry(), tmp_path, AgentType.COPILOT, "copilot") + _records, env = po.setup_plugins_from_config(cfg, create_dataset_entry(), tmp_path, AgentType.CLAUDE, "claude") + assert env == {"CLAUDE_CONFIG_DIR": str(tmp_path / ".bcbench" / "claude-home")} -def test_teardown_copilot_uninstalls_by_name(monkeypatch): - rec = _Recorder() - monkeypatch.setattr(po.subprocess, "run", rec) - installed = [po.InstalledPlugin("probe-plugin", "awesome-copilot", "probe-plugin@abc")] - - po.teardown_plugins("copilot", AgentType.COPILOT, installed) - - assert ["copilot", "plugin", "uninstall", "probe-plugin"] in rec.calls - assert ["copilot", "plugin", "marketplace", "remove", "awesome-copilot"] in rec.calls +def test_setup_failure_removes_partial_home_and_raises(tmp_path, monkeypatch): + def fake_clone(repo, commit, dest): + _make_marketplace(dest, name="awesome-copilot") -def test_teardown_claude_uninstalls_by_qualified_name(monkeypatch): - rec = _Recorder() - monkeypatch.setattr(po.subprocess, "run", rec) - installed = [po.InstalledPlugin("probe-plugin", "awesome-copilot", "probe-plugin@abc")] + monkeypatch.setattr(po, "clone_at_commit", fake_clone) - po.teardown_plugins("claude", AgentType.CLAUDE, installed) + def failing_run(args, **kwargs): + raise po.subprocess.CalledProcessError(1, args) - assert ["claude", "plugin", "uninstall", "probe-plugin@awesome-copilot"] in rec.calls - assert ["claude", "plugin", "marketplace", "remove", "awesome-copilot"] in rec.calls + monkeypatch.setattr(po.subprocess, "run", failing_run) + cfg = {"plugins": [_marketplace_entry_cfg()]} + with pytest.raises(AgentError, match="Plugin setup failed"): + po.setup_plugins_from_config(cfg, create_dataset_entry(), tmp_path, AgentType.COPILOT, "copilot") -def test_teardown_empty_is_noop(monkeypatch): - rec = _Recorder() - monkeypatch.setattr(po.subprocess, "run", rec) - po.teardown_plugins("copilot", AgentType.COPILOT, []) - assert rec.calls == [] + assert not (tmp_path / ".bcbench" / "copilot-home").exists() ``` - [ ] **Step 2: Run tests to verify they fail** -Run: `uv run pytest tests/test_plugin_operations.py -k "setup or teardown" -v` +Run: `uv run pytest tests/test_plugin_operations.py -k "setup" -v` Expected: FAIL — `AttributeError: module 'bcbench.operations.plugin_operations' has no attribute 'setup_plugins_from_config'`. -- [ ] **Step 3: Implement setup + teardown** +- [ ] **Step 3: Implement `setup_plugins_from_config`** Append to `src/bcbench/operations/plugin_operations.py`: ```python -def _run_plugin_cmd(cli_cmd: str, args: list[str], *, check: bool) -> None: - subprocess.run([cli_cmd, "plugin", *args], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, check=check) - - -def _uninstall_args(agent_type: AgentType, plugin: str, marketplace: str) -> list[str]: - # Copilot uninstalls by plugin name; Claude by "@" (both verified live). +def _home_env_var(agent_type: AgentType) -> str: match agent_type: case AgentType.COPILOT: - return ["uninstall", plugin] + return "COPILOT_HOME" case AgentType.CLAUDE: - return ["uninstall", f"{plugin}@{marketplace}"] + return "CLAUDE_CONFIG_DIR" case _: - raise AgentError(f"Unsupported agent type for plugin teardown: {agent_type}") + raise AgentError(f"Unsupported agent type for plugins: {agent_type}") + +def _run_plugin_cmd(cli_cmd: str, args: list[str], env: dict[str, str]) -> None: + subprocess.run([cli_cmd, "plugin", *args], env=env, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, check=True) -def setup_plugins_from_config(agent_config: dict, entry: BaseDatasetEntry, repo_path: Path, agent_type: AgentType, cli_cmd: str) -> list[InstalledPlugin]: - """Install every enabled plugin entry into the CLI, before the agent launches. - Returns the installed plugins (for recording and teardown). Raises AgentError on failure - (after tearing down any partial installs). Skips silently when no entry is enabled. +def setup_plugins_from_config(agent_config: dict, entry: BaseDatasetEntry, repo_path: Path, agent_type: AgentType, cli_cmd: str) -> tuple[list[str], dict[str, str]]: + """Install every enabled plugin entry into a fresh per-entry CLI config home. + + Returns (plugin_records, env_overrides). env_overrides sets the isolated config home + (COPILOT_HOME / CLAUDE_CONFIG_DIR) that the runner must also apply to the agent launch, so + concurrent matrix entries never share the user-scope plugin store. Returns ([], {}) when no + entry is enabled. Raises AgentError on failure (after removing the partial home). """ entries = [e for e in (agent_config.get("plugins") or []) if e.get("enabled", True)] if not entries: - return [] - - plugins_root = repo_path / _BCBENCH_ROOT / _PLUGINS_FOLDER - if plugins_root.exists(): - rmtree(plugins_root) # clean-before: wipe stale clones - plugins_root.mkdir(parents=True, exist_ok=True) - - installed: list[InstalledPlugin] = [] + return [], {} + + bcbench_dir = repo_path / _BCBENCH_ROOT + plugins_root = bcbench_dir / _PLUGINS_FOLDER + home = bcbench_dir / f"{agent_type.value}-home" + for path in (plugins_root, home): + if path.exists(): + rmtree(path) # clean-before + path.mkdir(parents=True, exist_ok=True) + + home_var = _home_env_var(agent_type) + env = {**os.environ, home_var: str(home)} + records: list[str] = [] try: for entry_cfg in entries: marketplace_dir, record_suffix = _materialize(entry_cfg, entry, plugins_root) marketplace_name = _read_marketplace_name(marketplace_dir) - - # self-heal: drop any stale registration left by a crashed prior run (best-effort) - for plugin in entry_cfg["plugins"]: - _run_plugin_cmd(cli_cmd, _uninstall_args(agent_type, plugin, marketplace_name), check=False) - _run_plugin_cmd(cli_cmd, ["marketplace", "remove", marketplace_name], check=False) - - _run_plugin_cmd(cli_cmd, ["marketplace", "add", str(marketplace_dir)], check=True) + _run_plugin_cmd(cli_cmd, ["marketplace", "add", str(marketplace_dir)], env) for plugin in entry_cfg["plugins"]: - _run_plugin_cmd(cli_cmd, ["install", f"{plugin}@{marketplace_name}"], check=True) - installed.append(InstalledPlugin(plugin=plugin, marketplace=marketplace_name, record=f"{plugin}@{record_suffix}")) - logger.info(f"Installed plugin {plugin}@{marketplace_name}") + _run_plugin_cmd(cli_cmd, ["install", f"{plugin}@{marketplace_name}"], env) + records.append(f"{plugin}@{record_suffix}") + logger.info(f"Installed plugin {plugin}@{marketplace_name} into {home}") except (subprocess.CalledProcessError, OSError) as e: - teardown_plugins(cli_cmd, agent_type, installed) + if home.exists(): + rmtree(home) raise AgentError(f"Plugin setup failed: {e}") from e - return installed - - -def teardown_plugins(cli_cmd: str, agent_type: AgentType, installed: list[InstalledPlugin]) -> None: - """Uninstall plugins and remove their marketplaces (best-effort; never raises).""" - if not installed: - return - for p in installed: - _run_plugin_cmd(cli_cmd, _uninstall_args(agent_type, p.plugin, p.marketplace), check=False) - for marketplace in dict.fromkeys(p.marketplace for p in installed): - _run_plugin_cmd(cli_cmd, ["marketplace", "remove", marketplace], check=False) - logger.info(f"Tore down plugins: {[p.record for p in installed]}") + return records, {home_var: str(home)} ``` -In `src/bcbench/operations/__init__.py`, add the imports and `__all__` entries (keep alphabetized within the block): +In `src/bcbench/operations/__init__.py`, add the import and `__all__` entry (keep alphabetized within the block): ```python -from bcbench.operations.plugin_operations import setup_plugins_from_config, teardown_plugins +from bcbench.operations.plugin_operations import setup_plugins_from_config ``` ```python @@ -615,8 +590,6 @@ from bcbench.operations.plugin_operations import setup_plugins_from_config, tear "setup_instructions_from_config", "setup_plugins_from_config", "setup_repo_prebuild", - "stage_and_get_diff", - "teardown_plugins", ``` - [ ] **Step 4: Run tests to verify they pass** @@ -628,19 +601,19 @@ Expected: PASS (all tests). ```bash git add src/bcbench/operations/plugin_operations.py src/bcbench/operations/__init__.py tests/test_plugin_operations.py -git commit -m "feat: install/teardown plugins via CLI commands (user scope, both agents)" +git commit -m "feat: install plugins into a per-entry isolated CLI config home" ``` --- -### Task 5: Wire plugin setup/teardown into the Copilot runner +### Task 5: Wire plugin setup into the Copilot runner **Files:** - Modify: `src/bcbench/agent/copilot/agent.py` - Test: `tests/test_plugin_operations.py` **Interfaces:** -- Consumes: `setup_plugins_from_config`, `teardown_plugins` (Task 4); `ExperimentConfiguration.plugins` (Task 1). +- Consumes: `setup_plugins_from_config` (Task 4); `ExperimentConfiguration.plugins` (Task 1). - [ ] **Step 1: Write the failing test** @@ -652,7 +625,6 @@ from unittest.mock import MagicMock, patch from bcbench.dataset import BaseDatasetEntry -@patch("bcbench.agent.copilot.agent.teardown_plugins") @patch("bcbench.agent.copilot.agent.setup_plugins_from_config") @patch("bcbench.agent.copilot.agent.parse_tool_usage_from_hooks", return_value=None) @patch("bcbench.agent.copilot.agent.parse_metrics", return_value=None) @@ -665,14 +637,15 @@ from bcbench.dataset import BaseDatasetEntry @patch("bcbench.agent.copilot.agent.build_prompt", return_value="do the task") @patch("bcbench.agent.copilot.agent.shutil.which", return_value="copilot") @patch("bcbench.agent.copilot.agent.subprocess.run") -def test_copilot_runner_records_plugins_and_tears_down( - mock_run, _which, _prompt, _mcp, _lsp, _instr, _skills, _agent, _hooks, _pm, _tu, mock_setup, mock_teardown, tmp_path +def test_copilot_runner_records_plugins_and_sets_home( + mock_run, _which, _prompt, _mcp, _lsp, _instr, _skills, _agent, _hooks, _pm, _tu, mock_setup, tmp_path ): from bcbench.agent.copilot.agent import run_copilot_agent from bcbench.types import EvaluationCategory mock_run.return_value = MagicMock(stderr=b"") - mock_setup.return_value = [po.InstalledPlugin("frontend-web-dev", "awesome-copilot", "frontend-web-dev@a1b2c3d4")] + home = str(tmp_path / ".bcbench" / "copilot-home") + mock_setup.return_value = (["frontend-web-dev@a1b2c3d4"], {"COPILOT_HOME": home}) entry = MagicMock(spec=BaseDatasetEntry) entry.instance_id = "microsoftInternal__NAV-1" @@ -680,23 +653,24 @@ def test_copilot_runner_records_plugins_and_tears_down( assert config.plugins == ["frontend-web-dev@a1b2c3d4"] mock_setup.assert_called_once() - mock_teardown.assert_called_once() + _args, kwargs = mock_run.call_args + assert kwargs["env"]["COPILOT_HOME"] == home ``` - [ ] **Step 2: Run test to verify it fails** Run: `uv run pytest tests/test_plugin_operations.py -k copilot_runner -v` -Expected: FAIL — `AttributeError`/`ImportError` for `setup_plugins_from_config` in `bcbench.agent.copilot.agent` (not imported/wired yet), or `config.plugins` is `None`. +Expected: FAIL — `setup_plugins_from_config` not imported/wired in `bcbench.agent.copilot.agent`, or `config.plugins is None`. - [ ] **Step 3: Wire the runner** -In `src/bcbench/agent/copilot/agent.py`, update the operations import line to include the new functions: +In `src/bcbench/agent/copilot/agent.py`, update the operations import line to include the new function: ```python -from bcbench.operations import setup_agent_skills, setup_custom_agent, setup_hooks, setup_instructions_from_config, setup_plugins_from_config, teardown_plugins +from bcbench.operations import setup_agent_skills, setup_custom_agent, setup_hooks, setup_instructions_from_config, setup_plugins_from_config ``` -Resolve the CLI binary **before** the plugin setup and reuse it for the launch. Replace the existing block that computes `config`/finds `copilot_cmd` so it reads: +Resolve the CLI binary **before** the plugin setup and reuse it. Replace the existing block that computes `config` / finds `copilot_cmd` so it reads: ```python tool_log_path: Path = setup_hooks(repo_path, AgentType.COPILOT, output_dir) @@ -707,7 +681,7 @@ Resolve the CLI binary **before** the plugin setup and reuse it for the launch. if not copilot_cmd: raise AgentError("Copilot CLI not found in PATH. Please ensure it is installed and available.") - installed_plugins = setup_plugins_from_config(copilot_config, entry, repo_path, AgentType.COPILOT, copilot_cmd) + plugin_records, plugin_env = setup_plugins_from_config(copilot_config, entry, repo_path, AgentType.COPILOT, copilot_cmd) config = ExperimentConfiguration( mcp_servers=mcp_server_names, @@ -715,7 +689,7 @@ Resolve the CLI binary **before** the plugin setup and reuse it for the launch. custom_instructions=instructions_enabled, skills_enabled=skills_enabled, custom_agent=custom_agent, - plugins=[p.record for p in installed_plugins] or None, + plugins=plugin_records or None, ) logger.info(f"Executing Copilot CLI in directory: {repo_path}") @@ -724,24 +698,19 @@ Resolve the CLI binary **before** the plugin setup and reuse it for the launch. Then delete the now-duplicate `copilot_cmd = shutil.which(...)` / `if not copilot_cmd: raise AgentError(...)` block that currently sits just before the `try:` (it has moved up). -Wrap the agent subprocess in a `finally` that tears the plugins down. Change the `try:` that runs the agent so it ends with: +Merge `plugin_env` into the agent-launch environment. In the `subprocess.run(...)` call, change the `env=` dict to include `**plugin_env`: ```python - return metrics, config - except subprocess.TimeoutExpired: - logger.exception(f"Copilot CLI timed out after {_config.timeout.agent_execution} seconds") - metrics = AgentMetrics(execution_time=_config.timeout.agent_execution) - raise AgentTimeoutError("Copilot CLI timed out", metrics=metrics, config=config) from None - except subprocess.CalledProcessError as e: - logger.exception(f"Copilot CLI execution failed with error {e.stderr}") - raise AgentError(f"Copilot CLI execution failed: {e}") from None - except Exception: - logger.exception("Unexpected error running Copilot CLI") - raise - finally: - teardown_plugins(copilot_cmd, AgentType.COPILOT, installed_plugins) + env={ + **os.environ, + "GITHUB_COPILOT_PROMPT_MODE_REPO_HOOKS": "true", + "GITHUB_COPILOT_PROMPT_MODE_WORKSPACE_MCP": "true", + **plugin_env, + }, ``` +(No `finally`/teardown — the per-entry home is discarded with the checkout; `setup_plugins_from_config` clean-befores any stale home.) + - [ ] **Step 4: Run test to verify it passes** Run: `uv run pytest tests/test_plugin_operations.py -k copilot_runner -v` @@ -751,26 +720,25 @@ Expected: PASS. ```bash git add src/bcbench/agent/copilot/agent.py tests/test_plugin_operations.py -git commit -m "feat: install plugins in the Copilot runner and tear down after the run" +git commit -m "feat: install plugins into a per-entry home in the Copilot runner" ``` --- -### Task 6: Wire plugin setup/teardown into the Claude runner +### Task 6: Wire plugin setup into the Claude runner **Files:** - Modify: `src/bcbench/agent/claude/agent.py` - Test: `tests/test_plugin_operations.py` **Interfaces:** -- Consumes: `setup_plugins_from_config`, `teardown_plugins`; `ExperimentConfiguration.plugins`. +- Consumes: `setup_plugins_from_config`; `ExperimentConfiguration.plugins`. - [ ] **Step 1: Write the failing test** Append to `tests/test_plugin_operations.py`: ```python -@patch("bcbench.agent.claude.agent.teardown_plugins") @patch("bcbench.agent.claude.agent.setup_plugins_from_config") @patch("bcbench.agent.claude.agent.parse_tool_usage_from_hooks", return_value=None) @patch("bcbench.agent.claude.agent.parse_metrics", return_value=None) @@ -783,14 +751,15 @@ Append to `tests/test_plugin_operations.py`: @patch("bcbench.agent.claude.agent.build_prompt", return_value="do the task") @patch("bcbench.agent.claude.agent.shutil.which", return_value="claude") @patch("bcbench.agent.claude.agent.subprocess.run") -def test_claude_runner_records_plugins_and_tears_down( - mock_run, _which, _prompt, _mcp, _lsp, _instr, _skills, _agent, _hooks, _pm, _tu, mock_setup, mock_teardown, tmp_path +def test_claude_runner_records_plugins_and_sets_config_dir( + mock_run, _which, _prompt, _mcp, _lsp, _instr, _skills, _agent, _hooks, _pm, _tu, mock_setup, tmp_path ): from bcbench.agent.claude.agent import run_claude_code from bcbench.types import EvaluationCategory mock_run.return_value = MagicMock(stdout=b'{"result": "ok"}') - mock_setup.return_value = [po.InstalledPlugin("frontend-web-dev", "awesome-copilot", "frontend-web-dev@a1b2c3d4")] + cfg_dir = str(tmp_path / ".bcbench" / "claude-home") + mock_setup.return_value = (["frontend-web-dev@a1b2c3d4"], {"CLAUDE_CONFIG_DIR": cfg_dir}) entry = MagicMock(spec=BaseDatasetEntry) entry.instance_id = "microsoftInternal__NAV-1" @@ -798,20 +767,21 @@ def test_claude_runner_records_plugins_and_tears_down( assert config.plugins == ["frontend-web-dev@a1b2c3d4"] mock_setup.assert_called_once() - mock_teardown.assert_called_once() + _args, kwargs = mock_run.call_args + assert kwargs["env"]["CLAUDE_CONFIG_DIR"] == cfg_dir ``` - [ ] **Step 2: Run test to verify it fails** Run: `uv run pytest tests/test_plugin_operations.py -k claude_runner -v` -Expected: FAIL — `setup_plugins_from_config` not wired in `bcbench.agent.claude.agent`, or `config.plugins is None`. +Expected: FAIL — `setup_plugins_from_config` not wired in `bcbench.agent.claude.agent`, or the `subprocess.run` call has no `env` kwarg (`KeyError: 'env'`). - [ ] **Step 3: Wire the runner** In `src/bcbench/agent/claude/agent.py`, update the operations import: ```python -from bcbench.operations import setup_agent_skills, setup_custom_agent, setup_hooks, setup_instructions_from_config, setup_plugins_from_config, teardown_plugins +from bcbench.operations import setup_agent_skills, setup_custom_agent, setup_hooks, setup_instructions_from_config, setup_plugins_from_config ``` Resolve the CLI binary before plugin setup and reuse it. Replace the block that builds `config` and finds `claude_cmd` so it reads: @@ -823,7 +793,7 @@ Resolve the CLI binary before plugin setup and reuse it. Replace the block that if not claude_cmd: raise AgentError("Claude Code not found in PATH. Please ensure it is installed and available.") - installed_plugins = setup_plugins_from_config(claude_config, entry, repo_path, AgentType.CLAUDE, claude_cmd) + plugin_records, plugin_env = setup_plugins_from_config(claude_config, entry, repo_path, AgentType.CLAUDE, claude_cmd) config = ExperimentConfiguration( mcp_servers=mcp_server_names, @@ -831,7 +801,7 @@ Resolve the CLI binary before plugin setup and reuse it. Replace the block that custom_instructions=instructions_enabled, skills_enabled=skills_enabled, custom_agent=custom_agent, - plugins=[p.record for p in installed_plugins] or None, + plugins=plugin_records or None, ) logger.info(f"Executing Claude Code in directory: {repo_path}") @@ -840,24 +810,23 @@ Resolve the CLI binary before plugin setup and reuse it. Replace the block that Then delete the now-duplicate `claude_cmd = shutil.which("claude")` / `if not claude_cmd: raise AgentError(...)` block that currently sits just before the `try:`. -Add a `finally` to the agent-run `try` block so it ends with: +The Claude `subprocess.run(...)` call currently passes no `env`. Add one that carries the isolated home: ```python - return metrics, config - except subprocess.TimeoutExpired: - logger.exception(f"Claude Code timed out after {_config.timeout.agent_execution} seconds") - metrics = AgentMetrics(execution_time=_config.timeout.agent_execution) - raise AgentTimeoutError("Claude Code timed out", metrics=metrics, config=config) from None - except subprocess.CalledProcessError as e: - logger.exception(f"Claude Code execution failed with error {e.stderr}") - raise AgentError(f"Claude Code execution failed: {e.stderr}") from e - except Exception: - logger.exception("Unexpected error running Claude Code") - raise - finally: - teardown_plugins(claude_cmd, AgentType.CLAUDE, installed_plugins) + result = subprocess.run( + cmd_args, + cwd=str(repo_path), + env={**os.environ, **plugin_env}, + timeout=_config.timeout.agent_execution, + check=True, + capture_output=True, + ) ``` +Ensure `import os` is present at the top of `src/bcbench/agent/claude/agent.py` (add it if missing). + +(No `finally`/teardown.) + - [ ] **Step 4: Run test to verify it passes** Run: `uv run pytest tests/test_plugin_operations.py -k claude_runner -v` @@ -867,7 +836,7 @@ Expected: PASS. ```bash git add src/bcbench/agent/claude/agent.py tests/test_plugin_operations.py -git commit -m "feat: install plugins in the Claude runner and tear down after the run" +git commit -m "feat: install plugins into a per-entry home in the Claude runner" ``` --- @@ -889,11 +858,13 @@ In `src/bcbench/agent/shared/config.yaml`, add this block after the `agents:` bl ```yaml # controls installing agent plugins (skills/agents/mcp/hooks bundles) into the CLI for the run. -# each enabled entry is installed via the CLI's `plugin marketplace add` + `plugin install` -# (user scope, both Copilot and Claude), then uninstalled after the run. Marketplace content is -# cloned at the pinned commit into /.bcbench/plugins/ (removed by the existing cleanup). -# NOTE: config injection (extraKnownMarketplaces/enabledPlugins) is trust-dialog-gated and -# ignored in headless mode, so we drive the real CLI commands instead. +# each enabled entry is installed via the CLI's `plugin marketplace add` + `plugin install` into a +# fresh per-entry config home (COPILOT_HOME / CLAUDE_CONFIG_DIR under /.bcbench/), so parallel +# matrix entries never share the user-scope plugin store and no teardown is needed. +# NOTE: config injection (extraKnownMarketplaces/enabledPlugins) is trust-dialog-gated and ignored +# in headless mode, so we drive the real CLI commands instead. +# NOTE: a fresh home authenticates via the env token the workflow sets (COPILOT_GITHUB_TOKEN; +# ANTHROPIC_API_KEY) — local plugin runs must have that token in the environment. # source: "marketplace" (repo + commit) or "local" (path under instructions/-/, # which must be a marketplace root containing .claude-plugin/marketplace.json). plugins: [] @@ -913,7 +884,7 @@ plugins: [] In `EXPERIMENT.md`, add a row to the config table (after the `mcp.servers` row): ```markdown -| `plugins` | _(empty)_ | List of agent plugins to install for the run (marketplace pinned to a commit, or local). Each enabled entry is installed via the CLI's `plugin marketplace add` + `plugin install` and removed afterward. | +| `plugins` | _(empty)_ | List of agent plugins to install for the run (marketplace pinned to a commit, or local). Each enabled entry is installed via the CLI's `plugin marketplace add` + `plugin install` into a per-entry isolated config home. | ``` And add this subsection after the "Custom instructions / skills / custom agents" subsection: @@ -926,7 +897,7 @@ And add this subsection after the "Custom instructions / skills / custom agents" - `source: marketplace` — `repo` (`owner/repo` or git URL) + `commit` (pinned for reproducibility) + `plugins` (names to install). - `source: local` — `path` (relative to `instructions/-/`, pointing at a marketplace root with `.claude-plugin/marketplace.json`) + `plugins`. -At runtime the marketplace is cloned at its commit into `/.bcbench/plugins/`, installed with the CLI's own commands (user scope), and uninstalled after the run. Installed plugins are recorded in the result's `ExperimentConfiguration.plugins` as `"@"` / `"@local"`. +At runtime the marketplace is cloned at its commit into `/.bcbench/plugins/` and installed with the CLI's own commands into a fresh per-entry config home (`COPILOT_HOME` / `CLAUDE_CONFIG_DIR` under `.bcbench/`), which keeps parallel matrix entries isolated. A fresh home authenticates via the env token the workflow already sets (`COPILOT_GITHUB_TOKEN`; `ANTHROPIC_API_KEY`) — local plugin runs must have that token set. Installed plugins are recorded in the result's `ExperimentConfiguration.plugins` as `"@"` / `"@local"`. ``` - [ ] **Step 3: Add a plugins scenario to the mock pipeline** @@ -967,7 +938,9 @@ git commit -m "docs+config: document plugins toggle, add config block and mock s ## Notes for the implementer -- **Do not** attempt to load plugins via `extraKnownMarketplaces`/`enabledPlugins` injection — verified (and confirmed by gist `alexey-pelykh/566a4e5160b305db703d543312a1e686`) to be ignored in headless mode. -- The full lifecycle (`marketplace add` → `install` → `uninstall` → `marketplace remove`) was verified live on both CLIs against a local marketplace: non-interactive, offline, exit 0, baseline restored. Copilot uninstalls by ``; Claude by `@` — `_uninstall_args` encodes this. -- `git fetch --depth 1 origin ` works on GitHub (arbitrary SHA fetch is allowed). If a target host disallows it, the clone will fail loudly and surface as `AgentError` — acceptable (fails the entry like other setup failures). -- Setup ordering: `setup_plugins_from_config` runs after `setup_hooks` and other `setup_*` — it does not touch `.github`/`.claude`, so instruction/skill setup is unaffected. +- **Do not** load plugins via `extraKnownMarketplaces`/`enabledPlugins` injection — verified (and confirmed by gist `alexey-pelykh/566a4e5160b305db703d543312a1e686`) to be ignored in headless mode. +- **Concurrency is why the per-entry home exists.** `copilot-evaluation.yml` runs entries with `max-parallel: 64` on the shared self-hosted `GitHub-BCBench` runner; the CLI plugin store is user-scope/global, so without a per-entry home two entries would corrupt each other's installs. The isolated home (`COPILOT_HOME` / `CLAUDE_CONFIG_DIR` under `.bcbench/`) removes the race and any teardown. +- The full plugin lifecycle and per-entry-home isolation were verified live: fresh `COPILOT_HOME` accepts `marketplace add`/`install`, `copilot skill list` shows the installed plugin's skill, and a real `copilot -p` session runs (exit 0, auth via `COPILOT_GITHUB_TOKEN`, no onboarding block), leaving the real `~/.copilot` untouched; fresh `CLAUDE_CONFIG_DIR` isolates install the same way. +- `git fetch --depth 1 origin ` works on GitHub (arbitrary SHA fetch allowed). A host that disallows it fails loudly as `AgentError` (fails the entry, like other setup failures). +- Setup ordering: `setup_plugins_from_config` runs after `setup_hooks` and the other `setup_*` — it does not touch `.github`/`.claude`. +- Copilot passes `--log-dir=` explicitly, so `process-*.log` (used for turn-count metrics) should still land in `output_dir` even with `COPILOT_HOME` redirected; confirm during Task 5. diff --git a/docs/superpowers/specs/2026-07-03-marketplace-and-local-plugins-experiment-design.md b/docs/superpowers/specs/2026-07-03-marketplace-and-local-plugins-experiment-design.md index d95ffd6c4..d20c524cc 100644 --- a/docs/superpowers/specs/2026-07-03-marketplace-and-local-plugins-experiment-design.md +++ b/docs/superpowers/specs/2026-07-03-marketplace-and-local-plugins-experiment-design.md @@ -21,11 +21,16 @@ launching the agent. This is deliberate: declarative `extraKnownMarketplaces` / commit we clone the marketplace ourselves into `/.bcbench/` and add it as a **local** marketplace. Multiple plugins can be enabled at once. -Both CLIs install at **user scope** (project scope is likewise ignored headless), -so both get the same **small, targeted** post-run teardown (`plugin uninstall` + -`marketplace remove`, verified reversible). Cloned content lives under `.bcbench` -(removed by the existing PR #651 cleanup). Installed plugins are recorded in -`ExperimentConfiguration`. +Because installs go to the CLI's **user-scope** store (project scope is ignored +headless) and execution-based categories run entries as a **parallel matrix on a +shared self-hosted runner** (see §2.5), each entry redirects the CLI's config home +to a **per-entry isolated directory** (`COPILOT_HOME` / `CLAUDE_CONFIG_DIR` under +`/.bcbench/`) for both the install commands and the agent launch. This makes +concurrent entries independent (no shared global state, no cross-entry race) and +**removes the need for teardown** — the isolated home is discarded with the +checkout. Auth in the fresh home comes from the env token the workflow already +sets (`COPILOT_GITHUB_TOKEN`; `ANTHROPIC_API_KEY`). Cloned content lives under +`.bcbench`; installed plugins are recorded in `ExperimentConfiguration`. ## 2. Background @@ -82,29 +87,54 @@ Copilot CLI `1.0.69-0`, Claude Code `2.1.161`. - Neither CLI has a `--ref`/`--commit` flag → we pin by cloning ourselves. - Claude exposes the same command set (`plugin marketplace add|remove`, `plugin install|uninstall`); use default **user** scope (see §2.3). -- Agent launch, model, MCP, permissions, auth are unaffected: the agent command - line is unchanged; the plugin is simply installed into the CLI beforehand. +- Agent launch, model, MCP, and permissions are still passed as explicit flags; + the agent command line is unchanged apart from the config-home env var (§2.5). +- **Per-entry isolated home works headless (verified):** with a fresh + `COPILOT_HOME`, `plugin install` succeeds, `copilot skill list` shows the + plugin's skill (content loads), and a real `copilot -p` session runs (exit 0, + auth via `COPILOT_GITHUB_TOKEN`) with no onboarding block; the real `~/.copilot` + is untouched. Fresh `CLAUDE_CONFIG_DIR` isolates install the same way. + +### 2.5 Concurrency: why a per-entry isolated config home +Execution-based categories (`bug-fix`, `test-generation`) run entries as a +**parallel matrix** (`max-parallel: 64` in `copilot-evaluation.yml`) on the +self-hosted `GitHub-BCBench` runner, where legs can share a machine. The CLI's +plugin store is **user scope / global** (`~/.copilot`, `~/.claude`) — the first +experiment lever to touch shared state (instructions/skills are repo-local, MCP is +a per-invocation flag, AL-LSP is repo-local `.bcbench`). Concurrent entries would +therefore race: two `plugin install`s writing the same global config, or one +entry's cleanup removing a marketplace/plugin an in-flight entry still uses. + +Fix: each entry redirects the CLI config home to a **unique per-entry directory** +(`COPILOT_HOME` / `CLAUDE_CONFIG_DIR` = `/.bcbench/-home`) for both +the plugin commands and the agent launch. Installs are then fully isolated per +entry, eliminating the race and **removing any need for teardown** — the home is +ephemeral (fresh checkout per matrix leg; cleaned before each run locally). Fresh +GitHub-hosted VMs (`code-review` on ubuntu-latest, `nl2al` on windows-latest) are +already isolated, but the per-entry home is applied uniformly. ## 3. Goals / Non-goals ### Guiding principle Keep it simple and reuse existing architecture. Use the CLIs' documented commands rather than reverse-engineering config or fighting the headless trust gate. Keep -cloned content repo-local under `.bcbench`. The teardown is minimal, symmetric -with setup, uses documented commands, and is scoped to exactly what we installed. +everything repo-local under `.bcbench` (cloned content **and** the per-entry +config home), so concurrent entries never share state and there is nothing global +to tear down. ### Goals - Config-driven, per-entry enable/disable, multiple plugins at once. - Marketplace plugins pinned to a git commit (reproducible); local plugins too. - Plugins installed by name via the CLI's real commands, on Copilot and Claude. -- No config-home redirection; residue removed by a targeted post-run cleanup. +- Concurrency-safe: per-entry isolated config home; no shared global state. +- No teardown and no residue in the developer's real config. - Installed plugins recorded in `ExperimentConfiguration`. ### Non-goals - Changing/migrating the AL-LSP plugin (left as-is). - Selecting which plugin agent *drives* the run (existing `--agent` covers it). - Relying on `extraKnownMarketplaces`/`enabledPlugins` injection (ignored headless). -- Redirecting `COPILOT_HOME` / `CLAUDE_CONFIG_DIR`; hand-writing CLI storage files. +- Hand-writing CLI storage files (undocumented schema — use the commands). - Dataset, category, or scoring changes. ## 4. Design @@ -113,46 +143,54 @@ with setup, uses documented commands, and is scoped to exactly what we installed New `operations/plugin_operations.py`: ```python -class InstalledPlugin(NamedTuple): - plugin: str # "frontend-web-dev" - marketplace: str # "awesome-copilot" - record: str # "frontend-web-dev@a1b2c3d4" (for ExperimentConfiguration) - def setup_plugins_from_config( agent_config: dict, entry: BaseDatasetEntry, repo_path: Path, agent_type: AgentType, cli_cmd: str, -) -> list[InstalledPlugin]: ... - -def teardown_plugins(cli_cmd: str, installed: list[InstalledPlugin]) -> None: ... +) -> tuple[list[str], dict[str, str]]: ... + # returns (plugin_records, env_overrides) ``` -`setup_plugins_from_config` clones/copies enabled plugins under `.bcbench`, runs -`marketplace add` + `install` (user scope), and returns what it installed (for the -result record and teardown). The agent launch command is unchanged. +`setup_plugins_from_config` creates a fresh per-entry config home under +`.bcbench`, clones/copies enabled plugins under `.bcbench`, runs `marketplace add` ++ `install` **into that home** (via `env`), and returns: +- `plugin_records`: `["@", "@local", …]` for the result, and +- `env_overrides`: `{"COPILOT_HOME": …}` / `{"CLAUDE_CONFIG_DIR": …}` (empty when + no plugin is enabled) that the runner merges into the agent-launch environment. + +There is **no** `teardown_plugins`: the isolated home is discarded with the +checkout, and a clean-before at the top of setup removes any stale home. ### 4.2 Materialize content into `.bcbench/` -- **Clean before / self-heal:** `remove_agent_plugin` for stale folders, and - defensively `uninstall` / `marketplace remove` our names (in case a prior run - crashed before teardown). +- **Clean before:** inline-`rmtree` `/.bcbench/plugins` and the per-entry + config home (do NOT import `remove_agent_plugin` — `operations` must not import + `bcbench.agent`, which would create a cycle). Then recreate both fresh. - **marketplace** (`repo` + `commit`): shallow-clone into - `/.bcbench/plugins/` and `git checkout ` (small - reused git helper). Read the marketplace `name` from its `marketplace.json`. -- **local** (`path`): copy into `/.bcbench/plugins/`. `path` resolves - under `src/bcbench/agent/shared/instructions//`. + `/.bcbench/plugins/` and `git checkout ` (`clone_at_commit`). + Read the marketplace `name` from its `marketplace.json`. +- **local** (`path`): copy into `/.bcbench/plugins/`. `path` resolves + under `src/bcbench/agent/shared/instructions//` and must be a + marketplace root (contains `marketplace.json`). - Failures raise `AgentError`. -### 4.3 Install via the CLI commands (user scope, both CLIs) -For each enabled entry, using the resolved CLI binary: -- ` plugin marketplace add /.bcbench/plugins/` +### 4.3 Install via the CLI commands (into the per-entry home) +With the config-home env var set to the per-entry directory, for each enabled +entry using the resolved CLI binary: +- ` plugin marketplace add /.bcbench/plugins/` - ` plugin install @` for each requested plugin -- Verify with ` plugin list`; failures raise `AgentError`. - -### 4.4 Teardown (symmetric, both CLIs) -In a `finally` around the agent run, for each installed plugin: -- ` plugin uninstall ` then ` plugin marketplace remove ` - (verified reversible; best-effort, logged, never masks the run outcome). -- `.bcbench` cloned content is removed by `remove_agent_plugin` / `git clean -fd`. -- The developer's real global config is returned to baseline. +- All plugin commands run with `env={**os.environ, : }`; failures + raise `AgentError` (after best-effort removal of the partial home). + +### 4.4 Isolation & cleanup (no teardown commands) +- The per-entry config home (`COPILOT_HOME` / `CLAUDE_CONFIG_DIR` = + `/.bcbench/-home`) makes every entry's marketplace/plugin state + private; concurrent matrix legs never collide, and there is nothing global to + undo. +- Cleanup is the clean-before `rmtree` of `.bcbench/plugins` + the home at the + next run, plus `git clean -fd` / fresh checkout. The developer's real + `~/.copilot` / `~/.claude` is never touched. +- Auth in the fresh home comes from the env token (`COPILOT_GITHUB_TOKEN`; + `ANTHROPIC_API_KEY`) the workflow already sets. Local plugin runs must have that + token in the environment. ### 4.5 Config schema (`config.yaml`) `plugins` is a flat list; each entry toggles independently. @@ -174,15 +212,13 @@ Only `enabled: true` entries are installed. `enabled` defaults to `true`. ### 4.6 Wiring into the runners ```python -installed = setup_plugins_from_config(config, entry, repo_path, agent_type, cli_cmd) -try: - result = subprocess.run(cmd_args, cwd=str(repo_path), ...) # unchanged agent launch - ... -finally: - teardown_plugins(cli_cmd, installed) - +plugin_records, plugin_env = setup_plugins_from_config(config, entry, repo_path, agent_type, cli_cmd) +... +env = {**os.environ, ...existing..., **plugin_env} # adds COPILOT_HOME / CLAUDE_CONFIG_DIR +result = subprocess.run(cmd_args, cwd=str(repo_path), env=env, ...) # unchanged args +... experiment_config = ExperimentConfiguration( - ..., plugins=[p.record for p in installed] or None, + ..., plugins=plugin_records or None, ) ``` @@ -206,35 +242,47 @@ plugins feature uses the commands; the `local` source could express AL-LSP later ## 5. Testing Mirror `test_mcp_config.py` / `test_custom_instructions.py` (subprocess + git mocked): - `tests/test_plugin_operations.py`: per-entry `enabled` (default true, disabled - skipped); marketplace clone/checkout; local copy into `.bcbench`; marketplace-name - read from `marketplace.json`; command construction for `marketplace add` / - `install` (user scope, both CLIs); `InstalledPlugin` records; `teardown_plugins` - builds correct `uninstall` / `marketplace remove`; clean-before self-heal; - `AgentError` on failure. + skipped → returns `([], {})`); marketplace clone/checkout; local copy into + `.bcbench`; marketplace-name read from `marketplace.json`; the per-entry config + home is created and passed as `env` to every `marketplace add` / `install` call + (`COPILOT_HOME` for Copilot, `CLAUDE_CONFIG_DIR` for Claude); returned + `(plugin_records, env_overrides)`; clean-before `rmtree` of `.bcbench/plugins` + + home; `AgentError` on failure (partial home removed). - Extend `test_experiment_configuration.py`: new `plugins` field + `is_empty`. +- Runner tests: assert `plugin_env` is merged into the agent-launch `env` and that + `config.plugins` is recorded. - Update mock `ExperimentConfiguration` scenarios (`commands/evaluate.py` `MockEvaluationPipeline`, result-serialization tests). ## 6. Documentation -- `EXPERIMENT.md`: add a `plugins` row + short schema/pinning note. +- `EXPERIMENT.md`: add a `plugins` row + short schema/pinning note, and the + env-token requirement for local plugin runs. - `config.yaml`: document the `plugins` block in comments. ## 7. Open questions / verification (small, non-blocking) -1. Confirm the Claude lifecycle (`marketplace add` + `install` + `uninstall` + - `marketplace remove`, user scope) end-to-end, mirroring the verified Copilot - lifecycle. High confidence; verify during implementation. -2. Ordering vs `setup_instructions_from_config`: run plugin setup after it. Low - risk (global config, not repo files), but keep the order explicit. -3. Migrate AL-LSP to a `local` plugin entry (deferred). -4. Optional agent-selection knob in a `plugins` entry (deferred; `--agent` covers it). +1. Confirm the fresh-`CLAUDE_CONFIG_DIR` headless **session** loads the plugin and + runs with `ANTHROPIC_API_KEY` (Copilot's fresh-`COPILOT_HOME` session is + verified; Claude's install-into-fresh-dir + isolation is verified — only the + authenticated session remains, and CI already runs Claude headless with the env + key). +2. Ensure Copilot's `--log-dir=` still lands `process-*.log` in + `output_dir` when `COPILOT_HOME` is redirected (used for turn-count metrics). + `--log-dir` is explicit, so this should hold; verify during implementation. +3. Ordering vs `setup_instructions_from_config`: run plugin setup after it. +4. Migrate AL-LSP to a `local` plugin entry (deferred). +5. Optional agent-selection knob in a `plugins` entry (deferred; `--agent` covers it). ## 8. References (verified) - Independent investigation: gist `alexey-pelykh/566a4e5160b305db703d543312a1e686` — `extraKnownMarketplaces`/`enabledPlugins` are trust-dialog-gated and ignored in headless mode; recommends explicit `plugin marketplace add` / `install` commands. - Live probes: full Copilot plugin lifecycle against a local marketplace - (non-interactive, offline, exit 0, "Installed 1 skill", baseline restored); a real - `copilot -p` session did **not** auto-install from repo-local settings. + (non-interactive, offline, exit 0, "Installed 1 skill"); a real `copilot -p` + session did **not** auto-install from repo-local settings; **fresh `COPILOT_HOME` + headless session runs (exit 0) and loads an installed plugin's skill, isolated + from the real config**; fresh `CLAUDE_CONFIG_DIR` isolates install the same way. +- Concurrency: `copilot-evaluation.yml` matrix `max-parallel: 64` on self-hosted + `GitHub-BCBench`; `types.py.runner` per-category runner labels. - PR #651 / commit `7eab2f9` (Haoran) — `agent/shared/plugin.py`. - `agent/copilot/agent.py`, `agent/claude/agent.py` — runner + `shutil.which` CLI resolution + subprocess pattern to reuse. From ae388167bbce2455efdeadac3c546fd7c52e8dc3 Mon Sep 17 00:00:00 2001 From: bcbench Date: Mon, 6 Jul 2026 10:41:38 +0200 Subject: [PATCH 3/3] docs: clarify uninstall/remove are verified-reversible but unused (per-entry home) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...026-07-03-marketplace-and-local-plugins-experiment-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-07-03-marketplace-and-local-plugins-experiment-design.md b/docs/superpowers/specs/2026-07-03-marketplace-and-local-plugins-experiment-design.md index d20c524cc..c4882753b 100644 --- a/docs/superpowers/specs/2026-07-03-marketplace-and-local-plugins-experiment-design.md +++ b/docs/superpowers/specs/2026-07-03-marketplace-and-local-plugins-experiment-design.md @@ -78,7 +78,7 @@ Copilot CLI `1.0.69-0`, Claude Code `2.1.161`. non-interactive, offline, exit 0 each: - `copilot plugin marketplace add ` → "Marketplace added successfully." - `copilot plugin install @` → "Plugin installed successfully. Installed 1 skill." - - `copilot plugin uninstall ` / `plugin marketplace remove ` → success; baseline fully restored. + - `copilot plugin uninstall ` / `plugin marketplace remove ` → success; baseline fully restored (verified reversible, but **not used** — see §2.5, we isolate via a per-entry home instead of tearing down). - Marketplace source accepts a **local path** (the marketplace ROOT — the dir containing `marketplace.json`); the CLI records a local source as `{"source":"directory","path":…}`. The plugin is selected `@`