docs: design + plan for marketplace & local plugins experiment toggle#733
docs: design + plan for marketplace & local plugins experiment toggle#733Groenbech96 wants to merge 3 commits into
Conversation
…gins experiment toggle Adds a design spec and a task-by-task implementation plan for a new experiment toggle that installs agent plugins (a marketplace pinned to a git commit, or a local marketplace) into the running CLI via its own 'plugin marketplace add' + 'plugin install' commands, records them in ExperimentConfiguration, and removes them after the run. Docs only; no code changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| design reuses that runtime-created, repo-local root to hold cloned marketplace | ||
| content. | ||
|
|
||
| ### 2.3 Why the commands (config injection does not work headless) |
There was a problem hiding this comment.
Plugins install at user scope (§2.3 says project scope is ignored headless), so the shared /.copilot//.claude store means concurrent runs can race — one entry's marketplace remove teardown can rip out a plugin another in-flight entry is still using; please state whether plugins-enabled runs must be serialized or add per-run locking/refcounting.
There was a problem hiding this comment.
Maybe it due to my limited knowledge on how experiments are executed.
Are they running on the same machine?
There was a problem hiding this comment.
For code-review every task should be a new VM. So it should not be affected.
But I remember for bug-fix sometimes it could be the same machine.
There was a problem hiding this comment.
Confirmed and fixed. Verified against copilot-evaluation.yml: entries run with max-parallel: 64 on the shared self-hosted GitHub-BCBench runner (bug-fix / test-generation), and the CLI plugin store is user-scope/global — so concurrent installs and one entry's marketplace remove racing another in-flight entry is a real problem, exactly as described. code-review (ubuntu-latest) and nl2al (windows-latest) get fresh VMs and are unaffected (thanks @Groenbech96 / @gggdttt for pinning that down).
Fix (pushed, commit 469ac69): each entry now installs into a fresh per-entry config home (COPILOT_HOME / CLAUDE_CONFIG_DIR under <repo>/.bcbench/), applied to both the plugin commands and the agent launch. Concurrent entries no longer share global state, so there is no cross-entry race — and no teardown at all (the home is discarded with the checkout; a clean-before removes any stale one). Verified live on both CLIs: a fresh COPILOT_HOME accepts marketplace add/install, copilot skill list shows the installed plugin's skill, and a real copilot -p session runs (auth via the COPILOT_GITHUB_TOKEN the workflow already sets) with the real ~/.copilot untouched. See spec §2.5 (Concurrency) and §4.4.
| agent_type: AgentType, cli_cmd: str, | ||
| ) -> list[InstalledPlugin]: ... | ||
|
|
||
| def teardown_plugins(cli_cmd: str, installed: list[InstalledPlugin]) -> None: ... |
There was a problem hiding this comment.
§4.1's signature teardown_plugins(cli_cmd, installed) is missing the agent_type param that the plan (Task 4/5/6) uses — it's needed because Copilot uninstalls by plugin and Claude by plugin@marketplace. Please sync the spec to teardown_plugins(cli_cmd, agent_type, installed).
There was a problem hiding this comment.
Fixed by removing teardown entirely. With the per-entry isolated config home (§2.5), there is nothing global to undo, so teardown_plugins is gone. setup_plugins_from_config now returns (plugin_records, env_overrides), and the runner merges env_overrides (COPILOT_HOME / CLAUDE_CONFIG_DIR) into the agent-launch env. Spec §4.1 / §4.6 and the implementation plan (Tasks 4-6) updated accordingly.
…iew) Addresses review: execution-based categories run entries as a parallel matrix (max-parallel 64) on a shared self-hosted runner, and the CLI plugin store is user-scope/global, so concurrent installs/teardowns would race. Fix: install into a fresh per-entry config home (COPILOT_HOME / CLAUDE_CONFIG_DIR under .bcbench), applied to both the plugin commands and the agent launch. This removes teardown entirely (isolated home discarded with the checkout) and moots the teardown_plugins signature comment. Verified live on both CLIs (fresh home: install + skill load + headless session, real config untouched). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…r-entry home) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2 participants
What
Docs only — adds the design spec and the implementation plan for a new experiment toggle: marketplace & local agent plugins. No code changes.
docs/superpowers/specs/2026-07-03-marketplace-and-local-plugins-experiment-design.mddocs/superpowers/plans/2026-07-06-marketplace-and-local-plugins.mdDesign summary
A config-driven
pluginstoggle (peer ofinstructions/skills/agents/mcp) that installs agent plugins into the running CLI so their skills are native during a run — measured like the other experiment levers.marketplace(repo + pinnedcommit) orlocal(path). Per-entryenabled; multiple at once.plugin marketplace add+plugin install, user scope, both Copilot & Claude). Config injection (extraKnownMarketplaces/enabledPlugins) is not used — it is trust-dialog-gated and ignored in headless mode (verified live + corroborated by an independent write-up).<repo>/.bcbench/plugins/.uninstall+marketplace remove);.bcbenchcontent wiped by the existing cleanup. No global/home residue.ExperimentConfiguration.plugins: list[str]("name@commit"/"name@local").Full plugin lifecycle was verified live on both CLIs (non-interactive, offline, reversible).
Plan
Seven TDD tasks:
ExperimentConfiguration.pluginsfield →clone_at_commitgit helper →plugin_operations.py(materialize + install/teardown) → Copilot & Claude runner wiring → config/docs/mock + full-suite verification.Note
This PR is the entry point for reviewing the approach before implementation — merging is optional.