agents cli: load remote Streamable HTTP MCP servers from mcp_servers (#1795)

Author: longquanzheng

agents/cli/README.md

@@ -60,6 +60,7 @@ claude: {}
 | `dex.tls.ca_file`, `dex.tls.server_name` | none | An `https` `dex.server` only. A PEM file of root CAs to trust as well, and the name the certificate must match. `DEX_TLS_CA_FILE` and `DEX_TLS_SERVER_NAME` override them. |
 | `connectors.mcp` | `auto` | The connectors MCP server. `auto` uses the one `dexcli dev` provides. Otherwise an `http` or `https` URL on `127.0.0.1`, `localhost`, or `[::1]`. See URLs below. |
 | `connectors.enabled` | every authorized connection | Which connections, and which of their operations, the agent may call. See below. |
+| `mcp_servers` | `{}` | Remote MCP servers whose tools the agent may call, by name, over Streamable HTTP. See Remote MCP servers below. |
 | `approval.always` | `[]` | Tool names, as the agent sees them, whose every call waits for approval. This adds to what the connectors and local tools already require. |
 | `workspace` | `~/dex-agent-workspace`; for `dex-agents-pi`, the directory the command starts in | The only directory the local file tools use, and the working directory of shell commands. |
 | `local_tools.files` | `true` | Read, write, list, and search files in `workspace`. Reads run at once; writes wait for approval. |
@@ -85,7 +86,8 @@ boundary. As in Pi, they run without approval unless `approval.always` names
 them; see [its README](typescript/pi/README.md#tools-and-approvals).
 
 Do not put secrets in the file. Provider keys come from the environment.
-Connector credentials stay in the connections you authorize in Dex Web. The
+Connector credentials stay in the connections you authorize in Dex Web.
+Remote MCP server headers take their secrets from environment variables. The
 schema rejects user info in URLs, but it cannot check the harness sections,
 which each harness command defines.
 
@@ -173,6 +175,66 @@ letter or a digit, and an acronym stays one word, so `listStoryIDs` is
 Two entries must not cover the same connection, so `slack` and `slack/work`
 cannot appear together. An empty list gives the agent no connector tools.
 
+### Remote MCP servers
+
+`mcp_servers` adds the tools of MCP servers that are not Dex connectors, such
+as a team's own tool server. Each key is the server's name, and each server
+speaks the Streamable HTTP transport. Stateless servers work: every request
+opens its own connection and protocol session.
+
+```yaml
+mcp_servers:
+  studio:
+    url: https://mcp.example.com/mcp
+    headers:
+      Authorization: "Bearer ${STUDIO_API_KEY}"
+  local-notes:
+    url: http://127.0.0.1:9000/mcp
+    trust_tool_annotations: true
+```
+
+| Field | Default | Meaning |
+| --- | --- | --- |
+| name (the key) | | 1 to 32 ASCII letters, digits, `_`, or `-`, starting with a letter. |
+| `url` | required | The server's endpoint: `https`, or `http` on `localhost`, `127.0.0.0/8`, or `[::1]` (`insecure_mcp_server_url`). No user info or fragment. It follows the URL rules of `dex.server` and is used as written. |
+| `headers` | `{}` | HTTP headers sent with every request to the server. |
+| `trust_tool_annotations` | `false` | Whether the server's `readOnlyHint` and `idempotentHint` annotations, and the effects it declares under the Dex MCP conventions, pick each tool's effect. |
+
+The agent sees each tool as `<server name>__<tool name>`, such as
+`studio__search_assets`, and `approval.always` names it that way. With
+`trust_tool_annotations: false`, every call waits for approval and runs at
+most once. With `true`, a tool the server marks read-only or idempotent runs
+without approval, unless `approval.always` names it, and may be retried; set
+it only for a server whose hints you trust. `dex-agents-pi` and
+`dex-agents-deepagents` still make every call that is not a read wait for
+approval, as they do for connector tools.
+
+Secrets belong in environment variables, never in the file. In a header
+value, each `${NAME}` is replaced by the value of the environment variable
+`NAME` when the command starts. `NAME` has ASCII letters, digits, and `_`, and
+does not start with a digit. Any other `$` stays as written. The value is used
+exactly as set, never trimmed or expanded again. `DEX_API_KEY`, `DEX_CREDENTIAL`, and `DEX_CREDENTIAL_FILE` count as not set,
+so a Dex credential never reaches another server. A variable that is not set
+or is empty stops the command with exit code 2 and a message that names it,
+such as
+`mcp_servers.studio.headers.Authorization needs the environment variable STUDIO_API_KEY, which is not set or is empty.`
+A value outside printable ASCII stops it the same way. A header value may
+also be literal text, but a literal secret then sits in the file: use a
+`${NAME}` reference for every secret. Only the Worker expands the values. The command never prints, logs, or stores them:
+the effective configuration, the session, and the log file keep the text as
+written. Error text that a server sends back, such as an HTTP 401 body or a
+tool error, is redacted on a best-effort basis: each header value, each of its
+parts between spaces, `=`, `:`, `,`, or `;`, and each variable value of at
+least 8 characters becomes `[redacted]` in the log file, the terminal, tool
+output, and errors. Other content the server returns, such as progress and
+task status messages and tool descriptions, is not covered, so give a token
+only to servers you trust with it. A Worker that runs a stored call checks it against its own
+configuration: for a server whose annotations it does not trust, the call must
+have had approval and a single-attempt effect, or it fails unsent.
+
+A server that cannot be listed offers no tools until it answers again; the
+log file has the error. Session settings cannot change `mcp_servers`.
+
 ### Images
 
 In the chat, `/image <path>` attaches an image file to your next message. A
@@ -242,7 +304,7 @@ session in Dex Web can edit its settings.
 
 Other top-level keys are ignored, so the settings can also hold keys that
 other readers use. This includes `dex`, `workspace`, `system_prompt_file`,
-`media`, and the harness sections, which belong to the file only. The command reports each
+`media`, `mcp_servers`, and the harness sections, which belong to the file only. The command reports each
 ignored key as a warning, but the turn still runs. Inside a known field, an
 unknown key is an error, so `connectors.mcp` fails.
 
@@ -257,7 +319,9 @@ cases every CLI core runs: YAML parsing, valid and invalid files with the
 expected effective configuration or error, session settings with the
 expected turn configuration and warnings, or error, and, in `dex_connection`,
 files and environments with the expected Dex connection settings, or the
-setting to fix and its exact message. Its `error_codes`,
+setting to fix and its exact message, and, in `mcp_servers`, files and
+environments with the expected remote MCP servers and their expanded headers,
+or the setting to fix and its exact message. Its `error_codes`,
 `warnings`, and `check_order` fields define how errors and warnings are
 reported.
 

agents/cli/config.schema.json

@@ -222,6 +222,17 @@
         }
       }
     },
+    "mcp_servers": {
+      "type": "object",
+      "default": {},
+      "description": "Remote MCP servers whose tools the agent may call over the Streamable HTTP transport, by server name: 1 to 32 ASCII letters, digits, underscores, or hyphens, starting with a letter. The agent sees each tool as <server name>__<tool name>, and approval.always names it that way. Every call waits for approval unless trust_tool_annotations lets the server's hints mark the tool a read or an idempotent write. Keep secrets out of this file: a header value can take them from environment variables with ${NAME} references, which the Worker expands when the command starts. Session settings cannot change it: they ignore an mcp_servers key with a warning.",
+      "patternProperties": {
+        "^[A-Za-z][A-Za-z0-9_-]{0,31}(?![\\s\\S])": {
+          "$ref": "#/$defs/mcp_server"
+        }
+      },
+      "additionalProperties": false
+    },
     "approval": {
       "type": "object",
       "additionalProperties": false,
@@ -475,6 +486,39 @@
         "$ref": "#/$defs/connector_entry"
       }
     },
+    "mcp_server": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": [
+        "url"
+      ],
+      "description": "One remote MCP server.",
+      "properties": {
+        "url": {
+          "type": "string",
+          "pattern": "^https?://(?:(?:[A-Za-z0-9-]+\\.)*[A-Za-z][A-Za-z0-9-]*|[0-9]{1,3}(?:\\.[0-9]{1,3}){3}|\\[[0-9A-Fa-f:.]+\\])(?::[0-9]{1,5})?(?:/[!-\"$-~]*)?(?![\\s\\S])",
+          "description": "The server's Streamable HTTP endpoint: an https URL, or an http URL on localhost, 127.0.0.0/8, or [::1], without user info or a fragment. The host rules and the CLI core checks are those of dex.server. The URL is used as written, never expanded, so it must not hold a secret."
+        },
+        "headers": {
+          "type": "object",
+          "default": {},
+          "description": "HTTP headers sent with every request to the server, such as Authorization, by header name: 1 to 256 HTTP token characters. A value is printable ASCII. Each ${NAME} in it is replaced by the value of the environment variable NAME when the command starts; NAME has ASCII letters, digits, and underscores and does not start with a digit. A variable that is not set, is empty, or holds anything but printable ASCII stops the command. Any other $ stays as written. A value may be literal; put each secret in a ${NAME} reference instead. The command never prints, logs, or stores the expanded values.",
+          "patternProperties": {
+            "^[!#$%&'*+.^_`|~0-9A-Za-z-]{1,256}(?![\\s\\S])": {
+              "type": "string",
+              "pattern": "^(?:[ -#%-~]|\\$(?!\\{)|\\$\\{[A-Za-z_][A-Za-z0-9_]*\\})*(?![\\s\\S])",
+              "description": "A header value: printable ASCII, where ${ always starts a ${NAME} reference."
+            }
+          },
+          "additionalProperties": false
+        },
+        "trust_tool_annotations": {
+          "type": "boolean",
+          "default": false,
+          "description": "Whether the server's readOnlyHint and idempotentHint annotations, and the effects it declares by the Dex MCP conventions, pick each tool's effect. false: every call of the server's tools waits for approval and runs at most once. true: a tool the server marks read-only or idempotent runs without approval, unless approval.always names it, and may be retried. Set it only for a server whose hints you trust."
+        }
+      }
+    },
     "shell_commands": {
       "type": "array",
       "uniqueItems": true,

agents/cli/python/core/README.md

@@ -438,6 +438,35 @@ read, and caches them for the harness:
   Connectors page, `<Dex Web URL>/v1/connectors`, and opens it only in an
   interactive terminal.
 
+## Remote MCP server tools
+
+`McpServersToolSource` offers the tools of the configuration's
+`mcp_servers` through `McpToolSource`, over Streamable HTTP. The agent sees
+each tool as `<server name>__<tool name>`. Its `source_name` is
+`mcp_servers`, and the command registers it even without servers, so it
+refuses a stored call of a server this Worker does not configure.
+
+- `resolve_mcp_servers(config, os.environ)` expands each `${NAME}` in a
+  header value when the command starts. A variable that is not set, is
+  empty, or is not printable ASCII raises `McpServerSettingsError`, whose
+  message names the setting and the variable, never the value; the command
+  prints it and exits with code 2. The Worker host does the same.
+- Each call adds the turn's `approval.always`. The tool Step checks it again
+  on the Worker that runs it and refuses a call that had no approval but now
+  needs one. For a server whose annotations that Worker does not trust, it
+  also refuses, unsent, a call without approval or with a retried effect.
+- The source replaces each header value, each part of it between whitespace,
+  `=`, `:`, `,`, or `;`, and each value of `mcp_server_secret_texts`, of at
+  least 8 characters, with `[redacted]` in errors, tool output, and the log
+  file. This is best-effort for error text only: progress and task status
+  messages and tool descriptions are not redacted. The log filter covers the
+  root logger's handlers that exist when the Worker connects the servers.
+- A listing that no server answers offers no remote tools and prints one
+  line, `MCP server tools are unavailable for now; each turn tries again.
+  See <log file> for the error.`, until a server answers again. The error
+  goes to the log file. The command lists every server once at startup, so a
+  harness whose tool set is fixed at build gets the tools that answered then.
+
 ## Local tools
 
 The local tool source offers these tools in their own Dex tool Steps. Paths
@@ -646,7 +675,7 @@ harness may replace only `FILES` and `SHELL`.
 | `workspace: Path` | The absolute workspace, which exists. |
 | `instructions: str \| None` | `system_prompt`, else the text of `system_prompt_file`, else `None` for the harness default. |
 | `harness_options: Mapping[str, Any]` | The harness's own section, `{}` by default. |
-| `tool_sources: Sequence[ToolSource]` | The connector and local tool sources, which apply each turn's settings themselves. Offer all of them. |
+| `tool_sources: Sequence[ToolSource]` | The connector, remote MCP server, and local tool sources, which apply each turn's settings themselves. Offer all of them. |
 | `media_store: MediaStore \| None` | The media store of `media.store`, holding the images of `/image` and of connector tool results. Pass it to an adapter that takes one, such as `media_store=context.media_store`. The CLI core always sets it; it is `None` only in a context a test builds without one. |
 | `resolve_turn(session_settings) -> ResolvedTurn` | A turn's narrowed configuration with its `model`, `instructions`, and `reasoning_effort`. Raises `ConfigError` for rejected settings. |
 | `resolve_tool_step_permissions(call=None) -> ResolvedTurn` | The permissions a replacement tool's own Step applies: this Worker's file narrowed by the session settings recorded for the call's turn. Pass the `DeferredToolExecution` from a runner, or nothing from a built-in loop function tool. Raises `RuntimeError` without a call outside a tool Step and a model call. |

agents/cli/python/core/src/dex_agents_cli_core/__init__.py

@@ -27,6 +27,7 @@
     ConnectorEntry,
     EffectiveConfig,
     MediaConfig,
+    RemoteMcpServerConfig,
     SessionsConfig,
     TurnConfigResolution,
     canonical_operation_name,
@@ -54,6 +55,13 @@
     session_flow_type,
 )
 from dex_agents_cli_core.local_tools import LOCAL_TOOL_SOURCE_NAME, LocalToolSource
+from dex_agents_cli_core.mcp_servers import (
+    MCP_SERVERS_TOOL_SOURCE_NAME,
+    McpServerSettingsError,
+    McpServersToolSource,
+    mcp_server_secret_texts,
+    resolve_mcp_servers,
+)
 from dex_agents_cli_core.tool_policy import (
     is_local_tool_group_enabled,
     is_shell_command_approval_required,
@@ -71,6 +79,7 @@
     "CONNECTORS_TOOL_SOURCE_NAME",
     "CONNECTOR_TOOL_META_KEYS",
     "LOCAL_TOOL_SOURCE_NAME",
+    "MCP_SERVERS_TOOL_SOURCE_NAME",
     "SESSION_ACTION_PERMISSION",
     "AgentHarness",
     "ConfigError",
@@ -81,7 +90,10 @@
     "HarnessContext",
     "LocalToolGroup",
     "LocalToolSource",
+    "McpServerSettingsError",
+    "McpServersToolSource",
     "MediaConfig",
+    "RemoteMcpServerConfig",
     "ResolvedTurn",
     "SessionEnvironmentIdentity",
     "SessionsConfig",
@@ -96,9 +108,11 @@
     "is_session_summary_indexed",
     "is_shell_command_allowed",
     "is_shell_command_approval_required",
+    "mcp_server_secret_texts",
     "namespace_session_flow_type",
     "parse_config_yaml",
     "resolve_config",
+    "resolve_mcp_servers",
     "resolve_turn_config",
     "resolve_turn_tool_policy",
     "run_agent_cli",

agents/cli/python/core/src/dex_agents_cli_core/cli.py

@@ -57,6 +57,7 @@
     sort_agent_sessions_by_last_activity,
 )
 from dex_agents.builtin_loop import ToolSource
+from dex_agents.mcp import McpStreamableHttpServerConfig
 from dex_agents_cli_core.chat import Terminal, TerminalChat
 from dex_agents_cli_core.config import (
     AUTO,
@@ -125,6 +126,12 @@
 )
 from dex_agents_cli_core.local_tools import LocalToolSource
 from dex_agents_cli_core.local_tools.shell import ShellProcessGroups
+from dex_agents_cli_core.mcp_servers import (
+    McpServerSettingsError,
+    McpServersToolSource,
+    mcp_server_secret_texts,
+    resolve_mcp_servers,
+)
 from dex_agents_cli_core.session_lease import (
     HeldSessionLease,
     SessionLeaseHolder,
@@ -865,6 +872,8 @@ class _RunInputs:
     workspace: Path
     resolver: TurnConfigResolver
     connection: DexConnectionSettings
+    mcp_servers: tuple[McpStreamableHttpServerConfig, ...] = ()
+    mcp_server_secret_texts: frozenset[str] = frozenset()
 
 
 def _resolve_connection(
@@ -920,7 +929,14 @@ def _prepare_run(
         report_ignored_settings,
         supported_reasoning_efforts=declared_reasoning_efforts(harness),
     )
-    return _RunInputs(config, media_store, workspace, resolver, connection)
+    try:
+        mcp_servers = resolve_mcp_servers(config, os.environ)
+        secret_texts = mcp_server_secret_texts(config, os.environ)
+    except McpServerSettingsError as error:
+        raise CommandError(error.message) from error
+    return _RunInputs(
+        config, media_store, workspace, resolver, connection, mcp_servers, secret_texts
+    )
 
 
 async def _run_plain_chat(
@@ -1091,8 +1107,9 @@ async def start(
         local_tools = _local_tool_source(
             config, inputs.workspace, harness, inputs.resolver, process_groups
         )
+        mcp_servers = await _connect_mcp_servers(inputs, reports)
         tool_sources: list[ToolSource] = [
-            source for source in (connectors, local_tools) if source is not None
+            source for source in (connectors, mcp_servers, local_tools) if source is not None
         ]
         identity = current_session_environment_identity(loaded.path, inputs.workspace)
         flow_type = namespace_session_flow_type(harness.flow_type, identity)
@@ -1343,6 +1360,35 @@ def describe_unavailable(error: ConnectorsMcpUnavailableError) -> str:
     return source
 
 
+async def _connect_mcp_servers(
+    inputs: _RunInputs, reports: _SessionWorkerReports
+) -> McpServersToolSource:
+    """Return the remote MCP server source, listed once so fixed tool sets get its tools.
+
+    It is returned even without servers, so it refuses stored calls of servers
+    another Worker has.
+    """
+    log_path = reports.log_path
+    unavailable_line = "MCP server tools are unavailable for now; each turn tries again."
+    if log_path is not None:
+        unavailable_line = f"{unavailable_line} See {log_path} for the error."
+    source = McpServersToolSource(
+        inputs.mcp_servers,
+        inputs.resolver,
+        report=reports.report_runtime,
+        report_at_startup=reports.report,
+        unavailable_line=unavailable_line,
+        media_store=inputs.media_store,
+        secret_texts=inputs.mcp_server_secret_texts,
+    )
+    if inputs.mcp_servers:
+        # Every record of this process may quote a server's error, which may echo a header.
+        for handler in logging.getLogger().handlers:
+            handler.addFilter(source.secret_redacting_log_filter)
+        await source.list_file_enabled_tools()
+    return source
+
+
 def _connectors_unavailable_hint(log_path: Path | None, web_url: str | None) -> str:
     """Say where the error and the connections are, for a line about an unavailable server."""
     page = f"{web_url}{CONNECTORS_WEB_PATH}" if web_url else "the Dex Web Connectors page"