Skip to content

Commit d9f3505

Browse files
feat: canonical fresh Copilot model listing with family-tier recommendations (#91)
* feat: add uncached account-backed Copilot model discovery * fix: publish Copilot client only after cancellable startup completes * fix: cancel and stop a pending Copilot startup when closing * feat: make canonical Copilot listing fresh with family-tier recommendations * fix: report only SDK-advertised catalog capabilities * fix: keep missing catalog output limits unknown --------- Co-authored-by: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
1 parent ed942e2 commit d9f3505

21 files changed

Lines changed: 1735 additions & 191 deletions

‎MIGRATION.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,18 @@
1+
# Unreleased: canonical model listing (PR91 successor)
2+
3+
`await provider.list_models()` remains valid and returns `list[ModelInfo]`, but
4+
now returns current SDK recommendations rather than a cached full inventory.
5+
Use `await provider.list_models(filters={})` for all current IDs. Optional
6+
`model_filters` configuration and per-call filters replace defaults; see README.
7+
Errors no longer become disk-cache success. Unknown catalog limits are `0`;
8+
inference's existing conservative budget policy is unchanged.
9+
10+
The unshipped candidate `discover_models()` extension is removed: use canonical
11+
`list_models(filters={})` plus local `get_model_list_info()` metadata instead.
12+
Consumers of that candidate must stop unpacking the old `{"models": ...}` envelope.
13+
Rollback to the prior provider restores its old cached-list behavior, not fresh
14+
account inventory. No Core/Foundation dependency or inference model pin changes.
15+
116
# Migration Guide: v2.7.x → v2.8.0
217

318
## Overview

‎README.md‎

Lines changed: 35 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -190,7 +190,7 @@ providers:
190190
191191
| Key | Default | Description |
192192
| --- | --- | --- |
193-
| `default_model` | `"auto"` | Model used when the caller does not specify one. Any ID from `list_models()` is valid. `auto` is Copilot's server-side router (the Copilot CLI's default): each turn goes to a model the service picks. It advertises no limits or reasoning levels, so the provider reports a conservative 128k context / 16k output, and a `reasoning_effort` is dropped for it once the model list is cached. Set an explicit model ID for a fixed window, reasoning control, or reproducible routing. |
193+
| `default_model` | `"auto"` | Model used when the caller does not specify one. Exact supported IDs remain usable even when omitted from recommendations. `auto` is Copilot's server-side router; the catalog includes it only if the SDK returns it. Unknown catalog limits are `0`, not the conservative inference policy budgets. Set an explicit model ID for reproducible routing. |
194194
| `reasoning_effort` | `"model default"` | Reasoning effort forwarded to models that support it (e.g. `low`, `medium`, `high`). Both this default and a caller-supplied value are best-effort: forwarded when the model supports it, dropped (field omitted, server uses its own default) when it does not — so a global or inherited effort never breaks delegation to a non-reasoning model. A malformed value (typo, mixed-case) still raises `ConfigurationError`. `"model default"` or unset defers to the model. Run `amplifier provider models github-copilot` for each model's levels. |
195195
| `enable_long_context` | `false` | Default to the model's long-context tier when the caller does not set a `context_tier`. Forwarded to the SDK as `long_context`; ignored by models without a long tier. |
196196
| `raw` | `false` | Include raw SDK payloads as a `"raw"` field in `llm:request` / `llm:response` events. See [Raw Payload Logging](#raw-payload-logging). |
@@ -399,7 +399,39 @@ For module structure, design decisions, and contract index see [docs/ARCHITECTUR
399399

400400
The provider translates all SDK errors to typed kernel errors before they reach the caller. Each `complete()` call uses an independent session — no state accumulates between requests. The shared client and disk model cache persist across requests by design.
401401

402-
On `list_models()` failure, the provider falls back to a disk cache (24-hour TTL) before raising `ProviderUnavailableError`.
402+
`list_models()` always uses the current bound client's typed SDK `listModels` RPC.
403+
It neither reads nor warms disk, provider or wrapper caches, and never substitutes
404+
static models. A genuine empty response returns `[]`; failures raise typed,
405+
sanitized errors. The separate legacy inference capability lookup is unchanged.
406+
Catalog capability flags come from the SDK; an absent flag is not a fabricated
407+
positive claim. Provider-level streaming support remains separate.
408+
409+
### Model recommendations and filters
410+
411+
```python
412+
await provider.list_models() # configured defaults (same as filters=None)
413+
await provider.list_models(filters={}) # ALL current SDK IDs, no recommendation policy
414+
await provider.list_models(filters={"include": ["claude-*"], "exclude": ["*-preview"]})
415+
await provider.list_models(filters={"include": []}) # no selected models
416+
```
417+
418+
The default `model_filters` is `{"include": ["*"], "exclude": [],
419+
"hide_dated": True, "latest_per_family": True}`. Set `model_filters` in provider
420+
configuration to replace it. An explicit per-call mapping replaces the entire
421+
configured policy, never merges it. Filters are copied before I/O; invalid keys,
422+
patterns and flags fail early. Globs are case-insensitive and OS-independent;
423+
exclusion wins and returned endpoint IDs remain byte-exact.
424+
425+
Latest-per-family compares numeric versions only within the same vendor, family,
426+
tier and route. For example, a **controlled test inventory**, not a live entitlement
427+
claim, retains GPT-6 Astra/Luna/Terra and GPT-5.10 Sol; Claude 5.5
428+
Opus/Sonnet/Haiku; Grok 4.7 and 4.7 Fast; and MAI Code 2 Flash and its picker route.
429+
It does not rank those tiers by price or quality. A preview remains distinct.
430+
An alias beats its matching snapshot; snapshot-only ties retain the newest date.
431+
Unrecognized names are conservatively retained, even by default. No model name
432+
is manufactured. `get_model_list_info()` describes all active policy locally
433+
using the version-1 selector metadata (`filters_schema`, `default_filters`,
434+
`unfiltered`, `override`). Filters never change explicit inference selection.
403435

404436
## Tool Sequence Repair
405437

@@ -520,7 +552,7 @@ python -m pytest tests/ -m live -v --tb=short
520552
| `Copilot SDK not installed` | Provider module not installed | Run `amplifier provider install github-copilot` |
521553
| `Not authenticated to GitHub Copilot` | Token not set | **Linux/macOS:** `export GITHUB_TOKEN=$(gh auth token)` **Windows:** `$env:GITHUB_TOKEN = (gh auth token)` |
522554
| `gh: command not found` | GitHub CLI missing | [Install gh CLI](https://cli.github.com/) |
523-
| Stale or wrong model list | Cached models | Delete `models_cache.json` from your cache directory: `%LOCALAPPDATA%\amplifier-provider-github-copilot\Cache\` (Windows), `~/Library/Caches/amplifier-provider-github-copilot/` (macOS), or `~/.cache/amplifier-provider-github-copilot/` (Linux). |
555+
| Missing model recommendation | Default filters or bound account | Try `list_models(filters={})` for unfiltered current inventory. Check the bound SDK account if still absent; catalog listing does not use disk cache. |
524556
| `Permission denied` on SDK binary | `uv` stripped execute bits | Provider auto-repairs on startup; if it fails, run `chmod +x <path-to-copilot-binary>` (Linux/macOS only) |
525557

526558
### Common Mistake
Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
"""Provider-local selectors: policy only, never an inventory or inference gate."""
2+
3+
from __future__ import annotations
4+
5+
import re
6+
from collections.abc import Mapping
7+
from dataclasses import dataclass
8+
from fnmatch import fnmatchcase
9+
10+
# Only recognize unambiguous version syntax. Everything else remains visible.
11+
# Keep the whole non-version name: astra/luna/sol/terra, opus/sonnet/haiku,
12+
# pro/flash/flash-lite, mini/codex, preview and picker routes are NOT equivalent.
13+
_VERSION = re.compile(
14+
r"^(gpt|claude(?:-[a-z]+)?|gemini|grok|mai-[a-z]+)-"
15+
r"(\d+(?:[.-]\d+)*)(?=-|$)(.*)$"
16+
)
17+
_O_VERSION = re.compile(r"^(o)(\d+)(?=-|$)(.*)$")
18+
_DATE = re.compile(r"-(\d{4}-\d{2}-\d{2}|\d{8})$")
19+
20+
21+
def _undated(model_id: str) -> tuple[str, str]:
22+
match = _DATE.search(model_id)
23+
return (model_id[: match.start()], match[1].replace("-", "")) if match else (model_id, "")
24+
25+
26+
def _family_version(model_id: str) -> tuple[str, tuple[int, ...], str] | None:
27+
base, date = _undated(model_id.casefold())
28+
match = _VERSION.fullmatch(base) or _O_VERSION.fullmatch(base)
29+
if match is None:
30+
return None
31+
prefix, version, suffix = match.groups()
32+
# Both claude-3.5-sonnet and claude-sonnet-4.5 identify the Sonnet family.
33+
# Preserve all suffixes, including unknown future tiers, rather than guessing.
34+
family = prefix + suffix
35+
numbers = tuple(int(part) for part in re.split(r"[.-]", version))
36+
return family, numbers, date
37+
38+
39+
@dataclass(frozen=True)
40+
class ModelFilters:
41+
"""Immutable copy of common selectors plus explicit recommendation policy."""
42+
43+
include: tuple[str, ...] = ("*",)
44+
exclude: tuple[str, ...] = ()
45+
hide_dated: bool = False
46+
latest_per_family: bool = False
47+
48+
@classmethod
49+
def parse(cls, value: Mapping[str, object]) -> ModelFilters:
50+
if not isinstance(value, Mapping):
51+
raise TypeError("model filters must be a mapping")
52+
if set(value) - {"include", "exclude", "hide_dated", "latest_per_family"}:
53+
raise ValueError("unknown model filter key")
54+
55+
def patterns(key: str, default: tuple[str, ...]) -> tuple[str, ...]:
56+
raw = value.get(key, default)
57+
if not isinstance(raw, (list, tuple)) or any(
58+
not isinstance(pattern, str) or not pattern.strip() for pattern in raw
59+
):
60+
raise ValueError(f"model filters {key} must be a list of nonblank strings")
61+
return tuple(raw)
62+
63+
def flag(key: str) -> bool:
64+
raw = value.get(key, False)
65+
if not isinstance(raw, bool):
66+
raise TypeError(f"model filters {key} must be a boolean")
67+
return raw
68+
69+
return cls(
70+
patterns("include", ("*",)),
71+
patterns("exclude", ()),
72+
flag("hide_dated"),
73+
flag("latest_per_family"),
74+
)
75+
76+
def select(self, ids: list[str]) -> list[str]:
77+
selected = {
78+
model_id
79+
for model_id in ids
80+
if any(fnmatchcase(model_id.casefold(), pattern.casefold()) for pattern in self.include)
81+
and not any(
82+
fnmatchcase(model_id.casefold(), pattern.casefold()) for pattern in self.exclude
83+
)
84+
}
85+
if self.hide_dated:
86+
folded = {model_id.casefold() for model_id in selected}
87+
selected = {
88+
model_id
89+
for model_id in selected
90+
if not ((parts := _undated(model_id.casefold()))[1] and parts[0] in folded)
91+
}
92+
if self.latest_per_family:
93+
# Version first, alias before its SAME-version snapshot, newest date
94+
# for snapshot-only ties. No cross-family quality/cost ranking.
95+
best: dict[str, tuple[tuple[int, ...], bool, str]] = {}
96+
parsed = {model_id: _family_version(model_id) for model_id in selected}
97+
for parts in parsed.values():
98+
if parts is not None:
99+
family, version, date = parts
100+
rank = (version, not date, date)
101+
best[family] = max(best.get(family, rank), rank)
102+
selected = {
103+
model_id
104+
for model_id, parts in parsed.items()
105+
if parts is None or (parts[1], not parts[2], parts[2]) == best[parts[0]]
106+
}
107+
return sorted(selected)
108+
109+
def as_dict(self) -> dict[str, object]:
110+
return {
111+
"include": list(self.include),
112+
"exclude": list(self.exclude),
113+
"hide_dated": self.hide_dated,
114+
"latest_per_family": self.latest_per_family,
115+
}
116+
117+
118+
def selector_schema() -> dict[str, object]:
119+
"""Fresh metadata, not references to live policy."""
120+
return {
121+
"type": "object",
122+
"additionalProperties": False,
123+
"properties": {
124+
"include": {
125+
"type": "array",
126+
"items": {"type": "string", "minLength": 1},
127+
"default": ["*"],
128+
"description": "Case-insensitive OS-independent globs; [] includes none.",
129+
},
130+
"exclude": {
131+
"type": "array",
132+
"items": {"type": "string", "minLength": 1},
133+
"default": [],
134+
"description": "Case-insensitive OS-independent globs; exclusion wins.",
135+
},
136+
"hide_dated": {
137+
"type": "boolean",
138+
"default": False,
139+
"description": "Hide YYYY-MM-DD/YYYYMMDD snapshots if their alias is selected.",
140+
},
141+
"latest_per_family": {
142+
"type": "boolean",
143+
"default": False,
144+
"description": (
145+
"Latest numeric version per recognized vendor/family/tier/route; alias before "
146+
"matching snapshot, newest snapshot otherwise. Unknown names remain visible."
147+
),
148+
},
149+
},
150+
}

‎amplifier_module_provider_github_copilot/models.py‎

Lines changed: 32 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,9 @@ def _format_long_millions(tokens: int) -> str:
115115
return f"{text}M"
116116

117117

118-
def copilot_model_to_amplifier_model(model: CopilotModelInfo) -> AmplifierModelInfo:
118+
def copilot_model_to_amplifier_model(
119+
model: CopilotModelInfo, *, catalog: bool = False
120+
) -> AmplifierModelInfo:
119121
"""Translate CopilotModelInfo to amplifier_core.ModelInfo.
120122
121123
Contract: sdk-boundary:ModelDiscovery:MUST:3
@@ -143,7 +145,12 @@ def copilot_model_to_amplifier_model(model: CopilotModelInfo) -> AmplifierModelI
143145
amplifier_core.ModelInfo (what kernel expects from provider.list_models())
144146
"""
145147
# Build capabilities list
146-
capabilities: list[str] = ["streaming", "tools"] # All Copilot models support these
148+
# Legacy inference/helper annotations stay unchanged. Authentic catalog
149+
# metadata advertises only SDK-reported flags; the SDK has no per-model
150+
# streaming flag. Provider-level streaming support is a separate contract.
151+
capabilities: list[str] = [] if catalog else ["streaming", "tools"]
152+
if catalog and model.supports_tools:
153+
capabilities.append("tools")
147154

148155
if model.supports_vision:
149156
capabilities.append("vision")
@@ -168,6 +175,7 @@ def copilot_model_to_amplifier_model(model: CopilotModelInfo) -> AmplifierModelI
168175
if (
169176
model.context_window_long > model.context_window_default
170177
and model.context_window_default > 0
178+
and (not catalog or model.max_output_tokens > 0)
171179
):
172180
# long= advertises the long tier's TOTAL capacity (prompt budget +
173181
# max output), whereas the Context column below reports only the
@@ -265,3 +273,25 @@ async def fetch_and_map_models(
265273
copilot_models = await fetch_models(client)
266274
amplifier_models = [copilot_model_to_amplifier_model(m) for m in copilot_models]
267275
return amplifier_models, copilot_models
276+
277+
278+
async def discover_and_map_models(client: Any) -> list[AmplifierModelInfo]:
279+
"""Map only a fresh SDK response, preserving typed discovery failures.
280+
281+
Unlike the legacy fetch helpers, this path neither wraps typed authentication/
282+
network errors as generic unavailability nor allows cached inventory. Metadata
283+
policy defaults cannot invent limits on this catalog path.
284+
"""
285+
from .error_translation import LLMError, load_error_config, translate_sdk_error
286+
287+
try:
288+
sdk_models = await client.list_models(force_refresh=True)
289+
models = [sdk_model_to_copilot_model(model, policy_defaults=False) for model in sdk_models]
290+
for model in models:
291+
if not isinstance(model.id, str) or not model.id.strip():
292+
raise ValueError("Malformed SDK model inventory: expected non-empty model ID")
293+
return [copilot_model_to_amplifier_model(model, catalog=True) for model in models]
294+
except LLMError:
295+
raise
296+
except Exception as exc:
297+
raise translate_sdk_error(exc, load_error_config()) from exc

0 commit comments

Comments
 (0)