Skip to content

Commit bc532a2

Browse files
committed
Rewrite README; add cold-start benchmark
1 parent 813e91c commit bc532a2

2 files changed

Lines changed: 244 additions & 84 deletions

File tree

‎README.md‎

Lines changed: 132 additions & 84 deletions
Original file line numberDiff line numberDiff line change
@@ -1,124 +1,172 @@
1-
# lm15 (Universal LM) — thin core, full plugin contract
1+
# lm15
22

33
[![PyPI version](https://img.shields.io/pypi/v/lm15.svg)](https://pypi.org/project/lm15/)
4-
[![Python versions](https://img.shields.io/pypi/pyversions/lm15.svg)](https://pypi.org/project/lm15/)
5-
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
4+
[![Python 3.10+](https://img.shields.io/pypi/pyversions/lm15.svg)](https://pypi.org/project/lm15/)
5+
[![MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
66

7-
A universal LM core optimized for low import/runtime overhead with provider plugins.
7+
One interface for OpenAI, Anthropic, and Gemini. Zero dependencies.
88

9-
## Core architecture
9+
| | lm15 | google-genai | litellm |
10+
|---|---:|---:|---:|
11+
| **install** | 72ms | 137ms | 184ms |
12+
| **import** | 95ms | 2,656ms | 4,534ms |
13+
| **total (install → response)** | **1,090ms** | **3,992ms** | **5,840ms** |
14+
| dependencies | 0 | 25 | 55 |
15+
| disk footprint | 408K | 41M | 155M |
1016

11-
- **Universal contract** (`types.py`): normalized request/response/stream/live types.
12-
- **Provider plugin contract** (`providers/base.py`): complete/stream/live/embeddings/files/batch/images/audio methods.
13-
- **Transport boundary** (`transports/*`): urllib and pycurl implementations.
14-
- **Capability resolver** (`capabilities.py`): static + optional hydration from models.dev.
15-
- **Model catalog bridge** (`model_catalog.py`): loads `https://models.dev/api.json`.
16-
- **Middleware pipeline** (`middleware.py`): retries/history/cache wrappers.
17-
- **Completeness harness** (`completeness/*`): fixture + live probes, score output.
17+
<sub>Median of 10 cold-start runs. Fresh venv, single completion against `gemini-3.1-flash-lite-preview`. [Benchmark source.](benchmarks/cold_start.sh)</sub>
1818

19-
## Support matrix (current adapters)
19+
```python
20+
from lm15 import LMRequest, Message, Part, build_default
21+
22+
lm = build_default()
23+
resp = lm.complete(LMRequest(
24+
model="claude-sonnet-4-5",
25+
messages=(Message(role="user", parts=(Part.text_part("Hello."),)),),
26+
))
27+
print(resp.message.parts[0].text)
28+
```
29+
30+
Switch models by changing the string. Same types, same streaming protocol, same error hierarchy. That's it.
31+
32+
> Yes, [we know](https://xkcd.com/927/).
2033
21-
| Provider | complete | stream | embeddings | files | batches | images | audio | live |
22-
|---|---:|---:|---:|---:|---:|---:|---:|---:|
23-
| OpenAI | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
24-
| Anthropic | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
25-
| Gemini | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
34+
## Why this exists
35+
36+
Every LLM wrapper either (a) ships a massive SDK per provider or (b) papers over differences with a lossy abstraction. lm15 takes a different cut:
37+
38+
- **Stdlib only.** No `requests`, no `httpx`, no `aiohttp`. Transport is `urllib` or optional `pycurl`.
39+
- **Frozen dataclasses all the way down.** `LMRequest` in, `LMResponse` out. No mutable builder chains, no hidden state.
40+
- **Nothing is hidden.** Provider-specific options pass through `Config.provider`. The normalized types cover the common surface; the escape hatch is always there.
41+
- **Plugin discovery via entry points.** Third-party providers install and register without touching lm15 core.
42+
43+
## Install
44+
45+
```bash
46+
pip install lm15 # stdlib transport (urllib)
47+
pip install lm15[speed] # + pycurl transport
48+
```
49+
50+
Set at least one provider key:
51+
52+
```bash
53+
export OPENAI_API_KEY=sk-...
54+
export ANTHROPIC_API_KEY=sk-ant-...
55+
export GEMINI_API_KEY=... # or GOOGLE_API_KEY
56+
```
57+
58+
## Usage
59+
60+
### Streaming
61+
62+
```python
63+
req = LMRequest(model="gpt-4.1-mini", messages=(...))
64+
for event in lm.stream(req):
65+
if event.type == "delta" and event.delta and event.delta.get("type") == "text":
66+
print(event.delta["text"], end="")
67+
```
2668

27-
## Quick start
69+
### Tool calling
2870

2971
```python
30-
from lm15 import Message, LMRequest, Part, build_default
72+
from lm15 import Tool, ToolConfig
3173

32-
lm = build_default(use_pycurl=True)
33-
req = LMRequest(model="claude-sonnet-4-5", messages=(Message(role="user", parts=(Part.text_part("Say hi."),)),))
74+
tools = (Tool(name="get_weather", description="...", parameters={...}),)
75+
req = LMRequest(model="gemini-2.5-flash", messages=(...), tools=tools)
3476
resp = lm.complete(req)
35-
print(resp.message.parts[0].text)
77+
# resp.message.parts may contain Part(type="tool_call", ...)
78+
```
79+
80+
### Embeddings, images, audio, files, batches
81+
82+
```python
83+
lm.embeddings(EmbeddingRequest(model="text-embedding-3-small", input=["hello"]))
84+
lm.image_generate(ImageGenerationRequest(model="gpt-image-1", prompt="a cat"))
85+
lm.audio_generate(AudioGenerationRequest(model="gpt-4o-mini-tts", input="hi", voice="alloy"))
86+
lm.file_upload(FileUploadRequest(...), provider="openai")
87+
lm.batch_submit(BatchRequest(...))
3688
```
3789

38-
## External plugin adapters (no core PR)
90+
### Middleware
3991

40-
LM15 auto-discovers installed entry-points in group `lm15.providers`.
92+
```python
93+
from lm15 import with_retries, with_cache, with_history
94+
95+
lm.middleware.add(with_retries(max_retries=3))
96+
lm.middleware.add(with_cache())
97+
lm.middleware.add(with_history())
98+
```
99+
100+
### External plugins
101+
102+
Any installed package can register a provider adapter:
41103

42104
```toml
43-
# in external package pyproject.toml
105+
# pyproject.toml of the plugin package
44106
[project.entry-points."lm15.providers"]
45107
myprovider = "lm15_x_myprovider:build_adapter"
46108
```
47109

48110
```python
49-
from lm15 import build_default
50-
51111
lm = build_default(discover_plugins=True)
52112
```
53113

54-
## Optional models.dev hydration
114+
### models.dev catalog
115+
116+
Hydrate model capabilities from the [models.dev](https://models.dev) catalog at startup:
55117

56118
```python
57119
lm = build_default(hydrate_models_dev_catalog=True)
58120
```
59121

60-
## Completeness
61-
62-
```bash
63-
python3 completeness/runner.py --mode fixture --fail-under 1.0
64-
python3 completeness/runner.py --mode live --fail-under 0.0
65-
```
66-
67-
Outputs:
68-
- `completeness/report.json`
69-
- `completeness/report.md`
122+
## Provider support
70123

71-
## Packaging and publishing (uv + twine)
124+
| Capability | OpenAI | Anthropic | Gemini |
125+
|---|:---:|:---:|:---:|
126+
| complete | ✅ | ✅ | ✅ |
127+
| stream | ✅ | ✅ | ✅ |
128+
| embeddings | ✅ | — | ✅ |
129+
| files | ✅ | ✅ | ✅ |
130+
| batches | ✅ | ✅ | ✅ |
131+
| images | ✅ | — | ✅ |
132+
| audio | ✅ | — | ✅ |
133+
| live | — | — | — |
72134

73-
```bash
74-
# build sdist + wheel
75-
uv run python -m build
76-
77-
# upload to TestPyPI
78-
# twine upload --repository testpypi dist/*
135+
## Architecture
79136

80-
# upload to PyPI
81-
# twine upload dist/*
137+
```
138+
LMRequest ──▶ UniversalLM ──▶ MiddlewarePipeline ──▶ ProviderAdapter ──▶ Transport
139+
│ │
140+
│ resolve_provider(model) │ build_request / parse_response
141+
▼ ▼
142+
capabilities.py providers/{openai,anthropic,gemini}.py
82143
```
83144

84-
## Known limitations
145+
- **`types.py`** — all request/response/stream types as frozen dataclasses
146+
- **`protocols.py`** — `LMAdapter` and `LiveSession` protocols
147+
- **`providers/base.py`** — `BaseProviderAdapter` with default `complete`/`stream` implementations over raw HTTP
148+
- **`transports/`** — `urllib` and `pycurl` backends behind a `Transport` protocol
149+
- **`capabilities.py`** — model→provider resolution + optional models.dev hydration
150+
- **`middleware.py`** — composable retry / cache / history wrappers
151+
- **`plugins.py`** — entry-point discovery for third-party adapters
85152

86-
- Realtime/live protocol parity is not fully implemented across all providers.
87-
- Completeness score is tied to the current fixture matrix; it is not a guarantee of full vendor API parity.
88-
- Provider-specific advanced options are partially passthrough (`config.provider`) and not all are normalized.
153+
## Docs
89154

90-
## Environment variables
155+
| Topic | Path |
156+
|---|---|
157+
| Getting started | [`docs/GETTING_STARTED.md`](docs/GETTING_STARTED.md) |
158+
| Core concepts | [`docs/CONCEPTS.md`](docs/CONCEPTS.md) |
159+
| Architecture | [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) |
160+
| Provider contract | [`docs/CONTRACT.md`](docs/CONTRACT.md) |
161+
| Error handling | [`docs/ERRORS.md`](docs/ERRORS.md) |
162+
| Streaming | [`docs/STREAMING.md`](docs/STREAMING.md) |
163+
| Writing an adapter | [`docs/ADAPTER_GUIDE.md`](docs/ADAPTER_GUIDE.md) |
164+
| Adding a provider | [`docs/ADD_PROVIDER_GUIDE.md`](docs/ADD_PROVIDER_GUIDE.md) |
165+
| Completeness testing | [`docs/COMPLETENESS.md`](docs/COMPLETENESS.md) |
166+
| Production checklist | [`docs/PRODUCTION_CHECKLIST.md`](docs/PRODUCTION_CHECKLIST.md) |
91167

92-
- `OPENAI_API_KEY`
93-
- `ANTHROPIC_API_KEY`
94-
- `GEMINI_API_KEY` or `GOOGLE_API_KEY`
168+
**Cookbooks:** [`docs/COOKBOOKS/`](docs/COOKBOOKS/) — 8 progressive examples from basic text to plugin development.
95169

96-
## Docs
170+
## License
97171

98-
### Core
99-
100-
- `docs/GETTING_STARTED.md`
101-
- `docs/CONCEPTS.md`
102-
- `docs/ARCHITECTURE.md`
103-
- `docs/CONTRACT.md`
104-
- `docs/ERRORS.md`
105-
- `docs/STREAMING.md`
106-
- `docs/COMPLETENESS.md`
107-
- `docs/PRODUCTION_CHECKLIST.md`
108-
109-
### Provider development
110-
111-
- `docs/ADAPTER_GUIDE.md`
112-
- `docs/ADD_PROVIDER_GUIDE.md`
113-
- `docs/COOKBOOK_TEMPLATE.md`
114-
115-
### Cookbooks (learning order)
116-
117-
- `docs/COOKBOOKS/01-basic-text.md`
118-
- `docs/COOKBOOKS/02-streaming.md`
119-
- `docs/COOKBOOKS/03-tools.md`
120-
- `docs/COOKBOOKS/04-multimodal.md`
121-
- `docs/COOKBOOKS/05-files-batches.md`
122-
- `docs/COOKBOOKS/06-reliability.md`
123-
- `docs/COOKBOOKS/07-external-plugins.md`
124-
- `docs/COOKBOOKS/08-models-dev-hydration.md`
172+
MIT

‎benchmarks/cold_start.sh‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
#!/bin/bash
2+
# Cold-start benchmark: fresh venv → install → import → client → API call.
3+
# Measures end-to-end latency for a single completion against Gemini.
4+
#
5+
# Requirements: uv, GEMINI_API_KEY set.
6+
# Usage: bash benchmarks/cold_start.sh [RUNS]
7+
set -e
8+
9+
RUNS=${1:-10}
10+
MODEL="gemini-3.1-flash-lite-preview"
11+
PROMPT="Reply with exactly: ok"
12+
WORKDIR=$(mktemp -d)
13+
CSVFILE="benchmarks/cold_start_results.csv"
14+
15+
trap "rm -rf $WORKDIR" EXIT
16+
17+
bench_one() {
18+
local label=$1 pkg=$2 script=$3
19+
20+
for i in $(seq 1 $RUNS); do
21+
rm -rf "$WORKDIR/venv"
22+
23+
t_start=$(python3 -c "import time; print(time.perf_counter())")
24+
25+
uv venv "$WORKDIR/venv" -q 2>/dev/null
26+
uv pip install -q $pkg --python "$WORKDIR/venv/bin/python" 2>/dev/null
27+
28+
t_installed=$(python3 -c "import time; print(time.perf_counter())")
29+
30+
result=$("$WORKDIR/venv/bin/python" -c "$script" 2>&1)
31+
32+
install_ms=$(python3 -c "print(f'{1000*($t_installed - $t_start):.0f}')")
33+
rest_ms=$(echo "$result" | grep TIMINGS | sed 's/.*import=\([0-9]*\)ms client=\([0-9]*\)ms call=\([0-9]*\)ms total=\([0-9]*\)ms/\1 \2 \3 \4/')
34+
import_ms=$(echo $rest_ms | cut -d' ' -f1)
35+
client_ms=$(echo $rest_ms | cut -d' ' -f2)
36+
call_ms=$(echo $rest_ms | cut -d' ' -f3)
37+
code_total=$(echo $rest_ms | cut -d' ' -f4)
38+
39+
echo "$label,$i,$install_ms,$import_ms,$client_ms,$call_ms,$code_total"
40+
done
41+
}
42+
43+
LM15_SCRIPT="
44+
import time, sys, os
45+
t0 = time.perf_counter()
46+
from lm15 import LMRequest, Message, Part, build_default
47+
t_import = time.perf_counter()
48+
lm = build_default(use_pycurl=False)
49+
t_client = time.perf_counter()
50+
req = LMRequest(model='$MODEL', messages=(Message(role='user', parts=(Part.text_part('$PROMPT'),)),))
51+
resp = lm.complete(req)
52+
t_call = time.perf_counter()
53+
print(f'TIMINGS import={1000*(t_import-t0):.0f}ms client={1000*(t_client-t_import):.0f}ms call={1000*(t_call-t_client):.0f}ms total={1000*(t_call-t0):.0f}ms', file=sys.stderr)
54+
"
55+
56+
LITELLM_SCRIPT="
57+
import time, sys, os
58+
t0 = time.perf_counter()
59+
import litellm
60+
t_import = time.perf_counter()
61+
t_client = time.perf_counter()
62+
resp = litellm.completion(model='gemini/$MODEL', messages=[{'role': 'user', 'content': '$PROMPT'}])
63+
t_call = time.perf_counter()
64+
print(f'TIMINGS import={1000*(t_import-t0):.0f}ms client={1000*(t_client-t_import):.0f}ms call={1000*(t_call-t_client):.0f}ms total={1000*(t_call-t0):.0f}ms', file=sys.stderr)
65+
"
66+
67+
GENAI_SCRIPT="
68+
import time, sys, os
69+
t0 = time.perf_counter()
70+
from google import genai
71+
t_import = time.perf_counter()
72+
client = genai.Client(api_key=os.environ['GEMINI_API_KEY'])
73+
t_client = time.perf_counter()
74+
resp = client.models.generate_content(model='$MODEL', contents='$PROMPT')
75+
t_call = time.perf_counter()
76+
print(f'TIMINGS import={1000*(t_import-t0):.0f}ms client={1000*(t_client-t_import):.0f}ms call={1000*(t_call-t_client):.0f}ms total={1000*(t_call-t0):.0f}ms', file=sys.stderr)
77+
"
78+
79+
echo "library,run,install_ms,import_ms,client_ms,call_ms,code_total_ms" | tee "$CSVFILE"
80+
81+
bench_one "lm15" "lm15" "$LM15_SCRIPT" | tee -a "$CSVFILE"
82+
bench_one "litellm" "litellm" "$LITELLM_SCRIPT" | tee -a "$CSVFILE"
83+
bench_one "google-genai" "google-genai" "$GENAI_SCRIPT" | tee -a "$CSVFILE"
84+
85+
echo ""
86+
echo "=== Medians ==="
87+
python3 -c "
88+
import csv
89+
from collections import defaultdict
90+
91+
rows = defaultdict(list)
92+
with open('$CSVFILE') as f:
93+
for r in csv.DictReader(f):
94+
lib = r['library']
95+
rows[lib].append({k: int(v) for k, v in r.items() if k not in ('library', 'run')})
96+
97+
def median(vals):
98+
s = sorted(vals)
99+
n = len(s)
100+
return s[n//2] if n % 2 else (s[n//2-1] + s[n//2]) // 2
101+
102+
print(f'| {\"\":15s} | {\"install\":>9s} | {\"import\":>9s} | {\"client\":>9s} | {\"call\":>9s} | {\"total\":>9s} |')
103+
print(f'|{\"\":-<17s}|{\"\":-<11s}|{\"\":-<11s}|{\"\":-<11s}|{\"\":-<11s}|{\"\":-<11s}|')
104+
for lib in ['lm15', 'google-genai', 'litellm']:
105+
data = rows[lib]
106+
install = median([d['install_ms'] for d in data])
107+
imp = median([d['import_ms'] for d in data])
108+
cli = median([d['client_ms'] for d in data])
109+
call = median([d['call_ms'] for d in data])
110+
total = median([d['code_total_ms'] for d in data])
111+
print(f'| {lib:15s} | {install:7d}ms | {imp:7d}ms | {cli:7d}ms | {call:7d}ms | {total:7d}ms |')
112+
"

0 commit comments

Comments
 (0)