When an alert fires, the fastest path to a fix is usually a memory: we have seen this before. That memory lives in post-mortems nobody rereads, Slack threads nobody can find, and PagerDuty logs nobody exports.
ops-recall ingests all of it and, when a new alert fires, reconstructs the incident: which past incidents this resembles, what the root cause turned out to be, what the team actually did about it, which of those conditions are true again right now, and the single best next action.
The same run as text
$ ops-recall demo
This alert looks 75% similar to INC-2197: checkout-api connection pool exhausted while an ad-hoc VACUUM FULL held locks on
orders_db.
Similar incidents
INC-2197 75% 3 months ago checkout-api connection pool exhausted while an ad-hoc VACUUM FULL hel
semantic 0.61 | lexical 1.00 | exact-keyword 1.00 | recency x0.93
INC-1842 39% 22 months ago checkout-api pool exhaustion during a non-concurrent index rebuild on
semantic 0.48 | lexical 0.60 | exact-keyword 0.60 | recency x0.71
Live telemetry
CONFIRMED A single session holds a lock on orders_db for minutes while dozens of queries queue behind it.
1 blocking session(s) on orders_db; longest is pid 51774 at 383s running 'VACUUM FULL orders', 44 waiters.
CONFIRMED checkout-api's HikariCP pool is pinned at max with a growing queue of pending connection requests.
checkout-api pool at 50/50 (100%) with 118 request(s) queued.
CONFIRMED No checkout-api deploy preceded the incident - the trigger came from the database side, not the application.
No deploys of checkout-api in the last 6h.
CONFIRMED checkout-api 5xx rate climbs above 20% within four minutes of the first pool timeout.
checkout-api error rate 37.0% (threshold 5.0%).
[...]
Recommended next action
1. Terminate the blocking backend.
$ SELECT pg_terminate_backend(48211)
This is what resolved INC-2197 (pool drained back to 6/50 active within 40 seconds, 5xx rate back to baseline by 14:36)
risk high | needs human approval
2. Re-run the space reclaim as VACUUM (ANALYZE) plus a scheduled pg_repack inside the Sunday window.
$ VACUUM (ANALYZE)
This is what resolved INC-2197
risk low | needs human approval
[...]
confidence 0.75 | reasoner template | retrieve 10.6ms -> check_evidence 0.2ms -> gather_quotes 3.3ms -> reconstruct 0.1ms
The [...] marks what the image leaves out: ops-recall demo also prints the root
cause hypothesis, the full reconstruction and the caveats. Both the image and this
text are generated from one real run by tools/screenshot.py, so neither can show
something the CLI does not print.
The last two sections are the point. Retrieval alone produces a plausible story; re-testing that story's own conditions against the live system is what makes it worth acting on — and what lets the system say "this looks like INC-2197 but nothing is actually blocking, so the resemblance is superficial" instead of confidently recommending the wrong fix.
python -m venv .venv && .venv/bin/pip install -e ".[dev]"
.venv/bin/ops-recall demo # the flagship reconstruction
.venv/bin/ops-recall search "consumer lag climbing"
.venv/bin/ops-recall probe check_db_locks database=orders_db
.venv/bin/ops-recall eval # score retrieval against the labeled set
.venv/bin/ops-recall serve # HTTP API on :8000
.venv/bin/python -m pytest # 121 tests, no networkOr in a container, with a real Qdrant behind it:
docker compose up --build # API on :8000, Qdrant on :6333
docker compose run --rm ops-recall demoEverything runs offline out of the box: the index is embedded Qdrant, the
embedder is deterministic, telemetry comes from a JSON snapshot, and the
reconstruction falls back to a deterministic assembler. Set
ANTHROPIC_API_KEY (or run ant auth login) and the narrative, the reasoning
about competing hypotheses, and the agentic follow-up checks switch on.
post-mortems ─┐
Slack ├─► ingest ─► correlate ─► index (Qdrant: dense + sparse)
PagerDuty │ │
wiki ─────────┘ │
▼
alert ───────────────────────────────► retrieve ──► no close match ──► "this is novel"
│
▼
check_evidence replay each retrieved incident's
│ probes against live telemetry
▼
gather_quotes what people actually said
│
▼
investigate Claude runs extra read-only checks
│
▼
reconstruct Claude writes the briefing
The pipeline is a LangGraph state machine (ops_recall/agent/graph.py).
Retrieval and evidence checking are deterministic and always run; the model is
reached last, with the facts already assembled. Every hop records timings and
counts in state["trace"], which is returned on the API so a surprising answer
can be traced to the step that produced it.
An alert contains two very different kinds of information, and one retrieval
strategy cannot serve both. ERR_5012 and db-prod-03 must match exactly —
embeddings put db-prod-03 and db-prod-07 in nearly the same place, which is
precisely wrong. "Requests are queuing and timing out" must match semantically
against "the pool never recovered on its own".
So three lanes, fused with renormalizing weights:
| Lane | What it is | Weight |
|---|---|---|
semantic |
Cosine over dense vectors | 0.5 |
lexical |
BM25 sparse vectors, native Qdrant sparse index | 0.3 |
keyword |
Fraction of the alert's extracted identifiers found in the incident | 0.2 |
similarity = (w_sem·semantic + w_lex·lexical + w_kw·keyword) / Σw × recency
Details that matter:
- The keyword lane drops out cleanly. An alert with no error codes in it has nothing for that lane to match, so its weight is redistributed rather than capping every result at 0.8.
- Both lanes are scored for every candidate. A lane only returns scores for the points it retrieved, so a result found lexically would otherwise have no semantic score. The candidate set is re-scored directly against both stored vectors, which makes every breakdown complete instead of lane-dependent.
- Each lane indexes different text. The dense vector is built only from title, symptoms, keywords and services — an alert can only describe how something presented, so the semantic lane matches symptom against symptom. Including remediation prose pulls the vector toward how the incident was fixed, which is the part a firing alert cannot resemble. The sparse vector indexes everything, including the commands inside remediation steps, so an exact token is findable wherever it appears.
- idf is per collection. Incidents and fragments have separate BM25 statistics; sharing one table let PagerDuty boilerplate distort the incident lane.
- The lexical score is absolute, not relative. It is normalized against the idf mass the query could achieve in this corpus, rather than against the best score in the result set — otherwise the top hit always looks perfect, even when the whole result set is poor.
Metadata filters (services, max_age_days, min_severity, exclude_ids) are
applied inside Qdrant, before scoring.
recency = floor + (1 - floor) · 0.5 ^ (age_days / half_life_days)
Defaults: half_life_days=365, recency_floor=0.6. Decay is a multiplier on
the fused score, never a term added to it, so it can only break ties between
comparable matches — it can never manufacture similarity that is not there. The
floor is deliberate: a three-year-old exact match is stale, not irrelevant, and
burying it is a worse failure than surfacing it with a caveat. In the demo above
it is what puts the 3-month-old INC-2197 (×0.94) decisively ahead of the
21-month-old INC-1842 (×0.71).
Every design choice above is an argument. evals/queries.yaml is the place they
get checked: 29 labeled alerts written the way a monitoring system emits them,
each paired with the incident a responder would want surfaced.
$ ops-recall eval
Retrieval quality
hit@1 0.92 hit@3 1.00 relevant@1 0.96 MRR 0.96
abstention 1.00 on novel alerts composite score 0.966 (29 cases)
By category
ambiguous 3/3 passed score 0.833
exact 13/13 passed score 1.000
novel 5/5 passed score 1.000
paraphrase 8/8 passed score 0.938
Two things make this more than a hit-rate number:
- Five cases have no correct answer. They are alerts about systems the
archive has never seen, and the right behavior is to return nothing above
min_similarity. They are scored asabstentionand folded into the composite, so a change that buys hit rate by lowering the threshold loses points. Optimizing retrieval without them produces a system that always has an answer, which is the failure this product exists to avoid. - Categories are scored separately.
exactalerts quote error codes and hostnames;paraphrasealerts describe the same failure in prose no post-mortem used;ambiguousalerts sit between two incidents in the same family. The aggregate hides which lane is doing the work.
hit@1 is strict — it wants the primary target first, not merely a member of
the right family. The gap to relevant@1 (0.92 vs 0.96) is exactly one case:
an alert mentioning CREATE INDEX should rank the migration incident above the
newer VACUUM one, and recency decay outvotes the evidence. That is the recency
tradeoff the section above describes, showing up as a number instead of a
worry.
The suite pins its own clock (as_of), because recency decay is a factor in
every score and an unpinned suite would drift downward as the corpus ages —
failing CI one day on a repository nobody had touched.
ops-recall eval --sweep grid-searches the lane weights over the simplex and
sorts by composite score. Running it is how the claims in this README stop
being claims:
| Configuration | Score |
|---|---|
| Best on the grid (5-way tie) | 0.983 |
| Configured 0.5 / 0.3 / 0.2 | 0.966 |
| Best without the lexical lane | 0.948 |
| Best without the semantic lane | 0.966 |
| Best without the keyword lane | 0.983 |
| Keyword lane alone | 0.874 |
| Lexical lane alone | 0.810 |
| Semantic lane alone | 0.776 |
The fusion argument survives: every lane alone is far worse than any blend of
them, which is the whole case for hybrid retrieval and is asserted as a test
(test_fused_ranking_beats_every_single_lane).
The keyword lane does not survive. Dropping it entirely scores 0.983 — at
the top of the grid, better than the shipped configuration. On this corpus the
BM25 lane already matches db-prod-01 exactly and gives it a high idf, so the
third lane is largely re-deriving what the second one found. The argument for
keeping it is about a corpus this one is too small to exhibit: with hundreds of
incidents, idf falls for hostnames that appear everywhere, and matching against
curated frontmatter keywords rather than full text stops being redundant. That
is a hypothesis, and it is now labeled as one rather than presented as a
finding.
The shipped weights are deliberately not set to the grid winner. The gap is
0.017, which on 24 ranked cases is a single case changing rank — fitting three
parameters to that is overfitting, and a portfolio number bought by tuning on
the test set is worth nothing. What the sweep is for is catching a weight edit
that falls off the plateau, which test_configured_weights_are_near_the_sweep_optimum
does.
CI gates on ops-recall eval --fail-under 0.90. A change to the tokenizer, the
embedder or the weights has to face the labeled set before it can merge.
min_similarity decides every abstention, and abstention is a third of what
this system promises. It is a property of the vector geometry, not a
constant, which is why it lives on the embedder rather than in the config
defaults.
Feature hashing puts unrelated text at ~0.0, so a threshold of 0.30 separates cleanly. A trained model puts all English at 0.3–0.4 before it has said anything: carry 0.30 across the swap and abstention collapses from 1.00 to 0.00, every novel alert drawing a confident-looking 0.36 match. The ranked metrics do not move at all while this happens, so nothing but the novel cases would have caught it — which is the argument for scoring abstention in the first place.
Measured on the labeled set under bge-small-en-v1.5:
| lowest true positive | highest false positive | gap | |
|---|---|---|---|
| hashing | 0.402 | < 0.30 | ≥ 0.10 |
| fastembed | 0.551 | 0.405 | 0.146 |
The trained model separates better — a wider margin — it just does so
somewhere else on the number line. Any threshold in 0.42–0.52 scores an
identical 0.966; the shipped value is 0.50, a round number inside that plateau
placed above center because the failure below it is a cliff and the failure
above it is a slope. OPS_RECALL_MIN_SIMILARITY overrides it; leaving it unset
is what makes a swap safe.
This is the central mechanism. A post-mortem declares the conditions that held during the incident, each bound to a telemetry probe:
signals:
- id: blocking-lock-orders-db
statement: A single session holds a lock on orders_db for minutes while dozens of queries queue behind it.
probe: check_db_locks
probe_args: {database: orders_db, min_seconds: 60}At alert time every retrieved incident's signals are collected, deduplicated
(incidents in the same family share signals; a condition seen in three incidents
is stronger evidence, not three checks), retargeted at the service the
current alert names (a signal recorded as "checkout-api's pool is saturated"
generalizes to "the affected service's pool is saturated"), and run. Each comes
back CONFIRMED, REFUTED, or UNKNOWN — never guessed. REFUTED is often
the most informative label on the page, because it rules a story out.
Try it: empty blocking_sessions in data/telemetry/current_state.json and run
ops-recall demo again. Same alert, same archive, opposite conclusion.
A probe is a question, not a data source. TelemetryProvider dispatches probe
names to probe_* methods and owns the argument validation; what answers them
is configuration. OPS_RECALL_TELEMETRY names the backends in precedence
order:
OPS_RECALL_TELEMETRY=prometheus+postgres+tls+fixture| Backend | Answers | Needs |
|---|---|---|
prometheus |
error rate, connection pool, service metrics, consumer lag, disk usage | PROMETHEUS_URL |
postgres |
blocking locks, replication slots | POSTGRES_DSNS |
tls |
certificate expiry | nothing |
fixture |
all nine, out of a JSON snapshot | the bundled file |
null |
nothing; every check comes back unknown | nothing |
No single system answers all nine, so a real deployment is always a
composition. First match wins, which is what the ordering is for: a
fixture at the end is the fallback for probes no live backend implements, and
the same fixture at the front would shadow every one of them.
A failing backend is not retried against the next. A probe that reports unknown because Prometheus is down has said something true, and substituting a fixture's answer for it would turn an outage in the monitoring stack into confident fiction in an incident briefing.
A backend that is named but not configured is a startup error, not a
warning. The failure that avoids is a service reporting telemetry=fixture in
/v1/stats while its operator believes it is reading production, and answering
every probe from a JSON file for a week.
pip install 'ops-recall[telemetry]' # or [prometheus] / [postgres]Prometheus. One PromQL instant query per probe. Metric names are not
portable — one team's http_requests_total is another's request_count — so
every query is a template and OPS_RECALL_PROMETHEUS_QUERIES points at a JSON
file that overrides any subset of them:
{
"error_rate": "sum(rate(requests_total{service=\"$service\",status=~\"5..\"}[5m])) / sum(rate(requests_total{service=\"$service\"}[5m]))",
"metric.goroutines": "sum(go_goroutines{job=\"$service\"})"
}Every probe argument is offered twice: $service for an = matcher and
$service_re, regex-quoted, for an =~ one — an unescaped db-01.eu.corp in
a regex matcher also matches hosts it is not. The defaults target the exporters
this corpus implies: a JVM service behind HikariCP, node_exporter,
kafka_exporter. An absent series is reported as
unknown, not as zero — a service nobody scrapes is not a healthy service, and
a 0% error rate for it would refute a signal that was never actually tested.
check_recent_deploys is deliberately absent from this backend. Deploy
history is not a metric in most installations, and a default query for it would
answer the most consequential question in the set with fiction. Compose
something that can answer it, or let it come back unknown.
PostgreSQL. The two probes that no metrics system answers well, because the
answer is a row in a catalog rather than a time series: which session is
blocking others and what it is running (pg_stat_activity joined through
pg_blocking_pids()), and how much WAL a replication slot is retaining
(pg_replication_slots). Read-only access is enough. OPS_RECALL_POSTGRES_DSNS
maps the database name the corpus uses to a libpq DSN, because the name a team
calls a database is rarely its connection string. Connections are opened per
probe and closed after — an incident tool that holds idle connections to a
database it is diagnosing for connection exhaustion has misunderstood its job.
Both statements run against a real PostgreSQL in CI, in a job that fails rather
than skips if the driver is missing. Catalog SQL is exactly the kind of code
that reads correctly and is not: the first version of the statement timeout
here used a bind parameter, which SET does not accept, and nothing short of
executing it would have said so.
TLS. The one probe that needs no infrastructure at all: connect, read the peer certificate, subtract. The interesting case is that an already expired certificate fails verification, so the handshake never completes and there is no date to read — at exactly the moment the answer is most urgent. That is reported as the probe holding, because "it expired" is what was being asked; other verification failures (wrong hostname, unknown issuer) stay unknown, since they say nothing about expiry either way.
ops_recall/agent/tools.py exposes the telemetry surface to Claude via the SDK
tool runner, plus search_incidents and get_runbook over the same corpus.
This covers the case the deterministic replay cannot: the alert resembles a lock
incident, the lock probe comes back refuted, and the useful next question — "is
the service leaking connections instead?" — is one no post-mortem wrote down.
Every tool is read-only. Nothing terminates a backend, rolls back a deploy or edits configuration. Remediation is proposed to a human with a risk label and an approval flag; that is a boundary, not an unimplemented feature.
The model receives a schema (ReconstructionDraft) covering the headline, root
cause hypothesis, narrative, recommended actions, confidence and caveats. It has
no schema slot for similarity scores, evidence statuses, incident ids or
timestamps — those are computed by the system and merged in afterwards, so a
retrieval score or a telemetry result cannot be invented. Structured output is
requested through client.messages.parse; the system prompt is invariant across
alerts and cached.
Drop files into data/seed/ (or point OPS_RECALL_DATA_DIR elsewhere) and run
ops-recall ingest.
postmortems/*.md— YAML frontmatter (id,title,severity,services,started_at,resolved_at,keywords,signals,pagerduty_incident,slack_thread) plus## Summary,## Symptoms,## Root cause,## Remediationsections. Remediation steps are numbered and tagged[diagnose]/[mitigate]/[fix], with an optional trailing`command`,— @actor, and(outcome: ...). The[fix]tag is what separates "what finally worked" from "what we tried first".slack/*.json—conversations.historyexports.pagerduty/*.json— incidents with their log entries.wiki/*.md— runbooks, indexed per##section.
Correlation is by explicit reference: the post-mortem's frontmatter declares its
PagerDuty incident and Slack thread, and those declarations let thousands of
unlabeled messages attach themselves to the right incident. Messages that name
an incident (INC-2197) link themselves. Nothing is linked by timestamp
proximity — ops channels run several conversations at once.
For continuous ingest, set OPS_RECALL_SLACK_BOT_TOKEN + OPS_RECALL_SLACK_CHANNELS
and OPS_RECALL_PAGERDUTY_API_KEY, then ops-recall ingest --live. The live
clients emit the same Fragment objects as the exports, deduplicated by source
id.
| Method | Path | Purpose | Auth |
|---|---|---|---|
POST |
/v1/alerts |
Reconstruct an alert. The main endpoint. | read |
POST |
/v1/alerts/pagerduty |
Same, behind a PagerDuty v3 webhook adapter. | signature |
POST |
/v1/search |
Rank incidents against free text, with score breakdowns. | read |
GET |
/v1/incidents, /v1/incidents/{id} |
Browse the archive. | read |
GET |
/v1/probes |
What the agent can ask the live system. | read |
POST |
/v1/reindex |
Rebuild the index from the configured sources. | admin |
GET |
/v1/stats |
Operational. | read |
GET |
/healthz |
Liveness. | none |
Set OPS_RECALL_API_KEYS and every endpoint above except /healthz wants
Authorization: Bearer <key>. OPS_RECALL_ADMIN_API_KEYS is a second tier that
/v1/reindex requires; admin keys are valid as read keys too, and with no admin
key configured there is one tier rather than an unreachable endpoint. A valid
key at the wrong tier is a 403, not a 401 — the credential is fine, retrying
with it will never help.
OPS_RECALL_API_KEYS='["s3cret"]' .venv/bin/ops-recall serve
curl -H 'Authorization: Bearer s3cret' localhost:8000/v1/incidentsWith no keys set the API is open, warns at startup, and reports
auth.enabled: false in /v1/stats — the same shape as a missing
ANTHROPIC_API_KEY degrading the reasoner instead of failing every request. It
keeps docker run ops-recall serve and the offline demo working unconfigured,
and it is not a deployment posture: /v1/search and /v1/incidents serve the
team's post-mortems, Slack quotes with author names, and hostnames, while
/v1/alerts spends model tokens per call.
PagerDuty cannot present a key you chose, so that route verifies the webhook
subscription's own secret instead: X-PagerDuty-Signature: v1=<hex>, an
HMAC-SHA256 over the raw request body, configured as
OPS_RECALL_PAGERDUTY_WEBHOOK_SECRETS. Several are accepted at once, so a
rotation does not need a deployment landing between the two keys. The body is
read as raw bytes because that is what the signature covers — re-serializing a
parsed payload would not reproduce them. A delivery with no signature falls back
to the bearer check, which is what replaying a captured payload by hand uses;
so configuring keys without a webhook secret locks PagerDuty out on purpose,
rather than leaving one unauthenticated write path.
Two things this is not. The signature covers the body and nothing else — there
is no timestamp in PagerDuty's scheme, so a captured delivery stays replayable
and the secret is the whole of the protection. And bearer tokens over plain HTTP
assume TLS terminates in front of the service. Authentication also does not
bound cost: an authenticated caller can still spend tokens in a loop, so
per-key rate limiting on /v1/alerts is the next thing this wants.
All settings take an OPS_RECALL_ prefix and can live in .env (see
.env.example).
| Setting | Default | Notes |
|---|---|---|
REASONER |
claude |
template for the deterministic fallback |
MODEL |
claude-opus-5 |
|
AGENTIC_CHECKS |
true |
Let the model run its own follow-up probes |
EMBEDDER |
hashing |
fastembed for local real embeddings, voyage for hosted |
FASTEMBED_MODEL |
BAAI/bge-small-en-v1.5 |
Any fastembed model; BGE/E5/Nomic get their instruction prefixes |
FASTEMBED_CACHE_DIR |
unset | Where the ONNX weights live; unset uses fastembed's default |
VOYAGE_DIM |
unset | Only for a model this build has no width for |
QDRANT_URL / QDRANT_PATH |
unset | Unset means in-memory, rebuilt at startup |
HALF_LIFE_DAYS |
365 |
|
RECENCY_FLOOR |
0.6 |
|
MIN_SIMILARITY |
the embedder's floor | Below this the answer is "this is novel". Unset is correct: the threshold is geometry-bound (0.30 hashing, 0.50 trained) |
TOP_K |
3 |
|
WEIGHT_SEMANTIC / _LEXICAL / _KEYWORD |
0.5 / 0.3 / 0.2 |
|
API_KEYS |
unset | JSON list of bearer tokens. Unset means the API is open |
ADMIN_API_KEYS |
unset | Tier required by /v1/reindex; also valid as read keys |
PAGERDUTY_WEBHOOK_SECRETS |
unset | JSON list of webhook signing secrets |
TELEMETRY |
auto |
Backends joined by +; auto is the bundled snapshot |
PROMETHEUS_URL / _TOKEN / _QUERIES |
unset | Required by the prometheus backend |
POSTGRES_DSNS |
unset | JSON object of database name to libpq DSN |
TLS_PORT |
443 |
A port in the probe's host argument wins over it |
docker compose up --buildBrings up the service on :8000 against a real Qdrant on :6333 — the
deployment shape the default configuration does not exercise, since embedded
Qdrant is the offline default. ./data is mounted read-only, so pointing it at
a real archive is a one-line change to the compose file.
The image is a two-stage build on python:3.12-slim, runs as a non-root user,
and has a HEALTHCHECK that reports ready only once the lifespan hook has
finished indexing — the process accepts connections before the index exists, so
"up" and "ready" are not the same event.
docker run --rm ops-recall demo # anything the CLI does
docker run --rm ops-recall eval
docker run -p 8000:8000 ops-recall serveEvery tool the container needs is inside it: no credentials are baked in, and
without an ANTHROPIC_API_KEY the service falls back to the deterministic
assembler and labels every response reasoner=template.
Two files, with different jobs:
pyproject.tomldeclares what the project supports — bounded ranges likepydantic>=2.9,<3. The upper bounds are the point: an unbounded floor means a clone a year from now resolves against a major release this code has never seen, and the failure lands on whoever tried the project with no signal that anything changed.requirements.lockrecords the exact transitive set it was last built and tested against, so an image or a CI run is reproducible rather than "whatever resolved that morning".requirements-dev.lockadds the test and lint tools.
The container installs from the lock; the test matrix installs from the ranges.
That split is deliberate — the matrix is the signal that the project still
works against current releases, and a locked-only CI would never tell you the
day that stopped being true. A separate locked install job installs the exact
set and runs the suite, because a lock nothing installs from rots quietly.
ruff and mypy are held to a single minor rather than a range. Both add rules
and sharpen inference between minors, so a floating version turns an unrelated
Tuesday into a red build on an untouched branch; upgrading them should be a
commit with the fixes in it.
Regeneration instructions are in the header of requirements.lock.
.github/workflows/ci.yml runs six jobs on every push and pull request:
tests across Python 3.11-3.13, ruff and mypy, the retrieval evaluation
against its floor, the telemetry catalog queries against a real PostgreSQL
service container, an install from the lockfile, and a container build that
starts the image and posts a real alert to it. Nothing in CI reaches a model or an embedding API — the workflow
pins OPS_RECALL_REASONER=template, so a dependency that started phoning home
would fail the offline claim rather than quietly cost money. The one live
dependency is that Postgres, which is a container the job starts itself.
ruff format is deliberately not run. Several tables and word lists here are
hand-aligned to be read, and the formatter collapses them; lint enforces
correctness rules, not layout.
The version lives in exactly one place, ops_recall.__version__. Setuptools
reads it from there, ops-recall --version prints it, and the OpenAPI document
carries it — so the number a responder reads off a running service is the
number that was built.
Cutting a release is four steps, and the tag is the trigger:
# 1. bump ops_recall/__init__.py, 2. move the Unreleased entries under the
# new heading in CHANGELOG.md, then:
git commit -am "Release 0.2.0"
git tag -a v0.2.0 -m "ops-recall 0.2.0"
git push origin main v0.2.0.github/workflows/release.yml picks the tag up and, in order: refuses it if
v0.2.0 disagrees with __version__ or the CHANGELOG has no section for it;
runs the whole of ci.yml against the tagged commit, so a release cannot be
greener than the branch it came from; builds the sdist and the wheel and
installs each into a clean environment to prove it is complete; pushes
ghcr.io/russellw/ops-recall:0.2.0 (and :latest); and publishes a GitHub
release whose notes are the CHANGELOG section, with the distributions attached.
Versions are semantic, with the usual 0.x caveat that the CLI flags, the /v1
shapes and the OPS_RECALL_* settings may still move in a minor release. A tag
with a hyphen in it — v0.2.0-rc.1 — ships as a prerelease and does not move
latest, which is the difference between a release candidate and something
docker pull hands an unsuspecting colleague.
The notes come out of CHANGELOG.md rather than out of git log: commit
subjects are written for whoever is reading the diff, and a release note is for
someone who will never look at it. tools/release_notes.py does the lifting,
and it is tested, because a release that fails at the last step has already
pushed an image.
What is stubbed here is the data, not the design. Each default has a working implementation on the other side of the same interface:
-
Embeddings. Three implementations sit behind one
Embedderprotocol.HashingEmbedder(the default) is a hashed bag of words and character trigrams: it captures morphology and co-occurrence but not meaning, and scores "pool starved" against "no free connections" at exactly 0.0. It exists so a fresh clone runs with nothing installed.FastEmbedEmbedderruns real ONNX weights locally — no key, no per-query network call, ~67MB cached on first use.VoyageEmbedderis the hosted option and the strongest of the three.pip install -e ".[fastembed]" && OPS_RECALL_EMBEDDER=fastembed ops-recall eval OPS_RECALL_EMBEDDER=voyage OPS_RECALL_VOYAGE_API_KEY=pa-... ops-recall eval
The local option exists because it is the only one CI can run, and running it produced a result worth reporting plainly:
Suite hashing fastembed (bge-small-en-v1.5) Composite 0.966 0.966 hit@1 / hit@3 / MRR 0.92 / 1.00 / 0.96 0.92 / 1.00 / 0.96 Semantic lane alone, MRR 0.910 0.744 A real semantic model does not improve this suite, and on its own it is distinctly worse. That is not a defect in the model — it is what a jargon-dense corpus looks like. Thirteen of the 24 answerable cases quote error codes and hostnames verbatim, and character-trigram hashing is a fuzzy exact-match engine, which is precisely the right tool for
ERR_5012anddb-prod-03. A general-purpose sentence encoder smooths those into a neighborhood whereERR_5012andERR_5013are neighbors. So the earlier claim in this README — that the 0.966 was "the weakest version of itself" because the semantic lane was hashed — was wrong, and is now measured rather than assumed. On this suite the semantic lane is not the bottleneck; the lexical and keyword lanes carry the fusion.The swap did surface one real bug, described under the abstention threshold.
-
Telemetry. Three live backends ship:
prometheus,postgresandtls, covering eight of the nine probes between them. See Where telemetry comes from — the composition, the query overrides and what each one deliberately declines to answer.FixtureTelemetryProviderremains the default so a fresh clone is demoable with no configuration, and remains useful in a composition as the fallback forcheck_recent_deploys, which is the one probe no general backend can answer: deploy history lives in whatever the team happens to use. Implementing it is a subclass with one method, and thedata["holds"]contract means the evidence layer does not move. -
Qdrant. Embedded by default. Point
OPS_RECALL_QDRANT_URLat a cluster and payload indexes are created automatically.
The corpus is the honest gap. Ten post-mortems, one Slack export, one PagerDuty dump and four wiki pages are seed data, written to exercise the system rather than recorded by a team that lived through them. Nobody has run this against a real incident archive, and the retrieval numbers should be read as a statement about the ranking rather than a forecast for anyone else's history.
Before it is reachable by anyone else, set OPS_RECALL_API_KEYS — see
Authentication. Shared bearer tokens are the right shape for
webhooks and internal tooling; per-user identity would mean an IdP and is a
different piece of work.
Also worth knowing: BM25 idf is corpus-global, so ingest re-indexes the whole corpus. At post-mortem volumes (hundreds to low thousands of documents) that is seconds; at a much larger scale it wants incremental idf updates.
- Similarity is explainable or it is worthless. "87% similar" is only actionable if a responder can see that it came from an exact error-code match rather than a vague prose resemblance. Every response carries the full breakdown, which is also why the two lanes are queried separately instead of fused server-side by Qdrant.
- Below the threshold, say nothing. Forcing a bad analogy onto a novel incident is the failure mode that destroys trust in a system like this. The cold-start path returns "this is novel" with whatever telemetry it could still gather.
- Recorded commands contain stale identifiers.
pg_terminate_backend(48211)refers to a process that died two years ago. The model is instructed to substitute a value from the telemetry it was given or name what the responder must look up; the deterministic path attaches a standing caveat. - Quotes are human commentary, not the loudest match. The fragment most similar to a firing alert is always the old alert text from PagerDuty — a near-copy of the query that tells a responder nothing. Authored fragments are selected first, capped per incident.
ops_recall/
models.py domain types (Incident, Signal, Alert, Reconstruction, ...)
config.py settings
service.py composition root
ingest/ postmortems, slack, pagerduty, wiki, pipeline
retrieval/ text, embeddings, sparse (BM25), temporal, store, ranker
telemetry/ probe protocol, fixture, prometheus, postgres, tls, composition
agent/ evidence, tools, prompts, investigate, reconstruct, graph
api/ FastAPI app and schemas
evals/ the labeled retrieval set and its metrics
cli.py
data/seed/ 10 synthetic incidents across 4 source systems
data/telemetry/ the live-system snapshot the probes read
evals/queries.yaml 29 labeled alerts, 5 of them deliberately unanswerable
tests/ 217 tests; the only ones needing a server skip without it
tools/screenshot.py redraws the README's terminal image from a real CLI run
tools/release_notes.py lifts a version's CHANGELOG section for the release
docs/demo.svg that image, and docs/demo.txt its plain-text twin
CHANGELOG.md what changed, and the source of the release notes
Dockerfile two-stage build, non-root, healthchecked
docker-compose.yml the service against a real Qdrant
requirements*.lock exact transitive pins; pyproject holds the ranges