Skip to content

About

Incident-response AI agent that uses hybrid RAG and live system data to find similar past incidents and identify whether their causes are recurring

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

ops-recall

ci

When an alert fires, the fastest path to a fix is usually a memory: we have seen this before. That memory lives in post-mortems nobody rereads, Slack threads nobody can find, and PagerDuty logs nobody exports.

ops-recall ingests all of it and, when a new alert fires, reconstructs the incident: which past incidents this resembles, what the root cause turned out to be, what the team actually did about it, which of those conditions are true again right now, and the single best next action.

ops-recall reconstructing a firing alert: two similar incidents with their score breakdowns, four confirmed telemetry checks, and the action that resolved the historical incident

The same run as text
$ ops-recall demo

This alert looks 75% similar to INC-2197: checkout-api connection pool exhausted while an ad-hoc VACUUM FULL held locks on
  orders_db.

Similar incidents
  INC-2197   75%    3 months ago  checkout-api connection pool exhausted while an ad-hoc VACUUM FULL hel
           semantic 0.61 | lexical 1.00 | exact-keyword 1.00 | recency x0.93
  INC-1842   39%   22 months ago  checkout-api pool exhaustion during a non-concurrent index rebuild on
           semantic 0.48 | lexical 0.60 | exact-keyword 0.60 | recency x0.71

Live telemetry
  CONFIRMED A single session holds a lock on orders_db for minutes while dozens of queries queue behind it.
            1 blocking session(s) on orders_db; longest is pid 51774 at 383s running 'VACUUM FULL orders', 44 waiters.
  CONFIRMED checkout-api's HikariCP pool is pinned at max with a growing queue of pending connection requests.
            checkout-api pool at 50/50 (100%) with 118 request(s) queued.
  CONFIRMED No checkout-api deploy preceded the incident - the trigger came from the database side, not the application.
            No deploys of checkout-api in the last 6h.
  CONFIRMED checkout-api 5xx rate climbs above 20% within four minutes of the first pool timeout.
            checkout-api error rate 37.0% (threshold 5.0%).

  [...]

Recommended next action
  1. Terminate the blocking backend.
     $ SELECT pg_terminate_backend(48211)
     This is what resolved INC-2197 (pool drained back to 6/50 active within 40 seconds, 5xx rate back to baseline by 14:36)
     risk high | needs human approval
  2. Re-run the space reclaim as VACUUM (ANALYZE) plus a scheduled pg_repack inside the Sunday window.
     $ VACUUM (ANALYZE)
     This is what resolved INC-2197
     risk low | needs human approval

  [...]

confidence 0.75 | reasoner template | retrieve 10.6ms -> check_evidence 0.2ms -> gather_quotes 3.3ms -> reconstruct 0.1ms

The [...] marks what the image leaves out: ops-recall demo also prints the root cause hypothesis, the full reconstruction and the caveats. Both the image and this text are generated from one real run by tools/screenshot.py, so neither can show something the CLI does not print.

The last two sections are the point. Retrieval alone produces a plausible story; re-testing that story's own conditions against the live system is what makes it worth acting on — and what lets the system say "this looks like INC-2197 but nothing is actually blocking, so the resemblance is superficial" instead of confidently recommending the wrong fix.

Quick start

python -m venv .venv && .venv/bin/pip install -e ".[dev]"

.venv/bin/ops-recall demo                       # the flagship reconstruction
.venv/bin/ops-recall search "consumer lag climbing"
.venv/bin/ops-recall probe check_db_locks database=orders_db
.venv/bin/ops-recall eval                       # score retrieval against the labeled set
.venv/bin/ops-recall serve                      # HTTP API on :8000
.venv/bin/python -m pytest                      # 121 tests, no network

Or in a container, with a real Qdrant behind it:

docker compose up --build                       # API on :8000, Qdrant on :6333
docker compose run --rm ops-recall demo

Everything runs offline out of the box: the index is embedded Qdrant, the embedder is deterministic, telemetry comes from a JSON snapshot, and the reconstruction falls back to a deterministic assembler. Set ANTHROPIC_API_KEY (or run ant auth login) and the narrative, the reasoning about competing hypotheses, and the agentic follow-up checks switch on.

How it works

post-mortems ─┐
Slack         ├─► ingest ─► correlate ─► index (Qdrant: dense + sparse)
PagerDuty     │                                    │
wiki ─────────┘                                    │
                                                   ▼
alert ───────────────────────────────► retrieve ──► no close match ──► "this is novel"
                                            │
                                            ▼
                                     check_evidence   replay each retrieved incident's
                                            │         probes against live telemetry
                                            ▼
                                     gather_quotes    what people actually said
                                            │
                                            ▼
                                      investigate     Claude runs extra read-only checks
                                            │
                                            ▼
                                      reconstruct     Claude writes the briefing

The pipeline is a LangGraph state machine (ops_recall/agent/graph.py). Retrieval and evidence checking are deterministic and always run; the model is reached last, with the facts already assembled. Every hop records timings and counts in state["trace"], which is returned on the API so a surprising answer can be traced to the step that produced it.

Hybrid search: three lanes

An alert contains two very different kinds of information, and one retrieval strategy cannot serve both. ERR_5012 and db-prod-03 must match exactly — embeddings put db-prod-03 and db-prod-07 in nearly the same place, which is precisely wrong. "Requests are queuing and timing out" must match semantically against "the pool never recovered on its own".

So three lanes, fused with renormalizing weights:

Lane What it is Weight
semantic Cosine over dense vectors 0.5
lexical BM25 sparse vectors, native Qdrant sparse index 0.3
keyword Fraction of the alert's extracted identifiers found in the incident 0.2
similarity = (w_sem·semantic + w_lex·lexical + w_kw·keyword) / Σw  ×  recency

Details that matter:

  • The keyword lane drops out cleanly. An alert with no error codes in it has nothing for that lane to match, so its weight is redistributed rather than capping every result at 0.8.
  • Both lanes are scored for every candidate. A lane only returns scores for the points it retrieved, so a result found lexically would otherwise have no semantic score. The candidate set is re-scored directly against both stored vectors, which makes every breakdown complete instead of lane-dependent.
  • Each lane indexes different text. The dense vector is built only from title, symptoms, keywords and services — an alert can only describe how something presented, so the semantic lane matches symptom against symptom. Including remediation prose pulls the vector toward how the incident was fixed, which is the part a firing alert cannot resemble. The sparse vector indexes everything, including the commands inside remediation steps, so an exact token is findable wherever it appears.
  • idf is per collection. Incidents and fragments have separate BM25 statistics; sharing one table let PagerDuty boilerplate distort the incident lane.
  • The lexical score is absolute, not relative. It is normalized against the idf mass the query could achieve in this corpus, rather than against the best score in the result set — otherwise the top hit always looks perfect, even when the whole result set is poor.

Metadata filters (services, max_age_days, min_severity, exclude_ids) are applied inside Qdrant, before scoring.

Temporal awareness

recency = floor + (1 - floor) · 0.5 ^ (age_days / half_life_days)

Defaults: half_life_days=365, recency_floor=0.6. Decay is a multiplier on the fused score, never a term added to it, so it can only break ties between comparable matches — it can never manufacture similarity that is not there. The floor is deliberate: a three-year-old exact match is stale, not irrelevant, and burying it is a worse failure than surfacing it with a caveat. In the demo above it is what puts the 3-month-old INC-2197 (×0.94) decisively ahead of the 21-month-old INC-1842 (×0.71).

Does the ranking actually work?

Every design choice above is an argument. evals/queries.yaml is the place they get checked: 29 labeled alerts written the way a monitoring system emits them, each paired with the incident a responder would want surfaced.

$ ops-recall eval

Retrieval quality
  hit@1 0.92   hit@3 1.00   relevant@1 0.96   MRR 0.96
  abstention 1.00 on novel alerts   composite score 0.966   (29 cases)

By category
  ambiguous    3/3 passed   score 0.833
  exact        13/13 passed   score 1.000
  novel        5/5 passed   score 1.000
  paraphrase   8/8 passed   score 0.938

Two things make this more than a hit-rate number:

  • Five cases have no correct answer. They are alerts about systems the archive has never seen, and the right behavior is to return nothing above min_similarity. They are scored as abstention and folded into the composite, so a change that buys hit rate by lowering the threshold loses points. Optimizing retrieval without them produces a system that always has an answer, which is the failure this product exists to avoid.
  • Categories are scored separately. exact alerts quote error codes and hostnames; paraphrase alerts describe the same failure in prose no post-mortem used; ambiguous alerts sit between two incidents in the same family. The aggregate hides which lane is doing the work.

hit@1 is strict — it wants the primary target first, not merely a member of the right family. The gap to relevant@1 (0.92 vs 0.96) is exactly one case: an alert mentioning CREATE INDEX should rank the migration incident above the newer VACUUM one, and recency decay outvotes the evidence. That is the recency tradeoff the section above describes, showing up as a number instead of a worry.

The suite pins its own clock (as_of), because recency decay is a factor in every score and an unpinned suite would drift downward as the corpus ages — failing CI one day on a repository nobody had touched.

What the sweep says about the weights

ops-recall eval --sweep grid-searches the lane weights over the simplex and sorts by composite score. Running it is how the claims in this README stop being claims:

Configuration Score
Best on the grid (5-way tie) 0.983
Configured 0.5 / 0.3 / 0.2 0.966
Best without the lexical lane 0.948
Best without the semantic lane 0.966
Best without the keyword lane 0.983
Keyword lane alone 0.874
Lexical lane alone 0.810
Semantic lane alone 0.776

The fusion argument survives: every lane alone is far worse than any blend of them, which is the whole case for hybrid retrieval and is asserted as a test (test_fused_ranking_beats_every_single_lane).

The keyword lane does not survive. Dropping it entirely scores 0.983 — at the top of the grid, better than the shipped configuration. On this corpus the BM25 lane already matches db-prod-01 exactly and gives it a high idf, so the third lane is largely re-deriving what the second one found. The argument for keeping it is about a corpus this one is too small to exhibit: with hundreds of incidents, idf falls for hostnames that appear everywhere, and matching against curated frontmatter keywords rather than full text stops being redundant. That is a hypothesis, and it is now labeled as one rather than presented as a finding.

The shipped weights are deliberately not set to the grid winner. The gap is 0.017, which on 24 ranked cases is a single case changing rank — fitting three parameters to that is overfitting, and a portfolio number bought by tuning on the test set is worth nothing. What the sweep is for is catching a weight edit that falls off the plateau, which test_configured_weights_are_near_the_sweep_optimum does.

CI gates on ops-recall eval --fail-under 0.90. A change to the tokenizer, the embedder or the weights has to face the labeled set before it can merge.

The abstention threshold

min_similarity decides every abstention, and abstention is a third of what this system promises. It is a property of the vector geometry, not a constant, which is why it lives on the embedder rather than in the config defaults.

Feature hashing puts unrelated text at ~0.0, so a threshold of 0.30 separates cleanly. A trained model puts all English at 0.3–0.4 before it has said anything: carry 0.30 across the swap and abstention collapses from 1.00 to 0.00, every novel alert drawing a confident-looking 0.36 match. The ranked metrics do not move at all while this happens, so nothing but the novel cases would have caught it — which is the argument for scoring abstention in the first place.

Measured on the labeled set under bge-small-en-v1.5:

lowest true positive highest false positive gap
hashing 0.402 < 0.30 ≥ 0.10
fastembed 0.551 0.405 0.146

The trained model separates better — a wider margin — it just does so somewhere else on the number line. Any threshold in 0.42–0.52 scores an identical 0.966; the shipped value is 0.50, a round number inside that plateau placed above center because the failure below it is a cliff and the failure above it is a slope. OPS_RECALL_MIN_SIMILARITY overrides it; leaving it unset is what makes a swap safe.

Signals and probes — re-testing history against the present

This is the central mechanism. A post-mortem declares the conditions that held during the incident, each bound to a telemetry probe:

signals:
  - id: blocking-lock-orders-db
    statement: A single session holds a lock on orders_db for minutes while dozens of queries queue behind it.
    probe: check_db_locks
    probe_args: {database: orders_db, min_seconds: 60}

At alert time every retrieved incident's signals are collected, deduplicated (incidents in the same family share signals; a condition seen in three incidents is stronger evidence, not three checks), retargeted at the service the current alert names (a signal recorded as "checkout-api's pool is saturated" generalizes to "the affected service's pool is saturated"), and run. Each comes back CONFIRMED, REFUTED, or UNKNOWN — never guessed. REFUTED is often the most informative label on the page, because it rules a story out.

Try it: empty blocking_sessions in data/telemetry/current_state.json and run ops-recall demo again. Same alert, same archive, opposite conclusion.

Where telemetry comes from

A probe is a question, not a data source. TelemetryProvider dispatches probe names to probe_* methods and owns the argument validation; what answers them is configuration. OPS_RECALL_TELEMETRY names the backends in precedence order:

OPS_RECALL_TELEMETRY=prometheus+postgres+tls+fixture
Backend Answers Needs
prometheus error rate, connection pool, service metrics, consumer lag, disk usage PROMETHEUS_URL
postgres blocking locks, replication slots POSTGRES_DSNS
tls certificate expiry nothing
fixture all nine, out of a JSON snapshot the bundled file
null nothing; every check comes back unknown nothing

No single system answers all nine, so a real deployment is always a composition. First match wins, which is what the ordering is for: a fixture at the end is the fallback for probes no live backend implements, and the same fixture at the front would shadow every one of them.

A failing backend is not retried against the next. A probe that reports unknown because Prometheus is down has said something true, and substituting a fixture's answer for it would turn an outage in the monitoring stack into confident fiction in an incident briefing.

A backend that is named but not configured is a startup error, not a warning. The failure that avoids is a service reporting telemetry=fixture in /v1/stats while its operator believes it is reading production, and answering every probe from a JSON file for a week.

pip install 'ops-recall[telemetry]'   # or [prometheus] / [postgres]

Prometheus. One PromQL instant query per probe. Metric names are not portable — one team's http_requests_total is another's request_count — so every query is a template and OPS_RECALL_PROMETHEUS_QUERIES points at a JSON file that overrides any subset of them:

{
  "error_rate": "sum(rate(requests_total{service=\"$service\",status=~\"5..\"}[5m])) / sum(rate(requests_total{service=\"$service\"}[5m]))",
  "metric.goroutines": "sum(go_goroutines{job=\"$service\"})"
}

Every probe argument is offered twice: $service for an = matcher and $service_re, regex-quoted, for an =~ one — an unescaped db-01.eu.corp in a regex matcher also matches hosts it is not. The defaults target the exporters this corpus implies: a JVM service behind HikariCP, node_exporter, kafka_exporter. An absent series is reported as unknown, not as zero — a service nobody scrapes is not a healthy service, and a 0% error rate for it would refute a signal that was never actually tested.

check_recent_deploys is deliberately absent from this backend. Deploy history is not a metric in most installations, and a default query for it would answer the most consequential question in the set with fiction. Compose something that can answer it, or let it come back unknown.

PostgreSQL. The two probes that no metrics system answers well, because the answer is a row in a catalog rather than a time series: which session is blocking others and what it is running (pg_stat_activity joined through pg_blocking_pids()), and how much WAL a replication slot is retaining (pg_replication_slots). Read-only access is enough. OPS_RECALL_POSTGRES_DSNS maps the database name the corpus uses to a libpq DSN, because the name a team calls a database is rarely its connection string. Connections are opened per probe and closed after — an incident tool that holds idle connections to a database it is diagnosing for connection exhaustion has misunderstood its job.

Both statements run against a real PostgreSQL in CI, in a job that fails rather than skips if the driver is missing. Catalog SQL is exactly the kind of code that reads correctly and is not: the first version of the statement timeout here used a bind parameter, which SET does not accept, and nothing short of executing it would have said so.

TLS. The one probe that needs no infrastructure at all: connect, read the peer certificate, subtract. The interesting case is that an already expired certificate fails verification, so the handshake never completes and there is no date to read — at exactly the moment the answer is most urgent. That is reported as the probe holding, because "it expired" is what was being asked; other verification failures (wrong hostname, unknown issuer) stay unknown, since they say nothing about expiry either way.

Agentic tool use

ops_recall/agent/tools.py exposes the telemetry surface to Claude via the SDK tool runner, plus search_incidents and get_runbook over the same corpus. This covers the case the deterministic replay cannot: the alert resembles a lock incident, the lock probe comes back refuted, and the useful next question — "is the service leaking connections instead?" — is one no post-mortem wrote down.

Every tool is read-only. Nothing terminates a backend, rolls back a deploy or edits configuration. Remediation is proposed to a human with a risk label and an approval flag; that is a boundary, not an unimplemented feature.

What the model is and is not allowed to author

The model receives a schema (ReconstructionDraft) covering the headline, root cause hypothesis, narrative, recommended actions, confidence and caveats. It has no schema slot for similarity scores, evidence statuses, incident ids or timestamps — those are computed by the system and merged in afterwards, so a retrieval score or a telemetry result cannot be invented. Structured output is requested through client.messages.parse; the system prompt is invariant across alerts and cached.

Adding your own data

Drop files into data/seed/ (or point OPS_RECALL_DATA_DIR elsewhere) and run ops-recall ingest.

  • postmortems/*.md — YAML frontmatter (id, title, severity, services, started_at, resolved_at, keywords, signals, pagerduty_incident, slack_thread) plus ## Summary, ## Symptoms, ## Root cause, ## Remediation sections. Remediation steps are numbered and tagged [diagnose] / [mitigate] / [fix], with an optional trailing `command`, — @actor, and (outcome: ...). The [fix] tag is what separates "what finally worked" from "what we tried first".
  • slack/*.json — conversations.history exports.
  • pagerduty/*.json — incidents with their log entries.
  • wiki/*.md — runbooks, indexed per ## section.

Correlation is by explicit reference: the post-mortem's frontmatter declares its PagerDuty incident and Slack thread, and those declarations let thousands of unlabeled messages attach themselves to the right incident. Messages that name an incident (INC-2197) link themselves. Nothing is linked by timestamp proximity — ops channels run several conversations at once.

For continuous ingest, set OPS_RECALL_SLACK_BOT_TOKEN + OPS_RECALL_SLACK_CHANNELS and OPS_RECALL_PAGERDUTY_API_KEY, then ops-recall ingest --live. The live clients emit the same Fragment objects as the exports, deduplicated by source id.

HTTP API

Method Path Purpose Auth
POST /v1/alerts Reconstruct an alert. The main endpoint. read
POST /v1/alerts/pagerduty Same, behind a PagerDuty v3 webhook adapter. signature
POST /v1/search Rank incidents against free text, with score breakdowns. read
GET /v1/incidents, /v1/incidents/{id} Browse the archive. read
GET /v1/probes What the agent can ask the live system. read
POST /v1/reindex Rebuild the index from the configured sources. admin
GET /v1/stats Operational. read
GET /healthz Liveness. none

Authentication

Set OPS_RECALL_API_KEYS and every endpoint above except /healthz wants Authorization: Bearer <key>. OPS_RECALL_ADMIN_API_KEYS is a second tier that /v1/reindex requires; admin keys are valid as read keys too, and with no admin key configured there is one tier rather than an unreachable endpoint. A valid key at the wrong tier is a 403, not a 401 — the credential is fine, retrying with it will never help.

OPS_RECALL_API_KEYS='["s3cret"]' .venv/bin/ops-recall serve
curl -H 'Authorization: Bearer s3cret' localhost:8000/v1/incidents

With no keys set the API is open, warns at startup, and reports auth.enabled: false in /v1/stats — the same shape as a missing ANTHROPIC_API_KEY degrading the reasoner instead of failing every request. It keeps docker run ops-recall serve and the offline demo working unconfigured, and it is not a deployment posture: /v1/search and /v1/incidents serve the team's post-mortems, Slack quotes with author names, and hostnames, while /v1/alerts spends model tokens per call.

PagerDuty cannot present a key you chose, so that route verifies the webhook subscription's own secret instead: X-PagerDuty-Signature: v1=<hex>, an HMAC-SHA256 over the raw request body, configured as OPS_RECALL_PAGERDUTY_WEBHOOK_SECRETS. Several are accepted at once, so a rotation does not need a deployment landing between the two keys. The body is read as raw bytes because that is what the signature covers — re-serializing a parsed payload would not reproduce them. A delivery with no signature falls back to the bearer check, which is what replaying a captured payload by hand uses; so configuring keys without a webhook secret locks PagerDuty out on purpose, rather than leaving one unauthenticated write path.

Two things this is not. The signature covers the body and nothing else — there is no timestamp in PagerDuty's scheme, so a captured delivery stays replayable and the secret is the whole of the protection. And bearer tokens over plain HTTP assume TLS terminates in front of the service. Authentication also does not bound cost: an authenticated caller can still spend tokens in a loop, so per-key rate limiting on /v1/alerts is the next thing this wants.

Configuration

All settings take an OPS_RECALL_ prefix and can live in .env (see .env.example).

Setting Default Notes
REASONER claude template for the deterministic fallback
MODEL claude-opus-5
AGENTIC_CHECKS true Let the model run its own follow-up probes
EMBEDDER hashing fastembed for local real embeddings, voyage for hosted
FASTEMBED_MODEL BAAI/bge-small-en-v1.5 Any fastembed model; BGE/E5/Nomic get their instruction prefixes
FASTEMBED_CACHE_DIR unset Where the ONNX weights live; unset uses fastembed's default
VOYAGE_DIM unset Only for a model this build has no width for
QDRANT_URL / QDRANT_PATH unset Unset means in-memory, rebuilt at startup
HALF_LIFE_DAYS 365
RECENCY_FLOOR 0.6
MIN_SIMILARITY the embedder's floor Below this the answer is "this is novel". Unset is correct: the threshold is geometry-bound (0.30 hashing, 0.50 trained)
TOP_K 3
WEIGHT_SEMANTIC / _LEXICAL / _KEYWORD 0.5 / 0.3 / 0.2
API_KEYS unset JSON list of bearer tokens. Unset means the API is open
ADMIN_API_KEYS unset Tier required by /v1/reindex; also valid as read keys
PAGERDUTY_WEBHOOK_SECRETS unset JSON list of webhook signing secrets
TELEMETRY auto Backends joined by +; auto is the bundled snapshot
PROMETHEUS_URL / _TOKEN / _QUERIES unset Required by the prometheus backend
POSTGRES_DSNS unset JSON object of database name to libpq DSN
TLS_PORT 443 A port in the probe's host argument wins over it

Running it

docker compose up --build

Brings up the service on :8000 against a real Qdrant on :6333 — the deployment shape the default configuration does not exercise, since embedded Qdrant is the offline default. ./data is mounted read-only, so pointing it at a real archive is a one-line change to the compose file.

The image is a two-stage build on python:3.12-slim, runs as a non-root user, and has a HEALTHCHECK that reports ready only once the lifespan hook has finished indexing — the process accepts connections before the index exists, so "up" and "ready" are not the same event.

docker run --rm ops-recall demo               # anything the CLI does
docker run --rm ops-recall eval
docker run -p 8000:8000 ops-recall serve

Every tool the container needs is inside it: no credentials are baked in, and without an ANTHROPIC_API_KEY the service falls back to the deterministic assembler and labels every response reasoner=template.

Dependencies

Two files, with different jobs:

  • pyproject.toml declares what the project supports — bounded ranges like pydantic>=2.9,<3. The upper bounds are the point: an unbounded floor means a clone a year from now resolves against a major release this code has never seen, and the failure lands on whoever tried the project with no signal that anything changed.
  • requirements.lock records the exact transitive set it was last built and tested against, so an image or a CI run is reproducible rather than "whatever resolved that morning". requirements-dev.lock adds the test and lint tools.

The container installs from the lock; the test matrix installs from the ranges. That split is deliberate — the matrix is the signal that the project still works against current releases, and a locked-only CI would never tell you the day that stopped being true. A separate locked install job installs the exact set and runs the suite, because a lock nothing installs from rots quietly.

ruff and mypy are held to a single minor rather than a range. Both add rules and sharpen inference between minors, so a floating version turns an unrelated Tuesday into a red build on an untouched branch; upgrading them should be a commit with the fixes in it.

Regeneration instructions are in the header of requirements.lock.

CI

.github/workflows/ci.yml runs six jobs on every push and pull request: tests across Python 3.11-3.13, ruff and mypy, the retrieval evaluation against its floor, the telemetry catalog queries against a real PostgreSQL service container, an install from the lockfile, and a container build that starts the image and posts a real alert to it. Nothing in CI reaches a model or an embedding API — the workflow pins OPS_RECALL_REASONER=template, so a dependency that started phoning home would fail the offline claim rather than quietly cost money. The one live dependency is that Postgres, which is a container the job starts itself.

ruff format is deliberately not run. Several tables and word lists here are hand-aligned to be read, and the formatter collapses them; lint enforces correctness rules, not layout.

Releases

The version lives in exactly one place, ops_recall.__version__. Setuptools reads it from there, ops-recall --version prints it, and the OpenAPI document carries it — so the number a responder reads off a running service is the number that was built.

Cutting a release is four steps, and the tag is the trigger:

# 1. bump ops_recall/__init__.py, 2. move the Unreleased entries under the
#    new heading in CHANGELOG.md, then:
git commit -am "Release 0.2.0"
git tag -a v0.2.0 -m "ops-recall 0.2.0"
git push origin main v0.2.0

.github/workflows/release.yml picks the tag up and, in order: refuses it if v0.2.0 disagrees with __version__ or the CHANGELOG has no section for it; runs the whole of ci.yml against the tagged commit, so a release cannot be greener than the branch it came from; builds the sdist and the wheel and installs each into a clean environment to prove it is complete; pushes ghcr.io/russellw/ops-recall:0.2.0 (and :latest); and publishes a GitHub release whose notes are the CHANGELOG section, with the distributions attached.

Versions are semantic, with the usual 0.x caveat that the CLI flags, the /v1 shapes and the OPS_RECALL_* settings may still move in a minor release. A tag with a hyphen in it — v0.2.0-rc.1 — ships as a prerelease and does not move latest, which is the difference between a release candidate and something docker pull hands an unsuspecting colleague.

The notes come out of CHANGELOG.md rather than out of git log: commit subjects are written for whoever is reading the diff, and a release note is for someone who will never look at it. tools/release_notes.py does the lifting, and it is tested, because a release that fails at the last step has already pushed an image.

Taking this to production

What is stubbed here is the data, not the design. Each default has a working implementation on the other side of the same interface:

  1. Embeddings. Three implementations sit behind one Embedder protocol. HashingEmbedder (the default) is a hashed bag of words and character trigrams: it captures morphology and co-occurrence but not meaning, and scores "pool starved" against "no free connections" at exactly 0.0. It exists so a fresh clone runs with nothing installed. FastEmbedEmbedder runs real ONNX weights locally — no key, no per-query network call, ~67MB cached on first use. VoyageEmbedder is the hosted option and the strongest of the three.

    pip install -e ".[fastembed]" && OPS_RECALL_EMBEDDER=fastembed ops-recall eval
    OPS_RECALL_EMBEDDER=voyage OPS_RECALL_VOYAGE_API_KEY=pa-... ops-recall eval

    The local option exists because it is the only one CI can run, and running it produced a result worth reporting plainly:

    Suite hashing fastembed (bge-small-en-v1.5)
    Composite 0.966 0.966
    hit@1 / hit@3 / MRR 0.92 / 1.00 / 0.96 0.92 / 1.00 / 0.96
    Semantic lane alone, MRR 0.910 0.744

    A real semantic model does not improve this suite, and on its own it is distinctly worse. That is not a defect in the model — it is what a jargon-dense corpus looks like. Thirteen of the 24 answerable cases quote error codes and hostnames verbatim, and character-trigram hashing is a fuzzy exact-match engine, which is precisely the right tool for ERR_5012 and db-prod-03. A general-purpose sentence encoder smooths those into a neighborhood where ERR_5012 and ERR_5013 are neighbors. So the earlier claim in this README — that the 0.966 was "the weakest version of itself" because the semantic lane was hashed — was wrong, and is now measured rather than assumed. On this suite the semantic lane is not the bottleneck; the lexical and keyword lanes carry the fusion.

    The swap did surface one real bug, described under the abstention threshold.

  2. Telemetry. Three live backends ship: prometheus, postgres and tls, covering eight of the nine probes between them. See Where telemetry comes from — the composition, the query overrides and what each one deliberately declines to answer. FixtureTelemetryProvider remains the default so a fresh clone is demoable with no configuration, and remains useful in a composition as the fallback for check_recent_deploys, which is the one probe no general backend can answer: deploy history lives in whatever the team happens to use. Implementing it is a subclass with one method, and the data["holds"] contract means the evidence layer does not move.

  3. Qdrant. Embedded by default. Point OPS_RECALL_QDRANT_URL at a cluster and payload indexes are created automatically.

The corpus is the honest gap. Ten post-mortems, one Slack export, one PagerDuty dump and four wiki pages are seed data, written to exercise the system rather than recorded by a team that lived through them. Nobody has run this against a real incident archive, and the retrieval numbers should be read as a statement about the ranking rather than a forecast for anyone else's history.

Before it is reachable by anyone else, set OPS_RECALL_API_KEYS — see Authentication. Shared bearer tokens are the right shape for webhooks and internal tooling; per-user identity would mean an IdP and is a different piece of work.

Also worth knowing: BM25 idf is corpus-global, so ingest re-indexes the whole corpus. At post-mortem volumes (hundreds to low thousands of documents) that is seconds; at a much larger scale it wants incremental idf updates.

Design decisions worth arguing about

  • Similarity is explainable or it is worthless. "87% similar" is only actionable if a responder can see that it came from an exact error-code match rather than a vague prose resemblance. Every response carries the full breakdown, which is also why the two lanes are queried separately instead of fused server-side by Qdrant.
  • Below the threshold, say nothing. Forcing a bad analogy onto a novel incident is the failure mode that destroys trust in a system like this. The cold-start path returns "this is novel" with whatever telemetry it could still gather.
  • Recorded commands contain stale identifiers. pg_terminate_backend(48211) refers to a process that died two years ago. The model is instructed to substitute a value from the telemetry it was given or name what the responder must look up; the deterministic path attaches a standing caveat.
  • Quotes are human commentary, not the loudest match. The fragment most similar to a firing alert is always the old alert text from PagerDuty — a near-copy of the query that tells a responder nothing. Authored fragments are selected first, capped per incident.

Layout

ops_recall/
  models.py            domain types (Incident, Signal, Alert, Reconstruction, ...)
  config.py            settings
  service.py           composition root
  ingest/              postmortems, slack, pagerduty, wiki, pipeline
  retrieval/           text, embeddings, sparse (BM25), temporal, store, ranker
  telemetry/           probe protocol, fixture, prometheus, postgres, tls, composition
  agent/               evidence, tools, prompts, investigate, reconstruct, graph
  api/                 FastAPI app and schemas
  evals/               the labeled retrieval set and its metrics
  cli.py
data/seed/             10 synthetic incidents across 4 source systems
data/telemetry/        the live-system snapshot the probes read
evals/queries.yaml     29 labeled alerts, 5 of them deliberately unanswerable
tests/                 217 tests; the only ones needing a server skip without it
tools/screenshot.py    redraws the README's terminal image from a real CLI run
tools/release_notes.py lifts a version's CHANGELOG section for the release
docs/demo.svg          that image, and docs/demo.txt its plain-text twin
CHANGELOG.md           what changed, and the source of the release notes
Dockerfile             two-stage build, non-root, healthchecked
docker-compose.yml     the service against a real Qdrant
requirements*.lock     exact transitive pins; pyproject holds the ranges

About

Incident-response AI agent that uses hybrid RAG and live system data to find similar past incidents and identify whether their causes are recurring

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages