ops-hand

(Python package and CLI are spelled opshand -- imports can't contain a hyphen.)

An LLM discovers how to complete a task in a real (deliberately legacy) back-office UI once. The successful run is compiled into a typed, versioned capability artifact. That artifact then replays deterministically -- no LLM in the loop -- with runtime error handling, and a real human-handoff path for when it can't safely proceed on its own.

See REPORT.md for the design write-up (architecture, artifact schema, determinism and error handling, heterogeneity/multi-tenant story, escalation model, safety model, cuts).

What's in this repo

Path	What
target_app/	Mock credit-union back-office app -- "CedarCore Servicing" -- deliberately legacy: frameset shell, table layout, no test IDs. Two tenant variants (/t/cedar, /t/summit). Runtime faults injectable via /_faults/{name}.
src/opshand/	The system: agent loop, artifact schema, replay engine, safety policy, escalation/handoff, evidence logging.
capabilities/	Saved capability artifacts (produced by discover, consumed by replay).
evidence/runs/	Structured logs + screenshots from real discovery and replay runs (see table below).
evidence/walkthrough.html	A narrated read-through of that evidence -- open it directly in a browser (self-contained, no server needed).
tests/	Runs the replay engine, safety policy, and redaction against the real target app (no LLM needed).

What's in /evidence/runs/

Five real runs, committed. Every number in REPORT.md traces back to one of these.

Run	What it proves
discover_1b46fcd1e9	A real LLM-driven discovery run against the live target app — 5 turns, including the model self-correcting its frame scope on turn 2. Produced the artifact in capabilities/.
replay_1e81db148e	Deterministic replay, no LLM. Returns SuccessResult with savings_balance = "$812.44".
replay_bf85158d1d	A business outcome, not a failure: member 00000 returns BusinessOutcomeResult(MEMBER_NOT_FOUND). The distinction the brief calls the most common design mistake.
replay_0409278760	Replay against an injected session-timeout fault → typed FailureResult (session_invalid), screenshots captured, escalated, blocked, then resumed by an operator — with the three actions they declared recorded in events.jsonl under handoff_resolved.
replay_974fb20338	The same artifact replayed against a different tenant variant (summit) with no re-record — the locator bundle degrades from role_name to name_attr past the renamed button.

Setup

Requires Python 3.12+ and uv.

uv sync
uv run playwright install chromium

Copy the env template and add an LLM key (only needed for discover -- replay never calls an LLM):

cp .env.example .env
# edit .env: set DEEPSEEK_API_KEY (default provider), or set
# OPSHAND_LLM_PROVIDER=anthropic and ANTHROPIC_API_KEY instead

Provider is swappable -- see src/opshand/agent/llm.py. DeepSeek was chosen because it speaks the OpenAI-compatible tool-calling wire format at low cost; Anthropic's Messages API is a one-line swap behind the same LLMClient protocol.

Demo path

1. Start the target app (separate terminal, stays running):

uv run uvicorn target_app.app:app --port 8000

2. Run a real LLM-driven discovery against it. This is the one step that requires an API key and makes real network calls -- it drives an actual Chromium browser through the frameset app, figures out the flow itself, and compiles a capability artifact on success.

uv run --env-file .env python -m opshand.cli discover \
  --goal "Look up member 12345 and read their current savings balance" \
  --url "http://localhost:8000/t/cedar/" \
  --name lookup_member_savings_balance \
  --description "Search for a member by member number and read their savings account balance." \
  --headed   # omit to run headless

On success this writes capabilities/lookup_member_savings_balance/v1.json and a full structured trace to evidence/runs/discover_<id>/.

3. Replay it deterministically -- no LLM, using the saved artifact:

uv run python -m opshand.cli replay lookup_member_savings_balance \
  --param member_id=12345 \
  --tenant-url "http://localhost:8000/t/cedar/"

Prints a structured SuccessResult with the extracted savings_balance.

4. See a business outcome (not a crash -- a legitimate typed result):

uv run python -m opshand.cli replay lookup_member_savings_balance \
  --param member_id=00000 \
  --tenant-url "http://localhost:8000/t/cedar/"
# -> BusinessOutcomeResult, outcome_code=MEMBER_NOT_FOUND

5. See a hard failure trigger escalation and a real human handoff, on the same live session:

curl -X POST http://localhost:8000/_faults/session_expired

uv run python -m opshand.cli replay lookup_member_savings_balance \
  --param member_id=12345 \
  --tenant-url "http://localhost:8000/t/cedar/"

The run detects the injected condition, stops, escalates, and blocks waiting for a human -- exactly as it would in production. It prints an operator console URL (http://localhost:8500/operator/<request_id>) and a live CDP session URL. Open the operator page (or just curl -X POST the resume endpoint it prints) to hand control back:

curl -X POST "http://localhost:8500/operator/<request_id>/resume?did=re-authenticated+at+teller+terminal&did=dismissed+stale-record+notice"

The blocked CLI process immediately unblocks and reports the handoff outcome — including how many operator actions were recorded. The did params (repeatable, plus an optional note) are what the operator declares they did while holding the session; they land on the intervention request and in evidence/runs/replay_<id>/events.jsonl under handoff_resolved, so the audit trail covers the human's turn, not just the automation's. See REPORT.md §5 for why these are operator-declared rather than sniffed off the browser session.

6. Replay the same artifact against a different tenant — same vendor product, different branding and control labels ("Member ID" / "Find Member" instead of "Member Number" / "Search"):

uv run python -m opshand.cli replay lookup_member_savings_balance \
  --param member_id=12345 \
  --tenant-url "http://localhost:8000/t/summit/"

This succeeds with no re-record. Check locator_used in the output: step_2_click resolves via name_attr here instead of the role_name it used on cedar — the ranked locator bundle degrading past the renamed button. That's the cross-tenant reuse story, demonstrated rather than asserted (see REPORT.md §4).

7. List the agent-invocable capability catalog:

uv run python -m opshand.cli list

Running without live services

tests/ exercises the full replay engine (success, business outcomes, hard failures, policy blocks) against the real target app with no LLM and no API key required -- pytest spins up the target app itself on a random free port:

uv pip install -e ".[dev]"
uv run pytest

The only step that genuinely needs a live LLM is discover; the saved artifact in capabilities/ and the evidence in evidence/runs/discover_*/ are the record of a real run already having happened (see REPORT.md for what's committed vs. what you'd regenerate).

Faults you can inject into the target app

curl -X POST http://localhost:8000/_faults/{fault}

fault	what it demonstrates
none	disarm (default)
slow_load	recoverable transient slowness
session_expired	hard failure -> escalation
app_error	hard failure (HTTP 500-style) -> escalation
interstitial	a surprise dialog the discovery agent must handle
validation	a business outcome (bad input), not a crash

Member 00000 -> not-found business outcome. Member 99001 -> access-denied business outcome (seeded as RESTRICTED status). See target_app/data.py for the full seed data.