Working AI agent: invoice triage on LangGraph

A working demo from winder.ai anchoring AI agent development services.

Anchors: https://winder.ai/services/ai-agent-development/

What this shows

This is a complete, runnable artificial intelligence (AI) agent that does one concrete job end to end: triaging invoices. It reads an invoice as raw text (the kind an optical character recognition, or OCR, step hands over), extracts the structured fields a finance team needs, validates them against business rules, self-corrects when validation fails, and posts the clean record to a mock enterprise resource planning (ERP) webhook. It is built on LangGraph as an explicit state graph, so the control flow (extract, validate, post, with a self-correcting loop and bounded retries) is visible rather than buried in a prompt. The non-obvious part is that this is a real loop over several tools, with state carried across steps and failure handling built into the graph, not a single large language model (LLM) call dressed up as an agent. Every model call goes through one provider (OpenRouter), and a held-out evaluation on synthetic invoices with exact ground truth proves the extraction is correct without an LLM judge.

Results (run 2026-06-19)

Metric	Dataset	Score	Model	Seed
Field-level extraction accuracy	20 synthetic invoices	100% (1.00)	claude-sonnet-4.6	7
Exact-invoice match rate	20 synthetic invoices	100% (1.00)	claude-sonnet-4.6	7
Post success rate	20 synthetic invoices	100% (1.00)	claude-sonnet-4.6	7
Self-correction rate	20 synthetic invoices	0% (0.00)	claude-sonnet-4.6	7
Mean cost per invoice	20 synthetic invoices	$0.0081 (1,476 tokens)	claude-sonnet-4.6	7

All five numbers come from one real run on 2026-06-19 (seed 7, 20 invoices, model anthropic/claude-sonnet-4.6, temperature 0). Header-field and line-item accuracy are both 100% as well. Mean spend was 1,476 tokens per invoice (1,167 in, 309 out) at about eight tenths of a US cent each, $0.16 for the whole run. Source: evals/results.json.

The self-correction rate is 0% because the model extracted every clean synthetic invoice correctly first time, so the repair loop never needed to fire. We did not manufacture a higher number; see Design decisions for why, and for how the self-correction and retry paths are proven instead.

Architecture

The agent is a LangGraph state graph with three nodes over one typed state object (TriageState): extract, validate and post. Conditional edges encode the policy. A clean validation routes to post; a failed validation routes back to extract with the errors as feedback (self-correction) until the extraction budget is spent; a failed post loops back to post (bounded retry) until it succeeds or the retry budget is spent. The single state object carries the loop's memory (the current best extraction, attempts so far, outstanding validation errors and token usage) across every step.

LangGraph fits because the hard part of this task is control flow with cycles, which is its core abstraction: a graph of nodes and conditional edges over typed state. The graph maps one to one onto the diagram above, so the code and the picture tell the same story, which matters more in a teaching demo than terseness. Routing is deterministic, driven by validation outcomes and retry budgets rather than by the model choosing its own next step, so runs reproduce. The model does the hard part (extraction and self-correction from feedback); validation and posting are the agent's own tools.

Only the extraction step calls a model, through LangGraph's chat-model binding pointed at OpenRouter's OpenAI-compatible endpoint, so one OPENROUTER_API_KEY covers everything and no second provider is involved. Each extraction is content-addressed and cached on disk, keyed by model, prompt version, invoice text and the current validation feedback, so re-running an invoice replays the cached result and never re-bills. The same cache makes both the evaluation and the walkthrough reproducible.

Design decisions

Two hard-to-reverse choices shaped the demo. They are recorded here so reviewers can see what was decided and why.

D1: Why invoice triage

Invoice triage is a fair illustration of agent work, not a single prompt dressed up. It needs a real loop over more than one tool (read, validate, post) with state across steps. The self-correction step is the non-obvious part: a one-shot extraction cannot check its own arithmetic against the line items or notice a missing tax identifier, whereas an agent that validates against business rules and repairs its own mistakes can. Failure handling is built into the graph (validation failures drive re-extraction, webhook failures drive bounded retries) rather than bolted on afterwards.

It also evaluates cleanly. The held-out set is synthetic invoices generated from known field values, so the ground truth is exact and the headline metric needs no LLM judge, which keeps the evaluation cheap, deterministic and free of judge bias. The data is fully public, carries no personally identifiable information (PII) and regenerates identically from a fixed seed, satisfying the public-data and determinism constraints this repo works under.

We rejected two alternatives. A research summariser would need reference summaries and an LLM judge for faithfulness, which is fuzzier, costlier and judge-dependent. Lead enrichment depends on scraping live websites, which is non-deterministic, fragile and legally fraught. Both fit the deterministic, public-data constraints worse than invoice triage.

D2: Why LangGraph

The invoice-triage loop is a stateful graph with cycles (validate, then repair and re-extract, or post, then retry), which is precisely LangGraph's core abstraction. The framework's shape matches the task's shape, so little of the code is ceremony unrelated to the problem, and the graph is inspectable and maps onto the architecture diagram. LangGraph is also provider-neutral, so we bind one OpenRouter-backed chat model and keep a single provider and key.

We rejected two alternatives. The OpenAI Agents SDK is ergonomic but designed around OpenAI as the provider, so naming it in a demo that forbids pinning to one provider is an editorial liability as much as a technical one. CrewAI's strength is multi-agent collaboration, which this deliberately single-agent task would leave unused, and conditional self-correction is less explicit in a crew than in a graph.

An honest note on the self-correction rate

The evaluation's self-correction rate is 0%: Claude Sonnet 4.6 extracted every clean synthetic invoice correctly on the first attempt, so the repair loop never needed to fire. We did not engineer harder or deliberately ambiguous invoices to manufacture a non-zero number, because that would either be gaming or make the gold labels contestable. The self-correction and bounded-retry paths are instead proven by the test suite (test_self_correction_repairs_a_bad_extraction, test_post_retries_until_it_succeeds) and shown live in the walkthrough through the --erp-failures flag, which forces transient ERP outages to drive the retry edge.

Video walkthrough

A roughly 74-second terminal capture of the agent running end to end on a real invoice: extract, validate and post, then the bounded-retry path when the mock ERP is forced to fail. The narration and the full transcript are in docs/video.md.

Higher-quality version: docs/walkthrough.mp4.

Run it yourself

You run the agent locally against your own OpenRouter key. We host no endpoints, so the only cost is your own token spend.

git clone https://github.com/winderai/winder-demos-working-agent
cd winder-demos-working-agent
cp .env.example .env   # add your OPENROUTER_API_KEY
uv sync
uv run invoice-triage --invoice data/samples/invoice_0001.txt

That triages one invoice end to end and prints the status, the extraction and post attempts, the token usage and the ERP receipt. To see the failure-handling path, force two transient ERP outages and watch the bounded retry climb:

uv run invoice-triage --invoice data/samples/invoice_0001.txt --erp-failures 2

The model and retry budgets are configurable from the command-line interface (CLI): --model takes any OpenRouter chat model with tool calling (default anthropic/claude-sonnet-4.6), and --max-extract-attempts and --max-post-attempts size the two loops.

Reproduce our evals

uv run python evals/run.py

This regenerates the 20-invoice synthetic set (seed 7), runs the agent on each invoice, scores field-level accuracy against the gold labels, prices the token usage from OpenRouter's live model pricing and writes evals/results.json. A single run costs approximately $0.16 in LLM tokens (about $0.008 per invoice). The run is idempotent: it skips a run already recorded with the same seed, count, model and code version, and the extraction cache means even a forced re-run replays cached calls and never re-bills. Continuous integration (CI) runs lint and tests only, never paid LLM calls.

Want help with this in production?

winder.ai builds, evaluates and ships AI agents for UK and US clients. Get in touch via the AI agent development services page or hello@winder.ai.