LabFlow Lite

LabFlow Lite is a full-stack TypeScript application that models a narrow scientific data workflow: a scientist creates a project, creates an experiment, uploads a raw instrument CSV dataset, the system parses and detects columns, an LLM proposes a field-level mapping to normalized scientific fields, a human reviews and approves each mapping, the dataset is normalized, and the results plus workflow history are displayed. It is a focused vertical slice rather than a LIMS, ELN, or bioinformatics suite, covering data traceability, event-driven processing, and AI workflow integration.

Core value: A scientist can upload a raw instrument CSV and trust that normalization only happens after they explicitly approve the AI-proposed field mapping. Every step is traceable and every decision is recorded.

---

The Workflow

LabFlow Lite implements a 7-step scientist workflow from project creation through normalized result review. The AI-human approval gate is the central design constraint of the system: normalization does not run until a human has reviewed the AI-proposed field mapping and explicitly submitted an approval. There is no path from an AI suggestion straight to a database write; every transformation traces back to a human decision.

flowchart TD
    A[Scientist creates Project] --> B[Scientist creates Experiment]
    B --> C[Scientist uploads raw instrument CSV]
    C --> D[System parses CSV\ndetects columns and sample values]
    D --> E[System calls AI to propose\nfield-level mapping]
    E --> F{Scientist reviews\nAI-proposed mapping\nfield by field}
    F -->|Override individual mappings\nor mark columns excluded| F
    F -->|Submit approval| G[Approved Mapping recorded\nnormalization gate opened]
    G --> H[System normalizes dataset\nusing approved mapping]
    H --> I[Normalized results stored\nwith full provenance]
    I --> J[Scientist views normalized\nresults and workflow history]
    D --> K[WorkflowEvent: columns_detected]
    E --> L[WorkflowEvent: mapping_proposed]
    G --> M[WorkflowEvent: mapping_approved]
    H --> N[WorkflowEvent: normalization_completed]

Loading

The diamond node (Step 5) is the human decision point. Its self-loop represents iterative field-by-field review: the scientist can override individual mappings or exclude columns before submitting the final approval. Only after that submission does an ApprovedMapping record exist, and only then can normalization run. Every significant transition (column detection, mapping proposal, approval, normalization) is also recorded as an immutable WorkflowEvent, giving the scientist a complete audit timeline for every experiment.

1. Project and experiment creation. Organizational scaffolding for a scientific run.
2. CSV upload. The scientist uploads a raw plate-reader export; a Dataset record is created.
3. Column detection. The system parses the CSV and extracts headers plus sample values.
4. AI mapping proposal. An LLM proposes a mapping from each raw column to one of the 11 normalized scientific fields, with a rationale per suggestion. This is advisory only.
5. Human review and approval. The scientist accepts, overrides, or excludes each mapping, then submits explicit approval. This is the gate.
6. Normalization. The dataset is transformed using the approved mapping; each row keeps a rawRowIndex back-reference to the original CSV.
7. Results and history. The scientist views normalized results and the full workflow timeline.

Full domain model, entity definitions, and the target normalized schema live in docs/domain-model.md. The complete step-by-step narrative lives in docs/workflow.md.

---

Architecture

LabFlow Lite is a pnpm monorepo with deployable app packages and shared library packages.

Package	Responsibility
apps/web	Frontend. Vite + React 19, TanStack Query for data fetching, React Hook Form + Zod for forms. Renders the project/experiment flow, CSV upload, mapping review UI, and results/history views.
apps/api	Backend API. Fastify + TypeScript. Owns CSV parsing, column detection, the AI mapping call, mapping approval, and the normalization endpoint.
apps/worker	Background worker. Node.js + TypeScript. Polls the Postgres-backed outbox table and processes WorkflowEvent records for reliable asynchronous delivery.
packages/db	Prisma schema (applied via db push) and the generated Prisma client. Single source of truth for the database shape.
packages/shared	Zod schemas and inferred TypeScript types shared by every app package. The single source of truth for domain types, validated at runtime and compile time.

Deliberate architecture choices:

• Outbox pattern on Postgres. Every WorkflowEvent is written in the same transaction as an OutboxEvent row, guaranteeing no event is lost even if the process restarts immediately after a state change. The background worker polls this table rather than relying on an external queue (SQS, EventBridge, Kafka). This proves event-driven processing locally before introducing external infrastructure.
• Stub identity. There is no real authentication in v1. A stub identity header simulates the scientist so the workflow, approval gate, and audit trail can be fully implemented and demonstrated without the scope of a real auth system.
• One fixed experiment type. The MVP normalizes plate-reader/absorbance-assay data against a single fixed 11-field schema (8 required, 3 optional). Configurable, multi-type schemas are intentionally deferred.

---

Local Setup

Requirements: pnpm only (no npm, yarn, or npx; use pnpm dlx for one-shot execution), Node.js >= 22.13.0, and Docker (for local Postgres).

Run these commands in order from the repo root:

pnpm install                          # install all workspace dependencies

pnpm db:up                            # start postgres:17-alpine via docker-compose.yml
                                       # (bound to 127.0.0.1 for local-only exposure)

pnpm db:generate                      # generate the Prisma client for packages/db

pnpm --filter @labflow/db db:push     # apply the Prisma schema to the running database
                                       # (this project uses schema-push, not `prisma migrate`)

pnpm -r typecheck                     # typecheck every package
pnpm -r test                          # run the full test suite across all packages
pnpm -r build                         # build every package

pnpm dev                              # run web, api, and worker in parallel for local dev

Environment variables

Copy .env.example to .env before running the commands above:

cp .env.example .env

.env.example is the source of truth for exact values and defaults. This section explains what each variable does, not what to type. DATABASE_URL must match whatever docker-compose.yml exposes for the local Postgres container.

• DATABASE_URL. The Postgres connection string shared by the API, worker, and Prisma client.
• API_PORT. The port the Fastify API listens on.
• WEB_PORT. The port the Vite dev server listens on.
• ANTHROPIC_API_KEY. A billed key for the live Claude mapping call, obtained from the Anthropic console. Only required for the live-Claude path below.
• AI_PROVIDER. Selects which AI mapping provider proposes the field-level mapping. See the two paths below; the behavior is not symmetrical, so read both.

AI_PROVIDER=mock (recommended for a first run). The full app runs end-to-end against the deterministic MockAiProvider, with no ANTHROPIC_API_KEY required, no API cost, and no live network call. This is the fastest way to see the whole workflow (upload, AI mapping proposal, approve, normalize, results, history) working in minutes, and it is the same path the test suite and CI run on.

• AI_PROVIDER unset, or set to anything other than mock. The app uses the live ClaudeAiProvider, which requires a valid ANTHROPIC_API_KEY and bills each mapping call to your own Anthropic account. This is the true default: leaving AI_PROVIDER unset does not fall back to the mock provider, so set AI_PROVIDER=mock explicitly if you want to avoid live calls entirely.

---

Test Commands

pnpm -r test                          # full suite across every package
pnpm --filter @labflow/shared test    # Zod schema and domain-type unit tests
pnpm --filter @labflow/api test       # API route, service, and outbox-lib tests
pnpm --filter @labflow/worker test    # outbox poll-loop tests
pnpm --filter @labflow/web test       # frontend component tests (React Testing Library)

Tests are written with Vitest (unit and integration) and React Testing Library (components). Some API and worker integration tests are gated on a live database connection and are skipped automatically unless DATABASE_URL is set to a running Postgres instance (e.g. via pnpm db:up above). This keeps the suite fast for pure logic changes while still exercising real transactional and outbox-atomicity behavior when a database is available.

---

Key Tradeoffs

These decisions are documented in full in docs/adr/0001-use-vertical-slice-scientific-workflow.md.

• Vertical slice over breadth. One complete path through the system (project creation through normalized results) was built and hardened before adding feature breadth. A narrow, fully working slice is easier to reason about and verify than a broad set of shallow, unproven components.
• AI-human approval gate is required. Normalization does not run until an ApprovedMapping exists. There is no trusted mode or bypass path. This is a correctness constraint rather than a UX convenience, because silent AI mapping errors in scientific data are difficult to detect and expensive to undo.
• Prisma over Drizzle. Chosen for its documentation and readability; not revisited without explicit discussion.
• Postgres-backed outbox before any external queue. Proves event-driven architecture and reliable async delivery locally, without taking on SQS/EventBridge/Kafka infrastructure before the core vertical slice is proven.
• Stub identity deferred to v2. No login UI, no sessions, and no OAuth in v1, which keeps scope on the data workflow. The approval-gate audit trail is informational rather than cryptographically verifiable until real auth is added.
• One fixed experiment type for the MVP. A single normalized schema (plate reader / absorbance assay, 11 fields) is enough to exercise the full mapping, approval, and normalization pipeline end to end. Configurable multi-type schemas are out of scope for v1.

---

Demo

Screen recording. Full walkthrough: create project, create experiment, upload CSV, review the AI-proposed mapping, approve, view normalized results, and view workflow history.