Skip to content
BranchesTags

Repository files navigation

Relay

A self-hosted AI chat and image-generation interface that proxies any OpenAI-compatible API, including OpenAI, Anthropic, Ollama, or your own endpoint. Think of it like open-webui or LibreChat, but lighter and easier to deploy.

Features

  • Multiple connections. Add as many API backends as you want (OpenAI, Ollama, Anthropic, or anything custom) and switch between them per chat.
  • Smart model routing. Turn on auto-routing and a classifier reads each prompt, sorts it into a category (coding, creative, reasoning, or fast), and sends it to the model you picked for that category. Because every category can point at a different connection, a single conversation can lean on different providers depending on what you ask. More on this below.
  • Context management you control. Pick how each chat handles a growing history: send everything, keep a recent token-budget window, or summarize older messages. A live gauge shows how full the model's context window is, and every message shows its token count, so you can see what is about to fall out of context instead of hitting a wall. Set it globally or per chat.
  • Chat. Streaming and non-streaming completions with full conversation history, editing, and regeneration.
  • Image generation. Works with gpt-image-1, dall-e-3, and dall-e-2, with controls that adapt to whichever model you select.
  • Usage tracking. Token usage is recorded per connection and per model in a local SQLite database, so you can see what you are spending.
  • Live logs in the UI. A toggleable log drawer (sidebar button or the Cmd/Ctrl backtick shortcut) streams the backend's logs in real time, with level filters and search, so you can see why a request failed without digging through docker logs.
  • Model-aware metadata. Context windows, pricing, and capabilities are resolved by the backend, which probes providers that expose them (Ollama, OpenRouter) and falls back to a maintained table. The frontend no longer hardcodes this.
  • Prompt library. Save prompts you reach for often and reuse them in a click.
  • One Docker image. The Go backend and React frontend ship together, served by nginx in a single container.

How model routing works

The old behavior forced every message in a conversation to a single model. Auto-routing replaces that with categories.

You set up four slots in Settings, and each slot points at a model on any of your connections:

  • Fast for short questions and simple tasks. This model also acts as the classifier.
  • Coding for writing, debugging, and refactoring code.
  • Creative for writing, brainstorming, and anything about tone.
  • Reasoning for math, logic, and multi-step problems.

When you send a message, Relay first does a quick local keyword check to catch the obvious cases for free. If the prompt is ambiguous, it asks the Fast model to classify it, then routes the message to the matching slot's model and connection. If the classifier is unreachable or times out, it falls back to whatever you configured, either the conversation's own model or the Fast slot. Each reply shows a small badge so you can see which category handled it.

None of this needs special provider support. Routing just decides which connection to use, and the backend handles the actual switch.

How context management works

Every chat turn sends the conversation history to the model, and that history keeps growing. Left unchecked it eventually exceeds the model's context window and the request fails, and along the way you re-pay for the whole transcript on every turn. Relay lets you decide how to handle that, globally in Settings or per chat from the header:

  • None. Send the full history. Simple, but it will fail once the chat outgrows the window.
  • Window (the default). Keep the system prompt plus the most recent messages that fit a token budget, and drop the oldest. Free and deterministic.
  • Summarize. When the history overflows, condense the dropped older messages into a short summary using a model you choose, then send that plus the recent messages. This keeps old context at the cost of an occasional summary call, so it is off by default and clearly marked.

The budget is a share of the model's context window (configurable), with some room reserved for the reply. To make all of this visible rather than silent, the chat header shows a gauge of how full the window is (it shifts toward red as it fills), and each message shows its estimated token count. Messages that the current strategy would drop are greyed out, so you can see what is leaving context before you send.

Context windows themselves come from the backend, which probes providers that report them and otherwise uses a maintained table. For unknown or self-hosted models you can set a window override in Settings.

Getting started

Option 1: Docker (recommended)

You need Docker installed.

docker run -d \
  -p 3000:80 \
  -e API_BASE_URL=https://api.openai.com \
  -e API_KEY=sk-... \
  -v relay_data:/data \
  --name relay \
  ghcr.io/johnbetancur/relay:latest

Then open http://localhost:3000.

Option 2: Docker Compose

git clone https://github.com/johnbetancur/relay.git
cd relay

cp .env.example .env
# Set API_BASE_URL and API_KEY in .env

docker compose up -d

The app runs at http://localhost:3000.

Option 3: Local development

You need Go 1.26 or newer and Node.js 22 or newer.

git clone https://github.com/johnbetancur/relay.git
cd relay

# Start the backend
cd backend
API_BASE_URL=https://api.openai.com API_KEY=sk-... go run ./cmd/server

# In another terminal, start the frontend
cd frontend
npm install
npm run dev

Frontend runs at http://localhost:5173 and the backend at http://localhost:8080.

You can also use Docker Compose with the dev profile to get hot-reload on both:

API_BASE_URL=https://api.openai.com API_KEY=sk-... \
  docker compose --profile dev up

Configuration

Everything is configured through environment variables.

Variable Default Description
API_BASE_URL https://api.openai.com Base URL of the upstream API
API_KEY (empty) API key sent as Authorization: Bearer
PORT 8080 Port the Go backend listens on
DB_PATH ./relay.db Path to the SQLite database file

When you run with Docker, keep the database around by mounting a volume to /data and leaving DB_PATH at its default (/data/vision.db in docker-compose.yml).

Connections

Relay supports multiple upstream connections, all managed from the UI under Settings then Connections. Each connection has:

Field Description
Name Display label
Base URL Upstream API root, for example https://api.openai.com or http://localhost:11434
API Key Optional, sent as Authorization: Bearer <key>
Type Hint for the UI: openai, anthropic, ollama, or custom
Default Whether this connection is pre-selected in new chats

The connection you set through the API_BASE_URL and API_KEY environment variables is the built-in fallback, used when no connection is selected.

Provider examples

OpenAI

Base URL: https://api.openai.com
API Key:  sk-...
Type:     openai

Anthropic

Base URL: https://api.anthropic.com
API Key:  sk-ant-...
Type:     anthropic

Ollama (local)

Base URL: http://localhost:11434
API Key:  (leave empty)
Type:     ollama

Any OpenAI-compatible API

Base URL: https://your-provider.com/v1
API Key:  your-key
Type:     custom

Image generation

The image page supports three model families and adjusts its controls for whichever you pick:

Model Sizes Quality options Style Multi-image
gpt-image-1 1024x1024, 1536x1024, 1024x1536 auto / high / medium / low n/a yes (up to 4)
dall-e-3 1024x1024, 1792x1024, 1024x1792 hd / standard vivid / natural no (n=1 only)
dall-e-2 1024x1024, 512x512, 256x256 standard n/a yes (up to 4)

Switching models resets quality, size, and count to valid defaults for that model.

Architecture

The backend is not just a passthrough. It holds your API keys server-side, sidesteps the CORS restrictions that would block a browser from calling providers directly, records token usage, and runs the tool-calling and document-extraction endpoints. Routing happens entirely in the frontend: it decides which connection a request should use and sets the X-Relay-Connection-ID header. The proxy reads that header, swaps in the right upstream URL and key, and forwards the call.

%%{init: {'flowchart': {'defaultRenderer': 'elk'}}}%%
flowchart TD
    subgraph Browser["Browser (React SPA)"]
        UI["Chat & Image UI"]
        Router["Auto-routing<br/>classifier"]
        UI --> Router
    end

    subgraph Container["Single Docker container"]
        Nginx["nginx :80"]
        subgraph Backend["Go backend :8080"]
            Conn["/api/connections<br/>CRUD"]
            Usage["/api/usage/by-model"]
            Meta["/api/connections/{id}/model-meta<br/>window, price, capabilities"]
            Logs["/api/logs/stream<br/>live SSE logs"]
            Agent["/api/agent/chat<br/>tool loop"]
            Docs["/api/documents/extract"]
            Proxy["/api/v1/*<br/>reverse proxy"]
        end
        DB[("SQLite<br/>connections + usage")]
    end

    Up1["OpenAI"]
    Up2["Anthropic"]
    Up3["Ollama / custom"]

    Router -->|"X-Relay-Connection-ID"| Nginx
    Nginx -->|"static files"| UI
    Nginx -->|"/api/*"| Backend
    Conn --> DB
    Usage --> DB
    Proxy -->|"injects key,<br/>records usage"| DB
    Meta -->|"probes for metadata"| Up3
    Proxy --> Up1
    Proxy --> Up2
    Proxy --> Up3
    Agent --> Up1
Loading
  • Frontend: React 19, Vite, Mantine UI, and Zustand.
  • Backend: Go with the Chi router, and SQLite through modernc/sqlite, so there is no CGO requirement.
  • Deployment: a single multi-stage Docker image. nginx serves the SPA and proxies /api/* to the Go process running alongside it in the same container.

API

The backend exposes a small REST API for managing connections, querying usage, and the proxy itself.

Connections

GET    /api/connections             List all connections
POST   /api/connections             Create a connection
GET    /api/connections/{id}        Get a connection
PUT    /api/connections/{id}        Update a connection
DELETE /api/connections/{id}        Delete a connection
GET    /api/connections/{id}/models     List models for a connection
GET    /api/connections/{id}/model-meta Model metadata (window/price/capabilities)
GET    /api/connections/{id}/stats      Usage stats for a connection
DELETE /api/connections/{id}/stats      Reset stats for a connection

model-meta with ?model=<id> resolves one model (probing the provider where supported); without it, it returns the full static metadata table for bulk use.

Connection object

{
  "id": "01J...",
  "name": "OpenAI",
  "baseUrl": "https://api.openai.com",
  "typeHint": "openai",
  "enabled": true,
  "isDefault": true,
  "createdAt": 1716000000,
  "updatedAt": 1716000000
}

apiKey is write-only. It is accepted on create and update, but never returned.

Usage

GET /api/usage/by-model    Token usage totals grouped by model

Agent, documents, and logs

POST /api/agent/chat        Tool-calling chat (streams tool steps then the answer)
POST /api/documents/extract Extract text from an uploaded file (PDFs are parsed)
GET  /api/logs/stream       Live backend logs as Server-Sent Events

Proxy

Every request to /api/v1/* is forwarded to the upstream API with the path rewritten to /v1/*. Pass X-Relay-Connection-ID: <id> to route the request through a specific connection.

Health

GET /healthz    Returns 200 OK when the backend is up

Building from source

# Frontend
cd frontend && npm ci && npm run build

# Backend
cd backend && go build -o relay ./cmd/server

# Or build the Docker image
docker build -t relay .

License

Apache 2.0. See LICENSE.

About

No description, website, or topics provided.

Resources

Readme
Apache-2.0 license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages