Low-Latency LLM Inference Server

End-to-end stack that streams 120 tokens-per-second from Llama-3-8B with forty percent lower p99 latency at thirty two concurrent requests.

---

Features

• vLLM backend Fast CUDA inference for 8B parameter model
• .NET 8 gRPC gateway Batched microservice with thirty two request batching
• RHEL 9 Ansible deploy Single command provisioning on AWS g5.2xlarge (A10G)
• Observability Prometheus scrape targets plus NVIDIA DCGM exporter and ready-to-import Grafana JSON
• Container images Podman-compatible Dockerfiles for gateway and backend

---

Architecture

    ┌──────────────┐       gRPC stream        ┌────────────────┐
    │  Clients     │  ─────────────────────→  │  Gateway .NET  │
    └──────────────┘                          │  micro-batch   │
            ▲                                 └────────┬───────┘
            │ HTTP JSON response                       │
            │                                          ▼
            │                          ┌───────────────────────┐
            └──────────────────────────│    vLLM GPU server    │
                                       └────────┬──────────────┘
                                                ▼
                                    NVIDIA DCGM exporter → Prometheus

---

Quick start

Prerequisites

• AWS EC2 g5.2xlarge or any A10G or higher GPU node
• RHEL 9 or compatible
• Ansible on local control machine

One-shot deploy

git clone [https://github.com/your-org/low-latency-llm-inference-server.git](https://github.com/your-org/low-latency-llm-inference-server.git)
cd low-latency-llm-inference-server
ansible-playbook -i infra/ansible/inventory infra/ansible/playbook.yml

• Gateway listens on port 5000
• vLLM backend listens on port 8000
• Prometheus on 9090, Grafana on 3000

Using the gRPC API

grpcurl -plaintext -d '{
  "prompt":"Hello",
  "max_tokens":32
}' localhost:5000 inference.Infer/Generate

Streamed tokens arrive in order.

Benchmark reproduction

1. Launch wrk or any gRPC load generator at thirty two parallel streams.
2. Collect gateway latency metrics from Prometheus query: histogram_quantile(0.99, rate(grpc_server_handling_seconds_bucket[1m]))
3. Confirm p99 latency reduction compared with REST baseline.

---

Repository layout

Path	Purpose
server/	.NET 8 gateway source and protobuf definitions
docker/	Container build files for gateway and vLLM
infra/ansible/	Inventory and playbook for RHEL provisioning
infra/prometheus.yml	Prometheus scrape config
dashboards/	Grafana dashboard JSON

Customization

• Change model by editing docker/Dockerfile.vllm.
• Adjust micro-batch size in server/Program.cs constant 32.
• Add scrape targets in infra/prometheus.yml.