BEV-Perception

Turn ordinary camera (and optionally Lidar) feeds into a live, top-down map of everything around a vehicle — cars, pedestrians, lanes, and obstacles — running on a plain CPU.

A lightweight, CPU-first Bird's-Eye View (BEV) road-scene perception system for embedded devices. It reconstructs the scene around the vehicle and draws it as a top-down map centered on the car (the Ego frame), the same way a self-driving car "thinks" about the world.

🤔 Why is this hard? (Read this first)

A camera gives you a flat picture. But driving decisions happen in the real world: "That truck is 9 meters ahead and drifting into my lane." Going from pixels to meters is the whole problem, and it hides two traps:

A single camera cannot truly measure distance. A photo is a projection — a toy car held close and a real car far away can occupy the exact same pixels. To guess distance from one camera, you must assume something (usually: "the ground is flat"). That assumption breaks on hills, bumps, and slopes, and the error grows the farther away you look (easily 1m+ at range). This is a fundamental limit of monocular vision — adding more cameras does not fix it.
Lidar measures distance honestly, but sees the world full of holes. A Lidar spins and shoots laser rings, so even on a perfectly clear day it leaves gaps: blank stripes between rings, a blind cone right next to the car, dark surfaces that don't reflect, and anything past its range.

So neither sensor alone is enough. The core idea of this project is a division of labor:

Lidar says where (precise position). Camera says what (rich semantics) and fills the geometric holes Lidar leaves behind.

And when there's no Lidar at all (a cheap dashcam chip), the system gracefully falls back to camera-only mode — still useful, just honest that its distances are estimates, not measurements.

🧠 The big ideas (and the subtle bits)

These are the design decisions that make the system work. Understanding them is understanding the project.

1. Everything lives in the Ego frame — never on the camera image

The cardinal rule: the final result is a top-down map in meters, centered on the car, not annotations drawn on a camera frame. Overlaying boxes back onto the camera is only for debugging.

Why bother? Because a real car has 5–6 cameras, each covering a slice of the 360° view. If the "brain" of the system thinks in the shared ground-plane coordinate system, then adding cameras is just adding more sources that pour into the same map — the fusion and tracking logic never changes. If instead you reasoned per-image, you'd have to re-stitch everything every time the camera layout changed. Decoupling from the number of cameras is the whole reason this scales.

2. Lidar is honest but sparse; camera is dense but guesses — so fuse them

Both sensors ultimately produce the same currency: a list of 3D points in the Ego frame (points_ego). Lidar points are precise. Camera points (from monocular depth) are dense but approximate. They pour into one shared grid — a module that neither knows nor cares which sensor a point came from. This "sensor-agnostic" seam is what lets you add or remove a sensor without rewriting the pipeline.

3. The honesty about holes: Group A vs Group B

This is the most subtle — and most important — distinction in the whole system.

Group A holes are gaps where Lidar simply didn't sample, but the camera still has a clear line of sight (between rings, the near blind cone). Camera depth can fill these. ✅
Group B holes are things physically blocked from view — the road hidden behind a truck. No sensor can fill these, because no sensor can see through solid objects.

A lot of naive systems quietly pretend to fill Group B and hallucinate obstacles (or empty space) that they can't actually see. This project deliberately only fills Group A and leaves Group B honestly unknown. Knowing what you can't know is a safety feature.

4. The occupancy grid remembers — log-odds accumulation

The world is noisy: a point might flicker in one frame and vanish the next. Instead of trusting each frame blindly, every grid cell holds a running log-odds belief ("how confident am I that this cell is occupied?"). Each frame nudges that belief up (saw something here) or down (saw through it). Consistent observations build strong confidence; one-off noise fades away. It's the same math a robot uses to build a map while driving through it.

5. Tracking happens in meters, not pixels

Most trackers follow boxes around the image. This one tracks objects in the Ego frame in meters, using a Kalman filter (a constant-velocity motion model that predicts where things go next) and Hungarian matching (optimally pairing this frame's detections to existing tracks). Two payoffs: (a) predicted trajectories come out in real-world meters, ready to feed planning; and (b) it's naturally multi-camera — an object crossing from one camera's view to the next is still one object at one ground position, so the track survives the handoff.

6. The scale bridge — Lidar teaches the camera its scale

Monocular depth models output relative depth ("this is farther than that") but don't know the true scale in meters. When Lidar is present, its precise ranges are used to fit a per-frame affine correction (a simple scale-and-shift) onto the camera's depth. The result: the camera's gap-filling points line up metrically with the Lidar points around them, instead of floating at the wrong distance.

📺 Reading the demo (three panels)

The nuScenes demo shows three synchronized views side-by-side:

Panel A (Left) — Camera view + debug overlay: the raw camera with results projected back on top for inspection only.
- Bounding boxes: dynamic objects (cars = green, trucks = orange, pedestrians = yellow) with a tracking ID and the real metric distance (x, y) in meters from the car.
- Lane lines: light-grey lines tracing detected lane boundaries.
- Height bars: vertical bars from the ground to the top of each obstacle. Solid = precise Lidar measurement; dashed = camera estimate (the visual language of "measured vs. guessed" runs through the whole demo).
Panel B (Middle) — BEV occupancy grid: the top-down map, the real output.
- Ego vehicle: a white triangle pointing forward at bottom-center; concentric rings mark every 5 meters.
- Teal cells: obstacles measured directly by Lidar (high accuracy — true measurement).
- Amber cells: obstacles estimated from camera depth (gap-filling — prediction), covering Lidar's structural blind spots (Group A holes).
- Tracks: colored dots = current positions, trailing lines = movement history, leading white dashes = the Kalman-predicted future path.
Panel C (Right) — BEV height map (Anchored-Z): absolute obstacle height above the road, from 0.4m to 6.0m.
- Colormap: Green (low) → Yellow (mid) → Red (high), with a reference colorbar on the right.
- Lidar is drawn at full opacity; camera estimates are faded — again, measured vs. predicted at a glance.

🏗️ Architecture

Three clean layers, deliberately decoupled from the sensor layout. The trick that makes it scale: Layers 2 and 3 have no idea how many cameras exist — they only see Ego-frame data.

graph TD
    subgraph "Layer 1: Per-Camera Pipeline (repeated per camera)"
        C1[Camera 1] --> P1[YOLO / UFLDv2 / Depth] --> Proj1[IPM / Calib Project] --> EgoDets1[Ego-frame Detections]
        C2[Camera N] --> P2[YOLO / UFLDv2 / Depth] --> Proj2[IPM / Calib Project] --> EgoDets2[Ego-frame Detections]
    end

    subgraph "Layer 2: Fusion & Occupancy (camera-count agnostic)"
        EgoDets1 & EgoDets2 --> Fuse[Ego NMS / Fusion]
        Fuse --> Tracker[Ego-space Kalman Tracker]
    
        Lidar[Lidar Point Cloud] --> LidarOcc[Lidar Occupancy] --> OccGrid[Unified Occupancy Grid]
        Proj1 & Proj2 --> DepthOcc[Depth Occupancy] --> OccGrid
    end

    subgraph "Layer 3: Render"
        Tracker --> Render[BEV Renderer]
        OccGrid --> Render
        Render --> BEV[BEV Top-Down Output]
        Render --> Overlay[Debug Overlay on Camera]
    end

Per-Camera Layer (runs once per camera): 2D detection, lane segmentation, and depth. Each result is projected into the shared Ego frame (meters, centered on the car) using that camera's own calibration.
Fusion Layer (independent of camera count): merges detections from all cameras, removes duplicates in overlap zones via Ego-frame NMS, tracks objects with an Ego-space Kalman filter, and accumulates the sensor-agnostic Occupancy Grid.
Render Layer: draws the top-down BEV map and (for debugging) projects tracked bounds back onto each camera.

🌟 Key Features

Embedded-First (CPU only): built for GPU-less targets using ONNX models quantized to INT8. No torch/ultralytics at runtime.
Sensor-Agnostic Occupancy Grid: one grid fed by Lidar points (primary position spine) and dense monocular Depth Anything V2 points (gap-filling / fallback).
Ego-Space Tracking: constant-velocity Kalman tracker with Mahalanobis-distance gating and Hungarian matching, entirely in metric coordinates — never in image space.
Scale Calibration Bridge: fits a per-frame affine correction onto monocular depth using sparse Lidar ranges, so camera points sit at the right metric distance.
Fail-Soft Degradation: when Lidar is absent, degrades smoothly to a self-contained camera-only IPM mode — the only mode deployable today.

🛠️ Tech Stack & Model Zoo

Dependencies

Core Processing: numpy, scipy (Kalman/Hungarian tracker)
Deep Learning Inference: onnxruntime (strictly CPU on deployment targets)
Image/Video I/O: opencv-python
Config & CLI: pyyaml, tqdm

Model Zoo (place in `models/`)

Object Detector: YOLOv8 (ONNX, nano/small, INT8-quantized)
Lane Detector: Ultra-Fast-Lane-Detection-V2 (UFLDv2 ONNX)
Semantic Segmenter: lightweight CNN (e.g. BiSeNet/PIDNet ONNX)
Depth Estimator: Depth Anything V2 (ViT-S ONNX, CPU-friendly)

🚀 Getting Started

Prerequisites

Python 3.10+
macOS or Linux

Installation

Clone the repository:

git clone https://github.com/your-username/BEV-Perception.git
cd BEV-Perception

Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate

Install dependencies:
```
pip install -r requirements.txt
```
Download the models (and, for the nuScenes demo, the dataset) — see Assets & Data Setup below.

💡 Usage

📦 0. Assets & Data Setup (models + dataset)

The heavy files live outside git (models/ and data/ are gitignored), packaged on Google Drive as two archives:

Archive	Extracts to	Size	Needed for
`models.zip`	`models/` (the `.onnx` weights)	~1 GB	all modes
`nuscenes.zip`	`data/nuscenes/` (nuScenes-mini: 6 cams + Lidar GT)	~5 GB	the nuScenes demo only

📁 Source folder: Google Drive

Note

Both gdown and the nuScenes dataset are dev/eval only — neither ships to the embedded runtime.

Option A — download helper (recommended):

# One-time: install the dev-only download tool (not a runtime dependency)
./venv/bin/pip install gdown

# Grab everything (models + dataset), extract to the right folders, and verify
./venv/bin/python tools/download_assets.py

# ...or just one:
./venv/bin/python tools/download_assets.py --only models
./venv/bin/python tools/download_assets.py --only nuscenes

The script skips anything already present; add --force to re-download. Model paths must match configs/pipeline.yaml.

Option B — manual: download the two zips from the Drive folder and extract them so the trees look exactly like this:

models/                         data/nuscenes/
├── yolov8s.onnx                ├── maps/
├── deep_anything_v2.onnx       ├── samples/     # keyframe images (6 cams) + LIDAR_TOP + 5 radars
├── ufldv2_..._320x1600.onnx    ├── sweeps/      # intermediate frames between keyframes
├── pidnet_s_cityscapes.onnx    └── v1.0-mini/   # metadata JSON (sample.json, calibrated_sensor.json, ...)
└── ...

Verify either/both at any time (checks the tree without downloading):

./venv/bin/python tools/download_assets.py --verify-only

The official nuScenes copy is also at nuscenes.org/download (v1.0-mini) if you prefer the source; extract it to the same data/nuscenes/ path.

🚗 1. Multi-Sensor Evaluation on nuScenes (Recommended)

Important

This is the primary demo and evaluation mode. It uses the nuScenes-mini dataset (6 surround 360° cameras + ground-truth Lidar) to show multi-camera semantic fusion and Lidar-primary occupancy mapping.

Run the full pipeline on nuScenes-mini:

./venv/bin/python tools/run_nuscenes.py --config configs/pipeline.yaml

Quantify IPM projection error against ground-truth Lidar distances (this is how "the truck looks ~15m away by IPM but Lidar says ~9m" becomes an actual number):

./venv/bin/python tools/eval_ipm_error.py

📹 2. Dashcam 1-Camera Demo (Camera-Only / Fallback)

Simulates a single front camera on a GPU-less, Lidar-less chip — the real embedded deploy floor. It runs purely on camera input using flat-ground IPM and monocular depth.

Run on a custom dashcam video:

./venv/bin/python demo.py --video data/sample_dashcam.mp4 --config configs/pipeline.yaml

--save out.mp4: save the output video.
--no-show: disable the GUI window (for headless / batch runs).

📐 Camera Calibration Guide for Custom Videos

BEV accuracy lives and dies by calibration. Rather than hand-guessing focal lengths and mounting angles (whose errors compound), use make_calibration.py to compute a single 3×3 ground homography that captures intrinsics and extrinsics together.

What's a homography? It's the exact mathematical link between "a point on the flat road in the image" and "that point's real position in meters." Give it a few known correspondences and it learns to map the whole ground plane.

Step 1: Export a sample frame from your video

# Save frame 0 to a scratch folder
./venv/bin/python tools/make_calibration.py --video data/sample_dashcam.mp4 --frame 0 --image scratch/calib_frame.png

Step 2: Calibrate (pick one method)

Method 1: Guided Mode (beginner-friendly) — Recommended Calibrate from the four corners of a known rectangular lane segment on the road:
```
./venv/bin/python tools/make_calibration.py --image scratch/calib_frame.png --out configs/camera/dashcam_front.yaml --guided
```
1. The terminal asks for: the lane width (e.g. a standard 3.5 m), the distance from the car's front to the near edge of the rectangle (e.g. 6.0 m), and to the far edge (e.g. 20.0 m).
2. An image window opens. Click the 4 corners of your lane segment in this order: Near-Left → Near-Right → Far-Left → Far-Right (on-screen hints included).
3. Press ENTER or q to process and save.
Method 2: Interactive Mode (arbitrary points)
```
./venv/bin/python tools/make_calibration.py --image scratch/calib_frame.png --out configs/camera/dashcam_front.yaml --interactive
```
Click at least 4 ground-plane points, press ENTER, then type each point's real Ego position (x y in meters, e.g. -1.75 20.0).

Method 3: Headless Mode (YAML point files) Pre-define pixel and ground coordinates in a YAML file:

./venv/bin/python tools/make_calibration.py --image scratch/calib_frame.png --points configs/points_dashcam_front.yaml --out configs/camera/dashcam_front.yaml

Step 3: Verify the result

The tool writes your calibration file plus a check image configs/camera/dashcam_front.verify.png, showing a metric BEV grid re-projected onto the camera view. If the grid lines don't sit cleanly on the road markings, re-run and click more precisely.

🧪 3. Running Unit Tests

Tests live under test/ and run fully offline:

./venv/bin/python -m unittest discover -s test -v

🗺️ Roadmap

Branch A: Multi-Camera Expansion
- Phase 1: 1-camera pipeline validation (YOLO + Tracker + Depth + IPM)
- Phase 2: Per-camera calibration config; simulated multi-camera fusion
- Phase 3: Real 5–6 camera surround setup with LSS/BEVFormer
Branch B: Lidar-Primary Fusion (Active focus)
- Stage 1: nuScenes multi-modal loader & camera-Lidar alignment check
- Stage 2: IPM quantitative error evaluation vs. Lidar ground truth
- Stage 3: Lidar occupancy rasterization & integration into the OccupancyGrid
- Stage 4: Dynamic scale bridge & smooth degradation arbiter

📁 Project Structure

.
├── bev_tracker/              # Fusion & Ego-frame object tracking
│   ├── bev_tracker.py        # Track life-cycle management + Hungarian data association
│   ├── fusion.py             # Multi-camera detection fusion & overlap suppression (Ego NMS)
│   ├── kalman.py             # Constant-velocity Kalman filter in 2D ego-metric coordinates
│   └── types.py              # Shared data structures (Detection, Track, Lane, Landmark)
├── configs/                  # Calibration & pipeline configuration
│   ├── camera/               # Per-camera YAML calibration (intrinsic & extrinsic)
│   └── pipeline.yaml         # Main run config (model paths, thresholds, grid sizes)
├── data/                     # Local data workspace (dashcam clip or nuScenes-mini, gitignored)
├── datasets/                 # Dataset parsing & loading
│   └── nuscenes_loader.py    # Custom JSON parsing of nuScenes (no official devkit dependency)
├── models/                   # Exported ONNX model weights
├── perception/               # Per-Camera Layer & sensor interfaces
│   ├── calibration.py        # Projection logic & coordinate transforms (2D pixel <-> 3D Ego)
│   ├── camera_pipeline.py    # Sequential per-camera pipeline projecting to Ego coordinates
│   ├── depth.py              # Depth Anything V2 monocular depth interface (ONNX CPU)
│   ├── depth_mono_fallback.py# Geometric distance approximation when Lidar is offline
│   ├── depth_occupancy.py    # Projects monocular depth into 3D Ego occupancy points
│   ├── detector.py           # 2D object detection (YOLOv8 ONNX CPU)
│   ├── height.py             # Approximates target height from projected geometry
│   ├── ipm.py                # Inverse Perspective Mapping for the flat ground plane
│   ├── lane_detector.py      # Lane detection with UFLDv2 (ONNX)
│   ├── lane_fit.py           # Polynomial lane fitting & smoothing
│   ├── lidar_occupancy.py    # Filters & rasterizes Lidar point clouds into Ego occupancy points
│   ├── lidar_provider.py     # Matches sparse Lidar ranges to camera 2D detections
│   ├── scale_bridge.py       # Per-frame affine calibration of mono-depth scale from Lidar
│   ├── semantic_seg.py       # Road-scene semantic segmentation (building, vegetation, pole)
│   └── static_extractor.py   # Extracts static boundaries from the seg mask onto the BEV grid
├── render/                   # Visualization & BEV rendering
│   ├── bev_renderer.py       # Core BEV renderer (vehicles, lanes, landmarks, grid)
│   ├── debug_overlay.py      # Projects Ego 3D bounds & lanes back onto the camera for debug
│   ├── height_color.py       # Maps height values to color arrays for static models
│   └── occupancy_grid.py     # Binary + log-odds temporal grid mapping (OccupancyGrid)
├── scratch/                  # Temporary outputs, frames/videos, throwaway scripts (gitignored)
├── test/                     # Unit tests
├── tools/                    # Evaluation & dev utilities
│   ├── download_assets.py    # Fetches models.zip + nuscenes.zip from Google Drive & verifies (dev-only)
│   ├── eval_ipm_error.py     # Quantifies IPM error vs. ground-truth Lidar distances
│   ├── run_nuscenes.py       # Main evaluation driver over nuScenes-mini
│   └── quantize_onnx.py      # Model quantization helper (INT8)
└── requirements.txt          # Python dependencies

Navigation Menu

Repository files navigation

BEV-Perception

🤔 Why is this hard? (Read this first)

🧠 The big ideas (and the subtle bits)

1. Everything lives in the Ego frame — never on the camera image

2. Lidar is honest but sparse; camera is dense but guesses — so fuse them

3. The honesty about holes: Group A vs Group B

4. The occupancy grid remembers — log-odds accumulation

5. Tracking happens in meters, not pixels

6. The scale bridge — Lidar teaches the camera its scale

📺 Reading the demo (three panels)

📌 Table of Contents

🏗️ Architecture

🌟 Key Features

🛠️ Tech Stack & Model Zoo

Dependencies

Model Zoo (place in `models/`)

🚀 Getting Started

Prerequisites

Installation

💡 Usage

📦 0. Assets & Data Setup (models + dataset)

🚗 1. Multi-Sensor Evaluation on nuScenes (Recommended)

📹 2. Dashcam 1-Camera Demo (Camera-Only / Fallback)

📐 Camera Calibration Guide for Custom Videos

Step 1: Export a sample frame from your video

Step 2: Calibrate (pick one method)

Step 3: Verify the result

🧪 3. Running Unit Tests

🗺️ Roadmap

📁 Project Structure

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
bev_tracker		bev_tracker
configs		configs
datasets		datasets
perception		perception
render		render
test		test
tools		tools
.gitignore		.gitignore
README.md		README.md
README_vi.md		README_vi.md
demo.py		demo.py
nuscenes_demo.gif		nuscenes_demo.gif
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation

BEV-Perception

🤔 Why is this hard? (Read this first)

🧠 The big ideas (and the subtle bits)

1. Everything lives in the Ego frame — never on the camera image

2. Lidar is honest but sparse; camera is dense but guesses — so fuse them

3. The honesty about holes: Group A vs Group B

4. The occupancy grid remembers — log-odds accumulation

5. Tracking happens in meters, not pixels

6. The scale bridge — Lidar teaches the camera its scale

📺 Reading the demo (three panels)

📌 Table of Contents

🏗️ Architecture

🌟 Key Features

🛠️ Tech Stack & Model Zoo

Dependencies

Model Zoo (place in models/)

🚀 Getting Started

Prerequisites

Installation

💡 Usage

📦 0. Assets & Data Setup (models + dataset)

🚗 1. Multi-Sensor Evaluation on nuScenes (Recommended)

📹 2. Dashcam 1-Camera Demo (Camera-Only / Fallback)

📐 Camera Calibration Guide for Custom Videos

Step 1: Export a sample frame from your video

Step 2: Calibrate (pick one method)

Step 3: Verify the result

🧪 3. Running Unit Tests

🗺️ Roadmap

📁 Project Structure

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Model Zoo (place in `models/`)