System for 3D models overlay and real-time surgical video tracking of organs.
This is an Unity3D project that communicates with a local Python server in order to overlay patients 3D models generated with TensorFlow (check out this repo aledelmo/APMRI-DNN) to the corresponding surgical operation video. Communication to the server established using ZeroMQ. Organs movements are real-time tracked from video frames using OpenCV. 3D models position is adjusted accordingly.
Before building the project download models and video using this link NOT_AVAILABLE and put them in Assets/Data folder.
- Build the app in Unity3D.
- Create the Python environment and install requirements
- Start the ØMQ server.
- Establish connection between app and server.
Unity3D 2019.4.18f1 or later
Python 3.8.6
If connection to the server is lost during tracking, the app must be restarted.
Alessandro Delmonte - alessandro.delmonte@institutimagine.org
This project is licensed under the Apache License 2.0 - see the LICENSE.md file for details