EEG-powered VR navigation using the Muse 2 headband and Unity. Imagine turning left, turning right, or walking forward to navigate the environment.
The native macOS app, BCI Navigator, guides you through headband setup, EEG recording, personal model training, and live control.
Real-time brainwave decoding — EEG, band powers, and model feedback.
VR navigation — Quest indoor navigation and prediction output.
| Brain Signal | Action in FreeNavScene |
Model support |
|---|---|---|
| Left motor imagery | Rotate left | Group and personal models |
| Right motor imagery | Rotate right | Group and personal models |
| Forward walking imagery | Move forward | Personal model trained with eligible Walk trials |
| Intentional blink | Detected; interaction not yet mapped | Blink classifier |
| Idle / baseline | No action | Personal models can learn idle from Relax trials |
The separate four-direction pipeline also supports backward movement with a compatible model. See Classification Results for offline evaluation scores.
- Muse 2 headband — 4 dry EEG electrodes at 256 Hz via Bluetooth
- Channels: TP9 (left temporal), AF7 (left frontal), AF8 (right frontal), TP10 (right temporal)
- Interface: BrainFlow SDK
- Download
BCINavigator-mac.dmgfrom GitHub Releases and install the app. Requires macOS 26 or later. - In this repository, run
uv sync --python 3.11. - Open the app's admin view with ⇧⌘A. Set Settings → Repo folder to your checkout and Python interpreter to
<checkout>/.venv/bin/python. - Sign in, connect the Muse 2, check sensor quality, and record a session.
- Under Real-time control → Get ready, choose Train Your Model or use the Group model.
- Start live control, then choose Open VR environment to launch BCI Indoor on your Mac.
Recording includes left/right imagery, forward Walk and Relax trials, and deliberate blinks, with recorded voice cues. Forward trials are eligible for training only when their block's imagery check approves them. Personal training performance still requires live validation.
See the app setup instructions for installation and the environment build guide for the Mac, Windows, and Quest packages.
Use this workflow to collect data with PsychoPy, train the historical classifiers, and inspect predictions in the PyQt feedback GUI.
# 1. Setup (Python 3.11; install uv first: https://docs.astral.sh/uv/getting-started/installation/)
uv sync --python 3.11
# 2. Get the recorded data (NOT committed to this repo)
# Download the Muse_Data zip from Google Drive, then unzip into data/:
# https://drive.google.com/drive/folders/1bHQ3Cm5NYfPvOb-EGZXeE6YcODsiLqth?usp=sharing
unzip ~/Downloads/Muse_Data-*.zip -d data/
# See data/README.md for the expected layout.
# 3. (Optional) Collect new data — requires a Muse headband
# --serial: Muse Bluetooth name (Muse-15C3 or Muse-12A6)
# Omit --serial to get an in-app device selection screen
# without hardware: add --simulate
uv run python src/pilot_data_collection/psychopy_recording/muse_psychopy_recording_structured.py --name test --number 9 --serial Muse-15C3
# 4. Train classifiers
uv run python all_ml_models/realtime_feedback/train_and_save_models.py
# 5. Run real-time feedback
uv run python all_ml_models/realtime_feedback/realtime_feedback.py # with Muse
uv run python all_ml_models/realtime_feedback/realtime_feedback.py --serial Muse-15C3 # specific Muse
uv run python all_ml_models/realtime_feedback/realtime_feedback.py --simulate # without hardware
# Optional: check the Muse connection by recording 10 seconds to temp/
uv run python src/pilot_data_collection/stream_save_muse.py
# Select a specific headband if more than one Muse is nearby:
uv run python src/pilot_data_collection/stream_save_muse.py --serial Muse-15C3Check TP9, AF7, AF8, and TP10 individually before recording:
uv run python -m src.signal_quality --serial 15C3
# Without a headband:
uv run python -m src.signal_quality --simulate --seconds 10The backend reports signal quality separately from transport/recording errors. The current recorder and Unity inference pipeline include live monitoring. See signal-quality setup, thresholds, and validation.
Standalone environment-one builds and instructions are in
releases/environment_1/. The Mac and Windows
apps support keyboard/mouse navigation without a headset or Unity Editor.
The Quest APK uses VR tracking. Python inference remains a separate process.
With the Muse 2 on and the Quest app running FreeNavScene, connect the Quest
over USB and forward its prediction socket:
adb reverse tcp:8765 tcp:8765From the repository root, start live EEG inference with the existing model:
uv run python -m pipeline realtime \
--model ml_pipeline/models/realtime_models.pkl \
--skip-personalize \
--serial 15C3 \
--mi-threshold 0.70 \
--blink-threshold 0.85The packaged APK connects to 127.0.0.1:8765. Use the USB adb reverse
command above to reach the prediction server on your computer. This APK does
not have a runtime Wi-Fi host selector. A direct LAN connection requires
changing the prediction host in unity/environment_1/ and rebuilding; see the
source and build instructions.
The desktop apps connect to the prediction server on the same computer.
--serial accepts either 15C3 or Muse-15C3. An explicit --model needs no
--participant. The included model is a legacy two-class bundle for left/right
turns. BCI Navigator personal models can add mi_forward and idle from eligible
Walk and Relax recordings. The separate four-direction pipeline
uses mi_forward, mi_backward, mi_rotate_left, and mi_rotate_right for
forward/backward movement and turning.
--mi-threshold 0.70 requires at least 70% model confidence before recognizing
a motor imagery direction. You can adjust it; the default is 0.55. It also
applies to the included left/right model.
--blink-threshold 0.85 requires at least 85% blink probability before
recognizing an intentional blink; the default is 0.60.
The Python server sends at most one valid navigation command every 3 seconds. The scene moves for up to 0.5 seconds after an accepted frame; unstable, low-confidence, stale, or disconnected input does not sustain movement.
BCI Navigator coordinates recording, personal training, and live inference through
Python helpers. Predictions stream over WebSocket port 8765 to Unity's
PredictionWebSocketClient and FreeNavController on desktop or Quest.
The alternative PyQt feedback workflow is shown below:
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Muse 2 EEG │────>│ BrainFlow SDK │────>│ Processing │
│ 4ch @ 256 Hz │ BT │ Ring Buffer │ │ Pipeline │
└─────────────────┘ └──────────────────┘ └────────┬────────┘
│
┌─────────────────────────────────┤
│ │
┌──────▼───────┐ ┌────────▼────────┐
│ Blink Detect │ │ Motor Imagery │
│ LDA, 500ms │ │ RF, 3s window │
└──────┬───────┘ └────────┬────────┘
│ │
└─────────────┬───────────────────┘
│
┌───────▼───────┐
│ PyQt6 GUI │
│ Real-time │
│ Feedback │
└───────────────┘
-
Hardware / Streaming — BrainFlow interface for Muse 2. Streams EEG to CSV or rebroadcasts over multicast for multiple consumers.
-
Experiment / Recording — BCI Navigator provides guided in-app recording with voice cues; the alternative PsychoPy workflow uses visual cues. Both save paired EEG and trial-log CSVs. See Data Collection and Experiment Procedure.
-
Analysis / ML — Signal processing (MNE bandpass/notch filtering, artifact removal), feature extraction (band power, Hjorth, CSP, time-frequency), and classification (LDA, SVM, RF). Three pipeline versions tested across 4 subjects.
macos_app/ # BCI Navigator app and Python helpers
pipeline/ # Collection, training, personalization, realtime CLI
ml_pipeline/ # Models and feature pipeline used by Quest inference
eeg_to_meta/ # Muse acquisition and prediction WebSocket server
unity/environment_1/ # Unity source for the packaged navigation environment
releases/environment_1/ # Standalone Mac, Windows, and Quest builds
all_ml_models/
training/ # Offline ML training pipelines
bandpower/
train_motor_imagery.py # v1: band power + Hjorth + CSP
timefreq/
train_motor_imagery_v2_timefreq.py # v2: + time-frequency features
eegnet/
train_motor_imagery_v3_eegnet.py # v3: + EEGNet-inspired features
realtime_feedback/ # Real-time feedback system
train_and_save_models.py # Train & save models for real-time use
realtime_feedback.py # PyQt6 GUI with live classification
train_blink_detector.py # Blink detection pipeline
models/
realtime_models.pkl # Trained model bundle for real-time use
src/
signal_processing/
eeg_filters.py # MNE-based bandpass, notch, artifact removal
pilot_data_collection/ # Pilot EEG data collection scripts
psychopy_recording/
muse_psychopy_recording_structured.py # Data collection experiment
stream_save_muse.py # Basic Muse streaming to CSV
rebroadcast_producer.py # Multicast EEG streaming (producer)
rebroadcast_consumer.py # Multicast EEG streaming (consumer)
muse_eeg_gui.py # Legacy PyQt6 GUI (superseded)
realtime_monitoring/
stream_realtime_bandpower.py # Live band power computation
visualize_data_live.py # Real-time EEG + band power plots
data/ # Downloaded from Google Drive — see data/README.md
README.md
Muse_Data/
Phase1/sub{NN}/ # Per-subject session
eeg_data_YYYYMMDD_HHMMSS.csv # Raw EEG (timestamp + 4 channels)
trial_log_YYYYMMDD_HHMMSS.csv # Trial metadata
metadata.yaml
Phase2/sub{NN}/ # Repeat sessions (currently sub03, sub05)
data_analysis/ # See data_analysis/README.md
_common.py # Shared discovery + loading helpers
raw/ # Per-subject raw-signal views
generate.py # plot_01/02/04
signal_level/ # PSD + band power, per/cross-subject
generate.py # plot_03/06/07/08 + cross_subject_*
ml/
results.md # Classification results write-up
The app records left/right imagery, forward Walk and Relax trials, and deliberate blinks. Defaults are 20 hand-imagery trials, 20 Walk + 20 Relax trials, and 10 blinks. The walking block includes instructions, practice, and an explain-back check; Walk and Relax trials last four seconds.
Sessions save paired EEG and trial-log CSVs in data/subNN/, plus
imagery_check_<timestamp>.json for walking checks. Personal training uses approved
walking blocks and saves each participant's model to
ml_pipeline/models/participants/subNN_app.pkl. Hand-only recordings retain the
left/right model. See the app guide for recording and training.
The PsychoPy workflow records two CSVs per session:
EEG data — continuous 4-channel EEG with timestamps aligned to BrainFlow's session clock:
timestamp, marker_code, marker_label, EEG_0(TP9), EEG_1(AF7), EEG_2(AF8), EEG_3(TP10)
Trial log — one row per trial with consistent schema:
trial_id, trial_type, label, start_time, end_time, event_marker, duration, session_id
Trial types: motor_imagery_left, motor_imagery_right, blink_intentional, baseline_quiet, baseline_active
Motor imagery epochs are interval-based (cue onset to spacebar press, 2–5s). Blink epochs are event-centered (500ms window around button press).
Historical evaluation: 4 subjects (sub02–sub05), 77 MI epochs, 190 blink epochs. Full breakdown in data_analysis/ml/results.md.
| Evaluation | Config | Classifier | Accuracy |
|---|---|---|---|
| Within-subject | all 4 channels | LDA | 88.7% |
| Cross-subject (LOSO) | all 4 channels | LDA | 86.0% |
| Evaluation | Pipeline | Config | Classifier | Accuracy |
|---|---|---|---|---|
| Within-subject | v1 Band Power | all 4 channels | SVM-rbf | 61.0% |
| Cross-subject (LOSO) | v3 EEGNet | temporal only | RF | 62.2% |
Three feature pipelines were compared:
- v1 — Band power + Hjorth + lateralization asymmetry + CSP
- v2 — v1 + sliding-window time-frequency (ERD/ERS dynamics)
- v3 — v2 + EEGNet-inspired fixed spatial/temporal features
Key findings: Blink detection outperformed motor imagery in this offline evaluation. Motor imagery is near chance with 4 subjects — needs more data or paradigm refinement. These scores do not establish live Quest performance.
The newer saved pipeline evaluation includes 11 subject IDs, 54 MI epochs, and 565 blink epochs. Random Forest MI scores are 52.39% within-subject and 45.66% LOSO; LDA blink scores are 89.41% and 82.07%, respectively. These are binary MI/blink offline results, not four-class navigation results.
The real-time GUI (all_ml_models/realtime_feedback/realtime_feedback.py) provides live brain state feedback:
- Blink detection — 500ms sliding window, LDA classifier, ~1s cooldown
- Motor imagery — 3s sliding window, Random Forest with CSP features
- Band powers — live delta/theta/mu/alpha/beta/gamma visualization
- Raw EEG — 4-channel waveform display (last 5 seconds)
Adjustable confidence thresholds via the GUI. Blink detection has priority over motor imagery each update cycle (5 Hz).
The WebSocket connection carries model predictions from Python to Unity so EEG processing runs on the computer while navigation runs on the Quest. Raw EEG stays in the Python pipeline; the prediction socket sends class labels, confidence scores, and pipeline state events.
| Socket | Direction | Purpose |
|---|---|---|
8765 — predictions |
Python → Unity | Navigation predictions and lifecycle events |
8766 — markers |
Unity → Python | Trial markers and run/session boundaries during data collection |
JsonWSServer broadcasts one UTF-8 JSON
message per WebSocket frame to connected clients. Unity's
PredictionWebSocketClient
connects to ws://<host>:8765 and passes predictions to FreeNavController.
With automatic connection enabled, it retries disconnected connections every
2 seconds. See Run the Unity Navigation System
above for USB and Wi-Fi setup.
Example left-turn prediction:
{
"type": "prediction",
"timestamp": 1711234567.89,
"predicted_class": "left_motor_imagery",
"confidence": 0.93,
"raw_probs": {"left": 0.93, "right": 0.07},
"stable": true,
"key_hint": "LeftArrow"
}- Class and confidence —
predicted_classidentifies the action;confidenceis a score from 0 to 1. Legacy left/right labels rotate the player. BCI Navigator personal models can addmi_forward; the separate four-direction models usemi_forward,mi_backward,mi_rotate_left, andmi_rotate_right. - Stability and timing — navigation predictions must be stable and meet the confidence threshold. The server sends at most one navigation command every 3 seconds and omits
idleframes. Unity rejects stale timestamps and stops movement when an accepted command expires. - Diagnostics —
raw_probscontains per-class probabilities.key_hintis advisory; navigation uses the class label directly, without injecting keyboard input. - Pipeline state — messages with
type: "state"report lifecycle transitions such aspersonalizing_start,personalizing_end, andrealtime_readyseparately from predictions.
After configuring the Quest connection, run the synthetic prediction server from the repository root:
uv run python tools/smoke_predictions.py
# For Wi-Fi, add: --host 0.0.0.0Wait for Quest connected, then enter forward, backward, left, or right.
Use low and unstable to check that rejected predictions do not move the
player. This checks transport and Unity movement independently of EEG decoding.
Enter q before starting live inference; both servers use port 8765.
| Key | Action |
|---|---|
| Space | Start / stop classification |
| Q | Quit |
Desktop MI preprocessing is applied per-channel:
- NaN/Inf sanitization
- Moving artifact removal (rolling z-score, 3-sigma threshold, 0.5s window)
- Linear detrend
- Notch filter (60 Hz powerline removal)
- Bandpass filter (1–40 Hz, 4th-order Butterworth)
The Quest runtime uses separate MI (1–40 Hz) and blink (0.5–10 Hz) bandpasses, with 2-second MI and 0.5-second blink windows; see eeg_to_meta/config.py.
Dependencies are declared in pyproject.toml and resolved in uv.lock. Install them with uv sync --python 3.11.
brainflow— EEG acquisitionmne— Signal filteringscikit-learn— ML classifiersPyQt6,pyqtgraph— Real-time GUIpsychopy— Experiment paradigmnumpy,scipy,pandas— Numericstorch— EEGNet features (v3 pipeline only)
| Name | Role |
|---|---|
| Ashee | BCI Lead |
| Spencer | BCI Lead |
| Alan | BCI Sub-team member |
| Michelle | BCI Sub-team member |
| James | BCI Sub-team member |
Cornell XR Project Team

