Skip to content

About

Multi-drone data center digital twin: thermal anomaly detection with OODA loop, Isaac Sim 5.1 + Pegasus + React dashboard

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

79 Commits

Folders and files

Repository files navigation

Project Starlink

Multi-drone data center digital twin — autonomous thermal anomaly detection with an OODA decision loop, built on NVIDIA Isaac Sim 5.1 + Pegasus + a Pandora-hybrid React dashboard.

Isaac Sim Pegasus ROS 2 Jazzy React 19 FastAPI TypeScript Vite Python NVIDIA RTX 4090 License: MIT

Foundation codebase for HelioViews.ai. Author: Mel Torres, CMU 12-831.


Demo

Dashboard demo — Day 7 Pandora-hybrid Replace this image with your operator-recorded class screen capture (Chrome at :5173 showing the live OODA cycle).


What this is

Project Starlink is a high-fidelity simulated digital twin of a Stargate-class data center, with an autonomous inspection drone that detects and responds to thermal anomalies. The simulation runs on NVIDIA Isaac Sim 5.1 with a Pegasus-flown Iris quadcopter, full PhysX collision, an aisle-aware path planner, and a stochastic 12-rack thermal field. Operator visibility is a live React dashboard fed by a FastAPI SSE bridge over a wire format pre-committed to NVIDIA's Omniverse Kit App Streaming schema.

What was built

Twelve sprint days, seven of them shipping a complete simulation + dashboard stack: Isaac Sim 5.1 in Docker on WSL2, ROS 2 Jazzy bridge with OmniGraph sensor publishers, a Pegasus-subclassed OODABackend running PID + SO(3) attitude control at 50 Hz with a PATROL → INVESTIGATE → RETURN state machine, twelve PhysX-collision-boxed server racks built from measured USD bounding-box geometry, an aisle-aware WaypointRouter with fault-tolerant emergency-fallback re-planning (validated in vivo), an MPC thermal optimizer driving cooling actions during INVESTIGATE phases, and a six-panel React dashboard with rotation-aware SSE tail handling WSL2 v9fs filesystem inode reuse. Ninety unit + integration tests, deterministic replay via seeded wind RNG and world.stop() lifecycle hardening, dual-runtime validation (Docker headless CI + Windows GUI demo).

What's novel

Three things distinguish Starlink from a tutorial-grade Isaac Sim project:

  1. Wire-format pre-commitment. Every event the simulation emits — thermal_alert, phase_change, telemetry, agent_diagnostic — uses the {event_type, payload} shape from NVIDIA's omni.kit.livestream.messaging extension verbatim. The dashboard's transport (today: FastAPI SSE; tomorrow: WebRTC via NVIDIA's OVAppStreamer) is decoupled from the schema. The Day 8/Stargate Kit App Streaming migration is a transport swap, not a schema change.
  2. Deterministic replay. A single --seed 42 flag gives byte-identical events.jsonl across runs (verified). The Day 6 hardening commit isolated this with a seeded global numpy RNG and world.stop() before world.reset() to tear down stale physics-step subscriptions.
  3. Empirically validated fault-tolerance. During Windows GUI dual-runtime testing, the drone hit a transit-zone safety assertion mid-cross-aisle under non-deterministic GPU-contended timing — and the emergency-fallback re-planner recovered autonomously to a safe patrol node. This is in-vivo evidence of constraint-satisfaction recovery under realistic rendering load, not a contrived benchmark.

Quick start

Headless Docker run (CI, testing, batch validation)

# WSL2 with Docker Engine + nvidia-container-toolkit
docker compose build
docker compose run starlink

# Inside the container:
bash scripts/run_tests.sh tests/                                            # 90 tests pass
/isaac-sim/python.sh scripts/build_datacenter_scene.py                      # one-time scene build
/isaac-sim/python.sh scripts/starlink_standalone.py --headless --num-steps 1000
/isaac-sim/python.sh scripts/validate_hover.py --log logs/ooda_flight_log.npz

See docs/setup-guide.md for the full Day 1 environment bring-up.

Live operator dashboard (Day 7 Pandora-hybrid — three terminals)

Port 8100 is not mapped in docker-compose.yml, so the FastAPI bridge runs on the WSL host (not inside the container). Chrome on Windows reaches both :5173 and :8100 via WSL localhost forwarding.

One-time setup:

python3 -m pip install --user -r palantir_integration/requirements.txt    # FastAPI, uvicorn, aiofiles
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
source ~/.bashrc
nvm install --lts                                                          # WSL-native Node, not Windows interop

Run:

# Terminal 1 — WSL — FastAPI SSE bridge on :8100
cd ~/project/starlink
STARLINK_EVENTS_LOG=/mnt/c/starlink_win/results/events.jsonl \
  python3 -m uvicorn palantir_integration.event_service:app \
  --host 0.0.0.0 --port 8100

# Terminal 2 — WSL — React dashboard on :5173
cd ~/project/starlink/webapp
export NVM_DIR="$HOME/.nvm" && . "$NVM_DIR/nvm.sh"
npm install                                                                # first run only
npm run dev
# Terminal 3 — Windows PowerShell — Isaac Sim launcher (deterministic replay)
cd C:\isaacsim
.\python.bat C:\starlink_win\scripts\starlink_windows.py --num-steps 5000 --seed 42

Open Chrome to http://localhost:5173. Start order does not matter — the SSE bridge is rotation-aware (inode + size-shrink detection works on both ext4 and WSL /mnt/c/ v9fs).

Architecture

Two-layer architecture with a deliberately frozen wire-format contract between them. See docs/architecture.md for the full component map (Layer 1 = today's Pandora-hybrid; Layer 2 = Day 8 Kit App Streaming target). OODA decision logic operates at three timescales:

  1. PID Flight Controller (50 Hz) — position tracking + wind-disturbance rejection
  2. MPC Thermal Optimizer (on-event) — per-rack cooling optimization during INVESTIGATE
  3. OODA State Machine (1 Hz) — PATROL → INVESTIGATE → RETURN with fault-tolerant fallback

Sprint summary (12 days × 6 hours)

Day Milestone Status
0 Project scaffolding ✅
1 Isaac Sim 5.1 + Pegasus v5.1 + ROS 2 Jazzy environment validated ✅
2 ROS 2 Jazzy bridge + OmniGraph sensor pipeline verified ✅
3 OODABackend — drone flies under PID + SO(3) attitude control ✅
4 Data center scene + AC wind disturbance + thermal visualization + PUE ✅
5 Full stochastic thermal model + MPC optimizer + dual-runtime validation ✅
6 Visual validation + multi-aisle PhysX (bbox-measured geometry, aisle-aware routing, fault-tolerant fallback) ✅
7 Pandora-hybrid React dashboard + FastAPI SSE bridge + wire-format pre-commit ✅
8–12 Kit App Streaming migration + paper production + final demo ⏭️ Deferred to Project Stargate

The Day 8–12 work — Kit App Streaming with OVAppStreamer, bidirectional WebRTC data channel, optional LLM chat panel, ACM sigconf paper, final demo recording — is captured for a follow-on sibling project. See docs/stargate-handoff.md for the coupling contract and kickoff prompt. Starlink itself ships at Day 7.

Repository structure

starlink/
├── CLAUDE.md                           # Living governing doc
├── PROJECT_STARLINK_COMPLETE_V5_1.md   # Authoritative master plan
├── README.md                           # You are here
├── Dockerfile, docker-compose.yml      # Isaac Sim 5.1 NGC container + GPU passthrough
├── thermalsight_ooda/                  # OODA stack — backend, PID, MPC, thermal field, router, event emitter
├── thermalsight_agent/                 # Agent scaffold (Future Work — moved to Stargate)
├── palantir_integration/               # FastAPI SSE bridge + (Day 12 stretch) Foundry client
├── webapp/                             # React 19 + TS + Vite Pandora-hybrid dashboard
│   └── src/{App.tsx, hooks/, panels/, theme/, types/, util/}
├── scripts/                            # Standalone + Windows launchers, scene build, validation
├── tests/                              # 90 tests — pure-Python OODA logic, no Isaac Sim required
├── scenes/, config/, assets/, deps/    # USD scenes, layout/scenarios, NVIDIA asset pack, Pegasus submodule
└── docs/                               # Architecture, day specs, retrospectives, Stargate handoff

License

MIT

Author + acknowledgments

Mel Torres — CMU 12-831 graduate, founder of HelioViews.ai.

Built on the shoulders of:

  • NVIDIA Isaac Sim 5.1 + PhysX 5.4 + Omniverse Kit 107.3 — physics, rendering, sensor sim.
  • Pegasus Simulator v5.1.0 — open-source quadcopter dynamics + flight backend ABC, used as the OODABackend parent class.
  • NVIDIA Data Center Asset Pack — 33 USD models for realistic rack geometry.
  • NVIDIA GTC 2026 lab session "Develop AI-Powered Operational Dashboards for Digital Twin Applications" (Ashley Goldstein + Justine Lin) — reference architecture for the Pandora-hybrid wire format and the Day 8 Kit App Streaming migration target.
  • NVIDIA NIM API — Nemotron Nano 30B-A3B endpoint, scoped for Stargate (Future Work).
  • CMU 12-831 Decision-Making Algorithms for Autonomous Systems — coursework framework that motivated the four-algorithm OODA stack (MPC, aisle-aware path planning, rule-based state machine, fault-tolerant re-planner).

About

Multi-drone data center digital twin: thermal anomaly detection with OODA loop, Isaac Sim 5.1 + Pegasus + React dashboard

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages