Skip to content

feat(backend): run the Python backend as a subprocess (drop embedded libpython) - #27

Merged
HichTala merged 9 commits into
HichTala:masterfrom
leiteszeke:up/02-subprocess
Oct 10, 2026
Merged

HichTala merged 9 commits into
HichTala:masterfrom
leiteszeke:up/02-subprocess

Conversation

@leiteszeke

@leiteszeke leiteszeke commented Jun 28, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #26 (macOS build fix). Review/merge that first.

Summary

Stop embedding libpython in the plugin. Instead, launch the user's Python as a
separate process and communicate with it over shared memory. This is a
cross-platform change motivated by macOS, where embedding the interpreter is
not viable.

Motivation

Linking libpython into the plugin and running the backend in-process has two
problems:

  • On macOS it clashes with OBS' own obs-scripting Python symbols, and
    importing numpy/torch in-process crashes.
  • It forces the plugin to be built against — and ship with — one specific
    Python ABI, and pins the user to that exact interpreter version.

Running the backend as a subprocess removes the symbol clash entirely and lets
the user point the plugin at any compatible Python that has the draw package
installed.

Changes

  • CMakeLists.txt — drop find_package(Python3) and the include/link
    step. The plugin no longer embeds an interpreter.
  • src/DrawDock.{cpp,hpp} — launch the backend with QProcess; stream its
    stdout into a log panel in the dock; manage deck lists from a
    decklists folder in the OBS plugin config dir; start/stop lifecycle.
  • src/shared_memory_wrapper.cpp — shared-memory transport between the
    plugin and the backend. Cross-platform: shm_open/mmap on Unix/macOS,
    boost::interprocess::windows_shared_memory on Windows (matching Python's
    multiprocessing.shared_memory).
  • src/SettingsDialog.{cpp,hpp} — select the Python prefix and the
    deck-list files used at launch.

Single-player. Multi-detector / per-player support (a second detector and
per-player deck lists) is deferred to its own PR.

  • data/locale/en-US.ini — strings for the new dock controls.

Cross-platform

  • The shared-memory layer has a Windows path (windows_shared_memory) and a
    Unix path (shm_open/mmap); QProcess is cross-platform.
  • The 3-platform CI on this PR (macOS / Ubuntu / Windows) is the build check.

Behaviour change (worth a maintainer's attention)

The plugin no longer bundles a Python interpreter on any platform. Users now
select a Python install that has the draw backend package. This PR updates the
README (and all translations — es/fr/jp/pt-br) accordingly: building from
source is now a universal build, and the macOS Python-backend setup is documented
for this subprocess model.

How it was tested

Environment: macOS 26 (Apple Silicon), OBS Studio 32.1.1, a Python prefix with
the draw backend installed.

This PR's scope is the process model, not the detection pipeline (the
capture + model code in src/draw.c is untouched here), so the test confirms
the backend now runs out-of-process and the plugin talks to it:

Build — cmake --preset macos + cmake --build build_macos → universal
build succeeds
(no libpython link; the previous arm64-only constraint from
embedding Homebrew's Python is gone).

Universal build succeeds

Backend runs as a separate process — Start DRAW spawns the Python backend;
its stdout streams into the dock's log panel (impossible with an in-process
interpreter), the plugin and backend connect over shared memory
(Shared memory found), and OBS does not crash — the libpython/obs-scripting
symbol clash on macOS is gone.

Backend subprocess running, shared memory connected

Detection itself is unchanged by this PR (same draw.c capture + model as
before), so it is not re-validated here.

…libpython)

Embedding libpython in the plugin clashes with OBS' own obs-scripting
Python symbols on macOS and breaks numpy/torch in-process. Instead of
linking libpython, launch the user's Python as a separate process and
talk to it over shared memory.

- CMakeLists: drop find_package(Python3) and the include/link step; the
  plugin no longer embeds an interpreter.
- DrawDock: spawn the backend with QProcess, stream its stdout into a log
  panel in the dock, manage deck lists from a decklists folder in the OBS
  plugin config dir, and handle the start/stop lifecycle.
- shared_memory_wrapper: shared-memory transport between plugin and
  backend (shm_open/mmap on Unix, boost windows_shared_memory on Windows).
- SettingsDialog: Python prefix selection and deck-list pickers.
- README + all translations (es/fr/jp/pt): document building from source
  (now a universal build) and the macOS Python-backend setup, described
  for this subprocess model.

Single-player; multi-detector (per-player) support is a separate change.
@HichTala

HichTala commented Sep 5, 2026

Copy link
Copy Markdown
Owner

Sorry for taking so long to review this PR, but this change is important since it essentially sets the direction for others so it needs to be tested carefully; it also involves (from my understanding) an update to the Python backend to support the addition of the new channel we need to take care of.

Unsurprisingly, it works well on Linux; we just need to run additional tests on Windows to make sure it works and that it’s still possible to provide the plugin with the built-in “all-in-one” Python module via the installer.
This is an important point, because many Windows users are not developers and are not familiar with "technology"; having to build the plugin from source could therefore be a barrier to use that we want to avoid.

Thank you so much for the huge work you provided, I'll do my best to merge this one this week-end.

Comment thread src/shared_memory_wrapper.cpp Outdated
@HichTala

HichTala commented Sep 6, 2026

Copy link
Copy Markdown
Owner

This PR is almost ready to be merged, we still need to update the python backend in the obs branch of the original repo draw2

@HichTala

HichTala commented Oct 8, 2026

Copy link
Copy Markdown
Owner

Python backend in the obs branch of the original repo draw2 has been updated:

This PR is ready to be merged, I'll just conduct one last test on windows to be sure everything works correctly as expected.

@HichTala

HichTala commented Oct 9, 2026

Copy link
Copy Markdown
Owner

That works perfectly fine on windows as expected. I'll just take advantage of this PR to also update Linux installation instructions, I'll do that tomorrow and merge that right after

@HichTala

HichTala commented Oct 10, 2026 •

Copy link
Copy Markdown
Owner

I'm merging this PR when all the actions are complete, thank you for the work, if I made any mistake in the doc or anywhere else, we can take advantage of the next PR to correct them, I'm reviewing it right away!

Edit: Also I'll publish a new release when all the PR have been merged

@HichTala
HichTala merged commit fcc967a into HichTala:master Oct 10, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants