Skip to content

Add core RTP-over-WebSocket (rtpws) for AudioBridge WebSocket media#3653

Open
mirkobrankovic82 wants to merge 4 commits into
meetecho:masterfrom
mirkobrankovic82:feature/audiobridge-rtpws
Open

Add core RTP-over-WebSocket (rtpws) for AudioBridge WebSocket media#3653
mirkobrankovic82 wants to merge 4 commits into
meetecho:masterfrom
mirkobrankovic82:feature/audiobridge-rtpws

Conversation

@mirkobrankovic82

@mirkobrankovic82 mirkobrankovic82 commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add Janus core rtpws listener for binary RTP over WebSocket (janus.jcfg rtpws section).
  • Wire AudioBridge media=websocket joins to rtpws peer bindings and websocket_media.url.
  • Honor the AudioBridge join codec (opus, pcma, pcmu, l16, l16-48) when creating the rtpws peer — call_info reports the negotiated codec, payload type, sample rate, and framing.
  • Add L16 linear PCM support (l16 @ 16 kHz, l16-48 @ 48 kHz) on plain RTP and media=websocket; decode/encode big-endian slinear and restrict L16 joins to a matching room sampling_rate.
  • Add optional per-participant ws_framing on media=websocket joins — rtp (default, full RTP packets, unchanged) or payload (raw codec payloads, no RTP header). In payload mode the core strips the RTP header outbound and synthesizes one inbound (monotonic seq/timestamp/SSRC), so external AI/STT/TTS clients can exchange plain codec bytes without parsing or emitting RTP.
  • Include ws-audio-test Go CLI and shell wrapper for plain RTP / WS media validation (--codec selects opus / pcma / pcmu / l16 / l16-48; --framing selects rtp / payload; G.711 and L16 tone encoding).

Notes

  • Requires --enable-websockets at build time.
  • AudioBridge rooms need allow_ws_participants = true at create time (cannot be enabled later via edit).
  • Clients must send RTP matching the server call_info (codec, PT, sample rate).
  • l16 requires a room sampling_rate of 16000 and l16-48 requires 48000; mismatched joins are rejected.
  • ws_framing is negotiated per join (no global config); payload framing pairs well with l16/l16-48 to hand STT/TTS providers raw 16-bit PCM directly.

Example (Opus)

go run . ws-stream   --janus-http http://127.0.0.1:8088/janus   --token "$TOKEN"   --room "$ROOM"   --duration 5s
23:23:56.642157 websocket_media.url = ws://127.0.0.1:8190/ws-audio?sid=5337f2ef-bd0a-42d2-ad7a-d7826889b5b7
23:23:56.971244 [ws-stream] connected ws://127.0.0.1:8190/ws-audio?sid=5337f2ef-bd0a-42d2-ad7a-d7826889b5b7
23:23:56.971895 [ws-stream] server: {"type":"call_info","codec":"opus","sample_rate":16000,"channels":1,"ptime_ms":20,"payload_type":100}
23:23:56.972449 [ws-stream] rx seq=1 ts=0 pt=0 ssrc=1372789150 (20 B)
23:23:57.542010 [ws-stream] rx seq=51 ts=48000 pt=0 ssrc=1372789150 (20 B)
23:23:58.543518 [ws-stream] rx seq=101 ts=96000 pt=0 ssrc=1372789150 (20 B)
23:23:58.973112 [ws-stream] tx=99 pkts (17028 B) | rx=122 pkts (2440 B) | rx_bad=0
23:23:59.543301 [ws-stream] rx seq=151 ts=144000 pt=0 ssrc=1372789150 (20 B)
23:24:00.542306 [ws-stream] rx seq=201 ts=192000 pt=0 ssrc=1372789150 (20 B)
23:24:00.972743 [ws-stream] tx=199 pkts (34228 B) | rx=222 pkts (4440 B) | rx_bad=0
23:24:01.544217 [ws-stream] rx seq=251 ts=240000 pt=0 ssrc=1372789150 (20 B)
23:24:01.995794 [ws-stream] tx=249 pkts (42828 B) | rx=273 pkts (5460 B) | rx_bad=0

Example (G.711 μ-law)

go run . ws-stream   --janus-http http://127.0.0.1:8088/janus   --token "$TOKEN"   --room "$ROOM"   --codec pcmu   --duration 5s
# call_info: {"type":"call_info","codec":"pcmu","sample_rate":8000,"channels":1,"ptime_ms":20,"payload_type":0,"framing":"rtp"}

Example (L16 / 16 kHz)

go run . ws-stream   --janus-http http://127.0.0.1:8088/janus   --token "$TOKEN"   --room "$ROOM"   --codec l16   --duration 5s
# call_info: {"type":"call_info","codec":"l16","sample_rate":16000,"channels":1,"ptime_ms":20,"payload_type":106,"framing":"rtp"}
# full RTP frames on the wire: 12-byte header + 640-byte L16 payload (320 samples * 2 bytes)

Example (payload framing, raw L16 for AI/STT/TTS)

go run . ws-stream   --janus-http http://127.0.0.1:8088/janus   --token "$TOKEN"   --room "$ROOM"   --codec l16   --framing payload   --duration 5s
# call_info: {"type":"call_info","codec":"l16","sample_rate":16000,"channels":1,"ptime_ms":20,"payload_type":106,"framing":"payload"}
# raw codec payloads on the wire (no RTP header): 640-byte L16 frames in and out
image

Add a Janus core rtpws listener for binary RTP over WebSocket (janus.jcfg
`rtpws` section) and wire AudioBridge `media=websocket` joins to rtpws peer
bindings and `websocket_media.url`.

- Honor the AudioBridge join codec (opus, pcma, pcmu); call_info reports the
  negotiated codec, payload type and sample rate.
- Keep the Janus session alive from ws-media RTP so HTTP-only join clients are
  not torn down by session_timeout while media is flowing (via the plugin
  touch_session callback, so the plugin build does not pull in ice.h).
- Optional rtpws::allow_ws_bind (default off) for browser-first
  connect-then-bind: a client may connect without ?sid=, receive the pod's
  server-name, and bind an AudioBridge-created sid over the same WS.
- Include ws-audio-test Go CLI and shell wrapper for RTP/WS validation.

Requires --enable-websockets at build time. Rooms need allow_ws_participants
set at create time.

Co-authored-by: Cursor <cursoragent@cursor.com>
@mirkobrankovic82
mirkobrankovic82 force-pushed the feature/audiobridge-rtpws branch from f805c53 to 7b0d27b Compare July 16, 2026 09:01
mirkobrankovic82 and others added 2 commits July 16, 2026 17:39
Support l16 and l16-48 on join, decode/encode, plain RTP and rtpws
call_info with room sampling_rate validation; extend ws-audio-test.

Co-authored-by: Cursor <cursoragent@cursor.com>
Add a per-participant "ws_framing" option to media=websocket AudioBridge
joins: "rtp" (default, full RTP packets on the wire, unchanged) or
"payload" (raw codec payloads, no RTP header). In payload mode the core
strips the RTP header outbound and synthesizes one inbound (monotonic
seq/timestamp/SSRC), so external AI/STT/TTS clients can exchange plain
codec bytes without parsing or emitting RTP. The negotiated framing is
advertised in the call_info handshake.

Pairs well with the L16 linear-PCM codec: payload framing + l16/l16-48
yields raw 16-bit PCM that most STT/TTS providers accept directly.

The ws-audio-test client gains --framing payload and --codec l16/l16-48.

Co-authored-by: Cursor <cursoragent@cursor.com>
…-payload

AudioBridge rtpws: L16 codec + optional payload-only framing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant