We recently talked about the efforts we devoted on tinkering with real-time media in MoQ, and in particular on how to bridge that to WebRTC and back. That post focused on our imquic library, the native demos we wrote to do real-time media, and the Janus branch hosting a preliminary WebRTC/MoQ gateway, which was basically the main topic of my latest CommCon presentation.
While at CommCon, though, new ideas for what could be done with MoQ came out, and one of them was a good ol’ push-to-talk kind of application. This got me thinking, and I decided to start concocting one when I’d be back: inspired by Marco’s presentation, I decided it would also be an opportunity to play with MoQ in a browser, rather than just in a native demo or via WebRTC.
What’s push-to-talk?
Don’t quote me on that, but in a nutshell, push-to-talk (or “PTT” as we’ll call it in the rest of this post, as I’m too lazy to write the full thing all the time) is a mechanism where you only send audio (often in broadcast to many recipients) while you hold a button down, rather than all the time as would normally be the case in a call or conference. Historically, that was for instance how two-way radio communication worked, using half-duplex lines, as in walkie-talkies.
While originally, and often still today, this used hardware devices and radio networks, there are of course plenty of applications and technologies mimicking the same kind of functionality on IP too, as it’s a commonly used feature, especially when a group of people need to be in touch and quickly synchronize. It’s the case, for instance, of popular applications like Mumble or Zello.
The broadcast and on/off nature of PTT can be more or less easily replicated with WebRTC in different ways (taking into account the need to try and minimize, or get rid of, renegotiations, which could happen way too frequently with speakers coming and going dynamically all the time), but I was mostly interested in seeing how it could be done with MoQ instead, so let’s have a look at that.
MoQ-to-talk
Before diving straight to the code, it’s worth spending a few minutes trying to better understand the scenario. As we mentioned, PTT typically involves a group of people, where anyone can, at any time, start talking for a bit (while they hold some sort of button) and be heard by everyone else, before disappearing until they have more to say. The bursty nature of these conversations implies a couple of things:
- we need some form of “presence”, as in a way to be notified when a new person starts speaking;
- we need a way to make sure that whatever a speaker is saying reaches all those that are interested;
- ideally we need something that takes care of subscriptions as quickly as possible, as often PTT is used for short messages that will end shortly after they started.
As we’ve seen in the previous blog post, MoQ comes with the concept of namespaces as a way to address, advertize and potentially discover publishers, where such namespaces are hierarchical in nature: within the context of a specific namespace, then, tracks can be made available where media will actually be flowing. This means that, as subscribers, all we need is a way to be notified when a namespace we’re interested in has some “activity” (e.g., a new publisher showing up or becoming active), in order to be able to tell the relay we’re interested in the media.
MoQ has a couple of requests that are of interest here, the first one being SUBSCRIBE_NAMESPACE: as the name suggests, this request tells the relay we’re interested in events related to a specific namespace; when a publisher sends a PUBLISH_NAMESPACE for a namespace that is “under” the one we subscribed to, we’ll be notified. Once this happens, it’s up to us to figure out if there are tracks available: one way to do that, as we saw in the previous post, is by subscribing to the catalog track (when available). That might tell us about tracks available at the publisher that we may want to subscribe to.
There also is another request that is partly related, and yet meant for a different use case. This request, called SUBSCRIBE_TRACKS, works pretty much like SUBSCRIBE_NAMESPACE, in the sense that we still tell the relay we’re interested in a specific namespace, but with a twist: rather than receiving notifications for any namespace of interest, we’re asking the relay to proactively send us a PUBLISH request for any track that becomes active under that namespace umbrella. The end result is that, as soon as a publisher contributes a track for any namespace under the one we’re subscribed to, the relay automatically pushes a subscription for that track towards us, rather than wait for us to react to an event and subscribe to that track ourselves.
Notice how in this case this only works because the publisher used PUBLISH to proactively start sending media to the relay: had the publisher used PUBLISH_NAMESPACE as before, no PUBLISH would have been sent to the subscriber, which means the subscriber would have needed to rely on the previous mechanism instead.
That said, as you can imagine, for a PTT kind of scenario this looks like a very efficient way of handling things, especially when you see how many steps SUBSCRIBE_NAMESPACE required for the same thing. Notice that the diagram above shows a PUBLISH getting to the relay after the relay got the SUBSCRIBE_TRACKS, but if there are tracks under the monitored namespace the relay is already aware of when a SUBSCRIBE_TRACKS comes in, then PUBLISH requests for those are triggered automatically all the same. This means that, no matter when you get it, you’ll be aware of future speakers but also current ones.
Now we know this is a pattern we can use for our demo, as basically all we need to do is:
- choose a “root” namespace all participants in the PTT group will need to refer to (e.g.,
push2talk); - have all participants send a
SUBSCRIBE_TRACKSfor that namespace when joining; - have participants be ready to handle incoming
PUBLISHrequests for live incoming audio; - have participants send a
PUBLISHfor a track (e.g.,audio) in a child namespace (e.g.,push2talk-Lorenzo) when they press the PTT button, and unpublish when the button is released (which will undo all subscriptions to that media stream).
This looks straightforward enough, so let’s give it a try!
Native demo first, because why not
In the previous blog post, I showed a couple of new demo applications I wrote specifically for playing with real-time media, that is:
imquic-moq-loc-send, a MoQ live audio/video publisher using LOC and catalogs;imquic-moq-loc-recv, the subscriber counterpart to receive live audio/video and decode/render it.
I couldn’t re-use those demos as they were because both those demos only do media in one direction: the former only sends, the latter only receives. Besides, both demos are conceived to only work with a single audio/video stream. In a PTT demo application, instead, we somehow need to play both roles, and for more than one stream, as we definitely need to be able to receive audio from other participants at any time, besides being prepared to send it when a button is pressed.
As such, I started working on a brand new demo application, smartly called imquic-moq-loc-ptt, that would do exactly that, by borrowing some of the code of both apps, and focusing on audio only. To make it more interactive and provide some visual cues, I wrote the application so that it would render some text any time a new user started talking in the group, and configured the UI so that the spacebar would be used as the PTT button: keeping it pressed would trigger the PUBLISH and the capture/delivery of live audio, while releasing it would get rid of the stream and get back to a muted state. The astonishing UI for the demo looks like this:
Under the hood, as soon as the application is started and manages to connect to the relay, a SUBSCRIBE_TRACKS is sent for the push2talk namespace, which means it will be notified when other people in the group start talking. When that happens, the application intercepts the incoming PUBLISH from the relay, accepts it, and decodes incoming audio (where objects are delivered via OBJECT_DATAGRAM) to render it. Just as an interesting tidbit, considering SDL2 is used for the UI, and SDL2 Audio for capture and playback, audio is mixed before it’s actually played: in fact, multiple people may end up speaking at the same time, but the SDL2 Audio API only allows for one stream at a time to be written to the audio device, which means it’s up to us to make sure that all active speakers are mixed in the single stream we’ll then play.
An example of how that looks like in action can be seen in the short video below:
That’s neat, isn’t it?
But a native demo is only partly fun… who has time to install dependencies and compile things? Besides, one of the points of MoQ is that, by leveraging WebTransport, it can be used in a browser too, exactly as WebRTC. How would that look like there?
MoQing a web demo
I vented enough, both during my CommCon presentation and in my previous blog post, about how you’re left to your own devices with MoQ. WebRTC gives you all you need out of the box, a full and complete media stack, mainly: MoQ doesn’t. This means that, if I want to replicate the same native demo in a browser too, it’s up to me to take care of:
- MoQ itself (WebTransport is, well, just the transport);
- audio capture+encoding (when sending) and decoding+rendering (when receiving);
- a lot of other things we won’t even bother about for this dumb demo (e.g., jitter buffers).
A few months ago I had tried dabbling with something like that, back when I tried experimenting with RTP Over QUIC (RoQ) on WebTransport: I thought it would be easy, using WebCodecs, to take care of both capturing and rendering media, using a format (RTP) I was already very familiar with.
Well, it turns out I was half right… The capturing part was indeed easy, as I could use getUserMedia as in WebRTC to get a MediaStream/MediaStreamTrack, and then MediaStreamTrackProcessor as an intermediary to get audio frames to encode using WebCodecs. The rendering part was the exact opposite of easy, it was a nightmare: while a MediaStreamTrackGenerator exists, it only works for video and not for audio, which means I couldn’t do the reverse of what I did for outgoing audio and simply pipe frames decoded with WebCodecs to a fake track I could render with an audio element. No, that would have been too easy, wouldn’t it? No matter what I tried, I couldn’t get AudioContext to directly work either, for who knows what other reason: I remember digging through a lot of posts and issues and feeling lost, and I wasn’t the only one. As such, at the time I simply gave up and cursed the skies.
During CommCon, though, Marco from Nimble Ape shared his experience streaming the event via MoQ, which included some nitty-gritty details about how he dealt with trying to implement a MoQ player in a web page. I found the presentation really interesting and well made, and while Marco himself admitted the media stack wasn’t the most efficient, it definitely did the trick, which was enough for me. Besides a nudge in the right direction with respect to audio rendering, it also convinced me to use the Moqtail TypeScript library to take care of the MoQ stack in the web demo, as it had worked nicely for him too.
As such, I started writing the code using Moqtail to connect and talk to the relay, which allowed me to use the same requests I used in the native demo, namely SUBSCRIBE_TRACKS to become aware of other speakers, and PUBLISH to send/receive media objects. This also allowed me to react to incoming PUBLISH requests to manage subscriptions, any time someone in the group started talking, and intercept incoming OBJECT_DATAGRAM objects: I’d process the payload of incoming objects to get access to the audio frames, decode them using WebCodecs, and then use Marco’s trick to render the audio. Bingo!
On the sending side, I used the same pattern as the native demo, meaning I hooked starting to send audio to the spacebar being pressed: this would trigger a PUBLISH to be sent to the relay, the getUserMedia + MediaStreamTrackProcessor dance to get access to audio frames, encoding the frames and then sending the objects. In hindsight, using the spacebar here too probably wasn’t the smartest choice (someone pointed out you can’t really use the spacebar that way when using a mobile browser, for instance), but for a basic demo that was enough.
The end result for a scenario similar to the one we saw in the native demo can be seen here:
As you can see, nothing fancy, but it does the trick!
Of course, it’s not all rainbows and unicorns. The media stack is pretty poor, since everything is done in the main thread, and the audio scheduling is pretty much a copy of what Marco showed in his presentation. A much better way to do that would involve Web Workers to take care of audio encoding and decoding in the background, for instance, and possibly involving some form of ring buffer to make audio playback smoother (which is basically what moq-encoder-player, a much more advanced MoQ media stack, does). That said, this was beyond my poor web development skills, and way more than I needed for a dumb demo, so I’m happy about how it came out nevertheless, at least as a proof-of-concept.
That’s cool, can I try it?
While the native demo is part of the imquic library repo, and so can definitely be tested easily as long as you build it, I haven’t put the ugly code for my web demo on any repo as of yet. I plan to do that, sooner or later, but in the meanwhile, in case you’re curious and want to play with it, you can test the demo online with the tiny relay I use for MoQ interop sessions using the link below:
You’ll have to use Chrome because of a few APIs that are only available there (like MediaStreamTrackProcessor, for instance).
Please notice that this is a very tiny VPS I mostly use to host some personal websites, and so don’t expect great performance out of it: I’d actually ask you not to abuse that tiny server at all, the poor thing! But nevertheless it should give you an idea of how the demo works using MoQ, especially if you’ll test it with a few other people rather than on your own.
What’s next?
Well, as a demo, this is pretty much done. The scenario intrigued me, I spent a little time figuring out how it could work, and I came up with a couple of demos, which allowed me to dip my toes in MoQ web development as well. That said, I don’t think the PTT demos (neither the native nor the web one) will evolve much beyond this stage, unless some interesting use for them comes up later on.
I’ll definitely use these efforts as a stepping stone towards more audio/video tinkering, though, especially in the web area. It could be cool to come up with new demos involving video too, for instance, in order to start playing with more advanced scenarios of different kinds (conferencing, broadcasting, caching, etc.) like our friends at Moqtail have been doing for a while already!
That’s all, folks!
I hope this was an interesting read and that you learnt something new: I certainly did while working on it! If you played with the native and/or online demos and have any feedback, or ideas about things you’d like to do with all this, please don’t hesitate to get in touch and let us know.
