I talked a lot about my QUIC, RTP Over QUIC (RoQ) and Media Over QUIC (MoQ) experiments, lately. This eventually led to the release of a dedicated open source library, called imquic, that could provide a testbed to play with them all. Just last week I made a presentation at CommCon to talk about my latest efforts in that sense: studying the parallels, if there are any, between MoQ and WebRTC, in order to try and figure out some common grounds for interoperability purposes. After all, the people behind these two worlds seem not to like each other very much, while I’d really like them to be friends.
How did that go? Well, that’s what this blog post is all about, so buckle up and keep on reading!
Looking at MoQ with WebRTC eyes
I am, first and foremost, a WebRTC developer: it’s what I’ve been focused on pretty much full time since day one of WebRTC, and some could say even before that! It’s what led me to author the Janus WebRTC Server, and do so many WebRTC related things for 15 years. That said, I’m also very excited about real-time communications in general, and so when I learned about MoQ I of course jumped at the opportunity of studying and tinkering with it. There are many blog posts you can go through, if you want to follow the different incremental steps that led me from figuring out what QUIC was in the first place, up to my first MoQ steps a couple of months later.
Of course, being a WebRTC developer meant that, especially at the beginning, I was looking at QUIC (and MoQ) with WebRTC lenses, both in terms of use cases, and protocol functionality. For those familiar with both protocols, it’s pretty clear that it’s not always possible to find exact matches and parallels, but it’s also true that some functionality are not that far from each other either, in terms of what they provide to end users and how. This is the perspective I planned to provide in my CommCon presentation, with the hope of making it easier for those more familiar with WebRTC to better understand how MoQ works, and if it could help them with anything, now or in the future.
That said, as I anticipated, there are people from both camps that don’t really seem to like each other. People invested in MoQ often vent about how much they dislike WebRTC, and WebRTC folks never hesitate to remark how they think MoQ is actually all hype, with no sign of any actual adoption in the wild. I joked a bit about that with a few memes, during my talk: I wonder if I missed anything!
But of course, behind every joke there’s always a bit of truth. It’s true that WebRTC can be a bit of a pain and a black box, sometimes, and that it’s not always easy to “bend” it to do what we want, especially in its lack of flexibility with respect to the type media you may want to exchange with it, or the codecs you want to use. At the same time, it’s also true that MoQ still has a lot to prove: it has a lot of potential, but it has nowhere near the same level of adoption that widespread solutions like WebRTC (and HLS) have, and it’s really up to debate whether QUIC will actually be a truly viable solution for real-time media or not.
That said, I’m not here to argue about any of that: I love WebRTC, I’m very intrigued by MoQ, and I very much want both to succeed. So let’s have a (very brief, and probably partly inaccurate) look at how MoQ works with some WebRTC lenses, shall we?
Hi WebRTC, it’s MoQ!
From an architectural perspective, one thing that’s immediately apparent is that the pub/sub nature of MoQ will be very familiar to those that know WebRTC: it’s basically very similar to how WebRTC SFUs work, with MoQ relays acting as SFUs (among other things), publishers that can contribute stuff to distribute, and subscribers that can be configured to receive it. You can complicate the architecture however you want (e.g., by making it look more like a CDN, with relays cascading across each other), but A-to-B that’s it in a nutshell.
Things start to change a bit when we look at signalling and how we address resources. WebRTC has a clear separation between signalling/negotiation, discovery/networking, security and media, while MoQ pretty much bundles everything together: MoQ is based on top of QUIC/WebTransport, which means a client-server paradigm that is fundamentally different than how WebRTC works (no need for ICE, to say one). In that sense, connection migration makes moving clients much easier to handle (no need for ICE restarts). QUIC is also always encrypted, so communications are secured out of the box, which also helps cut down on latency when establishing a connection (no need for separate ICE/DTLS steps). Finally, MoQ puts signalling and media together, in the same protocol: you use different QUIC streams to establish a session, signal your intention to publish/subscribe, exchange objects, and so on, which is a substantial difference from WebRTC, where signalling is not even standardized (only negotiation is, via SDP).
When it comes to signalling, most importantly MoQ starts from the assumption that stuff is published within the context of a namespace, which is a tuple that can be made of one or more fields, and is hierarchical in nature: this means that, for instance, I may publish stuff within a namespace that looks like “Meetecho / Room1234 / Lorenzo“, where the different fields of the namespace tuple provide context and hierarchy to the different resources. Within “Meetecho” there may be multiple applications/rooms, and within the same app/room there may be multiple contributors. How you use namespaces is entirely up to you, but it provides nice contextualization of resources, and in-protocol discoverability: in fact, if I want to dynamically figure out when resources become available, I can subscribe to a specific namespace, and I’ll be notified about everything that happens under that; e.g., if I subscribe to “Meetecho / Room1234“, I can be notified when other people besides me start advertising their presence there, and/or start publishing stuff.
Within the context of a namespace, I can then publish one or more tracks. A track is basically a media stream of some sort: it can be audio, video, text, images, data from a sensor, or whatever I want. It doesn’t really matter to MoQ: unlike WebRTC, which is more tied to audio/video when using RTP, and can be more “generic” when using data channels, in MoQ everything is a generic “object” that can be anything at all. There are of course ways to provide context for those objects (as otherwise recipients wouldn’t know what to do with them; we’ll get to this later), but as far as the MoQ transport protocol is concerned, they’re just generic octets of data, which definitely provide a great deal of flexibility to the protocol. Objects are always unique, with IDs that uniquely identify them within a track, and can be grouped (e.g., for video, it often makes sense to start a group with a keyframe, and have other objects be delta frames). A sequence of groups is basically what constitutes a track, a stream of data. These tracks are identified by unique IDs, again within the context of a specific namespace, but considering that both namespaces and track names can be quite long, any time a track is the aim of a specific subscription a unique (to the connection) numeric “track alias” is associated to it, so that a much more compact identifier can be put in objects for demultiplexing purposes.
If you think about it, this is pretty much what IDs like mid provide within the context of WebRTC: there are different identifiers that are used in WebRTC, but mid, as advertized in the SDP and then signalled in RTP packets, definitely provides the most efficient way to demultiplex traffic when a lot of RTP packets are sent on the same connection. This means that, in a nutshell, a track alias could be seen as the mid equivalent of the MoQ world, when it comes to figuring out what track a specific object belongs to.
Subscriptions in general are quite interesting too, as in MoQ they can take different forms, and can happen in different ways. With WebRTC SFUs, what usually happens is that a publisher may contribute media to an SFU proactively, so that the SFU is ready to react when subscribers express interest in any of those media streams. MoQ supports that too, using a request called PUBLISH that allows publishers to explicitly push objects towards the relay, but it also supports a different approach, where the publisher waits for an incoming SUBSCRIBE request before sending anything. This means that subscribers too can receive objects in different ways: they can either explicitly ask for something via SUBSCRIBE, or they may be offered something via PUBLISH instead. All this can be tied to that dynamic namespace discovery mechanism we talked about previously: rather than explicitly ask for something, I may prefer having the relay proactively tell me about anything that becomes available and offer to send it to me right away. This provides a nice flexibility, as far as available options are concerned.
One more interesting functionality that MoQ provides is the delivery of past objects via FETCH requests. With WebRTC we’re used to the concept of real-time media, where whatever we’re watching flow via RTP is, indeed, live traffic that’s happening now. That’s definitely part of what MoQ can do, but the specification was conceived with the concept of configurable latency, where any object that passes through a relay can be cached, so that it can be delivered again later on as a past asset, rather than a live one. This is an interesting feature for different reasons, as it means that via MoQ I can either get present/future object, or past ones: and by “past object”, I mean anything ranging from objects that were sent a few seconds ago, or days/months. The use cases for such a functionality are varying, as on one hand we could see FETCH as a MoQ alternative to an HTTP GET to retrieve something that happened before I may want to consume now (e.g., to go and retrieve the MoQ objects associated with a football match highlight), but on the other hand it could also be useful to provide more stability to a live stream (e.g., to get all objects starting from the latest keyframe so that I can fast-decode through them before handling live video, or to create a bit of a buffer if I’m ok with some more latency).
Whatever the use case or requirement, it’s definitely an interesting functionality, and one that, out of the box, we can’t really do with WebRTC: yes, it’s possible to replicate something similar by re-injecting old RTP packets and fixing RTP headers, but it’s not really the same thing, as it’s more of a fun hack than something that is actually a fundamental part of the protocol itself.
Which leads us to streaming formats. In WebRTC, we’re used to some structure in that sense: we know that audio and video will go over RTP, and we know a negotiation process will take place between the two parties in the conversation to basically “agree” on a set of ground rules, including which media streams will be exchanged, their direction, which codecs will be used, and so on and so forth. As we pointed out before, though, in MoQ anything is much more vague: what we send are generic objects, that can, in theory, be pretty much anything. This means we’re not working within the same framework as, e.g., RTP: we don’t know if the object will be audio or video, or even if we do, we have no clue how it will be encoded, serialized, or whatever. This doesn’t mean we have to be entirely clueless, though, or that we have to control both publisher and subscriber to know the formats in use or nothing will work. It basically falls down to figuring out how to advertize a streaming format, and how that can happen. Within the context of the IETF standardization efforts, one such way is described in the MoQT Streaming Format draft, which in a nutshell introduces two main concepts:
- a “catalog”, which is a JSON object that describes all tracks in a namespace, including the relevant details to process the associated objects;
- packaging formats, which describe how MoQ objects are serialized on the wire.
Considering the role of the catalog, we may see it as the WebRTC SDP equivalent in MoQ, as that’s where we put information about what each track is, how it’s encoded, etc. Unlike SDPs, though, catalogs are declarative in nature: there’s no negotiation involved, and so what’s written there about a track is what that track is, period. Besides, there’s no offer/answer taking place either: publishers generate a catalog, and publish it somewhere, even as part of a MoQ track itself (so that discovery of streaming formats can happen through the protocol as part of namespace discovery); subscribers can obtain that catalog somehow (again, even automatically via MoQ itself, before subscribing to media, or out of band), and that’s enough for them to discover existing tracks they may want to subscribe to, and know how to process incoming objects. Catalogs are also dynamic in nature, which means they’re live entities: tracks may come and go during a session, which means deltas can be generated to update them (once more, possibly via MoQ itself, since objects can be anything, including JSON text).
A sample catalog, as available in the current version of the draft, is the following, which shows an audio and a video track being available under a specific namespace, and their info:
{
"version": "1",
"generatedAt": 1746104606044,
"tracks": [
{
"name": "1080p-video",
"namespace": "conference.example.com/conference123/alice",
"packaging": "loc",
"isLive": true,
"targetLatency": 2000,
"role": "video",
"renderGroup": 1,
"codec":"av01.0.08M.10.0.110.09",
"width":1920,
"height":1080,
"framerate":30,
"bitrate":1500000
},
{
"name": "audio",
"namespace": "conference.example.com/conference123/alice",
"packaging": "loc",
"isLive": true,
"targetLatency": 2000,
"role": "audio",
"renderGroup": 1,
"codec":"opus",
"samplerate":48000,
"channelConfig":"2",
"bitrate":32000
}
]
}As you can see, it’s not that complex, and fairly straightforward to understand: it can get more complex than that (you can refer to the draft for more details), but its purpose is nonetheless quite clear. Another important piece of information it carries is related to the packaging, as knowing which codec is used for video, for instance, is not enough: we also need to know how it’s formatted in objects, in order to figure out how to process them and obtain a frame to decode. This is indeed where packaging formats come in. At the time of writing, there are mainly two available:
- CMSF (compliant with CMAF), mainly for segmented media in a format that feels closer to how HLS streaming works;
- Low Overhead Media Container (LOC), a more compact format that is more similar to RTP in “vibe” (and the one referenced in the example above for both tracks).
In both cases, what usually happens is that the whole frame (whether it’s audio or video) becomes part of a MoQ object, with some additional metadata that is either part of the payload itself, or an external property. This is quite a difference from how it works in WebRTC, where (especially for video) frames may be “chopped up” in multiple RTP packets, as they need to fit within the constraints of the MTU. Since MoQ works on top of QUIC, that’s not really needed, as any fragmentation of the payload can actually happen at the QUIC level when using STREAM as a way to send data (this is NOT true when using DATAGRAM, but that’s typically only used for audio, and audio frames are small enough that it’s not an issue). STREAM is closer to a “mini-TCP” of sorts, which is why, for real-time media, it’s important to use it wisely: if you put the whole track on a stream, you’ll bump into head-of-line blocking; a separate stream for every object is doable, but may be overkill; streams in general are flow- and congestion-controlled, which means they may get retransmitted and/or delayed. As such, prioritization of streams becomes an important feature of MoQ, especially if you want to keep latency low.
A WebRTC/MoQ cheat sheet (kinda?)
Long story short, what if we now wanted to summarize the main differences between WebRTC and MoQ, in order to allow someone familiar with WebRTC to figure out what could be a MoQ counterpart, or viceversa? As we anticipated, there isn’t always a 1-to-1 match, and even similarities are sometimes more apparent, than factual, but nevertheless during the CommCon presentation I prepared a slide that was aimed at exactly that.
As you can see, most of the things we discussed in this blog post are there, plus a few other things that weren’t, but there really isn’t enough time to go through all of them. Suffice it to say that some are differences that are more deeply routed in the fundamental nature of the protocol (e.g., QUIC is always client-server, and so NAT-related considerations are quite different than the ones WebRTC has to fight with most of the time), others in the fact that MoQ isn’t finished yet (e.g., the current lack of a proper way of doing simulcast/SVC, which may be done using either Dynamic Track Switching or SWITCH in the future). But, apart from that, for some specific use cases we can indeed imagine more than one parallel, which may make it easier to bridge the two protocols.
There’s one area where they differ quite a lot, though, and that’s in the ease of dealing with the client side. With WebRTC we’re used to having it easy: when writing a WebRTC web application, for instance, we mostly only need to worry about taking care of exchanging SDP offers/answers, and possibly some ICE candidates, but once that’s done, dealing with the media is something the browser (or more precisely libwebrtc, most of the times) does entirely for us: capturing, encoding and decoding media is done for us under the hood, as are functionality like jitter buffers, audio/video synchronization, and so on and so forth. The only thing we need to do, as web developers, is to react to events about tracks so that we can present remote content that has already been decoded and prepared for us.
This is most definitely NOT the case for MoQ, though… with MoQ, you’re pretty much left to your own devices. WebTransport does give you a way to exchange QUIC messages with the relay in a browser, but beyond that everything else is up to you: this includes implementing the MoQ transport stack, capturing media, encoding media, synchronizing it, implementing ring buffers, decoding media, rendering it, etc. It’s a titanic job, that can only partly be made easier by existing code (e.g., by relying on external libraries for MoQ itself). The media stack must be written from scratch, involving creative usages of WebCodecs, AudioContexts and so on, and there’s no way around that (Marco made an interesting overview on that at CommCon).
I do know that there are people that are actually looking forward to that: one of the main complaints people have with respect to WebRTC is that it’s a black box, and so that all the things it does for you under the hood actually deprive you of whatever control you may want to have on the process, control that you most definitely get back when you write the whole media stack from the grounds up yourself. I’m not one of those people, though. I don’t like having to reinvent the wheel, when it comes to media, and not many companies and organizations, beyond the colossal ones interested in MoQ right now, will have a ton of resources (or the required knowledge) to do all that properly without messing it up. This is one of the areas where I definitely see MoQ lacking, and a potential cause for delayed adoption. Who knows, maybe in the future there will be libraries that will make it easier: at the moment, if we exclude the MoQ stack itself (for which a couple are already available), that’s not the case, so we’ll see what will happen.
In the meanwhile…
Enough theory, prof, show us some demos!
Now that we know more about MoQ and how/where it differs more or less substantially from WebRTC, we can have a look at some more practical applications.
Experimenting with real-time audio and video using MoQ is something that I tried to do since day one: one of my first efforts, for instance, was getting moq-encoder-player, a WebTransport-based MoQ publisher and subscriber demo, to interop with WebRTC via Janus, which is something I talked about at length in a dedicated blog post a couple of years ago. That said, since then I focused mostly on strengthening the QUIC foundation and updating the MoQ implementation as the specification evolved. In the meanwhile a lot changed in the packaging formats I experimented with at the time, which meant I pretty much had to start almost from scratch to revive those efforts.
Considering that, in the meanwhile, moq-encoder-player also fell a bit behind (it was stuck on an older version of the MoQ transport draft, and to a deprecated packaging format), I decided to try and implement some native publisher/subscriber demos of my own that could deal with live audio and video streams. The choice of writing native code was basically driven by the fact that my main implementation, imquic, is a native library written in C, and that dealing with audio/video capture and rendering in a native application felt easier to me than figuring out how to implement a whole media stack in JavaScript (since, again, with MoQ we lose all the goodies that WebRTC gives us for “free” in that regard).
This led me to write two new simple demos:
imquic-moq-loc-send, an audio/video MoQ publisher that captures local devices (mic, webcam), encodes them in real-time, generates a catalog and uses LOC as a packaging format;imquic-moq-loc-recv, the counterpart MoQ subscriber that can consume catalogs, subscribe to live audio and video streams, decode and finally render them.
I configured the recipient to be able to both specify which tracks to subscribe to explicitly, or rely on the catalog for that. An example of the latter can be seen in the video below, which shows the publisher sending media, and then the subscriber only providing the namespace, and relying on the catalog to discover the track names and subscribe to them:
Once I got that working, the next step was to try and figure out a way to bring WebRTC in the picture: this demo, in fact, is basically a publisher sending live audio and video, and a subscriber getting it through a relay, which is most definitely something we can do with WebRTC too. The way to implement it, obviously, went through using what I know how to do best:
- writing a new plugin in Janus to deal with the WebRTC/MoQ translation;
- using the imquic library in that plugin to provide the MoQ functionality itself.
This was only the first step, of course. I knew I had to write a new plugin that could leverage imquic to implement the publisher and subscriber role, while at the same time exposing an API to WebRTC users to deal with the communication there and the WebRTC PeerConnection establishment. This definitely meant mapping some of the MoQ requests and events to signalling and negotiation on the WebRTC side. At the same time, though, I also needed to take care of properly bridging the live media, which as we discussed is not exactly trivial, as WebRTC and MoQ do things differently. This mainly meant being aware of some specific aspects:
- MoQ objects contain the whole frame, while multiple RTP packets may be needed for the same frame;
- LOC uses some metadata, RTP uses others, and there needs to be a way to translate to/from either;
- we use track aliases on the MoQ side, and these need to be properly mapped to mid/m-indexes on the WebRTC side;
- some tweaking of the media streams may be needed to, for some codecs (e.g., AVCC-to-Annex B and viceversa, when using H.264 for video);
- SDPs need to be converted to catalogs and vice versa, and subscriptions must be created accordingly;
- and so on and so forth.
I won’t bother you with the details of all I had to do (you can refer to the branch in the Janus repo where this effort is taking place, for that), but suffice it to say that this was only a subset, even though a considerable one, of all the things I had to take care of in order to implement a proper translation between the two protocols.
Long story short, though, I got to a point where it actually worked in both directions. The video below, for instance, shows the same native MoQ publisher as before sending media, but this time a WebRTC subscriber getting it instead, via the Janus MoQ plugin:
That’s really cool, isn’t it?
And of course we can do the other way around too, where we now publish media via WebRTC, have Janus turn that into a MoQ publisher for us, and then use the native LOC receiver to subscribe to live objects via MoQ and decode/render them:
What if we wanted to be a little frisky, now? We have WebRTC-to-MoQ working, and what else is there that uses WebRTC that’s all the rage? Well, WHIP, of course! So another interesting experiment we can try is using OBS to ingest live audio/video to Janus via WHIP (using our convenient janus-whip-server Node.js library). All we need to do is ensure that the WebRTC session established via WHIP is actually bridged to our MoQ plugin, rather than the VideoRoom. A tiny bit of code using Janode was all I needed to write to make that happen, and voilà: this is OBS sending WebRTC to the MoQ plugin via WHIP, and two different MoQ subscribers (one native, one WebRTC again via the MoQ plugin) consuming it. Pretty neat!
Again, super cool, don’t you agree?
The video is a bit delayed compared to before, this time, but that’s because OBS introduces a bit of latency when encoding the video with WebRTC (at least in the version I’m using). This is confirmed by the fact that both the native MoQ subscriber and the MoQ-to-WebRTC subscriber are pretty much experiencing the same amount of latency.
Really cool! What’s next?
Well, as an experiment and a proof-of-concept this was quite fun to work with, and already works reasonably well. That said, there’s of course a lot of work before this can be relied upon for more than a simple demo. Besides, the MoQ specification is far from complete, and so many changes will need to happen there, before we can call a specific version of this code “stable”.
That said, the very fact this demos work is proof of an interesting and important fact: that WebRTC and MoQ can and will interop with each other, at least for some specific use cases. This is quite important for a simple reason: while it’s true that MoQ may indeed soon succeed and become a widespread protocol, it’s not there yet; and even when it will be, I’m pretty sure WebRTC will be here to stay for quite some time, as it’s simply too widespread and way easier to use for many of the use cases it’s actually a very good protocol for. What does this mean then? Well, it means that you don’t have to throw away everything you have just because there’s a new protocol in town. MoQ is a very interesting solution, and has a lot of potential, but why not experiment with it, and possibly start deploying it, along what you currently have, rather than have it replace everything you have entirely?
Having a framework that allows you to still keep on using WebRTC, but at the same time seamlessly provide MoQ endpoints from the same sources, definitely allows you to start testing what could be the future of real-time media, all while keeping your infrastructure running as it is. At the same time, allowing MoQ endpoints to push media to your infrastructure, and having existing WebRTC endpoints be able to consume that content, would be quite an achievement, as it would open your platforms to more than one technology, and again with a look at the future and the present/past at the same time. This is particularly interesting when it comes to the real-time broadcasting industry, so for WHIP and WHEP (which as we’ve seen would be fairly easy to gateway), but even scenarios like conferencing or live communication may be an interesting target, if not for getting familiar with a new piece of technology.
Should you be interested in that, especially if you’re using Janus already, don’t hesitate to let us know, as we’d love to help!
That’s all, folks!
I hope you enjoyed this practical journey through MoQ, especially as seen from the eyes of a WebRTC developer (which probably many of you are as well), and I also hope I managed to convince you that yes, despite strong words on both sides, WebRTC and MoQ can be friends, and even work together. There’s definitely much more that we want with MoQ, with and without WebRTC in the picture, so if this post intrigued you, then by all means do come back soon, as we have exciting things cooking!
