We all know how important it is to monitor your WebRTC infrastructure, especially in terms of checking what happens with PeerConnections as they come and go. Everything always works great in a lab, but things can go horribly wrong the moment you’re in the wild, and having access to data that can help you check, for instance, why a PeerConnection failed, or what caused it to perform poorly, can be truly invaluable.
In Janus itself, there are historically two different ways of getting this data:
- using the Admin API, which is a pull-based mechanism to retrieve info on a specific handle/PeerConnection that’s currently active in Janus, or
- using Event Handlers, which is a push-based mechanism instead, where Janus is configured to continuously notify external services (like Homer) about whatever is happening (including live PeerConnection data) so that it can be monitored/analyzed externally.
Both these approaches work under the assumption we’re only interested in data collected from the Janus perspective, though, and people sometimes forget that browsers have excellent stat collecting capabilities as part of the WebRTC API standard. What if that’s what we wanted to use instead?
This is exactly the kind of answer and gap that rtcStats, a recently revived project by the well known masterminds of Tsahi Levent-Levi, Philipp Hancke and Olivier Anguenot, tries to address. Intrigued by the possibilities and the alleged simplicity of use, besides the fact that both client and server sides are available as open source software too, I wanted to see how hard it would be to, e.g., integrate in the Janus demos.
What’s rtcStats?
We mentioned how rtcStats tried to tackle WebRTC monitoring from a different perspective, that is the client side rather than the server side. This obviously makes a lot of sense for peer-to-peer applications, where there may not be a server dealing with media at all, but it also is very helpful in scenarios where a server is part of the picture. As a WebRTC developer, I’m well aware for instance of how incredibly helpful getting access to a webrtc-internals dump can sometimes be to finally identify a long standing issue. At the same time, I’m also well aware that such a dump is sometimes either very hard to collect, because it assumes some form of cooperation from the person/application experiencing the issue, and technical knowledge on their end to know how to use the browser’s webrtc-internals functionality anyway. Having some way to programmatically, and transparently, collect such information would as such be very helpful for a variety of reasons, and this in a nutshell is what rtcStats helps with.
People familiar with WebRTC may remember that rtcStats is not entirely a new name. A library called rtcstats-js existed already, a few years ago, as an effort created by Fippo and Gustavo Garcia to push periodic stats to a backend from web applications for analytics/debugging purposes. Even though initially conceived for usage at TokBox, the mechanism poved easy and effective enough that it was adopted by many applications, including Jitsi.
The key concept was straightforward and yet effective. Rather than configuring server side components to collect statistics, collecting statistics on the client side instead, using the powerful getStats API, and pushing them on a regular basis to a configurable backend via WebSocket, where a “story” of the PeerConnection can be collected. And in order to make the impact on web applications as low as possible, do it in a completely automated way, by basically “wrapping” the PeerConnection APIs to allow the library to be automatically aware of each new PeerConnection and its life cycle, while not requiring any change in the web application at all apart from a few lines of code to setup the library itself.
In time the project seemed to lose steam, and less and less updates were made to the library, until a few months ago efforts were made to revive and breathe new life into it. This led to a brand new repository that, this time, included basically all the different parts that make rtcStats work, that is:
- the
rtcstats-jsJavaScript module (client side); - the
rtcstats-serverWebSocket server to collect data from clients (as a standalone application or a Node.js library); - a basic
dump-importerto visualize collected dumps, either from rtcStats itself or webrtc-internals, for debugging purposes (which is basically the code of the good ol’ webrtc-dump-importer by Fippo).
This means that, while as before you can definitely rely on the rtcStats SaaS offering for collecting and visualizing data, as an open source project you can also host and configure everything on your own, and the way you like, and possibly rely on the rtcStats SaaS services only for the more advanced data visualization, and the deductions it can do for you automatically when looking at the data.
Again, the open source part was what I was intrigued with, so let’s dig into that.
rtcStats and the EchoTest demo get into a bar
As always, my go-to starting point for testing anything new is always the EchoTest demo. In its simplicity, it provides everything I need for testing purposes, and this made no exception. I wanted to check how easy it would be to, 1. add some rtcStats code to the EchoTest JS demo, 2. setup a basic rtcStats server locally, and 3. check the data I collected. Long story short, it was all much easier than I expected, but let’s go in order.
rtcstats-js
When it comes to the client side code, as we mentioned there’s a library called rtcstats-js we can use for the purpose. It’s important to use the one from the new repository, and not the old and now unmantained repo, as a lot of code changed in the meanwhile. The first thing I noticed is that this library is a JavaScript module: this is not a big deal, and can actually be an advantage for many (especially if you’re using Janode, for instance), but in the case of my dumb Janus demos it was a first tiny obstacle, as they’re conceived as plain CommonJS code instead. That said, at the same time it was not that big of a deal, especially considering I had already worked with modules in the past in the context of different experiments, like my tinkerings with SFrame and with the Lyra codec.
Turning the basic janus.js library to a module, in fact, is incredibly easy, since all you need to do is add an export { Janus }; at the very end. Once you do that, just changing the script type to module when you include it in the HTML code and importing the Janus object in the demo code is enough:
import {Janus} from './janus.es.js';
For importing the rtcStats library itself, there’s a bit more work to do if you don’t do a bundle. What I did initially was basically copying the rtcstats-js and rtcstats-shared folders to the Janus demos folder, and then included the relevant files via a script in the HTML code:
<script type="importmap">
{
"imports": {
"@rtcstats/rtcstats-js": "./rtcstats-js/rtcstats.js",
"@rtcstats/rtcstats-shared": "./rtcstats-shared/index.js"
}
}
</script>
The importmap structure is quite important, as it tells the code how to interpret those relative paths using mapping instead, especially in the code and due to the internal dependencies. Getting access to the library in the demo code is then as easy as importing the module like this:
import {wrapRTCStatsWithDefaultOptions} from '@rtcstats/rtcstats-js';
This was all I needed to start using the library as a module (besides loading the demo code as a module as well, but that’s a simple HTML change). That said, Fippo was extremely helpful and responsive during my tests, and he quickly explained how you can easily create a more “traditional” CommonJS bundle out of the rtcStats library code. In a nutshell, to bundle rtcstats-js and its dependencies you can do something like this:
npx webpack --entry ./packages/rtcstats-js/rtcstats.js --output-path ./packages/rtcstats-js/dist --output-filename rtcstats.bundle.js --mode production --output-library rtcstatswhich will produce a new rtcstats.bundle.js file that you can then include as any other JavaScript file in your project. In the context of the Janus demos, this made it much easier, as it meant I could ignore any (no matter how small) refactoring to use modules, and just include the new library instead:
<script type="text/javascript" src="janus.js" ></script>
<script type="text/javascript" src="echotest.js"></script>
<script type="text/javascript" src="rtcstats.bundle.js"></script>
with no need to import anything explicitly in the demo code. Long story short: whether you prefer working with modules or CommonJS, you should be covered!
No matter how you import the library, using it is quite straightforward. As we anticipated, in fact, the library is conceived to act in a very unobtrusive way: you don’t need to change any code to how you create, use and destroy PeerConnections, since the library will simply wrap parts of the WebRTC APIs so that it can intercept what’s happening automatically. As such, the only thing you really have to do is initialize it, and tell it where it should send the data. That’s it! A very simple fire and forget.
A simple example not involving authentication via JWT tokens (that we’ll have a look later) is this:
const trace = rtcstats.wrapRTCStatsWithDefaultOptions();
trace.connect('ws://' + window.location.hostname + ':8080');
In this basic snippet, I’m creating a new trace using wrapRTCStatsWithDefaultOptions, and then I’m connecting that trace to a specific WebSocket backend, in this case a simple rtcStats server instance listening locally on port 8080. We’ll have a look at the server part in a minute, but what’s important to highlight is that this is really all we need to do to perform a basic integration: the moment this runs, the WebRTC APIs are wrapped by the rtcStats library, which means that every new PeerConnection that is created and updated in that web page will be intercepted, and stats for it collected and sent to the remote backend. This means that it’s important to initialize the library at the right moment, that is before we start the first PeerConnection, or otherwise we might miss some important data. As we’ll see later when looking at JWT tokens, it will also be important to set it up when we have enough info on the context of the session, but we’ll get back to that in a minute.
Now that we have set up the client side, let’s see what we need to do on the server side instead.
rtcstats-server
The official repo comes with a very handy implementation of an rtcstats-server as well. This means we’re free to spawn our own backend for receiving and collecting data. If you check the code, you’ll notice there are basically two ways you can use it:
- you can use it as a stand-alone application, where the behaviour is controlled by a YAML configuration file, or
- you can use it as a Node.js library, where you can integrate an rtcStats server instance as part of a larger application, or simply use some parts you may need.
In both cases, the server instance provides simple ways to do many things. If you check the reference YAML configuration file, for instance, you’ll notice you can configure different ways to store the stats you collect (e.g., pushing them to an S3 bucket or storing them in a database), plus many other settings (e.g., port to bind to, optional authorization via JWT, obfuscation of addresses, etc.).
To keep things simple, I decided to simply rely on the standalone server, initially, and on local storage to test how it all worked. This meant tweaking the configuration file to also set deleteAfterUpload: false, as otherwise any capture would be deleted from the upload folder as soon as the user closed the connection to the backend. At this stage I didn’t configure any support for JWT tokens either (again, more on that part later).
That was all I needed to do, so as soon as I was ready I started the server via an npm start, and then launched the demo I had previously configured to connect to it, which caused some activity on the server:
Accepted new connection with uuid 0f22b12c-bf7c-4275-a24b-d0d3a70309c4Lo and behold, I could see a file with that name growing in the work folder, and as soon as I stopped the demo I saw it moved to the upload folder (where it stayed, thanks to my previous tweak to the configuration file):
lminiero@lminiero rtcstats-server $ ls -la upload/0f22b12c-bf7c-4275-a24b-d0d3a70309c4
-rw-r--r--. 1 lminiero lminiero 169489 Nov 28 12:46 upload/0f22b12c-bf7c-4275-a24b-d0d3a70309c4
Let’s have a look at what that file contains, shall we?
dump-importer
As we were saying initially, the repo comes with its own visualizer, which is basically the same webrtc-dump-importer that Fippo created some time ago, and that we’ve all grown to love (I certainly do!). All the code is part of the web page itself, so all you need to do to use it is serve it from a web server. Using it to open the file we just collected will show something like this:
Again, if you’re familiar with the tool, this will feel very familiar too (and if you’ve never used the tool before, you should, it’s amazing!). You basically get all the info you’d see if you had received a full fledged Chrome webrtc-internals dump, meaning all info on the PeerConnection life cycle, streams, tracks, candidates, state changes and so on and so forth.
Most importantly, you also get access to graphs, which can be incredibly helpful when debugging PeerConnections:
Long story short, with very little changes to the Janus EchoTest demo, we now have a lot of data we can inspect to check when there’s anything wrong on any of the PeerConnections that were created in that context, all without requiring any effort from the users of the demo themselves and hosting all the relevant parts ourselves (namely server and visualizer, in this case).
It’s worth pointing out that, while amazing, the importer visualization can be a bit crude, and won’t provide any insights of its own. If you create a free rtcStats account, you can also import the same stats files in a more advanced visualizer that’s hosted on their website as well, which looks a bit like this:
As you can see, much more fancy, and indeed full with interesting pieces of information, like more visual indications of how many PeerConnections were involved, how many streams, their direction, the device used for the session, etc. The visual graphs are easier to process as well, and the UI does provide a few helpful insights when you dig through the different items that are available. Apparently, if you subscribe to a paid account you can also get some automated insights and observations/deductions that the service makes out of the logs for you, but that’s something I haven’t tested myself, so I’ll have to take their word on that.
What about authorization?
In the previous section, we’ve gone full circle and showed how we can spawn a local rtcStats server, what changes we need to the client-side code to feed it with data, and finally how to visualize what we collected. There’s an important part missing, though, and that is authorization, as we most definitely don’t want anyone to just spam stuff to our server, and we want to make sure that only authorized entities are allowed to contribute data that we may want to analyze later.
Out of the box, the rtcStats API comes with support for specific JWT tokens. This means that a server can be configured to refer to a JWT secret key to use for authorizing clients: as such, clients will need to provide a valid JWT token when connecting to a server, or otherwise the connection will be rejected.
Enabling JWT token support on the server side is quite easy, as all we need to do is set a JWT secret in the configuration (the YAML file, for the standalone demo, or the config object when using the server as a library), e.g.:
authorization:
# JWT secret key to use for authorizing clients. If not set, no authorization is performed.
jwtSecret: janusrulez
At this point, all we need is a way to generate tokens that clients can use to connect.
A simple way to do that is using the generateAuthToken method in the rtcstats-server library utils, as shown in the example that is provided in the repo. Using the method is quite easy, as you’re only required to provide some fields that will be needed to produce the token (like user, a unique ID, and the conference for context), and then using the provided secret a token will be generated:
generateAuthToken({
user: 'example',
session: 'unique-id',
conference: 'krankygeek',
}, config.authorization.jwtSecret).then((token) => {
// Done
});
To keep things simple and keep on using the standalone server, I decided to create a very basic HTTP-based service that would take care of that on behalf of users. Very simply, users send an HTTP POST to this backend with the details that are needed, and the token is sent back in the response. Nothing fancier than that! Of course, in a production environment this token generation would be much fancier, and most importantly authenticated itself, but for the sake of a simple demo my approach was more than enough, and could be done in just a few lines of code:
import express from 'express';
import cors from 'cors';
import http from 'http';
import {generateAuthToken} from '@rtcstats/rtcstats-server/utils.js';
const jwtSecret = 'janusrulez';
const app = express();
const router = express.Router();
router.post('/jwt', (req, res) => {
// Generate a JWT token for this user (note: in a real setup,
// you'll want such a request to do authentication/authorization
console.log(req.body);
if(!req.body.user || !req.body.opaqueId || !req.body.room) {
res.status(400);
res.send('Invalid details');
return;
}
generateAuthToken({
user: '' + req.params.user,
session: '' + req.params.opaqueId,
conference: '' + req.params.room,
}, jwtSecret).then((token) => {
// Return the token
const jwt = { token: token };
res.writeHeader(200, { 'Content-Type': 'application/json' });
res.write(JSON.stringify(jwt));
res.end();
});
});
app.use(cors({ preflightContinue: true }));
app.use(express.urlencoded({extended: true}))
app.use(express.json());
app.use('/', router);
const server = http.createServer(null, app);
server.listen(8090);
As you can see, the server is incredibly basic, as it spawns a simple express server that listens on port 8090 and waits for POST messages to the /jwt path. Then it uses the info provided by the user (user, Janus opaqueID, and room) as source of data for generateAuthToken, which in turn uses the configured secret to produce a token we’ll return as part of a JSON object.
On the client side, this means that, when we set up the rtcStats integration, we need to get access to a token before connecting to the backend. As such, from the client side we’ll first need to perform a POST to the backend above with our info, and only when we get a token back, we can finally connect to the server and work as we did so far. A simple way to do that in the context of a dedicated function could be something like this:
// Helper function to setup rtcstats
async function janusSetupRtcstats(userInfo) {
const details = JSON.stringify(userInfo);
const fetchOptions = {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: details
};
const response = await fetch('http://' + window.location.hostname + ':8090/jwt', fetchOptions);
const jwt = await response.json();
const trace = rtcstats.wrapRTCStatsWithDefaultOptions();
trace.connect('ws://' + window.location.hostname + ':8080?rtcstats-token=' + jwt.token);
}
In a nutshell, we’re still creating the trace and connecting it to the server via WebSocket as before, but now we’re also providing a token via a query string argument (?rtcstats-token=XYZ), a token we retrieved via a fetch to the backend we described before. Having the details to send to the backend exposed as a function argument makes it simple to integrate in different demos, as the actual data we may want to send may be different depending on the demo.
In the EchoTest, for instance, we don’t really have a concept of users or conferences, but we can generate some artificially anyway (e.g., using the opaque ID we usually use in Janus for event handlers correlation as the unique ID). As such, initializing the rtcStats integration can be now summarized in a single line of code that looks like this:
// Before creating any PeerConnection, let's setup rtcstats
await janusSetupRtcstats({ user: 'Lorenzo', opaqueId: opaqueId, room: 1234 });
That’s all that’s needed. Now, any time we open this modified EchoTest demo, a proper token will be retrieved, and the connection to the server will be properly authenticated. Everything else happens exactly as we’ve already seen before, meaning that stats will be collected and stored on the server, for us to visualize and analyze any way we want.
A more interesting demo: VideoRoom
Now, tinkering with the EchoTest is nice, but not really representative of a typical scenario, since it involves just a single PeerConnection, and media that simply goes towards Janus and then comes back. Something that makes more sense to play with is a conference scenario, as it’s closer to what people will typically use Janus for, and as such the VideoRoom plugin is a good next step in this test, since it implements SFU functionality.
More interestingly, the Janus repo comes with two separate demos involving the VideoRoom, both implementing the same thing in slightly different ways. In fact, both demos implement a basic WebRTC multiparty conference in SFU mode, but while one uses the “legacy” approach of using multiple PeerConnections to subscribe to different participants, the other follows the more modern approach of using a single PeerConnection for the purpose instead, as such leveraging what we call the “multistream” approach. The same difference applies to sending media as well: in the legacy demo, if we want to send both our video and a screenshare, we’ll use two different PeerConnections even both are originated by us; in the multistream demo, a single PeerConnection is used for all content that we want to share, again leveraging the support Janus has had for a while of multiple tracks in the same PeerConnection.
In a nutshell, this means that when using the multistream demo, we’ll always only see two PeerConnections per participant (one for sending media, one for receiving it from others), no matter how many people are in the room, while in the legacy demo we’ll see new PeerConnections any time there are new contributions in the room. This makes it interesting to integrate rtcStats in both demos, to see if it manages to collect all data in both cases, and what it means when we’ll get to visualize it.
Integrating rtcStats support in these demos as well is actually quite straightforward, and not that different from what we did in the EchoTest. We again only need to include the new script in the HTML file, and we can use the same janusSetupRtcstats function we saw before to initialize rtcStats support in the web application code. What needs to change is the “trigger” to this initialization, as we now not only need to make sure we initialize the library before any PeerConnection is created, but we also need to wait until we have some more information, namely the username of the user and the room they want to join. A good place to do that, in both demos, is the registerUsername function, which is the basic method the demos use to prompt users for the name they’d like to use in the room they’re joining: this function is currently used as a trigger to then actually join the room and start publishing their contribution (and contextually subscribe to people who may already be in the room), and so is a good place to satisfy both our requirements.
Considering that this method returns a username value that will contain the desired display name, and that we already know the unique ID (opaqueId) and the conference we want to join (room), once we have that information we can initialize an rtcStats trace right before joining the room:
// Before joining the room, let's setup rtcstats
await janusSetupRtcstats({ user: username, opaqueId: opaqueId, room: myroom });
// Join the room
...
Let’s see what happens when we collect data in the two different flavours of the VideoRoom demo in a similar scenario, like a room with three other people in: we’ll first join ourselves, and then we’ll have three different participants join the room in sequence. As anticipated, this will definitely involve different PeerConnection topologies for the same amount of streams/data being exchanged, as in that scenario the multistream demo will create a single PeerConnection for sending, and single PeerConnection for receiving (everyone else), while the legacy demo will create separate PeerConnections (one for sending, three for receiving, in this case).
In both cases, rtcStats will collect a single file, where info on the whole session will be collected. This is what we get in the legacy demo, in terms of generic overview, number of PeerConnections, number of streams (and their direction/characteristics) and a snippet of the logs:
The data rtcStats gives us is pretty much what we expect: the overview tells us there was one audio and one video stream going out (our own), and three audio and video streams coming in (the three remote participants), plus some info on bitrates and other things of interest. When we look at PeerConnections, we have a confirmation we were using the legacy demo, since we see four PeerConnections (with their creation time and duration) and what kind of media they had: one was a PeerConnection with a single audio and video stream going out (again, our own), while the other three each had only a single audio and video stream coming in (three individual participants represented by dedicated PeerConnections). The Streams tab tells us more precisely which streams appeared in the room, their direction, their kind, the associated codec, and the PeerConnection they belonged to, all mapping quite precisely the info we saw from a general point of view in the Connections tab. Finally, if we check the logs, we can see how a new participant coming in results in the creation of a new PeerConnection, out of an SDP offer (and setRemoteDescription) triggered by Janus.
This is what we got in the multistream demo instead, replicating the same scenario:
The overview is pretty much the same as before, since again we have a single audio/video out, and three audio/video in, with similar data as far as the streams are concerned. The Connections tab immediately tells a different story, though, as we now only see two PeerConnections, with one that only has a single audio/video stream out (us, as before), and one that now has three audio and video streams coming in (the other participants, all on the same connection). The Streams tab provides more information on this mapping, as it shows how all the remote, incoming, streams are actually mapped to the second PeerConnection we saw, rather than belonging to separate PeerConnections as before: when they appeared and their duration is similar to before, as I replicated a similar scenario in terms of when people would start getting in the room. Finally, the Logs tab shows how now new participants joining doesn’t trigger a new SDP offer for a new PeerConnection, but an SDP offer update to the existing one, and so a renegotiation to add (or remove) streams to the conference.
Long story short, this is basically what we expected, but the important thing to get out of this is that we got all this information just by looking at a single dump we collected and analyzed. In this case I did the capture myself, but this may be coming from a completely different setup I may not be familiar with, and so having a complete picture of what a session looked like, its media and connection topology, the session updates, logs, graphs and whatnot can be incredibly valuable. And all without having to do much in the code itself!
How do I use this for my own Janus deployments?
The short answer would be: pretty much as I showed! In this case, I showed how you could add rtcStats data collecting capabilities to the demos just to have a shared reference, but using janus.js as a library (as the demos do) is not a prerequisite at all to use Janus, let alone rtcStats (which doesn’t even know what janus.js is). There are many companies and individuals that prefer using other approaches, like using Janode, or even just use WebRTC APIs directly and managing/wrapping the Janus signalling in ways they see fit. What you use to talk to Janus (or other servers, for that matter) is not really relevant: as we pointed out in the previous sections, the only important point is that you initialize the rtcStats code before you create any PeerConnection (to give it time to wrap the WebRTC APIs accordingly), and that you get access to the relevant info (user, unique ID, conference, and if needed JWT token) before connecting too. As long as you take care of that the right way, data will be properly collected for you to analyze later.
It really is a very straightforward process, especially if you’re ok with the defaults and you’re not interested in customizing the behaviour of the library (or the server). I sticked to the basics for the purpose of this test (which is why I needed very few lines of code), but if for any reason you need more tweaking, the repo on github comes with a lot of documentation on both server and client sides.
What’s next?
As far as rtcStats is concerned, I think that’s it, as it was very easy and quick for me to set it up and play with it, which suggests it will probably be as easy for you too. We may start using it more internally as well, but in case, I’ll only write more about it should I find interesting observations or use cases.
What I may focus more on in a future blog post, though, is the other perspective too, that is how you can collect data on the Janus side too, since on the client side rtcStats already covers all the ground that’s needed. Depending on the setup (e.g., especially when SIP gatewaying is involved), having a different perspective can be useful, and so it might be a good time to revisit how Event Handlers work and, why not, play a bit with the latest version of Homer, which Lorenzo (Mangani, there’s many Lorenzo’s in RTC!) hinted me about and sounds really cool. So stay tuned for that!
That’s all, folks!
I hope you appreciated this short and quick intro to how you can use rtcStats, open source or SaaS, with Janus to monitor and debug your sessions effectively. It definitely is a useful tool to have in your belt, especially if you want something quick and easy to integrate, and you’re ok with only collecting client-side statistics. Depending on how dirty you want to get your hands, you can either do everything on your own using the open source monorepo, or rely on what is offered via SaaS. One way or the other, I personally think it’s good news that the project has new life, and that it’s backed and mantained by people who really know what they’re doing.
Looking forward to hearing your experience with it too, especially if this blog post is what intrigued you to give it a try!
