The Design and Architecture of a WebRTC Application

(1)

Bachelor Thesis

15 credits

The design and architecture of a

WebRTC application

Simon Holm

Alexander Lööf

Supervisor: Mia Persson

Examiner: Joseph Bugeja

Degree: Bachelor’s degree 180 hp

Field: Computer science

Program: Information architecture &

Computer science

(2)

(3)

Abstract

The aim of this thesis is to investigate existing design patterns for WebRTC applications in order to achieve a scalable, performant and efficient WebRTC application that keeps streams unique. Further, this thesis shows how these can be implemented using JavaScript technologies. Through a literature study, we conclude that the design patterns full mesh using a signaling server and star topology with a media server that relays streams, called Selective Forwarding Unit (SFU). Both these design patterns have quality attributes that are desirable. We propose an approach of combining these patterns in the same application in order to achieve a scalable application that can fit a broad spectrum of use cases while being efficient. As full mesh is performant and cost-effective in comparison to an SFU but does not scale well with increasing number of participants, we investigate ways to optimize a full mesh session to use it as long as possible before converting a session to using an SFU. We came up with a way to optimize a full mesh session by limiting the bandwidth used for the media streams which reduces the CPU usage for the clients. The proposed approach of combining full mesh and an SFU is implemented based on a previous WebRTC application and a high-level description of how that was achieved is included in this thesis. We perform an experiment where we measure the client’s CPU usage using the above-mentioned approaches in order to reinforce our findings. The result show that limiting the bandwidth of media streams can increase the possible number of participants in a full mesh session and that it is possible to transfer an ongoing session from full mesh to an SFU and back again. We conclude that combining these patterns in the same application is a viable strategy when creating a WebRTC application.

Keywords: Real-time communication, WebRTC, peer to peer, performance, scalability, efficiency, design patterns, architecture.

(4)

(5)

List of Acronyms

WebRTC - Web Real-time Communication. A framework that provides browsers and mobile applications with Real-Time Communications via simple APIs.

SDP - Session Description Protocol. A protocol to negotiate sessions parameters, typically used in WebRTC.

NAT - Network Address Translation. A method of remapping one IP address space into another by modifying network address information in the IP header of packets while they are in transit across a traffic routing device. Used for conserving global address space due to IPv4 address exhaustion.

STUN - Session Traversal Utilities for NAT. A standardized set of methods, including a network protocol, for traversal of network address translator (NAT) gateways in applications of real-time voice, video, messaging, and other interactive communications.

TURN - Traversal Using Relays around NAT. A protocol that assists in a traversal of network address translators (NAT) or firewalls for multimedia applications. ICE - Interactive Connectivity Establishment. A technique used in computer networking to find ways for two computers to talk to each other as directly as possible in peer-to-peer networking.

WebSocket - A protocol that provides full-duplex communication channels over a single TCP connection.

SFU - Selective Forwarding Unit. A media server that relays streams.

REST - Representational State Transfer. An architecture style for designing networked applications.

MCU - Multipoint Control Unit. A device commonly used to bridge videoconferencing connections.

QoE - Quality of Experience. VOIP - short for “Voice over I.P”

SDK - Software development kit. A set of software development tools for a certain software package.

API - Application programming interface. A set of subroutine definitions, communication protocols and tools for building software.

(6)

1 Introduction

1.1 Motivation

The internet and modern information technology enable individuals and businesses to stay connected with each other and work together remotely using online meeting and collaborative applications. WebRTC is a technology that enables such applications to be developed for modern browsers by allowing the browser to communicate in real time utilizing peer to peer connections. This is achieved by sending control messages between the computers that should exchange streams. W3C’s WebRTC specification section 4.1 says that the “communications are coordinated by the exchange of control messages (called a signaling protocol) over a signaling channel which is provided by unspecified means” [1]. This means that it is up to the developers themselves to figure out how to solve this part of a WebRTC application which leaves room for innovation and research.

We attempted to use WebRTC in a previous project and developed a naive meeting application with a UI and functionality similar to Google Hangouts1_{. We successfully}

created a working meeting application that worked directly in the browser with no need for downloads and installations. We used Node.js to create a simple server to solve the unspecified signaling part described in the last section. However, when there were more than three participants in the meeting and someone shared their screen, the web browser would crash due to excessive CPU load. It turned out that our application did not scale and work for more than 2-3 meeting participants when video or screen sharing was enabled. It seemed that our implemented signaling server was not a scalable solution. The performance and scalability problems our previous project had together with WebRTC’s unspecified means regarding the signaling channel motivated us to do research in the field. To the best of our knowledge, there are no other studies that investigate the scalability of a WebRTC application while taking efficiency into consideration which we aim to do.

1.2 Aim and research questions

WebRTC only describes and implements how two browsers should communicate but is agnostic to the implementation of how the connection should be established, as described in the section above. A way to communicate WebRTC information and to establish a connection between peers is a necessary part of a WebRTC application. The problem and question are, therefore, how should a WebRTC based application, similar to the mentioned meeting application in section 1.1, be designed and architected to scale and support screen sharing and video for multiple participants as efficient as possible without causing excessive load and crash the browser. In order to find the answer, we must investigate, and research the existing design patterns used in today's WebRTC applications, if there are any. According to P.Kuchana a design pattern is “a documented best practice or core of a solution that has been applied successfully in multiple environments to solve a problem that recurs in a specific set of situations.” [2]. Specifically, we are looking for design patterns that promote scalability, performance, efficiency and maintain each stream

(8)

2

unique identity in a WebRTC application. If such design patterns exist, which one is most suited for our scope and how can such design pattern be implemented in an online meeting or collaborative application.

In our case, scalability and performance depend on each other. By scalability we refer to the possibility to increase the number of participants in a WebRTC session. By performance we refer to the client CPU load and quality of experience as the number of participants increase. The goal of a scalable WebRTC application is not achieved if the client performance suffers. Regarding efficiency we refer to the cost effectiveness as in runtime cost and computing power. Further, it is in our interest to maintain unique streams (i.e. stream identity) as to be able to tell where each stream is coming from and provide greater flexibility in the application's front-end as in enabling individual sound volumes and see who is talking.

The main aim of this study is to seek answers to the following research questions: RQ 1. What are the existing design patterns that promote scalability, performance and efficiency without losing stream identity in WebRTC applications?

RQ 2. How can the aforementioned design patterns be implemented in an online meeting application?

1.3 Approach and Limitations

We will conduct a literature study in order to investigate what previous research has found regarding WebRTC application design patterns. We will also take to the internet and the “gray studies”2_{, in order to gain a complete picture of how modern}

WebRTC application developers tackle the scalability and performance issues that can occur when the number of peers increase. We are also going to design and implement the results into our meeting application and finally perform an experiment on our meeting application to validate our findings. Apart from design patterns, we want to find out if there are other ways to reduce CPU load for each peer in a peer to peer WebRTC session.

In order to limit our scope, we will only focus on design patterns and techniques that promote scalability, performance and efficiency while maintaining each stream unique identity within a WebRTC application in order to achieve a cost-effective solution. Further, we will not consider the clients possible bandwidth limitations regarding download capacity but only focus on CPU usage for sending a video stream. Regarding how the design patterns can be implemented, we will only provide a high-level explanation of how this can be achieved in a WebRTC application. Thus, a step by step tutorial to create a WebRTC application will not be provided.

2_{Blog posts and forums}

(9)

3

1.4 Structure

This paper is organized in the following structure.

Chapter 2: Walks through the background of WebRTC and what is required. Chapter 3: Explains the methods we used for this research.

Chapter 4: Describes the results we came up with, including literature study, implementation and experiment.

Chapter 5: An analysis and discussion of the research and its results. Chapter 6: A conclusion of the research and options for future research. Chapter 7: References used during this research.

Chapter 8: List of acronyms. Chapter 9: Appendix

(10)

4

2 Theoretical background

2.1 An introduction to WebRTC

2.1.1 What is WebRTC

WebRTC is a new technology, and ECMAScript API, that enables browsers to communicate directly between one and another in real time. [1] This technology enables real-time media communication like voice, video and other data streams to be sent between modern browsers. [3]. WebRTC is developed and standardized through W3C and the Internet Engineering Task Force IETF together with industry leaders like Google and Apple [3][4]. It is free to use, and open source and its mission are [4]:

“To enable rich, high-quality RTC applications to be developed for the browser, mobile platforms, and IoT devices, and allow them all to communicate via a common set of protocols. The WebRTC initiative is a project supported by Google, Mozilla and Opera, amongst others.”

2.1.2 Requirements for WebRTC from a developer perspective

In order to realize a WebRTC application, one must establish a connection between the browsers that want to communicate. WebRTC provides the solution for the media plane but does not implement the signaling, i.e. how one clients’ information should be sent to remote peers in order to initiate a peer to peer connection [5]. This is referred to as signaling. Signaling can be implemented in many ways and therefore it is up to the developers themselves to choose a solution for this. An efficient solution could be a server using WebSockets. Websockets creates an open two-way connection between a browser and a server. This means that the browser and server interaction is event-based, and the browser does not have to poll the server for a reply every time data have to be exchanged [5]. This results in a very fast and interactive communication which fits very well for a signaling server. See figure 1 how such a WebRTC architecture looks like.

(11)

5

Figure 1: A WebRTC architecture with a signaling server

communicating necessary information between clients

A signaling server is not the only required piece to realize a WebRTC application. To send and receive data to another browser the client must have some specific information. The most important information is the client’s public IP address. Each client or browser must know this information about each other in order to receive an incoming connection from another client. Sometimes it’s difficult for the client to know its own public IP address, especially if the client is behind a firewall or NAT, which is very common these days [6]. To overcome this, WebRTC recommends the use of a STUN server. A STUN server allows clients to find out their own public IP address and which type of NAT they are behind. Sometimes it can be hard to fetch the information, depending on the NAT, and in that case, a TURN server can help the process. We won’t go too deep into the implementation of these servers, but a TURN server is a backup to a STUN server that will kick in if the STUN fails for a client by relaying the media streams [6].

The information gathered with STUN can then be used together with Interactive Connectivity Establishment (ICE) to coordinate the connection between clients [6]. ICE is a framework that WebRTC uses and ICE describes how the connection should work and helps the client discover information so that a peer to peer connection can be established. Under the hood what ICE is solving for us, is to systematically try all possible pairs of “candidates” (the information received from STUN server). ICE will try to discover which pair of addresses that will work. In practice, many combinations of information won’t work and that is why ICE helps to solve this [7]. Now we can extend our figure 1 in a more detailed flow with what is happening during a WebRTC connection between two clients. See figure 2 below.

(12)

6

Figure 2: A WebRTC architecture with a signaling server,

NAT, STUN server and TURN server

2.1.3 A generic WebRTC application flow

The WebRTC API enables the developer to implement a peer to peer application that shares streams like audio and video.

The procedure to start the setup of a peer to peer connection can be split into two phases, initialization phase and ICE negotiation phase. During the initialization phase, the client gathers all local information and creates an “offer” which has to be sent to the remote peer. To start the process, the client needs to collect a local stream that can be shared with a remote peer. The Media Stream Web API can be used for that. The Media Stream Web API is a powerful framework for handling and manipulating streams in the browser. The two main components in the Media Stream API are the interfaces MediaStream and MediaStreamTrack. As seen in figure 3, A MediaStream consists of one or several MediaStream Tracks and a MediaStream Track represents the actual media coming from a media source, e.g. a microphone or webcam [8]. These components also consist of many methods that will help manage the streams and the tracks.

(13)

7

Figure 3: Visualization of a MediaStream Object. Figure adopted from [https://dev.w3.org/2011/webrtc/editor/getusermedia-20120813.html]

With a MediaStream the client can start sending the stream using the WebRTC API. To create a new real-time connection, we use the primary interface in the WebRTC API, which is RTCPeerConnection [9]. This interface allows the application to start the procedure to connect with a peer, given that some pre-conditions are met (as described in section 2.1.2).

To begin using the RTCPeerConnection object it must first be created. The MediaStream Tracks that have been collected with the Media Stream API can be attached with help of the RTCPeerConnection.addTrack method. The attached media tracks will be sent to the remote peer when the full exchange of information and agreement is done. To start the exchange with a remote peer, an SDP must be generated and sent. Session Description Protocol (SDP), is a protocol WebRTC utilize to initiate a peer to peer connection between browsers. The SDP primarily consists of the type of media (video, audio, etc.) being sent, transport protocol, the format of the media, ports and necessary IP addresses needed in order to establish a peer connection [11]. The applications signaling layer should manage to send and to receive SDP’s. The RTCPeerConnection.createOffer method can be used to generate this information. The SDP can then be attached to the RTCPeerConnection object with a method called RTCPeerConnection.setLocalDescription and this method will also trigger the ICE Negotiation phase, which is the second phase. In the ICE Negotiation phase, the PeerConnection ICE agent will start to create ICE candidates. An ICE candidate is a piece of information that describes the protocol and routing needed for the PeerConnection to establish a connection [10]. Generally, there are three types of candidates that get generated in this process. Host Candidate, Server Reflex Candidate, Server Relay Candidate [12]. Usually, a STUN server is needed in order to traverse networks behind firewalls. i.e. if a client is behind a NAT. The ICE agent will reach out to STUN servers and receive a Server Reflex ICE candidate back. In some cases where STUN is not enough, as mentioned earlier in section 2.1.2, a TURN server can be used to relay the stream in order to overcome more complex network topologies. Therefore, TURN works like a fallback for STUN [6].

When both the initializing phase and ICE negotiation phase are done, the SDP and all the ICE Candidates must be delivered to the remote peer. This part of a WebRTC application is referred to as “signaling” and is up to the developer to figure out and implement as it is not a part of the WebRTC API [3].

(14)

8

When the remote peer has received the SDP offer from the other peer, the remote peer will go through pretty much the same phase as the other peer did. The remote peer will create its own RTCPeerConnection object because the RTCPeerConnection object will have all the information about the session and will represent an association with the other peer [3]. The remote peer will attach the incoming SDP to its RTCPeerConnection object and at the same time create an answer (with RTCPeerConnection.createAnswer method) which will generate an SDP to send back [26].

When both peers have received each other’s SDP information and the ICE servers has completed the agreement of ICE candidates, a peer to peer connection is fully established. Now, both peers can receive each other’s MediaTracks by adding MediaTracks to the RTCPeerConnection object with RTCPeerConnection.addTrack method [27].

A detailed flow of a peer to peer connection setup described above can be seen in figure 4.

(15)

9

3 Method

We decided to use a mixed method research approach as it enables us to use both qualitative and quantitative research methods that are suitable to answer our research questions.

Creswell [16] provides several reasons for choosing a mixed methods approach. One reason is that explaining quantitative results with qualitative data gives a more in-depth understanding of the quantitative results. Another good reason is to compare data derived from quantitative and qualitative studies to see how the data converge or diverge.

By using a mixed methodology, we can achieve a better result as the method generates more evidence that if one would only use a qualitative or quantitative method. The qualitative part of this paper is a literature study related to our case and the implemented WebRTC application based on the literature study findings. An experiment to evaluate our implementation will make up the quantitative part.

3.1 Literature study

The main and most important aspect was to understand WebRTC fundamentals in detail and gain an understanding of the technologies WebRTC is built around. We gained this knowledge by reading books like “Real-time communication with WebRTC” [3] and reading the official WebRTC documentation and code examples provided at WebRTC.org [4]. This knowledge helped us a lot with our research questions.

Using Google Scholar and ACM we discovered related work in the field [16, 19, 22, 24] that gave us insight in design patterns used and showed that there are many different approaches to realizing a WebRTC application depending on the applications use case and requirements.

In order to get a complete picture of WebRTC application development we investigated forums and blog posts where developers share their experience regarding WebRTC development.

3.2 Implementation

With the knowledge and findings gained from the research, we used our previous meeting application in order to improve it. We implemented some missing key pieces like dedicated STUN and TURN servers and performed a complete rewrite of the peer connection and signaling flow in order to correctly send and receive SDP offers between peers. The signaling server was also further improved with a cache for managing current connections in order to improve the stability of each peers WebSocket connection to the signaling server. These improvements made connections much more stable and reliable. The result was an application that worked well as a foundation for further development and upcoming performance improvements.

(16)

10

The entire tech-stack is built on JavaScript technologies and entirely written in TypeScript as seen in figure 5. The front-end is a React.js application written in TypeScript. For local state management, the library MobX is used. Data fetching is primarily built on the Apollo Client GraphQL framework but there are also some data fetched using REST. Socket.IO Client is used to communicate with the signaling server to establish peer connections for WebRTC.

The primary backend server is a customized Node.js GraphQL application that is utilizing a combination of Apollo Server and Express.js. Data is persisted in a MongoDB database in combination with the ORM Mongoose.

The signaling server is a Node.js application built with Socket.io, which is a lightweight real-time, bidirectional and event-based communication framework. The SFU server is also a Node.js application. This time using an open source project called MediaSoup which provides a C / C++ implementation of an SFU with a JavaScript API [11].

(17)

11

3.3 Experiment and implementation evaluation

When the discovered design patterns and performance optimizations were implemented, we performed an experiment in order to evaluate our implementation regarding scalability and performance.

To validate our obtained results (see sections 4.1.5 and 4.2), we performed four tests.

• Validate the scalability and performance issues of an unoptimized full mesh session.

• Validate if scalability and performance was improved in a bandwidth-optimized full mesh with a bandwidth stream limit to 500 kbit/s. • Validate if the scalability and performance was improved when using an

SFU.

• Validate that it is possible to swap between full mesh and SFU in an ongoing session in order to combine the two design patterns in the same application. For the first three tests, we took a similar approach as earlier WebRTC performance research [17][28], where we monitored the CPU usage using Windows Task Manager during an ongoing screen sharing session. Each test run through 8 steps where the number of steps corresponds with the number of peers in the session. At each step, we calculated the average CPU usage on the computer that was sharing screen by taking note of the CPU usage every 10 seconds for 1 minute. The computer played a 1-minute video each step to keep the test consistent. The tests were performed on a laptop3_{running Google Chrome.}

For the final test, we attempted to transfer an ongoing full mesh session to our SFU and back again. The purpose of this test was to simply verify that it works. We took note of eventual delays and interruptions of the ongoing session as to see if it is feasible to utilize both full mesh and SFU in the same application.

In order to perform the tests on ur application we had it deployed in a production like environment. The application’s frontend was deployed on Netlify4_{, which is a}

cloud-hosting service for websites. The server, signaling service and SFU were containerized using Docker5_{and deployed on the same instance, or “droplet”, on Digital Ocean}6_{. The}

“droplet” used was a level 1 which provides 1 GB memory, 25GB disk space and 1 CPU. The location of the Digital Ocean machine running the instance was Frankfurt.

3_{HP Pavilion - Intel(R) Pentium(R) CPU 4405U @2.10GHz, 4GB RAM, Windows 10} 4_{https://www.netlify.com/}

5_{https://www.docker.com/} 6_{https://www.digitalocean.com/}

(18)

12

4 Results

4.1 Literature study

Our conclusion after having conducted a literature study is that there are three primary design patterns that are implemented in production WebRTC applications.

• Full mesh topology realized with a signaling server • Star topology with an MCU in the middle.

• Star topology with an SFU in the middle.

Appear.in, a popular WebRTC based meeting application, use a full mesh topology [20]. Discord, a VOIP application dedicated to gamers, use a custom SFU solution and no peer to peer [21]. There are also several researches conducted [14][17][22] in regard to these design patterns and experimental hybrids between these [20]. 4.1.1 Full mesh - Peer to Peer

Full mesh is the most common topology for a WebRTC application [17]. As seen in figure 6 all peers have a connection with everyone in the network and the connections are entirely peer to peer [14]. This pattern is usually used for small scale conferencing and WebRTC applications with a low number of peers. The number of connections created in a WebRTC application based on full mesh can be calculated with the formula n(n-1)/2, where “n” is the number of peers [15]. Each client must encode and send their stream(s) to each remote peer in the network and thus the CPU load increase linearly as the number of peers grow. This makes it unsuitable for applications where larger groups of peers need to be connected at once. However, from the architecture point of view, it is an efficient solution as it puts the load on each client and does not introduce latency through a central server [17]. While the number of peer connections is low, a full mesh is a good choice from the application provider point of view.

(19)

13

4.1.2 MCU

MCU stands for Multipoint Control Unit and is in comparison to a full mesh connection a “star” topology. MCU is a central server and acts as a gateway as seen in figure 7. Because of that, all streams sent between peers will first go through the MCU unit before reaching out to the peers [16]. An MCU does not only work as a gateway but also as a media processing unit, where the streams get processed, encoded, decoded and mixed [19]. This process introduces latency that must be taken into account when considering introducing an MCU to your application. The number of streams from the MCU to a peer will therefore always be one. Transmitting one single stream reduces the CPU usage and bandwidth for each peer which makes the MCU design pattern a more scalable solution in comparison to full mesh. However, an MCU uses a significant amount of bandwidth and is, therefore, an expensive piece of a WebRTC application [18]. Due to its nature of mixing streams, an MCU cannot be used when unique stream identity is desired and is therefore not an option for our research. However, MCU may be a good choice when the need for a high number of participants, e.g. in a webinar scenario, is desired and the requirements for the presentation in the application front-end are low.

Figure 7: A star topology with an MCU in the middle

4.1.3 SFU

A Selective Forwarding Unit, SFU, is similar to the MCU as it acts as a central server that relays each peers’ streams. As illustrated in figure 8, the key difference compared to an MCU is that the SFU does not mix streams, it only relays them. Therefore, each individual stream identity is maintained, and latency is reduced when compared to MCU [19]. In an application that implements an SFU, each peer only sends their stream once and the SFU relays the stream to each of the remote peers. As a client, you may send one stream but receive many streams. Utilizing an SFU in a WebRTC application significantly reduces the CPU load on each peer [19].

(20)

14

Due to its nature of maintaining each streams identity and reducing CPU load on each peer, this design pattern fits the criteria for our research. However, much like the MCU, an SFU is expensive as it uses a lot of bandwidth and it would be beneficial to only utilize the SFU when it's necessary.

Figure 8: A star topology with an SFU in the middle

4.1.4 Overview comparing the discovered design patterns

Table 1: A comparison of the design patterns characteristics Full mesh using a

signaling server Star using an SFU Star using an MCU Scalability &

performance

Efficiency

Unique streams

(21)

15 4.1.5 Proposing a combined pattern

Out of these three design patterns, only two fit our limitation criteria regarding supporting unique streams. These are the full mesh with a signaling server and star topology with an SFU. The full mesh with a signaling server is a performant and cost-effective solution but it does not scale as discussed earlier. The star topology with an SFU in the middle scales a lot better compared to the full mesh but consume a considerable amount of bandwidth and is, therefore, more expensive to run in production. Neither of these two design patterns is a perfect match for our research limitation criteria regarding scalability, performance and efficiency while retaining the unique streams. As there are pros and cons with both design patterns, we propose a solution where these are combined in order to utilize the benefits of each in the same application and satisfy our research criteria.

As the full mesh pattern is performant with low latency and cost effective for the application provider it would be beneficial to utilize this design pattern as long as it's feasible. Googling and researching “gray studies”7_{, we discovered techniques to}

mutate the SDP in order to reduce the bandwidth used on a stream which will, in turn, reduce the CPU load for the client sending the stream. [23] Further searching led us to Muaz Khan’s open source GitHub repository “RTCMultiConnection” [24] where several examples of utility functions to mutate SDP are presented. Using these utility functions may increase the possible number of peers in a full mesh session.

We propose that a WebRTC session is initially started in a full mesh pattern using a signaling server before considering involving an SFU. We propose reducing each peer’s CPU usage by limiting the bandwidth and quality of streams in order to increase the number of possible peers in the full mesh. When the quality reaches unacceptable QoE we propose transferring the full mesh session to using an SFU. This will increase the number of possible peers in a full mesh allowing us to utilize the benefits of this pattern as much as possible and reducing the use of an SFU thus reducing the cost of running it.

Our proposed approach to combine these design patterns would resolve in a scalable, performant and efficient WebRTC application without losing unique stream identity.

4.2 Design and implementation

We used our earlier meeting application project as a starting point for the implementation of our proposed design pattern described above. The meeting application implemented a full mesh topology using a signaling server written in Node.js and Socket.io. The signaling server had the role of communicating the SDP and ICE Candidates between all peers in a session, thus completing the WebRTC application flow described in section 2.1.3. Simply put, the signaling server implements a series of event-listeners that react to events emitted from the client and emits messages to all other clients.

To give more insight into how the meeting application works and how the communication occurs between peers, a description of a typical flow is described

(22)

16 below.

1. A client clicks a button to start a meeting. This fires a request to the backend that generates a meeting ID. The ID is sent back to the client and the client gets redirected to a specific URL for the meeting that contains the ID.

2. When the client gets redirected to the meeting URL a request to the signaling server is triggered. The request creates a Socket.io “room” on the signaling server using the meeting ID as its identifier. The room represents the meeting and contains a list of the connected peers' user ID’s.

3. Whenever a new peer joins the meeting the peers' user ID is added to the Socket.io room in the signaling server. Several event listeners and handlers coupled to the newly joined peer are created, both on the client and the signaling server. The event listeners and handlers on the signaling server are purely for relaying SDP’s and ICE candidates between peers in a meeting. The event listeners and handlers on the client are for receiving the SDP’s and ICE Candidates to create the peer connection objects.

4. A message is emitted to all other connected peers in the room that a new peer has arrived and is ready to join the meeting (aka send and receive SDP’s and ICE candidates). The peers that receive the message, respond by going through the WebRTC API flow described in section 2.1.3 to set up a new peer connection together with the joining peer. The SDP offer and ICE Candidates generated from the initialization phase and ICE negotiation phase, described in section 2.1.3, gets sent and received through the signaling server via its events and result in established peer connections between all peers.

Our meeting application had a solid foundation for our proposed combined design pattern, but we still had to optimize the CPU usage by limiting the bandwidth and quality of streams. We also needed to integrate an SFU and make sure we could easily convert a meeting to and from using the SFU.

As mentioned earlier we wanted to remain on a full mesh as long as QoE allowed. As described in section 2.1.3, the SDP consists of the necessary information to establish a peer connection with another peer and the bandwidth limits for streams. By changing the bandwidth limits inside the SDP, we can manually decide how much bandwidth will be used. We implemented a simple algorithm that made use of Muaz Khans utility functions to set the bandwidth on the SDP [24]. Figure 9 shows a snippet of what this looks like in code. The CPU usage results from these SDP modifications can be seen in our experimental results in section 4.3.

(23)

17

Figure 9: Utility function that mutates the SDP to set custom bandwidth

To complete the implementation of the proposed design pattern we need a Selective Forwarding Unit. We used a framework called Mediasoup for that. Mediasoup, much like the WebRTC API, relies on third-party signaling for setting up connections so we used Socked.io for this. The Mediasoup API is based around the concept of rooms and, according to the documentation [25]:

“A room represents a multi-party conference with N participants (Peers). Within a Room, audio and video tracks generated by a client/browser are relayed to all the participants provides simple functions to create rooms for sessions, assign participants to rooms and relay streams between participants.”.

In other means, all we needed to do was to set up a Mediasoup room for a meeting ID and have the participants connect to it and send their streams to it. Mediasoup and its flow was very similar to the full mesh flow described earlier but made simpler as Mediasoup hides a lot of complexity behind its JavaScript API.

Setting up a Mediasoup room starts on the front-end where a client connects to our Mediasoup server via Socket.io. The client sends a payload to create a room on the server, that includes the meeting ID (same ID as described in the full mesh flow). A Mediasoup room is created on the server with the meeting ID. At the same time, the client creates a representation of the MediaSoup room object on the front-end using the Mediasoup SDK. Figure 10 and 11 show our implementation of the aforementioned features. The client can now join the Mediasoup room. The client

(24)

18

needs to join both the Mediasoup room on the server and the Mediasoup room object on the front-end. This is how Mediasoup works, the Mediasoup room object on the front-end is an association of the Mediasoup room on the server. By keeping this connection between the server- and front-end room Mediasoup can do its magical work by sending and receiving streams to connected peers.

Figure 10: A front-end function where a client creates and join a room

(25)

19

Because Mediasoup is an SFU there is no peer to peer connections in play and the client only have a connection to the Mediasoup server. Therefore, the Mediasoup API provides several events that are to be used to communicate necessary information between peers in a session, e.g. if a new MediaTrack is added or removed.

The SFU implementation was made rather easy thanks to Mediasoup and the only thing missing was the logic to transfer an ongoing full mesh meeting to the SFU. The goal is that a full mesh meeting in the signal service shall be replaced with an SFU meeting on the Mediasoup server. This was solved through a number of steps. First, we needed to close the full mesh connection and therefore we cleaned up the socket.io room on the signaling server and removed all connections between all peers in the meeting. Then we connect all peers in the meeting to a Mediasoup following the procedure described above. Going back from SFU to full mesh worked similarly but required the peers to connect with a small timeout between each peer in order to allow the full mesh to reconnect according to the flow described in section 2.1.3. Without the timeout, all peers would appear in the meeting at the same time and would create duplicate offers and answers in each peer connection which cause conflicts in the connection agreement.

Here, our implementation was complete, and an overview of our architecture is displayed in figure 12.

(26)

20

4.3 Experiment

The experiment with our implementation was made successfully and a visualization of the result from the three first tests is shown in figure 13. The y-axis represents the CPU usage, in percentage, of the computer that the test was running on. The x-axis represents the number of peers connected to the session.

The yellow line represents the first test with an unoptimized full mesh session. With the unoptimized full mesh, the CPU usage increased drastically and at 4 peers the CPU was limited, and input lag and stuttering occurred. Beyond 4 peers the screen sharing stream started to lag and stutter for all remote peers and the QoE was unacceptable.

The blue line represents full mesh with SDP modified to limit the bandwidth used. The CPU usage was considerably lower compared to the previous test which allowed about 7 - 8 participants with acceptable QoE.

The red line represents the SFU results where the CPU usage is consistent and stable at around 30% no matter if it was 1 peer or 8 peers.

Figure 13: CPU usage comparison from test result

These three tests show that the unoptimized full mesh has the worst possibility to scale regarding CPU usage with an increasing number of participants. It also shows

(27)

21

that modifying the SDP to reduce the bandwidth of the streams is a viable approach to increase the number of participants in a full mesh session. Finally, the use of an SFU is the best option regarding scalability and consistent CPU usage for the clients that are sending streams.

The fourth test was to verify the possibility to transfer an ongoing meeting from full mesh to SFU and back again. The test was a success and confirms that it is possible. A minor interruption of the streams for about 0-2 sec could be noticed when the transfer took place. This result shows that it is possible to utilize both full mesh and SFU in the same application.

(28)

22

5 Analysis and Discussion

5.1 Analysis

5.1.1 Literature study

The result from the literature study and our experiments show that a WebRTC application can be improved upon regarding scalability, performance and efficiency and that there are many different approaches. As the WebRTC API is rather simple in regards that it just provides an interface for the browser to send and receive streams it leaves a lot of freedom for the application developers to implement the signaling layer [3]. What design pattern and approach to select depends completely on the application requirements.

The purpose of the literature study was to answer RQ 1:

What are the existing design patterns that promote scalability, performance and efficiency without losing stream identity in WebRTC applications?

Our literature study shows that there has been a lot of research done in the field and that existing WebRTC applications utilize different design patterns to fit their specific needs. The design patterns that we discovered during our literature study are full mesh, MCU and SFU. Appear.in use full mesh [20] and Discord use SFU [21] which fit their needs well. For example, Discords use of SFU for a single voice stream per user does not need a lot of CPU for the user which is desirable when also playing a game as you want the computer's resources to be used to run the game. The design patterns all have their own benefits and limitations as discussed in section 4.1 and there was not one perfect design pattern that fits all our research question criteria. Hybrid solutions that combine different patterns in WebRTC applications have been proposed by others [22]. This led us to our proposal of a combination of full mesh and SFU as these design patterns together brings us close to fulfilling our criteria.

5.1.2 Implementation

The purpose of the implementation was to answer RQ 2:

How can the aforementioned design patterns be implemented in an online meeting application?

A high-level explanation of the implementation is described in section 4.2. We based our technical choices on technology we are most comfortable with which is the JavaScript and Node.js ecosystem. In order to provide a better developer experience, code readability, and catch bugs early we decided to use Typescript in the entire application. This was a great choice that helped us read and understand the code as the application grew.

The implementation was based on our previous meeting application as our research questions were based on improving it. Having a solid foundation and code base to work from saved us a lot of time and let us focus on implementing the proposed design pattern. We had already implemented basic signaling with socket.io so we

(29)

23

continued on that track. Socket.io fit very well with the event based WebRTC API and was, therefore, a good choice.

We implemented Muaz Khans SDP utility functions [24] with smaller modifications to fit into our application. With these functions, we managed to modify the bandwidth when using the signaling server and full mesh which gave the desired effect of lowering CPU usage.

For the SFU implementation, we discovered an open source project with a simple JavaScript API, called Mediasoup [13] that implements an SFU running on Node.js. Mediasoup also comes with a JavaScript SDK for the front-end to make communication with the SFU API easier. However, Mediasoup did not include typescript types which forced us to implement our own typings for the library which was a bit time-consuming. Once implemented Mediasoup worked very well and we could get a meeting up and running with very little code.

We successfully implemented our proposed design pattern, combining a full mesh and use an SFU when needed in the same application. Our tech stack seems to have been a great choice as they successfully solved the challenges the project involved.

5.1.3 Experiment

The experiment was performed in order to provide evidence for the performance issues that exist in a simple full mesh implementation using WebRTC and the performance gains with modified SDP and using an SFU.

As seen in the results, the unoptimized full mesh quickly increased CPU usage on the client sharing screen as the number of participants increased which proves the performance issues in this approach. As seen in the graph the CPU usage increase drastically at 4 participants which caused unacceptable performance. When modifying the SDP to lower the bandwidth used, the graph clearly shows improvement. At 7 participants the CPU usage was the same as 4 participants in the unoptimized. This proves that modifying the SDP is a viable option to increase the number of participants in a WebRTC session that utilizes a full mesh. The use of an SFU provided expected results showing a stable CPU usage no matter the number of participants. Judging by the test results an SFU is clearly the best option regarding scalability.

Finally, the successful test to transfer a full mesh to an SFU and back again in an ongoing session proves that it is possible to combine both patterns in the same application. However, the small interruption of the session must be accounted for when designing the application and deciding when to transfer between these.

The test results together back up our proposal of combining both full mesh with modified SDP and SFU in the same application to achieve a scalable, performant and cost-effective WebRTC application.

(30)

24

5.2 Discussion

Our proposed approach, to combine full mesh and SFU, enable a more flexible application that can suit a broad spectrum of use cases. Sessions with a low number of participants can utilize full mesh with its low latency and cost efficiency by being fully peer to peer. When there is a need to support a higher number of participants the SFU can take over the session, as demonstrated in the experiment. In this way, the SFU is only utilized when necessary which reduces the overall cost for the application infrastructure. By modifying the SDP to reduce bandwidth while in a full mesh a lower CPU usage and a higher number of participants could be achieved. However, we used a rather low bandwidth in our experiments, and we recommend testing and evaluating optimal bandwidth depending on the use case and number of participants. It is worth to note that we were only able to modify the SDP to set custom bandwidth in Google Chrome as it, at the time of writing, was not supported in Safari and Firefox. This must be taken into consideration when using this technique. Further, regarding the experiment we used a 1-minute video clip, as described in section 3.4, in order to make the screen sharing sessions consistent. However, this was not a realistic screen sharing scenario and perhaps more realistic results could have been achieved by performing a presentation following a script but then risk losing consistency. The experiment only reached 8 participants as at the time of testing we did not have any more devices to use for testing. This means we did not reach the limit for number of participants while using the SFU. Therefore, we do not have results showing how many participants the SFU can support. What makes our approach different from other WebRTC applications is that we can alternate between full mesh using a signaling server and SFU within the same ongoing session depending on the session needs and number of participants. Our approach is not limited to online meeting applications but can be applied to any type of WebRTC application that require unique streams and want to achieve real-time communication in group sessions. The media streams could be replaced with arbitrary data streams in order to achieve something else than a meeting application with webcam and screen sharing. If the WebRTC application needs to send CPU demanding data streams to a varying number of peers our approach can be a good fit.

The complexity of the application increases as it needs both a signaling server to enable full mesh and an SFU. All application features need to be implemented for both design patterns together with logic to transfer a session between the two. Maybe the simplicity of only using an SFU, despite its higher cost to run, outweighs the increased complexity of our proposed solution? This is a question one must consider before making a choice.

Regarding the SFU, we did some research on how to build one ourselves but most of the SFUs out there are implemented in languages we are not proficient in. Neither could we find any documentation on creating an SFU using only JavaScript. Therefore, we investigated the possibilities of using a third-party module or framework and came across the Mediasoup SFU framework. Using a framework often come with its benefits and drawbacks. Usually, one can save a lot of time up front but might run into problems when edge cases and customizations are needed. In our case, it was solely beneficial to use Mediasoup as we did not require any customization and could quite easily implement it in our prototype.

(31)

25

Rajab did similar research as us [17], where the research focus was to compare the different WebRTC topologies. Like us, experiments were made by measuring the CPU usage in both a full mesh and star topology session. Though Rajab only used audio tracks and an MCU server instead of an SFU server, we came up with pretty much the same results regarding the CPU usage increase in full mesh as the number of participants increase. Rajabs test results show that it is possible to be up to 6 - 7 participants in a voice call on a full mesh. Rajab limited the research to audio conferencing only and suggests further research to compare with video streams. Comparing Rajabs results with our we can see that video streams are more demanding regarding CPU usage due to the higher bandwidth needed for video streams compared to audio streams. Rajab suggests the use of an MCU in order to scale a WebRTC session. Both MCU and SFU implement a star topology that limits the streams sent by each client to one. To meet our criteria of unique streams, an SFU can be used instead of MCU.

The choice of a mixed methods approach for this research was good. By combining literature study with an experiment, we had to implement a WebRTC application to perform experiments on. This way we gained a deeper understanding of WebRTC and how its realized in an application. By performing experiments, we got evidence that supported our proposed approach of combining full mesh and SFU. However, it was very time consuming to implement the prototype application in order to perform the experiments. Thankfully we had our previous meeting application project as a starting point which saved us a lot of time. Without it, the scope may have been too big.

(32)

26

6 Conclusion and further research

6.1 Conclusion

This thesis aim was to find existing design patterns for WebRTC applications that promote scalability, performance and efficiency while keeping streams unique. A further aim was to show how these can be implemented.

Through a literature study, we discovered three existing design patterns used in WebRTC applications, full mesh, MCU and SFU. Regarding keeping streams unique some design patterns are not viable. e.g. MCU as the MCU mixes all streams into one [17]. We proposed a combination of full mesh and SFU, as they together provide attributes that promote our research criteria. By using full mesh with modified SDP, we can increase the number of possible participants in a WebRTC session. When full mesh reaches its limits, the session can be transferred to using an SFU. We showed how to implement the combined proposal in a prototype using JavaScript based technologies and explained on a high-level how it was achieved. We also performed experiments on our meeting application that validated that combining full mesh with modified SDP and SFU provide an answer to our research questions.

We conclude that implementing a WebRTC application using our proposed solution resolves in an application that fulfills the research criteria.

6.2 Further research

As mentioned in the limitation of this thesis we did not consider the clients bandwidth limitations. This might create problems for a client in a large session using an SFU as the bandwidth download requirements increases as the number of participants increase. This means that an SFU may not be the best solution for a session with a very high number of participants connected. It would be interesting to find out what the minimum bandwidth requirements are for a client in an SFU session depending on the number of participants and type of media stream.

Regarding modifying the SDP in order to reduce the bandwidth, we used a fixed bandwidth of 500kbit/s on screen sharing in our experiments. In order to achieve the best QoE, it would be interesting to research and test optimal bandwidth depending on the type of stream and number of participants. Some sort of algorithm to set optimal bandwidth could be developed and may be an interesting case for further research.

(33)

27

7 References

[1] “WebRTC 1.0: Real-time Communication Between Browsers,” W3C. [Online]. Available: https://www.w3.org/TR/webrtc/. [Accessed: 15-May-2019].

[2] P. Kuchana, Software architecture design patterns in Java. Boca Ratón, FL: Auerbach, 2004.

[3] S. Loreto, S. P. Romano, S. S. Laurent, and A. MacDonald, Real-time communication with WebRTC. Beijing: O'Reilly, 2014.

[4] “WebRTC Home,” WebRTC. [Online]. Available: https://webrtc.org/. [Accessed: 25-Jan-2019].

[5] “Signaling and video calling,” MDN Web Docs. [Online]. Available:

https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Signaling_and_video_calling. [Accessed: 01-Feb-2019]. [6] “What are STUN, TURN, and ICE?” Twilio. [Online]. Available: https://www.twilio.com/docs/stun-turn/faq. [Accessed: 01-Feb-2019]

[7] “Overview of ICE,” Interactive Connectivity Establishment (ICE): A Protocol for Network Address Translator (NAT) Traversal. [Online]. Available: https://tools.ietf.org/id/draft-ietf-ice-rfc5245bis-13.html#rfc.section.2. [Accessed: 07-Feb-2019].

[8] “Media Capture and Streams,” W3C. [Online]. Available: https://www.w3.org/TR/mediacapture-streams/. [Accessed: 19-Feb-2019].

[9] “RTCPeerConnection,” MDN Web Docs. [Online]. Available:

https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/RTCPeerConnection. [Accessed: 19-Feb-2019]. [10] “RTCIceCandidate,” MDN Web Docs. [Online]. Available: https://developer.mozilla.org/en-US/docs/Web/API/RTCIceCandidate. [Accessed: 15-Feb-2019].

[11] “SDP: Session Description Protocol,” IETF Tools. [Online]. Available: https://tools.ietf.org/html/rfc2327. [Accessed: 15-Feb-2019].

[12] “WebRTC connectivity,” MDN Web Docs. [Online]. Available: https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Connectivity. [Accessed: 15-Feb-2019].

[13] “Mediasoup,” mediasoup. [Online]. Available: https://mediasoup.org/. [Accessed: 26-Feb-2019].

[14] V. Kandari, “Implementation and Performance Optimization of WebRTC Based Remote Collaboration System ,” Faculty of Computing at Blekinge Institute of Technology, 2016. [15] “What is Mesh Topology?,” Computer Hope, 13-Nov-2018. [Online]. Available: https://www.computerhope.com/jargon/m/mesh.htm. [Accessed: 06-Mar-2019].

[16] J.W. Creswell. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. SAGE Publications, 2003.

(34)

28

[18] K. Fai Ng, M. Yan Ching, Y. Liu, T. Cai, L. Li, and W. Chou, “A P2P-MCU Approach to Multi-Party Video Conference with WebRTC,” Int. J. Futur. Comput. Commun., vol. 3, no. 5, pp. 319–324, 2014.

[19] “Conquering Scalable WebRTC Conferencing,” Dialogic, 2016.

[20] T. K. Husøy, “Topology in WebRTC Services,” B.S Thesis, Norwegian University of Science and Technology, Jun. 2015.

[21] Vass and J. Vass, “How Discord Handles Two and Half Million Concurrent Voice Users using WebRTC,” Discord Blog, 10-Sep-2018. [Online]. Available: https://blog.discordapp.com/how-discord-handles-two-and-half-million-concurrent-voice-users-using-webrtc-ce01c3187429. [Accessed: 12-Mar-2019].

[22] B. A. Jansen, “Performance Evaluation of WebRTC-based Video Conferencing,” Electrical Engineering at Delft University of Technology, Aug. 2016.

[23] “How to limit WebRTC bandwidth by modifying the SDP,” webrtcHacks, 01-Dec-2016. [Online]. Available: https://webrtchacks.com/limit-webrtc-bandwidth-sdp/. [Accessed: 12-Mar-2019].

[24] Muaz-Khan, “muaz-khan/RTCMultiConnection,” GitHub, 09-Apr-2019. [Online]. Available: https://github.com/muaz-khan/RTCMultiConnection. [Accessed: 12-Mar-2019]. [25] “Glossary,” Mediasoup. [Online].

Available: https://mediasoup.org/documentation/v2/glossary/. [Accessed: 12-Mar-2019]. [26] “RTCPeerConnection.createAnswer(),” MDN Web Docs. [Online]. Available: https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/createAnswer. [Accessed: 12-Mar-2019].

[27] “Signaling and video calling,” MDN Web Docs. [Online]. Available:

https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Signaling_and_video_calling. [Accessed: 13-Mar-2019]. [28] N. M. Edan, A. Al-Sherbaz, and S. Turner, “WebNSM: A Novel Scalable WebRTC Signalling Mechanism for Many-to-Many Video Conferencing,” 2017 IEEE 3rd International Conference on Collaboration and Internet Computing (CIC), 2017.

(35)

29

8 List of Acronyms