Evaluating remote and local web rendering of real-time interactive 2D graphics using Blazor

V¨

aster˚

as, Sweden

Thesis for the Degree of Bachelor in Computer Science 15.0 credits

EVALUATING REMOTE AND LOCAL

WEB RENDERING OF REAL-TIME

INTERACTIVE 2D GRAPHICS USING

BLAZOR

Alexander Andersson Tholin

atn17004@student.mdh.se

Examiner: Gabriele Capannini

M¨

alardalen University, V¨

aster˚

as, Sweden

Supervisor: Afshin Ameri

M¨

alardalen University, V¨

aster˚

as, Sweden

Company supervisor: Isak Savo,

ABB Power Grids Sweden AB, V¨

aster˚

as

Abstract

With the growing popularity of the web, companies are starting to extend current development to reflect this. When extending desktop applications to the web, it can be difficult to choose what tech-niques and technologies to use when solving a problem, as current solutions might not be directly applicable. Rendering high-performance interactive 2D graphics on the web can be achieved in mul-tiple ways. The rise in open standards such as the Canvas API allows the client to render natively in the browser, provided they can receive the full object state. There are some cases where this is simply not possible, where the object state is too large, or the client is lacking sufficient hardware. A possible solution is migrating the rendering of the graphic from the client to the server. However, remote rendering comes with new sets of issues as it often lacks high interaction capabilities, and would theoretically require more resources with multiple connections. This thesis will evaluate the performance differences and individual capabilities of remote and local rendering in terms of scala-bility and Quality of Experience using ASP.NET Core Blazor. The evaluation is done through the implementation of the four different solutions for the scenario. These implementations are based on Canvas and SVG using remote and local rendering. Different configurations of the performed tests, such as how much data should be rendered and how many clients are connected, were used to see how they affect response time and interaction latency. The results show that remote rendering performed better in all scalability tests, with remote SVG being the recommended approach. Due to implementation issues and lack of a proper testing environment, the number of concurrent clients was downsized. This caused problems when analyzing the results, and drawing concrete conclusions were difficult. In tests with increasing image size, the client solution suffered memory exceptions, preventing the local versions to be tested further. When testing interaction capabilities by measur-ing interaction latency, the SVG technology significantly outperformed Canvas, since SVG does not require a full re-render of the elements.

Table of Contents

2.2 Performance and load testing . . . 5

2.3 ASP.NET Core Blazor . . . 5

2.3.1 Blazor WebAssembly . . . 6 2.3.2 Blazor Server . . . 6 3. Related Work 8 4. Method 10 4.1 Test data . . . 10 4.2 Test environment . . . 11 4.2.1 Device specifications . . . 12

5. Comparing rendering strategies 13 5.1 Tests . . . 13

5.2 Implementation . . . 14

5.2.1 Generating test data . . . 15

5.2.2 Implementation issues . . . 16

6. Results 17 6.1 Scalability (clients) . . . 17

6.2 Scalability (image size) . . . 19

6.3 Interaction latency . . . 19

7. Discussion 21 7.1 Scalability (clients) . . . 21

7.2 Scalability (image size) . . . 21

7.3 Interaction latency . . . 22

7.4 Related work . . . 22

7.5 Limitations and issues . . . 23

8. Conclusions 24 8.1 Future work . . . 24

List of Figures

1 Example of a 2D figure rendered using DirectX. This is a screenshot from the Net-work Control desktop client. . . 2 2 Blazor WebAssembly. Source: [1]. . . 6 3 Blazor Server. Source: [1]. . . 7 4 Switch and transformer rendered in the Network Manager application, connected

to other components using lines. . . 11 5 Cropped screenshot of local rendering Canvas and SVG with 50 elements. Note that

not all 50 elements are visible. . . 15 6 Concurrent clients test using 250 elements with real-time updated data. Tested on

1 to 15 clients. . . 17 7 Concurrent clients test using 2,500 elements with real-time updated data. Tested

on 1 to 15 clients. . . 18 8 Concurrent clients test using 10,000 elements with real-time updated data. Tested

on 1 to 15 clients. . . 18 9 Increasing elements. Tested on one client. . . 19 10 Interaction latency with increasing elements. Tested on one client . . . 19

List of Tables

1 Testing devices hardware specifications . . . 12 2 Software version specifications . . . 12 3 Scalability (clients) configuration . . . 13 4 Steps for testing increasing clients. k = concurrent clients, n = # of elements . . . 13 5 Steps for testing increasing elements. n = # of elements . . . 14 6 Steps for testing interaction latency. n = # of elements. The actions taken will

simulate a pan movement. . . 14 7 Rendering method and technology description . . . 15

1.

Introduction

Applications for rendering 2D and 3D graphics usually require specific hardware components such as GPUs, forcing the application to run as a native executable. With the rapid growth of web technology, open standard solutions now provide graphics rendering functionalities in the browser [2, 3]. Early on, plug-ins like Adobe Flash would assist in hardware-accelerated rendering but typically requires the plug-in to be installed on the machine. The rise of native open standard methods, such as the Canvas API [2] and WebGL [3], led the once-popular plug-ins to lose support from large companies [4].

There are, however, cases where it is not possible to process data and render graphics on the client. Devices such as mobile phones might lack the processing power and network bandwidth to receive and render the incoming data. These problems open up a new realm of remote rendering, as opposed to local rendering, in which the server (or another system) processes the data, and the client receives the finished result ready to be displayed [5]. This method takes a significant load off the client, and displaying resource-heavy graphics can be done on thin clients such as a laptop or mobile phone. The main drawback of using remote rendering instead of local rendering is the loss in native interactivity since every input event requires a request to the server to receive the updated rendered result [6]. Another drawback is scalability since all of the processing is done on one system. With the server sending updates to a multitude of clients, there is a limit to where the server can not keep up.

This bachelor thesis is conducted together with ABB Power Grids, offering services in power technologies, providing systems, software, and service solutions across the world. ABB Power Grids is in the initial phase of extending its Network Manager software to the web, which is a central system for energy management, automation and communications with power distribution grids. Power utility companies are using the product in control rooms to monitor the transmission and distribution of energy in a country or region. Currently, this is a desktop application, requiring an extensive installation process to function. A feature of the current system is the ability to visualize stations in a 2D representation with rich interactivity and frequent real-time updates. This thesis aims to give an empirical basis for determining the proper rendering strategy (local or remote) to use when rendering 2D-figures on the web. Currently, the company generates image definitions from different object states. An example object could be an electrical component, and we define the example state as the component being either on or off. These definitions consist of various objects that contain symbols with specifications on how they are displayed. The sizes of these images can vary, but a simple model has around 200-300 elements. The current desktop implementation for rendering these images is implemented using DirectX. The example image in figure 1 shows the current result of the desktop implementation.

Currently, browsers do not provide native support for graphics APIs such as DirectX, and devel-opers will require finding a similar solution to solve the task of rendering graphics in the browser. When implementing this rendering feature using web technologies, there is a strong focal point on having the images update in real-time using a network socket connection. When an object state is updated, for example, when turning an electrical switch off, this should be propagated to the images displayed on the client. This propagation will trigger a re-render of the image, or parts of the image. Another essential aspect is focused on the QoE of the resulting product, as it should be possible to interact with the images with low latency. Hoxmeier and DeCesare [7] explain that it is reasonable that latency holds a strong relation to the complexity of the task and that users tolerate an approximate of 10 seconds interaction latency [5] for highly complex tasks, and around 50-150 milliseconds for operations such as zooming and panning. In the case of 2D graphics, loading an image can be considered a complex task depending on the size. Interacting (panning, zooming) is seen as a simple task and should be near-instant. As of now, there is no solid basis for deciding what strategy to render graphics in the browser while still maintaining good QoE.

1.1

Purpose

The purpose of this thesis is to evaluate different rendering strategies (remote and local rendering) of real-time visually critical interactive 2D images in terms of scalability and QoE) in a web-based environment. The work will assert the feasibility of different rendering strategies in a web environment with generated 2D images that are:

ˆ Large in size.

ˆ Updated in real-time.

ˆ Required to be interactive (pan, zoom).

With the current identified properties and requirements, the optimal solution to the problem would be something that:

ˆ is easily scalable.

ˆ has native interaction capabilities.

ˆ has low interaction latency and response time.

ˆ consumes the lowest amount of data sent for large image sizes.

Some of these properties fit within local rendering, while others fit for remote rendering. For example, scalability is an easier task in local rendering since the work is automatically distributed over multiple systems. As of now, no one solution fits these criteria, and the developers will have to compromise. The thesis work will investigate how the different choices affect these requirements, to give a solid basis on deciding which strategy and technology to use in the future. The technologies to be investigated are limited and chosen based on previous work on the subject.

Figure 1: Example of a 2D figure rendered using DirectX. This is a screenshot from the Network Control desktop client.

1.2

Problem formulation

The client-server model is a primary application structure in the modern web. While remote rendering is great for several reasons, with users demanding native interactivity, there are times where work must be distributed to the client [6]. The local rendering method is growing to be a solid method with the rise of native graphics APIs in the browser. Based on previous knowledge, the advantages and disadvantages of local rendering and remote rendering can be deduced. However, deductive reasoning is not always enough. The differences between local and remote rendering needs to be further investigated. The main goal of the thesis is to provide an answer to the question:

How does remote and local rendering differ in terms of scalability and QoE when rendering real-time interactive 2D graphics?

In cases where it is not possible to render on the client, remote rendering can be used. For applications where interactivity is a key feature, this becomes a problem. The thesis will investigate the following:

What are the possibilities of combining remote rendering with high interactivity in 2D graphics?

By investigating different technologies with remote rendering to see which one fits the scope of native interaction with low interaction latency, we can get a solid basis on recommending technology for this purpose.

1.3

Limitations

The current Network Manager application at ABB Power Grids is mainly developed in a .NET environment. When extending their current software to the web, they wish to continue this path. Their aim is to use ASP.NET and ASP.NET Blazor for developing all web applications and APIs. All of the tests will be performed in a local environment using commercial and workstation devices. One workstation will be dedicated for the server, while the other three are used for simulating clients. This environment might affect the final results, as three client devices might not be enough for simulating the amount of users needed.

The implementation will include two rendering strategies using a small number of technologies. The reason for this choice is mainly due to time limitations, where adding new techniques would increase implementation complexity. In addition to this, implementation simplicity is the main focus when choosing what technologies to use.

2.

Background

With the expanding growth of technologies in the web domain, such as the HTML5 standards [8], developers can utilize specialized hardware to enhance the experience of their applications. Developers can make use of features such as the Canvas API [2] and WebSockets [9] to create complex, interactive applications on the web. This section introduces fundamental knowledge and technologies used in the report. First, the different web technologies and communication frameworks. Second, we present different performance and load testing tools used in web testing. Lastly, we introduce the main web framework that is used for the implementation.

2.1

Web technologies

The Document Object Model (DOM) is an API for interacting with the document on a web page [10]. It is a programmatic interface for the representation of a web page that can be modified by scripting languages. Interacting with the DOM often means altering the HTML of a web page, such as changing the text of a paragraph. For example, this can be the result of a user event, such as responding to a click event.

The HTML5 standard is the latest addition to HTML. This version introduces the Canvas tag, which allows the web browser to manipulate image data and draw graphics natively. The Canvas API is an immediate-mode API [11], in which the application is managing the drawing commands. The Canvas API supports multiple rendering contexts [2,6]. The two primary rendering contexts are 2D and WebGL. The 2D context allows for rendering CPU-based 2D graphics, while WebGL allows for hardware-accelerated rendering. WebGL is mainly used for rendering 3D, but also works in a 2D environment. The main difference the 2D and WebGL contexts is the ease of use. The Canvas API allows for drawing simple primitives using a direct method, such as the f illRect() method. WebGL, however, works on a significantly lower level. The advantage of WebGL is the massive performance gains when working with complex calculations such as rendering 3D images and smooth animations.

The Canvas tag consists of a single DOM element. When drawing a primitive using the Canvas API, it is displayed on that element. If there is a change in the data, meaning the representation has changed, the canvas must clear and re-draw the image.

While bitmaps generated from the Canvas element are great for displaying static images on the web, it does not visually scale, and is not natively interactive. Scalable Vector Graphics (SVG) [12] addresses these issues, which is an open standard XML-based vector graphics format for the web. This format allows for infinitely scalable and interactive figures and images, which is highly applicable for the work. The main advantage to using SVG rather than Canvas is the DOM representation of the format. The SVG is an XML-based file format, meaning it has the same format structure as regular HTML [13], allowing all SVG content to be placed directly in the HTML. All of the primitives are then defined as a DOM element. For example, a rectangle primitive is represented with the <rect> HTML tag. If some data representing the SVG tag changes, the single primitive can be changed, allowing for fast re-rendering when a small number of elements change. In contrast to the Canvas API, SVG works in retained mode [11], where the draw commands are handled by the library, and not the application itself.

WebAssembly (WASM) is a portable assembly-like language designed for fast, safe and portable execution of programs [14, 15]. The main goal of the format is to allow for high-performance in web applications, and is one of the few natively supported languages among major we browsers. WebAssembly opens up new possibilities for creating new tools, techniques, and technologies, and allows native execution of common programming languages, like C and C++, in the browser [16]. While WebAssembly has direct memory access, it is only allowed to access a set boundary given by the browser [17]. WebAssembly works in a sandboxed environment, and has the exact same

limitations as Javascript code in the browser. This limitation prevents malicious code from being executed and access memory from other programs. Attempting to access memory that it does not have access to causes an out of bounds exception to be thrown.

2.1.1 Real-time communication

When creating real-time web applications, the client must receive updated information from the server continuously [18]. Polling is a conventional method of real-time communication, which consists of HTTP requests to the server to check if there are any updates [19,20]. If there is an update, the client receives it. If there is no update, the response is empty. The primary problem with this method is the number of requests sent by the client to check if there are any updates. The HTML5 standard introduces WebSockets [8,21,9], which is a two-way communication protocol based on network sockets for the web. This technology makes it possible to have an open connection between the server and client, allowing sending messages on an event-driven basis. The main advantage of WebSocket communication over polling is that the server can emit events instead of the client requesting updates. Another advantage is the mitigation of repeating headers sent for every request [22], reducing overhead. This way, only the necessary traffic will flow through the network.

2.2

Performance and load testing

Apache JMeter[23] is an application designed for load testing and performance measuring. The primary purpose of the tool is to simulate a heavy load on the server. For example, JMeter can be used to create 100 threads, each sending an HTTP request to a specific endpoint. The result would be the server processing 100 calls simultaneously. The main function of JMeter is simulating protocol-based requests. As far as an external server application is concerned, the JMeter requests are coming from real browsers. This is not the case, as JMeter does not execute or render anything that it receives. While JMeter is protocol-based, meaning it does not natively mimic browser behavior, it is possible using third-party plugins, such as the Selenium WebDriver plugin, to simulate a browser [24].

In many cases, testing browser requests on a protocol level is not enough. An example is end-to-end testing in web applications, where all parts of a feature, excluding the user, are tested. The Selenium WebDriver [25] can be used for this purpose, which provides an API for interacting with a controlled browser instance. All of the actions made by a real user can be simulated using the WebDriver API, is an interface for interacting with a browser. Most common browsers, such as Google Chrome [26] and Firefox [27], are implementing this interface.

2.3

ASP.NET Core Blazor

Blazor, developed by Microsoft, is a .NET framework for developing interactive client-side web components and UI [28]. Similar to other web UI frameworks, the goal is to make creating web applications easier for developers. A key feature of Blazor is that both client and server code can be written entirely in C#, allowing developers to share and re-use .NET libraries and code. As of now, Blazor can be hosted in two different ways. These two hosting models achieve the same rendered result, but with different approaches. These two approaches will be the primary definition of local and remote rendering throughout the work. It is important to note that an application created in a hosting model is not locked to that choice. Switching to another hosting model in the Blazor environment is effortless.

2.3.1 Blazor WebAssembly

Blazor WebAssembly is a single-page application framework for .NET web applications [1]. This hosting model makes use of WebAssembly to run compiled assembly-like code, which allows for running .NET runtimes directly in the browser.

Using Blazor WebAssembly, the .NET runtimes are served to the browser and are executed, which runs the web application [1]. Compared to conventional methods, directly running .NET runtimes gives fast execution speeds. It also allows for offline access since the application does not need to communicate with a server to operate.

Currently, there are a few issues that stand out when developing with Blazor WebAssembly. First, WebAssembly can not directly communicate with browser APIs such as the Document Object Model (DOM). The Blazor WebAssembly runtime uses Javascript interopability to handle DOM manipulation and make API calls [29]. The second problem is the payload size of the application. The application runtime is downloaded to the browser as a whole. While there are optimization techniques in place, large application sizes could lead to slow bootstrapping. Lastly, there are ongo-ing discussions regardongo-ing the performance of Blazor WebAssembly [30], even though WebAssembly is advertised and designed to be a fast alternative to native Javascript applications.

Blazor WebAssembly is in preview, and is not recommended to be used in a production environ-ment. Microsoft plans to release a supported version in May 2020 [31].

Figure 2: Blazor WebAssembly. Source: [1].

2.3.2 Blazor Server

As the name suggests, Blazor Server stores all of the business and UI logic on the server. Using a SignalR (WebSocket) connection, the server sends UI updates to the browser, which then updates the DOM. Browser events are sent to the server, in which it responds with a UI update request. This approach requires a constant connection to the server. As soon as the connection is lost, the application becomes unresponsive.

Just like WebAssembly, the server does not have direct access to the browser APIs, and must use Javascript interop handle the calls.

3.

Related Work

Remote rendering and visualization is not a new research subject in terms of efficient data visu-alization [5, 6]. Early on, computers were not powerful enough to process large amounts of data, or it was lacking the hardware to send over a network connection. This section will present pre-vious work on the general remote rendering subject as a whole, as well as similar problems and methodologies to the current task.

The survey by Shi and Hsu [5] investigates interactive remote rendering systems. Their work gives a complete overview of the state-of-the-art of current technologies and approaches. In their work, they identify four main advantages to remote rendering. The first point is the ability for devices with limited computing power to visualize intricate graphics. The second advantage is the caching of resources. If two or more clients request the same image, it does not have to be rendered for each request, assuming the result will be the same. The third point is the re-usability of the implementation. In many cases of local rendering, previous work can not be re-used if they are previously developed for a different platform. The last identified advantage is the prevention of piracy, as actual source content are not sent to the client in any way. For this reason, remote rendering is applicable in industries like cloud gaming [32,33,34].

Shi and Hsu identify two fundamental challenges in remote rendering: interaction latency and data transmission [5]. They define interaction latency as the time it takes from user interaction to the first frame displayed to the user. The second challenge is data transmission, which describes the transfer of the data through a communication channel. The work to be done in this thesis will focus on the interaction latency of the different strategies since it is directly related to the Quality of Experience (QoE) of the result. The conclusions they could make from the work was that researchers base their work on requirements from application specifications. Creating general solutions would require more advancements in frameworks, algorithms, and models. While Shi and Hsu briefly mention web-based remote rendering systems, they do not address interactive solutions using the HTML5 standards like Canvas and WebGL. They do not compare these solutions to conventional local rendering.

In the survey by Evans et al. [35], they investigate the general subject of 3D rendering on the web. Their work mentions the history of graphics rendering tools, and how there are less plugins for web-based 3D rendering than before. They explain that the reason for this is the support for the Canvas and WebGL standard on all major browsers. Evans et al. draw the general conclusion that web-based 3D content is on the rise, and the standards like WebGL are helping developers and companies join this world. They emphasize that model data sizes can be quite large, and that compression algorithms are a sub-field for rendering on the web that should be researched. While this is an important topic, this work will not go in-depth on any data transmission or compression algorithms.

Mwalongo et al. [6] review the current state-of-the-art of web-based visualization, and mention that there is a rise in these applications. They believe that this is due to the increased support by major browsers, and advancements in infrastructure and technology standards. They conclude that the need for remote visualization is growing with the size of data sets. They further explain that the main challenges are still latency and bandwidth. As for local rendering, the main reason to use this strategy is to improve interactivity since it reduces interaction latency. In their work, they focus on HTML5 standard technologies like Canvas and WebGL for local rendering, since they believe that other techniques are outdated. They conclude that the two rendering strategies both have their advantages and limitations, and that there is no best fit for all purposes.

There are several proposals for optimization techniques in remote rendering systems. Sun et al. [36] proposes a hybrid remote rendering method where the server is responsible for rendering the high-fidelity models. The client then draws the low-fidelity models and overlays them on the server-rendered result. The server renders key models, while the client can generate non-essential models like background objects. Another optimization technique from the authors of [37] makes use of 3D

image warping to eliminate interaction latency.

Wessels et al. [20] proposes a framework for remote visualization using the new HTML5 standard technologies. They explain that real-time visualization of large data can be achieved using the HTML5 Canvas and WebSocket API together with Base64 encoding for file transfer. Using Web-Sockets, the server sends rendered images as updates when necessary, and is idle when there are no updates. The usage of this technique allows for fast updates of the incoming image data. This framework will be considered for both the local and remote rendering part of the implementation. One issue with this solution is the lack of native interaction with the remotely rendered elements [6].

Many papers focus on proposing a new technique in web-based remote visualization, but they are often not explicitly measured and compared to conventional methods of rendering on the web. In the comparative study by Edvardsson [38], they investigate whether remote rendering could help in reducing the overall response time of an application when rendering interactive diagrams. The conclusion they could make from their experiment was that local rendering was the better option in terms of response time and data transmission size. Their method included implementing two prototypes providing the same result using different strategies: remote and local rendering. Response time and data transmission was measured for both of the techniques. However, their work does not include measuring whether large data sizes of concurrent clients have a significant impact on the resulting performance. Furthermore, it does not use any real-time communication components such as WebSockets or polling.

In the paper by Kee et al. [39], they study the different local rendering techniques available for interactive web visualization. They explain that there is no standard for visualizing on the web, and it can be challenging to choose which technology to choose. In their experiment, they test different technologies for interactivity and render speed. The conclusion they could make is that the HTML5 Canvas performed best in terms of performance, and was a clear winner for developing interactive visualizations.

Andersson and W¨orlund has conducted similar work [40] in their comparative study between server-and client-side rendering of web-based data visualizations. They implemented two prototypes, one for each rendering strategy, to measure the impact of render times in different scenarios, such as multiple client connections. They found that client rendering was consistent in lower render time, but had significantly larger data transmission sizes. In their work, both of the strategies use SVG to display the information on the client. They explain in their future work section that this might not be the best approach, and that different rendering strategies might have different optimal solutions.

Measuring time, particularly response time, has proven to be a valid metric for evaluating perfor-mance in web applications. In QoE, the equivalent for this would be interaction latency, which is also quite common [5, 41, 37, 42, 36]. In related works studying remote web rendering systems, there is little done in measuring the scalability in terms of concurrent connections. By summarizing the related work in the remote rendering subject, there are clear advancements in the development of new frameworks and techniques. However, there is not much work done on evaluating the dif-ferent strategies for remote and local rendering. This work will not propose new technologies or techniques, but will rather provide a strong basis on the best-fitting strategy and technology for a specific application use case. Furthermore, during the phase of studying related works, we found no research related to ASP.NET Core Blazor, increasing the scientific relevance of the work.

4.

Method

The work explores the subject of different rendering strategies in depth. The chosen method for this work is an experimental approach. The work will consist of two implementations of the same application. One of the prototypes will render graphics on the server using remote rending, while the other will receive object state data and present it on the client using local rendering. The work will evaluate the different rendering strategies in terms of scalability and QoE. Iyer, Gupta, and Johri [43] explain that scalability in web technologies is tied to the response time of the application combined with the number of concurrent users. However, measuring the scalability in terms of concurrent connections in web-based remote rendering does not seem to be a popular subject. Related works show that response time and data transmission size have proven to be a highly used statistical metric in measuring the performance of web applications, and interaction latency is a popular metric in remote rendering systems. By taking inspiration from similar and related work, this work will measure the response time and interaction latency of the implementations.

Both of the rendering strategies in the experiment will use WebSockets for real-time communication between the server and the client. The number of elements sent over the WebSocket connection will be measured to evaluate the data transmission sizes of the two rendering strategies. For remote rendering, this data is the resulting rendered image. In local rendering, it is the object state that the client should render. By measuring different figure sizes, we can model the scalability of the system in terms of data transmission size. The resulting outcome can be rendered using different technologies. In the experiment by Andersson and W¨orlund [40], they mention that testing different implementations for the strategies is a next step in the research of web visualizations. They explain that while some technologies might be suitable for one strategy, it might prove to be the worst match for the other approach, giving a biased result. This work will test two different formats for each strategy: SVG and Canvas. The main reason for these technologies is the interaction possibilities, and the implementation simplicity. While a conventionally rendered bitmap might be interesting, it does not naturally fit within the requirements of being able to interact with the rendered objects. For the Canvas implementation, the 2D context will be used, rather than the WebGL context. Since the 2D context works on a higher level, it is easier to create a fair implementation between Canvas and SVG. In SVG, a line component would be the equivalent of calling the drawLine() method in the Canvas API. While WebGL has better theoretical performance, this will not be tested due to time limitations.

The tests were chosen based on related work on remote rendering subject as a whole, as well as closely similar work. It is clear that the main problem with web rendering is data transmission and response times between the client and the server. In interactive remote rendering, the main issues is the interaction latency. The tests were split to three categories. The first one is testing one device with an increasing number of elements to see the direct impact of data transmission. The second test focuses on increasing concurrent connections to measure how this affects the response time. The last test covers the interaction latency of the rendered images to evaluate the performance of re-rendering the images. The average of 20 performed iterations will be the result of each test.

4.1

Test data

Images displayed in production systems are often dynamically generated from an input source, which consists of multiple electrical components (such as lines, switches, etc.). Each element has a collection of properties that affect how the final image must be rendered (e.g., a line component requires a list of other objects to which it is connected) and metadata (e.g., a name and description of the component). Therefore, the overall amount of data to be accessed for rendering an image is heavily affected by the object state data.

ˆ Switch: A black square with two overlapping intersecting lines. ˆ Transformer: Two vertically overlapping circles.

ˆ Line: A line connecting two positioned elements (Switch or Transformer).

(a) Switch (b) Transformer

Figure 4: Switch and transformer rendered in the Network Manager application, connected to other components using lines.

Each of these elements has unique data for them to function. For example, the line data requires a list of other elements that it is connected to. All of the components have some metadata, such as a name and description of the component. The reason for this is to increase the overall size since a real-world situation would have significantly more data than just a few state properties.

4.2

Test environment

All of the tests are conducted using a total of four devices; one for the server and three for the clients. The primary client is only running a single instance of a browser. Since this is the client that will be taking measurements, the aim is to have as little interference as possible. In the case of needing multiple clients connected, they will be equally distributed on the other devices. Since this work includes measuring rendering and interaction in the browser, it is not sufficient to use a protocol-based tool, such as an HTTP testing tool. Another reason is that Blazor Server uses a SignalR connection handling all events between the browser and the server. The data passed through this connection is of a binary format. From initial testing, we found that it is not a simple task to effectively simulate this data without reading and understanding the source code.

In similar work by Edvardsson [38], they use the browser developer tools to measure the perfor-mance of different techniques. While this approach is appropriate for testing the perforperfor-mance of a single client, it does not support measuring multiple clients. This work will make use of the JMeter scalability testing tool, together with the Selenium WebDriver plugin [23,24]. This plugin allows us to simulate multiple browser instances to load, render, and interact with a webpage and measure the time of the operations. All of the tests will be simulating a Google Chrome browser instance to execute the appropriate actions. While it would be sufficient to perform the test on several browsers, this will not be done due to time limitations. However, during the implementa-tion phase, the applicaimplementa-tion will be checked using several other browsers to make that there are no major performance differences.

Using a simulated browser to execute actions and measure time opens up a new wave of problems for the test environment. With protocol-based testing, there is little computational effort from the computer, and it is possible to simulate hundreds of requests at once successfully. The browser simulation approach, on the other hand, is far more computationally demanding. During initial testing, it was clear that instantiating more browsers than the number of cores on the processor would cause performance issues. The current environment does not have enough performance

to simulate hundreds of requests. Due to this limited environment, we are only able to test a maximum of 15 concurrent connections.

4.2.1 Device specifications

All of the tests will be using four devices: three clients and a server. In the case of testing the performance of concurrent clients, using multiple browser windows on a single machine might cause performance issues, especially when the number exceeds the processor cores. To mitigate this issue, we will add two additional devices and distribute the rest of the browsers evenly. Figure 1 specify the hardware specifications of the devices. The client device marked with primary is the device running the JMeter instance. This device is only running a single browser window to minimize the possibility of interference with the performance measurements. All of the devices are running locally on the same network to prevent external interference.

Type Processor Memory Network

Server Intel(R) Core(TM) i7-9850H @ 2.6GHz 48 GB LAN Client (primary) Intel(R) Core(TM) i5-8600K @ 3.60 GHz 16 GB LAN Client Intel(R) Core(TM) i7-6700HQ @ 3.50 GHz 8 GB WLAN Client Intel(R) Core(TM) i5-5257U @ 2.50 GHz 8 GB WLAN

Table 1: Testing devices hardware specifications

Software Version

ASP.NET Core 3.1.201

ASP.NET Core Blazor Server 3.1.3

ASP.NET Core Blazor WebAssembly 3.2.0-preview4.20210.8

Apache JMeter 5.2.1

Apache JMeter Selenium/WebDriver Support 3.1

Google Chrome 83.0.4103.61

ChromeDriver 83.0.4103.39

5.

Comparing rendering strategies

This section will present the practical work, which includes an in-depth explanation of the tests performed, the implementation, and what issues occurred during this phase.

5.1

Tests

Three different tests were designed to evaluate and compare different rendering strategies in terms of performance. Each test will be measured for every approach and technology (remote Canvas, remote SVG, local Canvas, local SVG). The measurements derived is the average of 20 iterations.

ˆ Scalability (clients) was measured by connecting multiple clients to the same server. The server would serve the same image to all clients. On a set interval of 1 second, the image would update some elements on the image, and send them back to the client for a re-render. The results from this test will help with answering the first research question. Table 3 shows all of the configurations for the test.

Number of elements Concurrent connections 250, 2 500, 10 000 1

250, 2 500, 10 000 5 250, 2 500, 10 000 10 250, 2 500, 10 000 15

Table 3: Scalability (clients) configuration

Below are the procedural steps for the simulation. Note that once the simulation is initialized, all actions with numbers as steps are taken automatically.

Step # Action

Before test. Initialize and render k browser windows with an image of size n. The server sends constant updates to these instances on a set interval.

1. Start timer.

2. Invoke render event for image of size n. 3. Wait for image to render.

4. Stop timer.

Table 4: Steps for testing increasing clients. k = concurrent clients, n = # of elements

ˆ Scalability (image size) was measured in the same way as client scalability, but increasing the number of elements rather than concurrent connections. To get a better measurement of change over image size, the number of items increase from 250 to 4750, incrementing by 250. The results of this test will help with answering the first research question. Below are the procedural steps for the simulation. Note that once the simulation is initialized, all actions are taken automatically.

Step # Action 1. Start timer.

2. Invoke render event for image of size n. 3. Wait for image to render.

4. Stop timer.

Table 5: Steps for testing increasing elements. n = # of elements

ˆ Interaction latency is measured by simulating a mouse pan movement and measuring the time it takes to re-render the new image. By simulating a browser window, it is possible to move the mouse an exact number of coordinates for each test. The results from this test will help with answering the first and second research questions. Below are the procedural steps for the simulation. Note that once the simulation is initialized, all actions are taken automatically.

Step # Action

1. Render image of size n. 2. Start timer.

3. Move the mouse to the center of the screen and hold the mouse 1 button. This action will begin the pan event 4. While holding the mouse 1 button, move (200, 200) pixels.

5. Release mouse 1 button. This action stops the pan event, which will cause a re-render. 6. Wait for image to update.

7. Stop timer.

Table 6: Steps for testing interaction latency. n = # of elements. The actions taken will simulate a pan movement.

5.2

Implementation

The implementation is inspired by the report by Wessels et al. [20], where they use WebSockets and Canvas to superimpose a raster image.

When implementing the different versions, the main focus was on creating reusable components that could be shared across the two Blazor hosting models, to ensure that the implementations were comparable to a high degree. Both of the rendering strategies make use of the same SVG and Canvas components, and the Blazor hosting models are what defines them as a server- and client-side solution. While there is importance in the visual similarity between the technologies and approaches used, the work is more focused on the performance. Therefore, slight deformities and differences between the techniques are allowed.

The implementation was split into three Visual Studio projects: localRenderingWASM (Blazor WebAssembly), remoteRendering (Blazor Server), and the shared RenderingDataService (C# class library). All of the shared components, such as the Data classes, Canvas, and SVG components, and WebSocket hub, are stored in the class library. The main reason for this is the reusability of components and interfaces on the other projects.

Name Description Data sent Remote SVG Retrieves the graphical element states, composes

an SVG object, and sends it to the client to ren-der.

Composed SVG

Remote Canvas Retrieves the graphical element states. For each element, use Javascript interop [29] to call the 2DContext methods to render the element.

2DContext draw methods

Local SVG Server retrieves the graphical element states and sends it to the WebAssembly client, which com-poses an SVG object and renders.

Object state

Local Canvas Server retrieves the graphical element states and sends it to the WebAssembly client. The client uses Javascript interop [29] to call the 2DCon-text methods to render the element.

Object state

Table 7: Rendering method and technology description

5.2.1 Generating test data

The server contains a SignalR (WebSocket) hub, which handles the data feed to the clients. This hub stores a list of elements, which we define as the object state. Clients can request to receive an image with a specified number of nodes, which returns a JSON encoded array. The client can also request to subscribe to the update event. This event will fire on a 1-second interval, and will update a number of elements and send them back to the client.

For testing purposes, the generated image is stored in-memory. All requests return and update the same list instance of the elements. In the current environment, this is an adequate solution. The main problem arises when different clients request an image of a varying number of nodes, which would cause the list to re-generate for each request. To combat this, when performing the tests, all of the clients will request an image of the same size.

(a) SVG (b) Canvas

Figure 5: Cropped screenshot of local rendering Canvas and SVG with 50 elements. Note that not all 50 elements are visible.

Fig. 5 shows the finished implementation of local SVG and Canvas. It should be noted that there is a slight difference in the visual representation of the two technologies, but will not affect performance in a significant manner.

5.2.2 Implementation issues

While implementing the different strategies and technologies, some issues occurred, which directly affects the resulting tests and measurements.

The idea of the remote Canvas is to send paint events by Javascript interop on a WebSocket connec-tion. These events are initialized by the server and received by the client for execuconnec-tion. The client would execute the commands, which results in the rendered image. The current implementation sends one Javascript interop paint call per element so that the number of calls grows linearly to the number of elements. Sending repeated, or too many, in a short period paint calls (i.e. sending updates on a set interval) to the client would cause congestion, resulting in a connection loss be-tween the server and client. The time when the connection is lost does not seem consistent, and could depend on multiple factors. This behavior causes it to be reflected in the results, where the remote Canvas values could cut off at different points.

Blazor WebAssembly contains a small amount of Javascript code that handles the initial bootstrap-ping and running the WebAssembly. When the WebAssembly client makes browser API calls, such as WebSocket events, it is directed through this Javascript. When implementing local SVG and Canvas using WebAssembly, requesting a large number of elements (around 2000) would cause a memory out-of-bounds exception [44,17], specifying that an error occurs when converting the object to a WebAssembly string. Batching the data was expected to help, it was not the case, and the exception was still thrown. This behavior causes it to be reflected in the results, where the local strategies are not able to be tested on elements over 2000.

6.

Results

The following chapter shows the resulting data from the tests performed. More information about how these tests and measurements are made can be seen in section 5.1.

The graphs show response time, which we define as the time from when the client requests some data to when it is updated and displayed in the browser. In other words, this is the combination of request time and render time.

6.1

Scalability (clients)

CONCURRENT CLIENTS (250 ELEMENTS)

Remote SVG Local SVG Local Canvas

Figure 6: Concurrent clients test using 250 elements with real-time updated data. Tested on 1 to 15 clients.

Figure 6 shows the results of the scalability (clients) test using 250 elements. Due to issues with remote Canvas specified in 5.2.2, it is not included in this result. All of the techniques used are following the same pattern that increasing the number of clients does not have a significant effect in a configuration of 250 elements. The results show that remote technique has a lower response time than the two local versions.

714 748 790 770 0 500 1000 1500 2000 2500 1 5 1 0 1 5 R ES P O N SE T IME ( MS ) CONCURRENT CLIENTS

CONCURRENT CLIENTS (2,500 ELEMENTS)

Remote SVG

Figure 7: Concurrent clients test using 2,500 elements with real-time updated data. Tested on 1 to 15 clients.

Figure 7 shows the results of the scalability (clients) test using 2,500 elements. Due to issues with remote Canvas and WebAssembly specified in 5.2.2, it is not included in this result, meaning that only remote SVG is displayed. As with 250 elements, there is no significant difference between the number of clients. 2 669 2 997 3 596 4 029 0 1000 2000 3000 4000 5000 1 5 1 0 1 5 R ES P O N SE T IME ( MS ) CONCURRENT CLIENTS

CONCURRENT CLIENTS

(10,000 ELEMENTS)

Figure 8: Concurrent clients test using 10,000 elements with real-time updated data. Tested on 1 to 15 clients.

Figure 8 shows the results of the scalability (clients) test using 10,000 elements. As with 2,500 elements, the implementation issues are still present, causing remote SVG to only be displayed. The remote SVG version follows a clear linear pattern as the number of client grows.

6.2

Scalability (image size)

INCREASING ELEMENTS

Remote SVG Remote Canvas Local SVG Local Canvas

Figure 9: Increasing elements. Tested on one client.

Figure 9 shows the results of the scalability (image size) test using a single client. Due to im-plementation issues mentioned in 5.2.2, both of the local rendering techniques cut off at 2,000 elements, and local Canvas cut off at 3,500. All of the techniques follow a linear pattern. The values from the remote SVG stay significantly flatter than other versions.

6.3

Interaction latency

INTERACTION LATENCY

Remote SVG Remote Canvas Local SVG Local Canvas

Figure 10: Interaction latency with increasing elements. Tested on one client

Figure 10 shows the results of the interaction latency test using a single client. Due to implemen-tation issues mentioned in 5.2.2, both of the local rendering techniques cut off at 2,000 elements,

and remote Canvas cut off at 1,000. Unlike the other performed tests, the results do not show any clear pattern shared between all of the techniques. The interaction latencies of both the SVG versions seem to not be affected by the change in element count. Both implementation stay at a constant interaction latency of 300 milliseconds.

7.

Discussion

The following section will discuss the findings from the results, the significance of the results, and how they compare to similar work and their outcomes.

7.1

Scalability (clients)

The first thing that can be seen is the lack of remote Canvas on all of the results. As explained in section 5.2.2, this is due to the technique congesting the connection with Javascript interop requests, causing it to disconnect from the server. When testing with multiple connections, this problem occured more frequently. Since the test relies on multiple browser instances running all at once, a disconnect from a browser would invalidate the results since the number of connections is not correct.

The other noticeable difference between the graphs is the lack of some techniques and technolo-gies when testing on a higher number of elements. The main reason for this, explained in the implementation issues section, is that Blazor WebAssembly is unable to handle data where the size exceeds 2,000 elements.

All of the applicable versions scale 250 elements without any effort. When increasing the number of nodes to 2,500, remote SVG is still able to keep up, and scaling up to 15 users does not seem to affect the response time significantly. The results clearly show that the server handling 15 concurrent connections is entirely feasible when sending a request of reasonably small sizes. When increasing the number of elements to 10,000, the results start to differ. For remote SVG, the response time is almost doubled when increasing concurrent clients from 1 to 15. It would seem reasonable that the response time is increased due to higher work on the server since remote rendering handles more actions on the server. Due to the implementation issues mentioned, these results can not be compared to other versions. This problem makes concluding these results a difficult task.

When attempting to answer the research question ”How does remote and local rendering differ in terms of scalability and QoE when rendering real-time interactive 2D graphics?”, this section will focus on the scalability aspect. The goal of the tests performed is to evaluate how the combination of image size and concurrent connections affect response time, not whether the server can handle 15 connections at all. With any respected web framework, there is no doubt that the server should be able to handle 15 concurrent connections with reasonable request sizes. As explained in the method section, the current testing environment does not allow for more concurrent clients, which leads to focusing on increasing the image data size by altering the number of elements. The fact that 3 out of 4 versions are not able to handle large image sizes causes problems when attempting to answer how they differ, specifically in terms of client scalability.

7.2

Scalability (image size)

The results clearly show an increase in response time when increasing the number of elements, which is a reasonable expectation since more data points are explicitly related to the overall size of the image. While the increase for all versions are relatively linear, they increase at drastically different rates. From the results, we can see that both remote versions have lower response times than their locally rendered counterpart. As mentioned, all of the results follow the same linear pattern, but provide different response times. The main reason for this is likely the re-use of components. However, using the same component does not result in the same response time when using different rendering strategies.

The graph also clearly shows the result of the remote Canvas and WebAssembly implementation issues mentioned in section 5.2.2. Both of the local rendering strategies stop at 2,000 elements,

while remote Canvas stops at 3,500. These results help in answering the first research question regarding image size scalability. From the results, it is clear that choosing a remote rendering approach would result in overall lower response times than local rendering. It is also clear that in situations that require a large number of objects, remote rendering is the recommended approach since the implemented client-side rendering approach is not capable of rendering more than 2,000 elements.

7.3

Interaction latency

The results from the interaction latency test is more grouped in technology rather than rendering approach. The reason for this is likely the implementation of different techniques. Since SVG is placed in the DOM, it only requires a transform property on the root object, and not a full re-render. In contrast, the Canvas API renders in a raster format, meaning that it requires a re-render to compose a new image. Naturally, this approach would take longer in terms of interaction latency, which can be seen in the results. For this reason, remote SVG is the top contender together with local SVG, having a constant response time of around 300 milliseconds. Hoxmeier and DiCesare [7] explain that this could be considered a common frequent task, which we believe matches up quite well. It is unclear why the interaction latency for these SVG technology is constant, but we speculate that this is overhead from the JMeter testing simulation when waiting for an element to update. When setting up the test, we specify that the JMeter instance should wait until a specific element in present and visible in the browser. It is possible that the check will occur on a set interval, for example every 300 milliseconds.

The results from this section will assist in answering both of the research questions. From the above discussion, it is clear that SVG is the superior technology over Canvas for interactive 2D graphics using the Blazor framework. When answering the first research question in terms of QoE, we can make see that there are no distinguishable features between the two rendering strategies. Whether local or remote rendering is chosen, they will not differ in terms of interaction latency. However, the choice of technology makes a significant difference, which helps in answering the second research question. In our result, we see that remote SVG has outstanding interaction performance compared to remote Canvas. With these results in mind, when implementing a remote rendering interactive system for 2D graphics using Blazor, SVG is the recommended version over the two.

7.4

Related work

In the general state-of-the-art evaluation of different web rendering strategies and technologies used, this work might not be directly applicable. A significant part of the results is the cause of performance issues with Blazor and its hosting models, as explained in the background section. In similar work, there seem to be a pattern that remote rendering systems are not as performant as the local counterpart. In our case, this seems to be the opposite. In the experiment by Edvardsson using a Desktop PC for rendering 1,000 data points on a 1920x1080 local Canvas [38, Figure 40], they receive average response times of 20-70 milliseconds. Compared to our results, client-side Canvas using 1,000 elements took over 2 seconds, while the server-rendered Canvas took under 2 seconds. Another work by Andersson and W¨orlund [40] explore web visualization of 2D graphs using remote and local SVG. In their results using fixed data samples, their results show an average response time of 100 milliseconds for rendering 1,000 data points [40, Figure 5.2]. Using the same size, the results from our work show around 500 milliseconds for remote SVG, and 1,500 milliseconds for local SVG. While these results might not be directly comparable due to factors such as the difference in element vs. data point size, we speculate that it is mostly due to the nature of the Blazor hosting models.

Blazor is a relatively new web framework, which uses special techniques. From the results, we see a clear lack of performance in Blazor WebAssembly, giving the complete opposite results to

related works. The Blazor WebAssembly hosting model is currently in preview, and the developers recommend that it should not be used in production applications. As the developers keep updating the framework, performance increase might be significant, decreasing the relevance of these results. Blazor is designed to work the same, no matter what hosting model it is running on. As a result, in the case that local rendering using Blazor WebAssembly proves to be a better choice in the future, it is easily interchangable.

7.5

Limitations and issues

In the starting phase of the work, the initial plan was to use the JMeter tool to simulate WebSocket connections, allowing a higher number of concurrent connections, making scalability testing a breeze. It was later discovered during the implementation phase that it was not a simple task to simulate Blazor Server WebSocket connections. The main reason was that the Blazor Server uses unique identifiers in the binary message. When attempting to intercept and copy these messages, the server would not respond since the identifier doesn’t match. This problem ultimately forced us to simulate a browser, which forced the scale down to a manageable amount of connections. We believe that there are better solutions to this problem, such as using the source code of Blazor Server to generate messages with a valid identifier. We did not research this due to the time limitations of the work, and we would have to settle with simulating browsers. Instead of having multiple connections sending small requests, a lower number of browsers would be sending large requests in hopes of getting the same result.

In the end stages of implementing local rendering using WebAssembly, rendering using a large number of elements caused a memory exception, as explained in section 5.2.2. This error was caused by converting between a string to a WebAssembly string. Different optimization techniques, such as sending the elements in batches, were implemented in hopes of mitigating this issue. These efforts gave no results, and the error still occurred. Due to time limitations, this was decided to be a Blazor WebAssembly issue. If this problem was resolved, we could possibly see different results and be able to draw other conclusions.

The idea of a remote rendering using Canvas was interesting. Related work show that this is not a conventional method, and generating bitmaps is far more common. Due to time limitations, our work does not make use of the bitmap technique. Instead, the Canvas paint calls are sent to the client. For example, if the server wanted the client to display a line, it would request the client to call the function that draws a line. As expected, this sends a lot of requests through the network, introducing the issues discussed in section 5.2.2. As seen by the results, the point where the error occurs is not consistent, which is a highly undesirable result.

8.

Conclusions

The work has evaluated the difference in performance and QoE between remote and local rendering using SVG and Canvas in an ASP.NET Core Blazor environment. The general conclusion is that, for all performed tests, remotely rendered SVG is a clear winner when combining rendering strategy and technology. When the decision is based on which rendering approach to use, remote rendering is highly preferable for all situations with a large number of data. The main reason for this is the inability of Blazor WebAssembly, which is used in local rendering, to handle large amounts of data due to limited memory access. When developing a Blazor-powered system that requires high interaction capabilities, the recommended technology to use is SVG, regardless of the rendering approach (local or remote).

The results derived from the work are the opposite to many related works, where the remote solutions do not result in lower response times. We conclude that the reason is the performance of the Blazor framework. For this reason, the significance of the results are highly affected on whether the framework is used or not. While this work shows that remote rendering using Blazor Server has significant performance over Blazor WebAssembly, this could change in the future since Blazor WebAssembly is currently in preview. Due to the fact that code written for both of the hosting models are completely interchangable, the decision on what strategy to use is not final. If the performance improves for Blazor WebAssembly, it is possible to change later.

8.1

Future work

All of the tests were conducted on a single browser version, Google Chrome. Future work on evaluating different rendering techniques using Blazor should include different browsers, which would help in making the test results more general, and increases the overall legitimacy of the work. In addition, all of the tests were performed using similar types of devices. It would be interesting to see how the different devices handle the rendering.

To further research in the area of 2D graphics using Blazor, similar and more extensive tests need to be performed further in the future. As Blazor is a new framework with constant updates and improvements, it is possible that the results derived from this work are soon invalidated. During development, we soon realized that Blazor WebAssembly had issues with large amounts of data. In further updates of the Blazor framework, this could potentially be fixed, allowing further testing of Blazor WebAssembly. For the work, only a small number of technologies were implemented. A clear step forward would be testing the WebGL context for the Canvas API. Another step forward would be following related works by generating bitmaps on the server.

When evaluating the scalability of multiple clients, the number of clients must be high enough to which the server cannot keep up. Due to unexpected issues with Blazor Server and time limitations, this work performs scalability tests by simulating browsers, which decreases the number of concurrent clients. Due to a limited testing environment of 3 client devices, the number of clients is not enough to give any significant results. By simulating the WebSocket connection directly, a single testing device can handle multiple concurrent connections. An example on how this can be achieved is by examining the source code on how the Blazor Server sends binary messages through the SignalR connection.

All of the tests were performed in a local testing environment. In a production environment, there is a high possiblity that these devices would be connected over the internet, which would result in a slower connection. For future tests, the server should be deployed to a production environment, giving more accurate results. This is especially true for increasing the image size.

References

[1] Microsoft, “Introduction to ASP.NET Core Blazor,” 2020. [Online]. Available: https: //docs.microsoft.com/en-us/aspnet/core/blazor/

[2] Mozilla, “Canvas API,” 2006. [Online]. Available: https://developer.mozilla.org/en-US/ docs/Web/API/Canvas API

[3] The Khronos Group, “WebGL - OpenGL ES for the Web,” Jul. 2011. [Online]. Available:

https://www.khronos.org/webgl/

[4] Adobe Blog, “Flash & The Future of Interactive Content,” Jul. 2017. [Online]. Available:

https://theblog.adobe.com/adobe-flash-update/

[5] S. Shi and C.-H. Hsu, “A Survey of Interactive Remote Rendering Systems,” ACM Computing Surveys, vol. 47, pp. 1–29, May 2015.

[6] F. Mwalongo, M. Krone, G. Reina, and T. Ertl, “State-of-the-Art Report in Web-based Visualization,” Computer Graphics Forum, vol. 35, no. 3, pp. 553–575, 2016. [Online]. Available: http://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.12929

[7] J. A. Hoxmeier and C. Dicesare, “System Response Time and User Satisfaction: An Experi-mental Study of Browser-based Applications,” 2000.

[8] I. Hickson, “A vocabulary and associated APIs for HTML and XHTML,” 2010. [Online]. Available: https://www.w3.org/TR/2010/WD-html5-20100624/

[9] Mozilla, “The WebSocket API (WebSockets),” 2010. [Online]. Available: https: //developer.mozilla.org/en-US/docs/Web/API/WebSockets API

[10] ——, “Introduction to the DOM - Web APIs — MDN,” 2005. [Online]. Available:

https://developer.mozilla.org/en-US/docs/Web/API/Document Object Model/Introduction

[11] QuinnRadich, “Retained Mode Versus Immediate Mode - Win32 apps.” [Online]. Available: https://docs.microsoft.com/en-us/windows/win32/learnwin32/ retained-mode-versus-immediate-mode

[12] W3C, “Scalable Vector Graphics (SVG) 2,” 2018. [Online]. Available: https://www.w3.org/ TR/SVG2/

[13] G. Vladi´c, N. Mili´c, ˇZ. Zeljkovi´c, and D. Avramovi´c, “Evaluating Web browser graphics rendering system performance by using dynamically generated SVG,” Journal of Graphic Engineering and Design, vol. 3, no. 1, pp. 15–22, May 2012. [Online]. Available:

https://doaj.org

[14] A. Rossberg, B. L. Titzer, A. Haas, D. L. Schuff, D. Gohman, L. Wagner, A. Zakai, J. F. Bastien, and M. Holman, “Bringing the web up to speed with WebAssembly,” Communications of the ACM, vol. 61, no. 12, pp. 107–115, Nov. 2018. [Online]. Available:

http://doi.org/10.1145/3282510

[15] A. Mø ller, “Technical perspective: WebAssembly: A quiet revolution of the web,” Communications of the ACM, vol. 61, no. 12, p. 106, Nov. 2018. [Online]. Available:

http://doi.org/10.1145/3282508

[16] Emscripten Contributors, “Main — Emscripten 1.39.14 documentation.” [Online]. Available:

https://emscripten.org/

[17] L. Clark, “Memory in WebAssembly (and why it’s safer than you think) – Mozilla Hacks - the Web developer blog,” 2020. [Online]. Available: https://hacks.mozilla.org/2017/07/ memory-in-webassembly-and-why-its-safer-than-you-think

[18] K. E. Ogundeyi and C. Yinka-Banjo, “WebSocket in real time application,” Nigerian Journal of Technology, vol. 38, no. 4, pp. 1010–1020, Dec. 2019. [Online]. Available:

https://www.ajol.info/index.php/njt/article/view/191780

[19] V. Pimentel and B. G. Nickerson, “Communicating and Displaying Real-Time Data with WebSocket,” IEEE Internet Computing, vol. 16, no. 4, pp. 45–53, Jul. 2012.

[20] A. Wessels, M. Purvis, J. Jackson, and S. Rahman, “Remote Data Visualization through WebSockets,” in 2011 Eighth International Conference on Information Technology: New Gen-erations, Apr. 2011, pp. 1050–1051.

[21] “HTML5 WebSocket - A Quantum Leap in Scalability for the Web.” [Online]. Available:

http://www.websocket.org/quantum.html

[22] M. Wenzel, L. Gericke, R. Gumienny, and C. Meinel, “Towards cross-platform collaboration - Transferring real-time groupware to the browser,” in Proceedings of the 2013 IEEE 17th International Conference on Computer Supported Cooperative Work in Design (CSCWD), Jun. 2013, pp. 49–54.

[23] Apache Foundation, “Apache JMeter - Apache JMeter—,” 2020. [Online]. Available:

https://jmeter.apache.org/

[24] A. Pokhilko, “Documentation :: JMeter-Plugins.org,” 2020. [Online]. Available: https: //jmeter-plugins.org/wiki/WebDriverTutorial/

[25] “SeleniumHQ Browser Automation.” [Online]. Available: https://www.selenium.dev/

[26] “ChromeDriver - WebDriver for Chrome.” [Online]. Available: https://chromedriver. chromium.org/

[27] “Geckodriver — Firefox Source Tree Docs 78.0a1 documentation.” [Online]. Available:

https://firefox-source-docs.mozilla.org/testing/geckodriver/

[28] Microsoft, “Blazor — Build client web apps with C# — .NET,” 2018. [Online]. Available:

https://dotnet.microsoft.com/apps/aspnet/web-apps/blazor

[29] ——, “Call JavaScript functions from .NET methods in ASP.NET Core Bla-zor,” 2020. [Online]. Available: https://docs.microsoft.com/en-us/aspnet/core/blazor/ call-javascript-from-dotnet

[30] julienGrd, “[Blazor WebAssembly] Serious performance issues · Issue #21085 · dotnet/asp-netcore,” 2020. [Online]. Available: https://github.com/dotnet/aspnetcore/issues/21085

[31] Microsoft, “ASP.NET Blog — Blazor Server in .NET Core 3.0 scenarios and performance,” Oct. 2019. [Online]. Available: https://devblogs.microsoft.com/aspnet/ blazor-server-in-net-core-3-0-scenarios-and-performance/

[32] Y.-C. Chang, K.-T. Chen, C.-C. Wu, C.-J. Ho, and C.-L. Lei, “Online game QoE evaluation using paired comparisons,” in 2010 IEEE International Workshop Technical Committee on Communications Quality and Reliability, CQR 2010, Jul. 2010, pp. 1–6.

[33] K.-T. Chen, Y.-C. Chang, P.-H. Tseng, C.-Y. Huang, and C.-L. Lei, “Measuring the latency of cloud gaming systems,” in MM’11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops, Nov. 2011, pp. 1269–1272.

[34] P. Eisert and P. Fechteler, “Remote Rendering of Computer Games.” Jan. 2007, pp. 438–443. [35] A. Evans, M. Romeo, A. Bahrehmand, J. Agenjo, and J. Blat, “3D graphics on the web: A survey,” Computers & Graphics, vol. 41, pp. 43–61, Jun. 2014. [Online]. Available: