Faculty of Computing
Blekinge Institute of Technology SE-371 79 Karlskrona Sweden
A closer look at WebAssembly
Denis Eleskovic
26
thAugust, 2020
ii (45 )
This diploma thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in partial fulfillment of the requirements for the diploma degree in Software Engineering. The thesis is equivalent to 10 weeks of full time studies.
Contact Information:
Author:
Denis Eleskovic
E-mail: deel18@student.bth.se University advisor:
Andreas Arnesson
Department of Software Engineering
Faculty of Computing
Blekinge Institute of Technology SE-371 79 Karlskrona, Sweden
Internet : www.bth.se
Phone : +46 455 38 50 00
Fax : +46 455 38 50 57
1 (45 )
ABSTRACT
WebAssembly is a new emerging technology for the web which offers a low-level bytecode format for other languages to compile to. The aim of the technology is to effectively speed up the performance of the web, as well as to offer a way for developers to port existing libraries and code that were written in static languages such as C/C++ or Rust. The technology is currently supported in all the major browsers of today.
This study takes a closer look at how the technology works and how it is compiled and executed across two of the major browsers as compared to JavaScript. Furthermore, a smaller experiment was conducted where AssemblyScript was used to compile to WebAssembly, with AssemblyScript being a typescript-to-WebAssembly compiler. The two technologies were then tested in terms of runtime performance and consistency across two browsers, operating systems as well as with different optimizations.
The study showed that the technology goes through ahead-of-time-compilation and optimization through a compiler toolchain that depends on the language used as opposed to JavaScript which makes use of just-in-time-compilation. Furthermore, the fundamental structure for WebAssembly proved to be limited in order to be able to better link directly to machine code and through this offer performance benefits. The experiment conducted showed that WebAssembly performed better when it came to pure calculations but fell behind JavaScript when performing operations on an array. The reason is most likely due to the overhead of passing data structures between the two technologies. The experiment further showed that WebAssembly was able to offer a more consistent and predictable performance than JavaScript.
Keywords: WebAssembly, JavaScript, JITC, AOT
2 (45 )
TABLE OF CONTENTS
1
INTRODUCTION ... 31.1
BACKGROUND... 31.2
PURPOSE ... 31.3
SCOPE ... 42
QUESTIONS TO BE ANSWERED ... 53
METHOD ... 73.1
LITERATURE REVIEW ... 73.2
EXPERIMENTAL STUDY ... 84
LITERATURE REVIEW ... 114.1
INTRODUCTION ... 114.2
WEBASSEMBLY ... 114.2.1
Semantic Phases ... 124.2.2
AssemblyScript ... 134.3
JAVASCRIPT ... 134.3.1
Compilers ... 134.3.2
Just-in-time ... 144.3.3
Profile-based optimizations ... 154.3.4
V8 Engine... 154.3.5
Spidermonkey Engine ... 164.4
CONCLUSION ... 175
EMPIRICAL EXPERIMENT ... 195.1
RESULTS ... 195.2
CONCLUSION ... 306
ANALYSIS AND DISCUSSION ... 316.1
WEBASSEMBLY AND JAVASCRIPT IN THE BROWSER ... 316.2
STRENGTHS AND WEAKNESSES ... 336.3
PERFORMANCE ... 356.4
CONSISTENCY ... 367
CONCLUSION ... 387.1
KEY CONCLUSIONS: ... 398
FUTURE WORK ... 40REFERENCES ... 41
3 (45 )
1 I NTRODUCTION 1.1 Background
As the web has developed over the years, developers have always had to keep up with the latest technological advancements meant to make web applications faster and more efficient.
JavaScript has proven powerful enough to solve most issues, with it being the only natively supported language on the web. As the need for speed and efficiency has rapidly escalated, JavaScript has shown it does not possess the necessary performance to produce apps which require near native performance. Consequently, four of the major browser companies came together to create WebAssembly with the goal of bringing safe, fast and portable code to the Web [1]. WebAssembly is a low level bytecode; a compilation target for several different languages meant to enable developers to create apps at near native performance and can be used in more intensive use cases, making CAD applications, video/image editing and scientific visualization and simulation available directly in the browser [1].
1.2 Purpose
The release of WebAssembly has garnered interest in the developer community, with many choosing to benchmark the technology using various methods and comparing it against plain JavaScript [2], [3], [4], [5]. Consequently there have been many benchmarks performed for the different languages supported by WebAssembly but few have been performed using AssemblyScript [6]. AssemblyScript is a strict subset of Typescript, which compiles to WebAssembly, effectively making AssemblyScript a Typescript-to-WebAssembly compiler [7]. AssemblyScript enables existing JavaScript developers to be able to take advantage of WebAssembly in applications without needing to learn a new language; this can have a major effect on how the web develops for the foreseeable future as JavaScript is the most used language [8].
Therefore the aim of this paper is to provide greater insight into how WebAssembly operates
in order to shed more light on the technology and to better understand where it fits in on the
web. Furthermore, there have been various benchmarks performed using Assemblyscript but
they have not looked at performance across browsers, operating systems and also how the
technology is affected by the JavaScript engine [6].
4 (45 )
1.3 Scope
In order to limit the scope of this research, the thesis will not look at if WebAssembly can
improve the load time of a web application or the effects of building entire websites through
WebAssembly. The paper looks to provide a greater insight into WebAssembly and where it
fits into the picture on the web. Furthermore, a small set of benchmarks will be used to test
the technologies in different aspects in the two of the major browsers of today as well as
across two operating systems while looking at how each JavaScript engine affects the
technologies. Furthermore, in order to limit the scope of browsers used for testing. Two of the
most popular browsers on desktop computers were chosen for usage in the benchmark [9].
5 (45 )
2 Q UESTIONS TO BE ANSWERED
This study aims to take a closer look at WebAssembly in order to better understand how it works, for what problem the technology was created to solve, but also how it fits into the existing ecosystem on the web. Furthermore, a set of benchmarks will be selected to further experiment with the technology in different browsers, operating systems and different states of optimizations.
The questions to be answered in this study are the following:
RQ1. How are WebAssembly modules compiled and executed in the Firefox and Chrome browsers compared to JavaScript?
WebAssembly was built for the web, meaning it is supposed to run alongside JavaScript in the browser. Since WebAssembly is a low level bytecode that was created as a compilation target for other languages and is supposed to run in the browser, the research question aims to investigate how programs can be compiled to WebAssembly and executed in a browser environment as compared to JavaScript. The Firefox and Chrome browsers were chosen for the benchmark as they are the most popular browsers used on desktops [9].
RQ1.1. What are the strengths and weaknesses of WebAssembly?
The aim of this research question is to try and better understand in what areas the technology excels and where it might fall behind, further building on the first research question posed.
Since the technology is not a complete programming language, it would not be fair to directly compare it to JavaScript. Hence, the focus will be on what the technology excels at and what it might not be suited for.
RQ2. Does WebAssembly offer an improved runtime over JavaScript when executed in the Firefox and Chrome browsers?
The goal with this research question is to compare the execution time for different
benchmarks and pit the technologies against each other in the chosen browsers and across
operating systems as well as different states of optimizations. The reason being to further
6 (45 )
evaluate if the underlying operating system affects the performance, but also how WebAssembly is affected by the JavaScript engine.
RQ2.1. Does WebAssembly offer a more consistent and predictable performance over JavaScript when executed in the Firefox and Chrome browsers?
The above research question expands on and builds on the first research question as there
have been mentions about WebAssembly offering a faster and even more consistent
performance due to how it is compiled but not any direct testing performed to verify the
claim [3]. Hence, the aim of the research question is to compare the technologies based on
consistency and predictability in performance.
7 (45 )
3 M ETHOD
3.1 Literature review
A literature study will be conducted with a focus on how WebAssembly operates as opposed to JavaScript and with this also provide more insight into the strengths and weaknesses of the emerging technology. It is necessary to obtain more information about how respective technology works in order to fairly compare them. Furthermore there is also a need to provide a clearer view into what kind of problem WebAssembly aims to solve and where the technology fits into the picture. WebAssembly is also a rather new technology which is still evolving; as such it is necessary to search the internet for further information on how it works. However, this can also prove to limit the study in its reliability as there might not be as much valid material available on the technology. Thus, less reliable sources may be included as sources of information as a means of procuring additional information about the technology. Through this, the paper will aim to answer RQ1 and RQ1.1 effectively.
The literature study will be performed by searching various databases on the internet through the use of the search engine Google for different papers regarding the technologies. The words to be used will be: “WebAssembly”, “WebAssembly compiler”, “JavaScript engine”,
“JavaScript benchmark”, “JavaScript JIT” and “JIT-compilation”. If the search for papers
proves insufficient then the search of alternative sources will be conducted in order to fill any
gaps in knowledge that might occur. The criteria to be used in order to identify relevant
literature will be looking at if the information provided is from a scientific paper, contains
information that is relevant to the technologies and how old the information is. Once the
information has been collected, it will be compiled and analyzed in order to clarify the
information and present an answer to RQ1 and RQ1.1 respectively.
8 (45 )
3.2 Experimental study
In order to acquire an accurate measurement of the performance of a given technology one would have to tackle the challenging task of creating an extensive suite of benchmarks, these would then need to be tested across different browsers, operating systems and devices. This is an enormously time-consuming task which would not be feasible, given the time limit of this thesis. Hence, a smaller set of benchmarks were selected based on previous benchmarks performed on the technology and then tested in two of the major browsers of today but also across two operating systems [10], [11]. The benchmark suite was based on previous benchmarks which were used to test the SpeedyJs compiler. As the SpeedyJs compiler operates on a similar basis as AssemblyScript, benchmarks were chosen for this reason.
SpeedyJs also uses Typescript as a way to type check code against the WebAssembly specifications [11].
The experiment was conducted in order to gauge if WebAssembly offered a more advantageous performance and if the technology also offered a more consistent performance over JavaScript. The benchmarks included a variety of tests which were pre-configured for SpeedyJs and which were narrowed down for this experiment. The reason for narrowing down the benchmarks and not using all of them is due to time constraints, as all the benchmarks needed to be performed across browsers and operating systems. The benchmarks did not adhere to the code syntax of AssemblyScript, hence they required rework as well as creating a new environment where the AssemblyScript benchmarks could be compiled. The modules were compiled using the full runtime and the files were not bundled.
The algorithms included IsPrime, Fib, Merge sort, Array Reverse and Nsieve. IsPrime is an algorithm for calculating if the input provided is a prime number, while Fib is used to calculate the Fibonacci number of 40. Array Reverse calculates the reverse of an array containing 10 000 elements up to 999 times. The Merge sort test is implemented to sort an array containing 10 000 elements using the Merge sort algorithm. The Nsieve test calculates the prime numbers in the range of 2 - 39 9999 using the sieve of Eratosthenes [11].
In order to obtain reliable results as the execution of certain tests would finish too quickly and
to minimize the margin of error the iterations were increased for each algorithm. The
benchmarks were iterated individually. The Merge sort, Nsieve, Prime number algorithms
9 (45 )
were iterated 1000 times, the Array Reverse algorithm was iterated 100 times, the Fibonacci experiment was iterated one time. This was then done 30 times in both a warm and cold browser session, similarly to how a previous benchmark was conducted [10]. For the cold run aspect of the experiment, the browser was closed between each run in order to ensure that the compiler did not manage to optimize the code, while the warm run session allowed the compiler to optimize the code by running the benchmark one time before measurement took place. Furthermore, the browser was kept open between each run in the warm run session.
The reason for this aspect of the experiment is to analyze and see if WebAssembly is able to gain more in performance when optimized as compared to JavaScript, in essence seeing how WebAssembly is affected by the JavaScript engine. Furthermore the tests were executed in two of the most used browsers on desktops today, Firefox and Chrome [9]. The settings in Firefox were also changed in order to prevent the browser from requesting to close the window if the execution took too long and to prevent the browser from running in safe mode in the event of a crash.
The JavaScript implementations of the selected benchmarks were tested first in order to establish a baseline for which WebAssembly could be tested against. The WebAssembly implementations of the algorithms used AssemblyScript. Both the technologies were implemented to the same degree as much as possible. However, some limitations to keep in mind at the moment are that AssemblyScript is not able to effectively pass complex data structures between the two technologies at this point in time. Hence, the instantiation of the array to be used by the algorithms performing operations were done inside of the AssemblyScript module. This was done instead of using additional libraries to enable the passing of data structures as it would add an additional overhead to the WebAssembly implementation [3].
The AssemblyScript performance included the overhead of jumping between the two
technologies but also the time it takes for the browser to download and instantiate the
WebAssembly file [3]. A technique which streams a WebAssembly module, meaning the
engine can begin working on the module as it is being downloaded, was utilized in the
experiment [3]. Furthermore the experiment was performed in a local environment in order to
minimize the effects of an internet connection as this could have affected the total time
needed by the browser to begin parsing the file.
10 (45 )
The experiment was conducted on an AMD Ryzen 5 2600X 3.60 GHz machine with 16GB of
memory dual-booting Windows 10 version 1909 and Ubuntu 20.04 in the Firefox browser
version 78.0.1 (64-bit) and Chrome version 83.0.4103.116 (64-bit). The results were
compared based on their geometric mean and standard deviation, which was calculated using
the Geometric mean formula, and the Standard deviation formula.
11 (45 )
4 L ITERATURE REVIEW 4.1 Introduction
In order to better understand how the two technologies differ, it is important to first understand what WebAssembly is and how it works in comparison to JavaScript, in both how WebAssembly is supposed to offer better performance but also in how it is compiled and able to target the web platform as a whole.
4.2 WebAssembly
WebAssembly is a new type of intermediate language in binary form for the web that is meant to be run alongside JavaScript in the browser, it provides some new features and also helps improve performance [1], [11], [12]. In order for the different JavaScript engines to be able to execute WebAssembly modules, the browser companies have extended the capabilities of their respective virtual machine, effectively enabling WebAssembly to be executed in the browser [12]. The binary format was created to help speed up compute- intensive tasks and to ensure consistent performance as opposed to regular JavaScript [1].
However, WebAssembly is not meant to be written by hand but instead it is aimed at being a compilation target for other languages. The technology also does not support garbage collection at this point in time but is rather a long term goal, currently this is taken care of by the language compiling to WebAssembly. This means that code written in multiple languages which could not run on the web before have now become available; effectively enabling developers to port existing code from such languages as C/C++ and Rust to the web [12].
In general compilers work by taking a source language and converting it into machine language which the host CPU can then use to execute the instructions provided.
WebAssembly is in this regard rather similar to an assembly language, the exception being
that each assembly language (x86, ARM) is specifically tied to certain machine architecture
while WebAssembly is not. The reason behind this is because you never know what kind of
architecture will be used by the end-user. Hence there is a need to provide a type of
architecture agnostic assembly language [13]. WebAssembly takes it another step closer to
actual machine code than JavaScript source code due to it having a more direct mapping to
machine code. When a script is executed, the browser will download the WebAssembly and
compile it directly to machine code for the underlying hardware, effectively making the
12 (45 )
execution of WebAssembly faster than normal Just-in-time compilation of the JavaScript source due to skipping the need for parsing and optimizing the source code [13].
WebAssembly is built around certain concepts in order to be able to encode and provide its low-level binary format. The technology is a bit limited in terms of value types as it only works with four basic types. These are all IEEE standard floating-point arithmetic numbers.
Each of these types of values can have a 32 and 64 bit width. There is also no distinction between signed and unsigned integers [14]. The technology handles instructions by basing its computational model on a stack machine which works by a last-in, first-out principle. The instructions will perform operations on data by popping the arguments from the operand stack and pushing results back in [14].
When writing code for WebAssembly, there may be instructions provided that under certain conditions produce traps. When this happens, WebAssembly will abort execution since traps cannot be handled directly by WebAssembly. Instead information will be passed to the host environment where the problem can be raised [14].
The concept for functions work by taking a sequence of values as parameters and performing operations before returning results made up of a sequence of values. In the current version of WebAssembly, functions are only able to return one value [14].
WebAssembly makes use of linear memory, which is an array of raw bytes. The memory is created with an initial size but can be changed as needed. When a program is executed, it can store values in the memory but it is able to also access it at any byte address [14].
4.2.1 Semantic Phases
The modules are distributed in a binary format where the format is later decoded and made into an internal representation of a module. In a real implementation this would instead be compiled directly to machine code. In order for WebAssembly modules to work it needs to be valid. The validation phase performs checks for varied conditions to ensure the module is both safe and executable. One condition which is especially important is to perform type checking of functions and the sequences in their body. [14]
When the module has been decoded and validated it is finally ready to be executed. The
execution of a module can be seen as creating a module instance - meaning everything which
13 (45 )
is needed for execution is first instantiated, such as memory, tables and globals. Once this is performed the module can then be executed by invoking an exported function in a module instance. [14]
4.2.2 AssemblyScript
AssemblyScript is a strict subset of Typescript which is currently being developed with the aim of providing the advantages of WebAssembly to existing JavaScript developers. The AssemblyScript compiler works by compiling to WebAssembly through Binaryen [2].
Binaryen is a compiler used for compiling to WebAssembly. It is a library written in C++
aimed at making compiling fast and effective. It performs both minimizations of the code base as well as optimization. Binaryen is able to utilize multi-threading, meaning it can both build intermediate representations and optimize it at the same time [18].
4.3 JavaScript
JavaScript is an object-oriented programming language that is used in web pages to make them more dynamic. The language supports multiple paradigms such as object-oriented, procedural and declarative [16]. Due to its dynamic nature it utilizes just-in-time-compilation [17]. The process of compiling JavaScript code to machine code for execution varies on the browser the code is being run in, with the SpiderMonkey engine being run in the Firefox browser and the V8 engine being run in Chrome [18]. However, there is a general approach that the major browsers implement in order to speed up the performance of the browser. The JavaScript engines adopt a very complex multi-tier architecture. The architecture is composed of several execution tiers including the interpreter and the compiler, these tiers then operate on a function-by-function basis on the code to speed up the execution of code [19].
4.3.1 Compilers
In order to execute code, the code first needs to be translated from the language it was implemented in and into a machine language that can be run by the computer. For this there are mainly two methods [20].
The first method involves using an interpreter. An interpreter steps through the code and
executes it on a line-by-line basis. The interpreter prefers to directly evaluate expressions and
execute statements. An interpreter may need to process the same expressions and statements
14 (45 )
several times compared to a compiler, hence the reason for it being considered slower.
However, an interpreter is easier to move to another machine and they are faster to get up and running and run the program as well. Interpreters are quick to start but slow to execute [20].
The second method is to use a compiler. A compiler translates code written by human programmers using a high-level programming language into low-level machine code that can be understood by the machine. During the process of compiling the program, the compiler will also use this time to spot mistakes in the syntax and report them. The reason for using a high-level language is that it is closer to the way humans think about problems, the compiler can also effectively spot any issues with the programs but high-level languages also enable programmers to port the same code to many different machine languages, resulting in the program being available on multiple machines. Compilers need to compile the code before actually executing the program; compilers are slow to start but fast to execute [20].
4.3.2 Just-in-time
Just-in-time compilers (JITC) were designed to compile code during execution. It is an attempt at combining the better of two worlds, which are an interpreter and a compiler. To provide a low-level overview they tend to work as follows;
The JITC tends to be split up in three tiers. Starting from the first tier, the parser translates the source code which was provided for an invoked JavaScript function to bytecode. The bytecode is then executed in order to ensure a quick start to the JavaScript execution. From here the execution is repeated several times, depending on the engine it will repeat it a set number of times before considering the code “warm”. When the bytecode is considered
“warm”, the baseline JITC will trigger and start to compile the bytecode to machine code
with minimal optimizations. When this happens the baseline JITC will also collect some
profile information by inserting instrumentation code, should the baseline code be executed
enough times to collect stable profiles - the number varies between each browser's engine,
then the function that was being executed will be considered “hot”. This will in turn initiate
the third phase in the compiler. The already optimizing JITC will begin to recompile the
bytecode using various optimizations, the most powerful one being profile-based
optimization [19].
15 (45 )
4.3.3 Profile-based optimizations
JavaScript is a dynamically-typed language which means that the types of different variables in the code are interchangeable during execution; even properties can be dynamically inserted and deleted. Due to this, the compiler constantly needs to evaluate what type of object or variables are being passed. After some time the compiler will try to predict subsequent code execution based on patterns collected in previous executions, so called profiles. The profile information is related to the dynamic behaviors during execution. The JITC then proceeds to generate code to fit these profiles, attempting to predict which behavior will be repeated in future executions [19].
JavaScript engines tend to use hidden classes, a list of properties and their offsets within the object. This is used to represent the shape of an object, which means every object has a pointer to its hidden class. The hidden class is used when we want to access a specific property from the object [19]. This can later be used, depending on the code and what the JITC observes when attempting to optimize the code to either optimize the execution by ignoring the hidden class - called inline caching, since it has observed that the function is always called with the same object and will now expect the same identical object in the future. The JITC will try to validate each assumption it makes by inserting guard code. The address of the hidden class is cached and then compared to the current object; if it differs it signals to the compiler that another differently shaped object has been passed as an argument.
Thus, the JITC will begin to recover - called deoptimization, it will halt execution at that point in the optimized code and proceed to return to a similar position in the baseline code.
This time without any speculation and proceed with the execution [19].
4.3.4 V8 Engine
The V8 engine works by taking the JavaScript source code and passing it through their parser to create an intermediate representation in the form of an Abstract Syntax Tree (AST) [19].
An Abstract Syntax Tree is a tree-like representation of the structure of source code. Each
node in the tree holds a construct within the source code [21]. The AST is then consumed by
the first interpreter called Ignition. Ignition will proceed to generate unoptimized machine
code or otherwise called as bytecode, bytecode being an abstraction of machine code. The
bytecode is then executed several times while being analyzed. After a while it will be passed
to the optimizing compiler, TurboFan, which will then optimize the code and seek to de-
16 (45 )
optimize based on the conditions not being met and return to the slower baseline code and restart the process [21].
Figure 1 - Generic pipeline of code compilation and execution in the V8 engine [4].
The interpreter Ignition consists of bytecode handlers that handle specific bytecodes and then pass those along to the handler in order to process the next bytecode. The bytecode handlers themselves are written in an architecture agonistic way through assembly code. This means the interpreter can be created once and then rely on the TurboFan compiler to generate machine instructions for every architecture that is supported by the V8 engine. [22]
4.3.5 Spidermonkey Engine
The SpiderMonkey engine uses a different approach for its JITC. The source code is passed
to the parser, which generates an AST. The AST is then used by the interpreter to generate
bytecode which is then passed to the optimizing compiler with profiling data. The differing
thing from the V8 engine is that the SpiderMonkey engine has two optimizing compilers
instead of one. The source code goes through the interpreter first to be optimized into the
Baseline compiler. The Baseline compiler will then produce “somewhat” optimized code and
also analyze the code. This will then be passed to the heavily optimizing compiler called
IonMonkey. Should any of the assumptions made fail then IonMonkey will fall back to the
Baseline code. [18]
17 (45 )
Figure 2 - Generic pipeline of code compilation and execution in the SpiderMonkey engine [4].
4.4 Conclusion
JavaScript is designed to be run directly in the browser and in order to make it as fast as
possible it needs to employ a complex multi-tier architecture in order to parse and optimize
the code being executed. The architecture depends on the given browser which the JavaScript
code is being executed in, such as Chrome which uses the V8 engine for compiling code and
Firefox which uses the SpiderMonkey engine. In general they use both an interpreter and a
compiler in order to get the better of two worlds. The code will execute fast through the
interpreter and then be analyzed and optimized based on profiling data by the compiler before
being translated into the machine language suitable for the underlying machine. These steps
are all performed while the code is being executed, which is why it is called Just-in-time-
compilation. WebAssembly is a binary format which is a compilation target for other
languages. Hence, it is compiled ahead-of-time and undergoes optimizations before being
downloaded in the browser. Due to the binary format being a closer mapping to actual
machine code than regular JavaScript, it is able to make a short jump directly to machine
code without the need to be parsed like a regular JavaScript file, the modules tend to also be
smaller in size which makes it faster to download. WebAssembly is built around underlying
concepts in order to effectively be able to map directly to the low-level machine code.
18 (45 )
WebAssembly mainly uses 4 types of integers, which are of the type float. Hence, the technology is more suitable for pure calculations. However, while the technology is able to provide the low-level format it is then not as optimized for dealing with complex data structures and building out parts of a website. Furthermore, the technology makes use of a stack machine as a way to handle instructions. The stack machine operates on a last-in, first- out basis where it takes out arguments and performs operations before returning the value into the stack. WebAssembly also makes use of linear memory, where it can store values in any byte address. This is also used when communication is needed between JavaScript and WebAssembly, such as passing data structures between the technologies, which in turn can add additional overhead to execution. A summary of the key similarities and differences mentioned above can be found in figure 3.
Figure 3 - Summary of comparison between WebAssembly and JavaScript.
19 (45 )
5 E MPIRICAL EXPERIMENT 5.1 Results
Introduction
The data was compiled and the different geometric means and standard deviations were
calculated for each data set and the results are presented in the form of bar graphs for each
operating system, browser and type of execution conducted. The benchmarks were run 30
times each in both a cold and warm browser environment. The geometric mean was used to
calculate the average runtime for a benchmark and the standard deviation was used to
determine the inconsistency for every test; a high standard deviation would indicate that the
values are spread out over a higher range, in essence indicating that the test is not as
consistent and vice versa.
20 (45 )
WebAssembly and JavaScript in Ubuntu/Firefox (Cold)
The geometric mean which was calculated after compiling the data indicates that the results of the cold run experiment are mixed. The cold run data set shows that for the algorithms involving data structures, such as Array Reverse and Merge sort, WebAssembly did not offer an advantage in performance. However, for the algorithms involving more calculations, WebAssembly managed to outperform and match the speed of JavaScript. In the prime number test, WebAssembly had a better performance by a smaller amount compared to the improvement which was seen in the Fibonacci test, as seen in figure 4. The technology had a slightly worse performance in the Nsieve test.
Figure 4 - Runtime for algorithms in the Firefox browser on Ubuntu (Cold)
21 (45 )
WebAssembly and JavaScript in Ubuntu/Firefox (Warm)
The warm run experiment performed on Firefox yielded similar results as seen in figure 5. In the benchmarks working on arrays, WebAssembly did not offer any speed up over JavaScript. It did however see some marginal improvement in the runtime compared to JavaScript. The Merge sort test implemented in JavaScript was the only one which performed poorer than its initial cold run test. The experiment involving the Fibonacci algorithm showed that WebAssembly managed to gain a smaller advantage over JavaScript as seen in figure 5, while IsPrime was slightly slower. In the prime number tests, JavaScript managed to improve its performance from the initial cold run sequence to marginally better than WebAssembly.
Figure 5 - Time taken to complete given algorithms in the Firefox browser on Ubuntu (Warm)
22 (45 )
WebAssembly vs JavaScript in Ubuntu/Chrome (Cold)
The tests conducted in the Chrome browsers showed that in the benchmarks which worked with arrays, WebAssembly was slower. The JavaScript implementation of the array algorithms proved to have better performance than even their Firefox equivalent.
WebAssembly still showed similar performance in the Merge sort and Array Reverse tests as its Firefox equivalent. WebAssembly also performed better in the Prime number and Fibonacci tests as opposed to the Merge sort, Array Reverse and Nsieve tests where JavaScript was faster, as seen in figure 6.
Figure 6 - Time taken to complete given algorithms in the Chrome browser on Ubuntu (Cold)
23 (45 )
WebAssembly vs JavaScript in Ubuntu/Chrome (Warm)
The warm run implementation of the benchmark proved to provide similar results as its cold run equivalent. The JavaScript Merge sort implementation, which is seen to the far left in figure 7, proved to provide a better performance than WebAssembly in this instance. When it came to the Array Reverse implementation, JavaScript performed significantly better than its WebAssembly version. For the more calculation based algorithms, WebAssembly managed to provide better performance in the Fibonacci and Prime number tests, while for the Nsieve test WebAssembly fell behind. The difference between the cold run and warm run implementation provided a small increase in performance for both technologies across the tests, with an exception being the Array Reverse WebAssembly implementation which proved to decrease performance by a small margin.
.
Figure 7 - Time taken to complete given algorithms in the Chrome browser on Ubuntu (Warm)
24 (45 )
WebAssembly vs JavaScript in Windows/Firefox (Cold)
The results obtained from the tests conducted on the Windows 10 operating system indicate that similarly to the Ubuntu tests, WebAssembly performed poorer than its JavaScript equivalent when it came to the Merge sort and Array Reverse tests. However, when looking at the Nsieve, Fibonacci and Prime number tests the results are a bit more even, as demonstrated in figure 8.
Figure 8 - Time taken to complete given algorithms in the Firefox browser on Windows (Cold)
25 (45 )
WebAssembly vs JavaScript in Windows/Firefox (Warm)
In the cold run aspect of the benchmark, WebAssembly was a little bit slower in the Nsieve tests than JavaScript but in the subsequent test it manages to overcome the small gap and is even proving to be slightly faster while JavaScript regressed in the same test. The differences in the cold and warm runs are miniscule as both technologies improved their performance slightly, with the exception of Array Reverse and IsPrime for WebAssembly. The Merge sort for JavaScript also showed poorer performance as seen in figure 9.
Figure 9 - Time taken to complete given algorithms in the Firefox browser on Windows (Warm)
26 (45 )
WebAssembly vs JavaScript in Windows/Chrome (Cold)
The experiment conducted on the latest Chrome version in Windows 10 showed WebAssembly still falling behind in terms of performance when it came to the array sorting algorithms as seen in figure 10. The JavaScript implementation of the array sorting algorithms proved to have a significantly better performance in the Chrome browser compared to WebAssembly. For the rest of the tests, WebAssembly was slower in the Nsieve test but was faster in the Fibonacci and Prime number test.
Figure 10 - Time taken to complete given algorithms in the Chrome browser on Windows (Cold)
27 (45 )
WebAssembly vs JavaScript in Windows/Chrome (Warm)
For the warm run implementation, WebAssembly once again proved to be slower when it came to sorting arrays as seen in figure 11. Similarly for the Nsieve test, WebAssembly was a bit slower but faster when it came to the Fibonacci and Prime number test. While the JavaScript implementation of the various algorithms showed slight improvement in terms of performance from their cold run, WebAssembly’s performance degraded to a smaller degree across all tests except for the Prime number algorithm .
Figure 11 - Time taken to complete given algorithms in the Chrome browser on Windows (Warm)
28 (45 )
Ubuntu standard dev
Cold-js Cold-Wasm Difference Warm-js Warm-Wasm Difference
Merge sort Firefox 78 21,92 55,86 -33,94 15,62 19,40 -3,78
Merge sort Chrome 83 7,82 27,70 -19,88 8,36 3,35 5,01
Array Reverse Firefox 78 27,53 254,29 -226,76 19,28 34,36 -15,08
Array Reverse Chrome 83 27,60 59,92 -32,33 20,36 38,61 -18,25
Nsieve Firefox 78 28,39 8,23 20,16 15,43 3,81 11,62
Nsieve Chrome 83 10,19 13,52 -3,33 14,92 4,28 10,64
Fib Firefox 78 29,86 13,76 16,10 12,51 2,12 10,39
Fib Chrome 83 28,40 20,57 7,82 6,61 2,41 4,19
IsPrime Firefox 78 5,39 4,22 1,17 2,57 2,19 0,38
IsPrime Chrome 83 5,16 3,45 1,71 3,98 0,46 3,52
Consistency
The standard deviation calculated from the data set obtained through the tests on the Ubuntu operating system while performing a cold run, shows that WebAssembly is more consistent in performance when it comes to the algorithms dealing with calculations, the only exception being the Nsieve test in the Chrome browser. The deviation was higher for the tests that worked with arrays.
The warm run saw similar performance from WebAssembly, where it provided more consistency across the tests with calculations. The technology also improved its performance in the Nsieve test as compared to JavaScript which worsened in performance in comparison to its cold run for the Chrome browser but saw improvement in the Firefox browser.
The tests which were conducted on Windows using a cold run show that WebAssembly had less consistency when it came to the array sorting algorithms, the exception being the Array Reverse test which in the Firefox browser provided a performance closer to the mean. The tests involving calculations proved to provide a better consistency than their JavaScript equivalent with the exception of the Nsieve test where JavaScript was more consistent.
In the warm run implementation, WebAssembly showed better consistency across most of the tests, with the exception being the Array Reverse test conducted in the Chrome browser.
Table 1 – Standard deviation as calculated across the browsers and algorithms for the Ubuntu OS
29 (45 )
Windows standard dev
Cold-js Cold-Wasm Difference Warm-js Warm-Wasm Difference
Merge sort Firefox 78 14,85 15,49 -0,64 30,17 15,81 14,36
Merge sort Chrome 83 27,23 72,48 -45,25 30,46 16,82 13,64
Array Reverse Firefox 78 366,49 63,32 303,17 70,20 19,41 50,79
Array Reverse Chrome 83 35,06 40,05 -4,99 13,40 22,11 -8,71
Nsieve Firefox 78 15,14 48,24 -33,09 12,94 2,76 10,18
Nsieve Chrome 83 13,46 23,77 -10,31 14,14 6,95 7,19
Fib Firefox 78 11,17 4,00 7,17 8,87 1,70 7,18
Fib Chrome 83 44,28 18,64 25,63 25,10 2,11 22,99
IsPrime Firefox 78 4,10 2,78 1,32 11,48 8,93 2,54
IsPrime Chrome 83 4,55 2,26 2,29 8,82 1,34 7,47
Table 2 – Standard deviation as calculated across the browsers and algorithms for the Windows OS