A comparison of Hybrid and Progressive Web Applications for the Android platform

A comparison of Hybrid and

Progressive Web Applications for the Android platform

Faculty of Computing, Blekinge Institute of Technology, 371 79 Karlskrona, Sweden

fulfilment of the requirements for the degree of Bachelor of Science in Software Engineering. The thesis is equivalent to 10 weeks of full time studies.

The authors declare that they are the sole authors of this thesis and that they have not used any sources other than those listed in the bibliography and identified as references. They further declare that they have not submitted this thesis at any other institution to obtain a degree.

Contact Information:

Author(s):

Denis Eleskovic

E-mail: deel18@student.bth.se

University advisor:

Dr. Nauman Ghazi

Department of Software Engineering

Faculty of Computing Internet : www.bth.se

Blekinge Institute of Technology Phone : +46 455 38 50 00

SE–371 79 Karlskrona, Sweden Fax : +46 455 38 50 57

The Hybrid approach of development has for a long time been the dominating way to develop cross-platform applications targeting both the web and mobile. In recent years, a new combination of technology has appeared called Progressive Web Ap- plication (PWA) which aims to combine Native capabilities with best practices of the web to deliver a new Native-like experience to users without the need of Native wrappers. So far PWAs have proven to be the inferior choice when it came to per- formance and platform support. The purpose of this study is to compare the two technologies based on a literature review and evaluate the current performance across three parameters in an experiment - battery consumption, CPU utilization and time to first activity. Two applications were developed using each respective technique, with the Apache Cordova framework being used for the Hybrid approach and the Re- act framework being used to implement PWA features. The results showed that the Hybrid approach is better in the majority of tests, offering more in terms of platform API access and providing better performance while only being slower when it came to time it took to first activity; but something to consider is that the PWA approach was not far behind. The conclusion this study arrived at was that PWAs have devel- oped significantly since previous studies and is almost able to match Hybrid apps in terms of APIs and performance, but that Hybrid apps are still the preferred choice when it comes to performance. Further development and a wider adaptation of the PWA specification could very well change the way developers choose to approach mobile app development in the future as well as a potential for bringing the web closer to the mobile platform.

Keywords: Progressive Web Apps, Hybrid apps, Battery consumption, CPU uti-

lization, Launch time.

Abstract i

0.1 List of Acronyms . . . . 1

1 Introduction 2 1.1 Background . . . . 2

1.2 Purpose . . . . 2

1.3 Scope . . . . 3

2 Questions to be answered 4 3 Method 6 3.1 Literature review . . . . 6

3.1.1 Experiment . . . . 7

3.1.2 Experimental Goal . . . . 7

3.1.3 Experimental Variables . . . . 8

3.1.4 Experimental Hypotheses . . . . 8

3.1.5 Data analysis . . . . 9

3.1.6 Technical Implementation . . . . 9

3.1.7 Test Execution . . . . 12

4 Literature Review 14

ii

4.2 Hybrid apps . . . . 14

4.3 Progressive Web Apps . . . . 16

4.3.1 Features . . . . 17

4.3.2 Criteria . . . . 18

4.4 Related work . . . . 19

4.4.1 Introduction . . . . 19

4.5 Summary . . . . 20

5 Results 23 5.1 Introduction . . . . 23

5.2 Impact on Battery Consumption (RQ2) . . . . 24

5.2.1 Impact on CPU utilization (RQ2.1) . . . . 25

5.3 Impact on Activity Launch time (RQ2.2) . . . . 27

5.3.1 Summary . . . . 29

6 Analysis and Discussion 30 6.1 Technical Differences . . . . 30

6.2 Performance . . . . 32

6.2.1 Battery consumption . . . . 32

6.2.2 CPU utilization . . . . 34

6.2.3 Launch activity . . . . 36

7 Threats to validity 38 7.1 Internal Validity . . . . 38

7.1.1 History . . . . 38

7.1.2 Maturation . . . . 39

7.1.3 Selection . . . . 39

iii

7.2.1 Interaction of setting and treatment . . . . 39

7.2.2 Interaction of history and treatment . . . . 40

7.3 Construct Validity . . . . 40

7.3.1 Inadequate preoperational explication of constructs . . . . 40

7.3.2 Mono-operation bias . . . . 40

7.4 Conclusion Validity . . . . 40

7.4.1 Low statistical power . . . . 41

7.4.2 Violated assumptions of statistical tests . . . . 41

7.4.3 Reliability of measures . . . . 41

8 Conclusions and Future Work 42 8.1 Future work . . . . 44

References 45

iv

0.1 List of Acronyms

PWA Progressive Web Application . . . . i

CPU Central Processing Unit . . . . 2

ADB Android Debug Bridge . . . . 13

IoT Internet of Things . . . . 44

Introduction

1.1 Background

The Hybrid approach of development has long been the go to method for developing applications that combine the best of two worlds, web applications and Native func- tionalities. By drawing from existing web technologies developers are able to merge their web based application with Native functionality while supporting several dif- ferent platforms with one code base. The apps can then be distributed through each respective mobile platform by making use of their app stores. In recent years a new approach has been introduced, namely Progressive Web Apps which offers an alter- native solution for developers. Progressive Web Applications PWA aim to provide Native features such as offline support, background synchronisation and home-screen installation [1]. In essence this technology combines the best practices of web devel- opment offering a native-like experience for the end user [2].

1.2 Purpose

Currently the mobile device market is booming with more than 5 billion people possessing a mobile phone, and 57% of those phones are smartphones. Hence, there exists a need to perform research into technology that affects the experience of so many end users in order to enable further development within the field [3]. This study aims to contribute to this large field and provide further value to the many developers which have specialized in this area to be able to make better decisions when using these technologies. The potential issues which could arise with this work comes down with problems measuring the correct thing, as there are many variables which can come to affect the outcome of the experiment conducted in this study, relating to battery consumption, Central Processing Unit (CPU) usage and time to first activity. Hence a rigorous study will be performed that will collect information which can be leveraged from previous works in order to identify as many variables

2

as possible and account for these for the study to provide reliable results.

1.3 Scope

The paper will not attempt to experiment with all types of applications, instead the study will investigate how a minimal data driven application using the the PWA specification can be compared to an equivalent Hybrid application in terms of battery usage, CPU usage but also the time taken to first activity. A minimal application is chosen as there are currently no open-source alternatives which had both a Hybrid implementation as well as a PWA implementation that could be used for testing.

Furthermore, there was the problem of time, as there is not enough time to develop

a full-scale business application. Thus, this will most likely affect the scalability of

the test results which would mean that the results do not necessarily translate to

real life applications. The study will also not try to identify any specific development

patterns which may affect the battery usage, e.g not recommend a specific design

pattern for working with the technologies. The current functionalities are also based

on the underlying platform used, hence this paper will only look at what is available

on the Android platform when applying PWA and Hybrid technologies.

Questions to be answered

This study aims to investigate the effects of the development method on mobile apps through the development and testing of both a PWA app and a Hybrid app. The focus selected was on battery consumption, CPU utilization and the time to taken to first activity. Since these parameters have been selected and tested extensively in earlier papers, hence making them a natural selection for the experiment conducted in this study. These parameters are further explained under each respective research question. [4, 5, 6]

The following research questions have been adapted for this study:

RQ1. What are the current differences between PWAs and Hybrid apps in terms of development method and Native capabilities?

The aim of this question is to gain an understanding of the differences between the methods of mobile application development in order to be in a better position of evaluating and identifying different variables that could come into play when experimenting with the technologies as well as serving as a backdrop for further analysis.

RQ2. To which extent does a development method affect the battery usage in the context of a PWA and Hybrid app?

The battery is a finite resource which needs to be managed carefully when developing applications targeting the mobile platform.

RQ2.1. To which extent does the development method affect the CPU utilization in the context of PWA and Hybrid apps?

The aim of this question is to investigate if the development method affects the CPU usage as this can indirectly affect the battery consumption of the app.

RQ2.2. To which extent does the development method affect the time to first activity in the context of PWA and Hybrid apps?

4

The time it takes to launch the first activity of the app and begin rendering the first view of the app is a very important factor as indicated in previous studies [7]. The studies showed that users in general do not like to wait and just a small increase in the time it takes to load a website/app could affect the customer retention, resulting in them switching to another website or app [7].

The paper aims to provide answers to these questions in order to provide developers

with a better understanding of the impact of development methods on battery usage,

CPU utilization and app launch time to be able to use that knowledge in the future

when approaching new projects in relation to the mobile platform.

Method

This section will present which methods were used in order to answer the research questions of the study.

3.1 Literature review

In order to be able to compare the technologies underlying both Progressive Web Apps as well as Hybrid apps it is first necessary to understand these technologies at their core. Hence, a qualitative literature review was conducted into what en- compasses the PWA and the Hybrid development methods. In addition, it was also necessary to obtain this knowledge in order to be able to fairly compare the tech- nologies, as well as to be able to answer RQ1 effectively.

Furthermore, this information was then used to better understand the current state of research performed in terms of PWA vs Hybrid as well as battery consumption, CPU utilization and launch time. Consequently, this review formed the foundation for the development of the testing in relation to the applications.

The literary review was conducted by searching various databases on the internet for different papers regarding the technologies. The following databases were used:

• Summon@BTH

• Libris

• ACM

• Digital Library

• Google Scholar

6

The keywords and sentences that were used to find relevant literature were the fol- lowing:

• Progressive Web Application battery consumption

• PWA and Hybrid applications

• Hybrid applications battery consumption

• Hybrid apps

• Cross-development methods

• Mobile cross-development effect on battery usage

Furthermore, a snowballing procedure was used to find relevant literature with the references from papers in the first database used as starting point [8, p.23]. Moreover, the criteria used in order to determine the relevance of the information obtained was to look at if the obtained information came from documentation in relation to the technologies, scientific papers, conference papers or a relevant published book, and had information which was relevant to the technology as well as the age of the information. Once the information was collected, it was compiled and analyzed in order to clarify and narrow down the relevant literature before being presented in the literary review section. [8, p.23]

3.1.1 Experiment

An experiment was carried out in order to be able to answer the RQ2s effectively.

Moreover, the literature review was used as a basis for creating the tests as creating and designing new tests is enormously time consuming and not feasible given the time limit of this thesis. Hence, similar tools and parameters were reused and then tested in two PWA apps as well as a Hybrid app. The parameters were based on pre- vious work conducted in the field relating to PWAs and cross-platform development frameworks. [4, 5, 6]

3.1.2 Experimental Goal

The goal of the experiment was to evaluate the impact development methods have on the energy consumption, the CPU usage and time to first launch of mobile apps.

To be more specific, the paper investigates if a PWA app would draw more energy,

use more CPU than a Hybrid app and if a PWA would be slower when launching its

first activity in an app. Table 3.1 demonstrates a more formal definition of the goal

by applying the Goal-Question-Metric technique. [8, p.85-87]

Analyze cross-platform development approaches

for the purpose

of evaluating its impact on with respect to

energy consumption, CPU usage and time to first launch

from the point

of view of developers In the context of mobile apps

Table 3.1: Goal-Metric-Question technique

Putting it all together, the following goal was created:

Analyze cross-platform development approaches for the purpose of evaluating its impact on the energy efficiency, CPU utilization and time to first launch from the perspective of a developer in the context of mobile apps.

3.1.3 Experimental Variables

The independent variable selected for this experiment is the development method, with two different treatments: the PWA method and the Hybrid method. To enforce these methods, the PWA app is executed while the dependent variables are measured before closing it and executing the Hybrid app. The dependant variables selected are three, one for each defined question. [8, p.92]

Battery consumption: This variable is measured in percentages through Battery Historian. It represents the amount of energy consumed by the two different applica- tions when they are started, performing actions (see section 3.1.8), and subsequently closed.

CPU utilization: Measured as a percentage, representing the percentage of the total CPU used during a given time frame for starting the app, performing actions (see section 3.1.8) and then closing the app.

Activity launch time: Measured in milliseconds, representing the time elapsed between pressing on an app and the app launching its first activity.

3.1.4 Experimental Hypotheses

The following hypotheses have been formulated in order to answer the previously

defined research questions [8, p.91,92].

Given that μP W A is the average energy consumption when executing an app devel- oped using the PWA approach and μHybrid is the average energy consumption when executing an app developed using the Hybrid approach, then the null and alternate hypotheses for RQ2 are as follows:

H

e : μP W A = μHybrid H

e : μP W A = μHybrid

Given that μP W A is the average CPU utilization when executing an app developed using the PWA approach and μHybrid is the average CPU utilization when executing an app developed using the Hybrid approach, then the null and alternate hypotheses for RQ2.1 are as follows:

H

c : μP W A = μHybrid H

c : μP W A = μHybrid

In a similar fashion, given that μP W A is the average time to first activity when executing an app developed using the PWA approach and the μHybrid is the average time to first activity when executing an app developed using the Hybrid approach, then the null and alternate hypotheses for RQ2.2 are as follows:

H

t : μP W A = μHybrid H

t : μP W A = μHybrid

3.1.5 Data analysis

In the first step of the data analysis, a qualitative data analysis was performed on the data obtained from the experiments by calculating the mean, standard deviation and displaying the low and high values for each dependent variable, as seen in the tables presented under section 5.2 [8, p.66-68]. In order to verify if the data is normally distributed, histograms were created for each dependent variable. Subsequently, the t-test (which is a statistical test) was applied to test the statistical hypothesis related to each of the RQ2s [8, p.138,139], with a confidence level of 0.05% being applied.

Moreover, due to the data sets in the experiment being unrelated, an unpaired t-test was used [9].

3.1.6 Technical Implementation

In order to be able to experiment with different approaches, and as there were no

open source solutions found in the wild which employ both approaches, a need was

identified for developing solutions independently for the experiment. To this end, a

simple single-page-application which makes randomized requests to a single API was

developed using both the React library and the Cordova framework; making it the

same application with two different approaches, with the React version seen in figures

3.1 and 3.2 and the Cordova version seen in figures 3.3 and 3.4 [10]. The reason for

using React is that it is currently considered one of the most popular libraries when

developing web solutions [11]. This app was created with the intention of representing

the general PWA, hence the app was developed with the PWA specific criteria in

mind [12]. The app makes use of a service worker for caching content, a manifest for

adding the app to the homescreen but is also served over HTTPS as well as using the

HTTP/2 protocol when served over the network, the app is being hosted through

DigitalOcean.

Figure 3.1: First half of the PWA implementation

Figure 3.2: Second half of the PWA implementation

The second implementation makes use of the Apache Cordova hybrid framework for developing mobile apps [13]. The reason for using the Cordova is due to it being considered a progenitor to the other Hybrid frameworks of today and often serves as a base for other frameworks, e.g Phonegap and Ionic both make use of Cordova in one way or another to provide Native functionality [14]. The existing React app was rewritten to exclude the PWA aspects before being compiled with the Apache Cordova framework, this was done in order to be able to independently measure the two technologies performance. The application can be seen in figures 3.3 and 3.4.

In essence, the PWA is a simple single-page-application while the Hybrid approach

makes use of the same code base but is in this instance wrapped in the Apache

Cordova framework. Furthermore, no optimization techniques have been applied for neither implementation.

Figure 3.3: First half of the Hybrid implementation

Figure 3.4: Second half of the Hybrid implementation

3.1.7 Test Execution

The experiment was conducted on a mobile device, the specifications can be seen in

table 3.2. The PWA is hosted through DigitalOcean, with the content being served

over HTTP/2, and has been added to the home screen through Google Chrome (ver- sion 86.0.4240.198) and Mozilla Firefox (version 83.1.0), as well as the Hybrid app being downloaded onto the device by compiling through Cordova. In order to mea- sure the impact on battery usage and the CPU usage, the tool Android Debug Bridge Android Debug Bridge (ADB) is used to send commands to the mobile device [15].

For the experiment, the command dumpsys and Battery Historian is combined with the ADB tool in order to produce a report which contains data about the different apps running on the device [16, 17]. Hence, before each run the device is connected and the recording start point is reset in order to clean out the statistics being col- lected before beginning a new run of the experiment. Furthermore, the command am_activity_launch_time was used to analyze the time taken for the app to start its first activity [18].

OS Android OS, v8.0 (Oreo)

Chipset HiSilicon Kirin 658

CPU 4x2.1 GHz Cortex-A53 & 4x1.7 GHz Cortex-A53

GPU Mali-T830MP2

Memory 4 GB

Display 1080 X 1920 pixels

Table 3.2: Hardware specifications of the mobile device used for the experiments

Each run of the experiment is manually launched in the mobile device, where the app under test makes 20 API request in 60 seconds before the app is closed. When each run is completed, the device is connected through the ADB and a report is generated which is then opened with Battery Historian showcasing the data collected during the 60 seconds. This is then repeated 30 times for each development method. As there was no suitable method found for measuring time while the app is running and executing API requests, an online stopwatch was used instead. Due to this method being inherently error-prone due to human intervention, the tests are repeated 30 times and the means are presented as results.

The mobile device was prepared by doing the following before performing the testing in order to minimize their impact on the test :

• All background apps were closed

• All notifications disabled

• The battery was fully charged

• The screen light of the mobile phone was set to 25%

Literature Review

4.1 Introduction

In order to better understand the current differences between the development meth- ods, it is important to first understand each methods underlying technology and concepts.

4.2 Hybrid apps

The way to develop apps has changed dramatically over the years. Initially, there were no marketplaces for users to get apps from; however, in March of 2008 the App Store was introduced by Apple, and in October of the same year the Play store was released for Android. These stores separated online web sites and native apps and bound them to their respective platform. As technologies progressed, a new way of development was presented to the world in the form of Apache Cordova in 2009.

Apache Cordova was subsequently sold, re-branded and has become one of the most popular development platforms for Hybrid applications. [19]

Hybrid applications are platforms which allow developers to create one codebase for all the other different platforms, without the need to specialize in any particular platform, e.g Android, iOS or Windows. The underlying principle is that you write code as you would for any regular website, the code is then wrapped in a native shell which allows you to target any platform as you see fit. Hybrid apps grew in popularity in comparison to mobile web apps as they managed to offer an offline experience as well, meaning that the app can function without an internet connec- tion and essentially moving away from limiting an app to the web to instead be an actual app. A hybrid application is run inside the existing webview which is located inside of the native application. The bridge between the native part and webview is what in turns gives the application access to native like features such as sensors and

14

notifications [19]. It does this by providing a native wrapper where the web-based code is contained, as well as a JavaScript API that bridges the service requests from the web-based technology to the platform API. The advantages of using a Hybrid approach include but are not limited to:

• Only one codebase

• Support for offline mode

• Creates a broader user base as the application can be accessed on several plat- forms

• Can be distributed to users through respective app stores

• Apps are able to access certain native capabilities as illustrated in table 4.1 below.

Accelerometer 

Camera 

Capture 

Compass 

Connection 

Device 

Events 

File 

Geolocation  InAppBrowser 

Media 

Notification  Splashscreen 

Status Bar 

Storage 

Vibration 

Table 4.1: Platform APIs available for the Cordova framework. [20]

Nonetheless, Hybrids apps suffer from restricted hardware features as well as a de- crease in performance [19]. In addition, some weaknesses of the Hybrid approach are the following:

• Hybrid performance tends to be inferior in terms of performance due to the execution happening in the browser engine.

• The bridge created between the technologies can create a bottleneck when

accessing platform specific APIs.

• The user experience offered can be problematic across different devices, as some do not make use of the on screen back button.

• The application functionality and performance is also tied to the JavaScript bridge capabilities.

• Needs to be approved before being available in the app store which can limit the app in terms of functionality. [19]

4.3 Progressive Web Apps

A Progressive Web App is a combination of different web technologies that enable existing websites to act and look like a native application. Existing web applications can leverage this new combination of technologies in order to provide new features which would not be possible otherwise, the underlying idea being that PWAs are based on a progressive implementation strategy. This means that developers start with the lowest common denominator for browser functionality. From there, the app is gradually improved in accordance to the different browser functionalities available, so instead of expecting the end user to have the latest browser, the implementation would work for everyone. Certain browsers might contain more complex functions but the essential ones should still work in order to provide a minimal function experience;

an app might start as a simple static HTML-page before further complexity is added, such as animations or asynchronous network access. PWAs are delivered to the end user through the use of a link that can be shared with others [21]. Furthermore, developers can take advantage of this new technology rather easily, by simply creating a manifest JSON file and registering a service worker will make the browsers recognize the application as a Progressive Web App. The definition of what a PWA is can be summarized to three characteristics:

• Fast - the first time usage of a PWA should offer an immediate, responsive and enjoyable experience.

• Installable - PWAs should be able to be installed just like any other native application, and it should be available directly on the home screen or on a users desktop.

• Engaging - PWAs should follow existing guidelines when it comes to offering the best UX and UI, and this should work irrespective of network conditions or the underlying hardware. This in turn may offer a limiting experience in some scenarios but the app should never have just an empty screen which tells the user to go online or update. [21]

At present, PWAs have following access in terms of hardware as illustrated in table

4.2.

Accelerometer 

Camera 

Capture 

Compass 

Connection 

Device X

Events 

File 

Geolocation  InAppBrowser -

Media 

Notification  Splashscreen 

Status Bar 

Storage 

Vibration 

Table 4.2: Platform APIs available for PWAs [22]

4.3.1 Features

The PWA specification is something which has been promoted by Google, in doing so they have specified features that PWAs are distinctly able to offer over other solutions:

• Offline support - The app is able to work without an internet connection, the usage of the app needs to be minimally interrupted by the user not having an internet connection.

• Push notifications - The app has the capability of making use of notifications.

• Home screen - The app is installable at will by the end user.

• Background sync - The app has the ability to synchronize in the background.

• Storage - The app is able to estimate the storage that it uses and also know how much storage is left.

• Web share - The app has the ability to make use of the native sharing widget.

• Cross-platform capabilities - The app is able to work across the major browsers.

• Page identifiers - The app has a unique address which makes it linkable to other pages on the internet.

• Payments - The app is capable of making use of the web payment API for when

users want to make payments. [19]

4.3.2 Criteria

Furthermore, for any application to be classified as a PWA, it first needs to meet the following criteria:

• HTTPS - The app needs to be served over a secure connection due to security reasons. The app can in itself be rather simple, it can be just a website that runs in a browser tab. [23]

• Web app manifest - The app needs to contain a web app manifest which lists the metadata. This is a JSON file which describes the PWA and should among other things contain the following: the name of the PWA, if it opens in land- scape or portrait mode, icons, what access the app needs to the device, etcetera.

The manifest is necessary in order to trigger the prompt for end-users to install the PWA for later usage. [23]

• Service Workers - The app needs to register at least one service worker in order to be able to work with requests as well as cache content. A service worker works in the background when the user uses the PWA. The developer is able to define what content gets saved on the device and what parts of the app to reload when the user has a working internet connection again, as well as which notifications to push and when to update the app. [13]

• Application shell - PWAs make use of a fast loading application shell that is the first piece of the interface to render. The idea is to save the shell around the content between reloads while the dynamic content is presented inside it.

Hence, this part of the interface is made up of static resources; because it loads quicker than the rest of the content in the app and is saved across reloads, it manages to provide an immediate user experience. [19]

Moreover, some of the current strengths of PWAs are the following:

• The app is usable without an internet connection.

• PWAs require HTTPS to function, which in turn make them secure.

• Is accessible through a link, no need to go through an app store.

• It is easy to implement in existing websites.

• Saves on development as the app is made available to many different platforms due to being served through the browsers. [19]

Correspondingly, some of the weaknesses are as follows:

• PWAs do not have full access to features available in mobile devices.

• PWas are not available in app stores.

• PWAs does not offer as good performance as Hybrid applications.

• PWAs are not fully supported in all browsers nor by all platforms. [19]

4.4 Related work

4.4.1 Introduction

This section presents and summarizes previous research papers which have been published in similar areas to this thesis and are thus deemed relevant.

Comparison between Progressive Web App and Regular Web App

This study built two test apps and compared them based on performance through lighthouse. It concluded that PWAs were not fully support yet in all browsers at that time. The study further showed that the HTTPS requirement for PWAs actually slowed it down on the first visit, however on a repeat visit it outperformed the Regular Web App. The author recommends using the newer HTTP/2 protocol on the servers as the performance otherwise can decrease. [2]

Comparing Native and Hybrid Applications with focus on Features The study focused on comparing Hybrid functionality with Native and concluded that Native apps are better suited for more hardware intensive apps while Hybrid is in general a good choice for content centric apps. [24]

An empirial analysis of energy consumption of cross-platform frameworks for mobile development.

The paper measured the effect of different cross-platform frameworks on battery usage. The battery usage was measured through the use of a power monitor. The authors concluded that the usage of cross-platform techniques always implied an increase in energy consumption. [5]

Evaluating the impact of caching on the energy consumption and perfor- mance of PWA

This study focused on testing the effect of caching on energy consumption and per-

formance in real world PWAs which were developed by third-party developers. The

apps were tested with an unpopulated cache and with a populated cache in order to

determine their effects. The conclusion that the authors arrived at was that cache

does not have a significant impact on battery usage and can also help improve PWA

load times. [12]

Comparing Progressive Web Applications with Native Android Applica- tions

The study performed an experiment by comparing PWAs with Native apps and found that Native is still faster in terms of performance involving access to hardware. The ImageCaptor API was used and showed that PWA was slower when utilizing this API but faster when using the Geolocation API. [25]

Comparison of mobile application development technologies

The paper compared different approaches to mobile development and summarised the key differences between them. At the end, the different technologies were weighed against each other and the author recommended different approaches for different types of applications. [26]

Progressive Web Apps: The possible web-native unifier for mobile devel- opment

The authors developed two cross-platform applications and one PWA for performance comparisons. The paper concluded that at the time, PWAs were a relatively new technology which has the potential for unifying the fragmented web but due to it not having the same level of access to APIs, the technology is adapted on a case-by-case basis. [1]

Dawning of PWA: Edging out the pitfalls of traditional mobile develop- ment

The authors summarized the key differences between the Native/Hybrid/PWA ap- proaches of development and argued for how PWA had brought forward a new dawn for mobile development. [19]

Progressive web application assessment using AHP

The study makes use of the Analytical Hierarchy Process in order to determine which type of development method is best suited for a given application based on three different constraints. The constraints were based around application size, multi- platform support and offline mode. For these constraints, the paper evaluated the PWA approach as being on top. However, it still presented some shortcomings of the PWA technology, such as needing further development in terms of browser support as well as a need for storage similar in context to the App stores of regular mobile apps. [23]

4.5 Summary

The research conducted regarding the two development methods have agreed on

that the PWA method is a promising new combination of technologies which has

the potential to disrupt the way we go about developing mobile apps but that needs further development and wider adaptation by browsers. At the time when the re- search was conducted the PWA specification still had gaps in terms of API access, API performance as well as not being widely supported in browsers which made it a less favorable method to consider when developing applications. Another thing to consider is that cross-platform development methods would also always increase the battery usage in applications. Furthermore, the authors agree that the best method is best adapted on a case-by-case scenario, such as Native being favored for apps reliant on performance while Hybrid was recommended for content focused apps.

The Hybrid approach combines web technologies with native in order to provide a native like experience. The technologies manages this by creating a bridge between the technologies so that a developer may make use of native APIs but in doing so it also creates a performance overhead as the developer will be working through several abstraction layers. The Hybrid approach also offers offline mode in a restricted manner as it still requires an internet connection. Furthermore, Hybrid technologies make use of webview which is an embedded browser.

Progressive Web Apps are a combination of different best practices for the web that make use of APIs for features which used to be limited to only native applications.

In order for an app to be classified as a PWA it needs to meet certain criteria; the app must be served over HTTPS, it must have a Web app manifest, it must make use of service workers and make use of an app shell which can be considered as best practice. The combination of technologies is based on underlying concepts, such as being fast, being installable and being engaging. At present PWA are almost able to provide the same features as the Hybrid approach, with only a handful of features being inaccessible, but as the technologies move forward these things will most likely be implemented as well. Furthermore, PWAs are now able offer offline experiences as well as they are able to cache content.

The PWA approach is rapidly catching up to the Hybrid style of development with

smaller differences in terms of hardware and platform specific access while differ-

entiating in terms of underlying technology as PWAs make use of best practices of

the web and the functionality of the browser while the Hybrid approach leverages

the benefit of web technologies but instead wraps it in a Native wrapper for Native

capabilities. Furthermore, a summary of what is currently considered the strength

and weaknesses of the two technologies can be seen in table 4.3.

Strengths & Weaknesses of Hybrid and PWA Development

method

Strength Weakness

Hybrid + Unified code base - Worse performance

+ Offline mode - Platform API bottleneck

+ Multi-platform access - Problematic UX across devices

+ App stores - App store approval

+ Extensive native capabilities - Performance/functionality tied to the JavaScript bridge

PWA + Offline Mode - Weak feature access

+ Requires HTTPS - Unavailable in App stores + Independent of App store

+ Easy implementation - Worse performance + Reduces development time/-

cost

- Not fully supported by all browsers/platforms

Table 4.3: Summary of strengths and weaknesses of Hybrid/PWA

Results

5.1 Introduction

This section presents the results of the empirical experiment conducted for each respective research question.

23

5.2 Impact on Battery Consumption (RQ2)

Figure 5.1: Graph showing the means for the battery consumption experiment

The different apps tested are listed in the figures as PWA tested in the Firefox

browser (PWA-F), PWA tested in the Chrome browser (PWA-C) and lastly the Hy-

brid application is listed as simply Hybrid. The descriptive statistics of the collected

battery usage across the different treatments can be seen in figure 5.1. The arith-

metic mean which was calculated after compiling the data obtained indicate that

the trend is in favour of the Hybrid approach, as the Hybrid approach uses the least

amount of battery as illustrated in figure 5.1. Furthermore, the lower standard de-

viation of 0.017 shows that the collected measurements are concentrated around the

mean, which has a value of 0.109%. The Chrome version of the application is still

rather close to the same level of battery usage as the Hybrid approach coming in at

0.119%. The standard deviation calculated at 0.206 further indicate that the values

are closer to the mean. The Firefox version came in last, falling a bit behind the

others with a mean value of 0.206% and a standard deviation of 0.0206 which is a

bit more scattered.

Battery Statistics

Type Highest Lowest Median Standard

deviation PWA-

Firefox

0.28 0.17 0.205 0.206

PWA- Chrome

0.17 0.09 0.12 0.021

Hybrid 0.14 0.08 0.11 0.017

Table 5.1: Statistics for the battery consumption experiments

5.2.1 Impact on CPU utilization (RQ2.1)

Figure 5.2: Graph showing mean CPU percentage used in the CPU utilization exper- iments

The test involving the CPU utilization was calculated over a period of 60 seconds and

repeated 30 times. The mean indicates that the Hybrid approach uses the least CPU

resources over 60 seconds as seen in figure 5.2, with a result of 0.056%. The Chrome

version of the PWA showed some good performance as well, with it getting close to

the same level of CPU usage as the Hybrid approach with a mean of 0.07%. The

Firefox version of the app fell behind by a larger margin to both previous versions with a result of 0.11%. Looking to the table 5.2 containing further statistics related to the data set show that the standard deviation is low for all three approaches, implying that the data is closer to the mean.

CPU utilization

Type Highest Lowest Median Standard

deviation PWA-

Firefox

0.133 0.100 0.112 0.008

PWA- Chrome

0.083 0.060 0.068 0.006

Hybrid 0.073 0.048 0.056 0.006

Table 5.2: Statistics for the CPU utilization experiments

5.3 Impact on Activity Launch time (RQ2.2)

Figure 5.3: Graph showing means for the activity launch time experiment

In the last test involving the launch activity time, the Hybrid version with a mean

of 919.06ms of the app fell behind the PWA versions as seen in figure 5.3. With

the Chrome version being the fastest with 171ms and the Firefox version being a bit

slower with 631.76ms. The descriptive statistics shown in table 5.3 further show that

the standard deviation is low for the Firefox version but a bit higher for the other

samples, indicating that the data is closer to the mean for the Firefox version while

being a bit more scattered for the rest which is probably related to how the different

technologies go about initiating their processes behind the scene.

Launch Statistics

Type Highest Lowest Median Standard

deviation PWA-

Firefox

743 585 624.5 0.021

PWA- Chrome

198 149 169 11.69

Hybrid 970 886 914.5 21.55

Table 5.3: Statistics for the activity launch time experiments

5.3.1 Summary

Overall, the trend is that the Hybrid approach of development was better in the

majority of tests, by managing to use less battery and providing less CPU utilization

than its equivalent PWA implementation. It did however fall behind when it came

to the time it took to launch the first activity of the application. Furthermore, the

PWA Chrome version of the app was not far behind the Hybrid approach, with

it even being superior when it came to time to launch activity test. The Firefox

implementation came in last in every test except the one involving launch time to

first activity, where it was still bested by the Chrome implementation. The differences

between the browsers indicate that they are optimized for different things. Seeing

as the PWA work of development is a method which first saw adaptation at Google,

with the other browser vendors following suit over time, it is reasonable to assume

that the PWA method would work better with the Chrome browser.

Analysis and Discussion

This section presents the analysis performed on the results gathered from the exper- iment as well as a discussion in regards to the results.

6.1 Technical Differences

RQ1. What are the current differences between PWAs and Hybrid apps in terms of development method and Native capabilities?

PWAs and Hybrid apps differ in how they go about utilizing the resources available for them on the mobile platform. With the PWA approach making use of a combina- tion of different web technologies in order to offer the best experience across several platforms and the Hybrid approach being the progenitor of such an approach. PWAs have taken it a step further however, with focusing on adding new features which were not available previously, such as service workers and being directly link-able instead of having to go through any app store where a developer is more restricted in what they can use in their apps. This in turn can have great effect on how we go about using apps in the future, enabling users to skip the process of going through the app store and instead be able to access apps through a shareable link instead.

While it opens up the market for other applications which might not have been ac- cepted before it also links the mobile platform closer to the web platform by enabling the use of links as a form of acquiring apps.

The PWA approach is further built on top of good practices for the web, which helps it in leveraging existing techniques for good user experience but also adding on top of this to provide better Native experience as well. This would then be the advantage to consider when using PWAs as they are meant to take the user experience one step further than Hybrid, as the Hybrid approach has struggled with providing the same user experience as Native apps. Furthermore, comparing what is currently available for the two different approaches in terms of platform APIs it is clear that there is not a major difference. Meaning that the old argument that the Hybrid approach is

30

able to target more platform APIs might not be true in the future as clearly the gap

is closing.

6.2 Performance

6.2.1 Battery consumption

RQ2. To which extent does a development method affect the battery usage in the context of a PWA and Hybrid app?

The following section presents the different samples which were used when experi- menting with battery consumption.

Table 6.1 shows the different samples from the battery consumption experiment compared to each other. The two different variations of the PWA are compared against their Hybrid equivalent, as well as presenting the p-value of a T-test and if these results were statistically significant.

Figures 6.1, 6.2, 6.3 showcase the frequency of the different results obtained in the form of a histogram, this is done in order to create a better overview of how varied each result set was for the battery consumption experiment. The p-value calculated being 0.0124 and 0.0471 for each respective experiment indicate that we are able to reject the null hypothesis. Hence the alternative hypothesis that states that an app which is created following the PWA development method will consume a different amount of energy than an app developed using the Hybrid method on our mobile device can be accepted.

Sample 1 Sample 2 P-Value Statistically significant PWA-

Firefox

Hybrid 0.0124 Yes

PWA- Chrome

Hybrid 0.0471 Yes

Table 6.1: P-value of a T-test with the different samples of the battery consumption experiment

The data which was extracted and compiled from the battery consumption experi-

ment indicate that at the moment the Hybrid approach is the one which consumes the

least energy, with the PWA-Chrome version of the app managing to almost reach

the same level. The PWA-Firefox version of the app was the one using the most

energy. Hence, the data shows that if we chose to follow the PWA approach of devel-

oping a cross-platform app it is necessary to account for differences in the underlying

browsers used by the end users. However, given that technology is constantly evolv-

ing, one could argue that given enough time that the PWA development method will

mature and be better optimized by the browser vendors. Furthermore, due to the

Chrome implementation being so low in terms of battery consumption that it almost

matches the Cordova implementation, they have managed to in essence show that

Figure 6.1: Histogram of battery consumption for the PWA-Firefox version

Figure 6.2: Histogram of battery consumption for the PWA-Chrome version

Figure 6.3: Histogram of battery consumption for the Hybrid version

PWAs are able to compete with Hybrid apps. This is to a certain degree surprising,

as one would assume that due to the need of a PWA to rely on its underlying browser

that it would actually use a lot more energy than a Hybrid app which is Native-like

and has a more direct mapping to the underlying platform through the use of its

Native wrapper.

6.2.2 CPU utilization

RQ2.1 To which extent does the development method affect the CPU utilization in the context of PWA and Hybrid apps?

Table 6.2 showcases the different samples from the CPU utilization experiment when compared to each other. Similarly, the two variations of the PWA are compared against the Hybrid implementation as well as showing the p-value of a T-test which was performed. As the p-values calculated were both below 0.001, which indicate that the tests were statistically significant, we are able to reject the null hypothesis. Thus, the claim that an app using the PWA approach will utilize a different percentage of CPU compared to an app using the Hybrid approach can be accepted.

Sample 1 Sample 2 P-Value Statistically significant PWA-

Firefox

Hybrid <0.001 Yes PWA-

Chrome

Hybrid <0.001 Yes

Table 6.2: P-value of a T-test with the different samples of the CPU utilization experiment

The data extracted and compiled from the CPU utilization experiment show that the Hybrid approach is using the least CPU during 60 seconds, with the PWA Chrome approach coming in quite close to the same level. The Firefox version performed the worst in this test. This being similar to the battery usage experiment. The data indicates that if we want our app to use the least amount of CPU over a given time, then it would be best to use the Hybrid approach. It is however important to note that the PWA approach is not far behind, and this is while requiring the underlying browser. Hence, the development method adopted in terms of CPU utilization does not necessarily create a large divide between two apps in terms of CPU performance.

However, it should be noted as well that the type of application being tested is not

a CPU bound app, hence the results might not necessarily carry over to other apps

which stress the CPU heavily. The figures 6.4, 6.5 and 6.6 further illustrate the

spread of the values obtained from the experiment.

Figure 6.4: Histogram of CPU utilization for the PWA-Firefox version

Figure 6.5: Histogram of CPU utilization for the PWA-Chrome version

Figure 6.6: Histogram of CPU utilization for the Hybrid version

6.2.3 Launch activity

RQ2.2 To which extent does the development method affect the time to first activity in the context of PWA and Hybrid apps?

Table 6.3 presents the different samples from the launch activity experiment when compared to each other. The table compared the two PWA implementations of the app against their Hybrid equivalent. Furthermore, figures 6.7, 6.8, 6.9 showcase the different samples in the form of histograms for a better view of the distribution. The p-value calculated from the T-test performed, with both the result being <0.001, indicate that the experiments are statistically significant. Which enables us to reject the null hypothesis and accept the alternative hypothesis which says that an app using the PWA approach will not take the same time to start an app activity as another app using the Hybrid approach of development.

The presented results for the experiments are a bit limited as they have only been tested on one device, the Huawei P10 Lite, running Android 8.0. Hence, the dif- ferences observed could be significantly different if tested on older devices and less powerful devices or even more powerful devices. Furthermore, differences in tech- nical frameworks that are being used in each implementation is also expected to make a difference and further research into the differences in these across common parameters would also be beneficial.

Sample 1 Sample 2 P-Value Statistically significant PWA-

Firefox

Hybrid <0.001 Yes PWA-

Chrome

Hybrid <0.001 Yes

Table 6.3: P-value of a T-test with the different samples of the activity launch time experiments

The data compiled from the experiment conducted on launch to first activity show

that in this test the Hybrid approach turned out to be slower than its equivalent

PWA application. Where the PWA-Chrome implementation was the fastest, Firefox

version came in second and the Hybrid version was the slowest. Hence, based on the

data gathered we can deduce that in order to provide the best user experience, in

the context of the experiment variable used, would be to use the PWA approach of

developing mobile apps.

Figure 6.7: Histogram of activity launch time for the PWA-Firefox version

Figure 6.8: Histogram of activity launch time for the PWA-Chrome version

Figure 6.9: Histogram of activity launch time for the Hybrid version

Threats to validity

This section covers the evaluation of the soundness of the experiment performed, four different types of threats to validity have been analyzed.

7.1 Internal Validity

Internal validity threats are related to the experiment design and its subsequent execution [8, p.106,110].

7.1.1 History

The threat of history is related to the fact that time passes between different treat- ments being applied at different times thus there is a risk that the time will affect the results as the circumstances will not be the same. When designing the experiment, two options were considered for executing and collecting the results. The first would be to execute and collect the results in an incremental manner, which would result in the tests being performed over a series of days and in different environments. The other option was to perform the tests and collect all the necessary data in one sit- ting. The second option was chosen in order to ensure the same environment and conditions for the two subjects being tested, thus mitigating this threat. Trial tests were performed before the start of recording of each test in order to ensure that the testing environment was stable and that there were no unknown variables disturbing the tests during their execution [8, p.106,110].

38

7.1.2 Maturation

The maturation threat has to do with the subjects under test reacting differently as time passes. In order to mitigate this threat, the 20 API requests were performed over the time of 60 seconds. Furthermore, the data which was collected by Battery historian was reset between each individual run of the tests in order to ensure that the correct data was being collected. The tests were repeated 30 times for each development method, with all the 30 runs being completed for one development method before moving to the other [8, p.106,110].

7.1.3 Selection

The threat of selection has to do with subjects selected for an experiment, and their effect on the final results. The internet was first searched for possible candidates that could be used in the experiment, the intent was to use real-life applications in order to provide more validity to the experiment. Due to being unable to find any open-source application which had both a Hybrid implementation and a PWA implementation, the decision was then made to create two minimal applications for testing purposes instead. Selection may thus play a role in this experiment as it might not translate over to the real world in scale. In order to mitigate this threat, the apps developed for the experiment were created as close to a regular data-driven application as possible given the time available [8, p.106,110].

7.2 External validity

External threats to validity are those which limit the ability to generalize the results from the experiment [8, p.110,112].

7.2.1 Interaction of setting and treatment

This threat is related to using an unrelated environment and settings for the exper-

iment. In order to reduce this threat, a modern mobile device was used which was

connected through WiFi which is the usual setting in which users make use of mo-

bile apps. The latest Mozilla Firefox and Google Chrome browsers were used in the

testing in order to ensure the tests were as relevant to toady’s technology as possible

[8, p.110,112].

7.2.2 Interaction of history and treatment

This threat is related to any events happening on the specific day which might influ- ence the test results. In order to mitigate this threat the experiment was conducted on a weekday during regular working hours, this was done in order to emulate a more realistic testing environment as regular users are rarely the only users of a WiFi network [8, p.110,112].

7.3 Construct Validity

Construct validity threats are related to the relationship between the theory, obser- vations made and if the observations can be generalized [8, p.110,112].

7.3.1 Inadequate preoperational explication of constructs

The constructs were defined ahead of time, thus mitigating this threat. The GQM approach was used in order to define the goal, which then helped develop the research questions for this paper. Furthermore, the hypothesis, dependant, independent vari- ables as well as treatments were all selected during the initial planning phase of the experiment [8, p.110,112].

7.3.2 Mono-operation bias

This threat is related to the independent variable and its treatments, the underlying threat being that if you use one version of a program at a single point in time, you may not be capturing the full range of the concept. In order to mitigate this threat the independent variable was given two treatments, the PWA and the Hybrid treatment [8, p.110,112].

7.4 Conclusion Validity

Conclusion threats to validity are focused on issues related to the ability to establish

the correct conclusion based on the relationship between the treatments used in an

experiment and the subsequent results [8, p.104,106].

7.4.1 Low statistical power

In order to mitigate the threat of low statistical power, the experiment was repeated 30 times for each dependent variable in order to ensure that there was enough data collected. Thus, two applications were used with each of its dependent variable tests being repeated 30 times which resulted in a total of 180 runs. This number can be increased in the future in order to provide even more accurate results. Furthermore, more applications can be developed while reusing the same measurement base [8, p.104,106].

7.4.2 Violated assumptions of statistical tests

To mitigate this threat, a qualitative data analysis was performed in order to identify if the data sets were normally distributed. Then if the data sets turned out to be normally distributed, an unpaired T-test would be used [8, p.104,106].

7.4.3 Reliability of measures

Mitigating the effect of factors such as background processes, screen brightness and network connectivity can be difficult. Therefore, the screen brightness was set to 25%

and the battery was fully charged when performing the experiments. Furthermore,

all the tests were performed in one run, ensuring that the phone was close to the

router to minimize the effect of the network and ensuring there were no apps running

in the background [8, p.106,110].

Conclusions and Future Work

PWA apps are able to offer good user experience due to its underlying UX and UI design principles as well as the concepts it is based upon, as the technology has advanced so too have more platform APIs become available for usage through greater browser support. This has come to a point where a PWA is almost able to access the same features as a Hybrid app would. This in turn brings it a step closer to Hybrid apps in terms of what it is capable of doing and provides the possibility of even outdoing them all together in the future. The current difference in performance between the two major browser vendors also further suggests, that while the PWA approach is a technology that has the potential to unify the way of development for the web it also indirectly showcases the current fragmentation apparent on the web.

The experiments conducted show that the Hybrid approach of mobile app develop- ment is to be favored in the majority of cases. Something to keep in mind however, is that the Google Chrome version of the app did not fall that much behind in terms of battery consumption and CPU utilization while even outdoing the Hybrid based app in terms of time to launch activity. PWA has come a long way in terms of performance matching the Hybrid approach, and is even able to almost match the Hybrid approach in parameters which are important for the mobile platform. How- ever, something to keep in mind that these results do not necessarily carry over to all the different frameworks which can be employed when developing a PWA or a Hybrid application. Due to the application that was developed for this experiment being minimal it is also important to consider that these results to not necessarily carry over the bigger applications.

Another aspect to consider is the business effect these development methods might have, for a long time the consensus has been to make use of the Hybrid approach in order to cut down on development time in order to be able to launch a product as soon as possible. It is however interesting to entertain the idea that the development time could potentially be reduced even further through the use of PWA principles, which would also reduce the costs of development. However, this is something which should be adapted on a case-by-case basis as the PWA approach is not a technique which is able to provide advantages to all types of applications. To conclude, PWAs

42

have their own specific usages and they are catching up to Hybrid frameworks quite

quickly but the preference of which to use are ultimately tied to the developers which

will need to weigh the pros and cons of the technologies.

8.1 Future work

As this work only performs a set of minimal tests for the technologies, it would be beneficial to either reuse some older tests that were made when the PWA technology was in its infancy or to identify additional current aspects that are important and perform further testing of the technology as it progresses as well as doing it across multiple devices. Furthermore, it would also be interesting to measure performance of mobile app develop methods in Internet of Things Internet of Things (IoT) devices.

Further experiments could also be conducted to test the performance of PWA desktop

apps as well as trying out the different frameworks which fall under the more broader

definition of development methods.

[1] Biorn-Hansen, Andreas, Tim A. Majchrzak, and Tor-Morten Grønli. "Progressive webapps: The possible web-native unifier for mobile development." In International Conferenceon Web Information Systems and Technologies, vol. 2, pp. 344-351.

SCITEPRESS, 2017.[Online]. Available: https://www.scitepress.org/Papers/2017 /63537/63537.pdf. [Accessed:Sept. 27, 2020][2]

[2] F. Said Tahirshah, ‘Comparison between Progressive Web App and Regular Web App’, 2019, [Online]. Available: http://www.diva-portal.org/smash/get/diva2:

1334458/FULLTEXT02.pdf. [Accessed: Dec. 11, 2020]

[3] I. Malavolta, G. Procaccianti, P. Noorland, and P. Vukmirovic, ‘Assessing the Impact of Service Workers on the Energy Efficiency of Progressive Web Apps’, May 2017, In IEEE/ACM 4th International Conference on Mobile Software En- gineering and Systems (MOBILESoft), Buenos Aires, 2017, pp. 35-45, [Online].

Available: http://www.ivanomalavolta.com/files/papers/Mobilesoft_2017.pdf. [Ac- cessed: Dec. 11, 2020]

[4] A. Biørn-Hansen, T.A. Majchrzak, and T. Grønli, ‘Progressive Web Apps for the Unified Development of Mobile Applications’, 19 June, 2018, [Online]. Available:

https://www.researchgate.net/publication/325823248_Progressive_Web _Apps _for _the _Unified _Development _of _Mobile _Applications. [Accessed: Dec. 10, 2020]

[5] M. Ciman, and O. Gaggi, ‘An empirical analysis of energy consumption of cross- platform frameworks for mobile development’, 2017, In Pervasive and Mobile Com- puting, 39, 214-230, [Online]. Available: http://localwww.math.unipd.it/ gaggi/- doc/pmc2016.pdf. [Accessed: Dec. 11, 2020]

[6] D. Li, S. Hao, J. Gui, and W. G. J. Halfond, ‘An Empirical Study of the En- ergy Consumption of Android Applications’, 2014, In IEEE International Conference on Software Maintenance and Evolution, Victoria, BC, 2014, pp. 121-130, [Online].

Available: https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.711.8458&rep

=rep1&type=pdf. [Accessed: Dec. 11, 2020]

[7] L. Adams, E. Burkholder, and K. Hamilton, Micro-Moments:Your Guide to Win-

45