Evaluating alternatives to native development for handheld computers

(1)

EVALUATING ALTERNATIVES TO NATIVE DEVELOPMENT FOR

HANDHELD COMPUTERS

Bachelor Degree Project in Information Technology Basic level 30 ECTS

Spring term 2020 Alfred Claesson

Supervisor: András Marki Examiner: Joe Steinhauer

(2)

Abstract

Selecting a suitable mobile application development framework is challenging.

Multiple frameworks exist with respective strengths and weaknesses, and evaluations comparing the different frameworks are lacking. Each company and developer has its own demands and preferences, thereof, no single development framework poses as a suitable solution for every situation.

This study is aimed at contributing to the current foundation of research by examining Android application development for handheld computers. By selecting promising cross-platform frameworks and comparing these with the native development framework for Android this study shows that cross-platform development frameworks can provide the tools required to build sufficient applications for handheld computers.

This is particularly relevant for the developers affected by the termination of support for Windows Mobile. The findings of this study show that the frameworks Flutter and Xamarin both pose as promising alternatives to native development.

Keywords: Android, Benchmarking, Kotlin, Flutter, Xamarin, Handheld Computers

(3)

Acknowledgements

I would like to thank Netred AB for providing me with the opportunity to carry out this study. By providing the necessary means and by supporting me throughout the study, they played a central role in actualizing it. Due to their support, this study was made possible.

I would also like to express my gratefulness towards the people involved in this study at the University of Skövde. I would like to thank my supervisor in particular, for all the time that he has put in to help me with the study.

(4)

1. Introduction

The increase in smartphone sales creates the need for efficient methods of developing mobile applications. Various approaches have been presented for developing applications for multiple operating systems simultaneously, also known as cross-platform development.

(Biørn-Hansen, Grønli, & Ghinea, 2018). This study evaluates and compares two cross- platform development frameworks, Flutter and Xamarin, with native development, which refers to development for one specific platform, in the context of handheld computers. The majority of smartphones and a large number of handheld computers use Android as an operating system (Gartner, 2018). Android is therefore the platform targeted in this study.

This study is carried out in collaboration with Netred AB, an IT consulting firm developing applications for handheld computers among other things. They require further investigations in application development for handheld computers due to their clients replacing their Windows Phone devices with Android devices.

The cross-platform frameworks available all have their respective strengths and weaknesses (Tunali & Zafer, 2015). As a result, no single framework can be ruled to be the best option for each situation. Inadequate performance is often listed as one of the major drawbacks of cross-platform frameworks (Willocx, Vossaert, & Naessens, 2015). The various drawbacks of cross-platform development proposes the need for further research in whether or not a company or a developer with a certain set of requirements and aims should choose to develop applications natively or using cross-platform frameworks. As with cross-platform development, developing applications natively imposes certain drawbacks. In addition to the fact that one application has to be built multiple times in order to be available on multiple platforms, native development also requires developers to have a high degree of experience and detailed knowledge (Heitkötter, Hanschke, & Majchrak, 2013).

This study examines promising options to develop Android applications for handheld devices. To enable the examination of promising alternatives to native development, criteria are established both for selecting and evaluating cross-platform frameworks. The range of available frameworks complicates this process, as there are several aspects to consider. For each of the various approaches of cross-platform development, several cross-platform frameworks exist (Biørn-Hansen, Grønli, & Ghinea, 2018).

When the selection process is finished, the selected frameworks are evaluated according to the evaluation criteria, and are then compared with each other as well as with the native development framework for Android. The study aims to display and evaluate the various advantages and disadvantages of each development approach in order to gain better understanding of the current state of the cross-platform landscape and its application areas.

It also aims to show if cross-platform development can be used instead of native development without significant negative impacts on the developed applications. The context in which the frameworks are evaluated is on a Honeywell Dolphin CT60, which is a handheld computer, and also to an extent on a Samsung Galaxy S10, to extend the generalizability of the results.

(6)

2

2. Background

Today there are several strategies available for developing Android applications. These strategies can be divided into two broad categories: native development and cross-platform development. The former refers to development where only a single platform is targeted, such as iOS or Android. Tools and frameworks provided by the targeted platform are used during development with these strategies, and the application is written in natively supported languages. The latter of the two categories, cross-platform development, consists of various approaches for creating applications for multiple platforms using a common code base (Heitkötter, Hanschke, & Majchrzak, 2013). El-Kassas, Abdullah, Yousef, & Wahba (2017) present a compilation of proposals for classifying the sub-categories of cross-platform development. The literature of the research area is inconsistent in its taxonomy, but it is most common, as Biørn-Hansen, Grønli, & Ghinea (2018) state, to categorize cross-platform frameworks into the following categories:

 The hybrid approach:

This approach leverages web development tools to create the application. It embeds the functionality in a native application and provides tools for communication between the two (Biørn-Hansen, Grønli, & Ghinea, 2018).

 The interpreted approach:

This approach provides tools for bridging and interpreting code in order to make use of native user interface components and native functionalities (Biørn-Hansen, Grønli, & Ghinea, 2018).

 The cross-compiled approach:

This approach refers to frameworks which compile a common language into executable native bytecode for each platform (Biørn-Hansen, Grønli, & Ghinea, 2018).

 The model-driven approach:

This approach allows developers to provide abstract models of their desired functionality and user interface. Model-driven framework translate models into native code for each platform (El-Kassas et al., 2017).

 The progressive web apps approach:

This approach provides the ability to install and use web applications similarly to how native applications are used, but with limited access to native features (Malavolta, 2016).

There are advantages and disadvantages with all approaches. The most performant applications, apart from native applications, are built with the cross-compiled approach, also known as the generated approach (Xanthopoulos & Xinogalos, 2013). An overview of available frameworks and a thorough explanation of each the strategies is presented by Biørn-Hansen, Grønli, & Ghinea (2018).

(7)

3

2.1. Android

Android OS is the most used mobile operating system globally (Gartner, 2018). It supports a wide range of devices and provides a set of features which can be used by programmers to develop applications for these various devices. The operating system is based on the Linux Kernel which provides necessary features for hardware manufacturers to use when building their devices. Hardware components for each device are manipulated through the hardware abstraction layer which figures as an API for the higher-level interfaces accessible to developers. Each application is assigned with an instance of Android Runtime, formerly known as Dalvik, which provide virtual machines created to execute applications with low memory usage. The low memory usage is achieved due to the use of DEX bytecode, which the Android Runtime compiles from Java sources. Android Runtime supports both just-in- time (JIT) compilation and ahead-of-time (AOT) compilation. For developers, the features that the Android platform provides are available through the Java API and the C and C++

libraries provided. There is also a set of system applications which can be used at will (Android, 2020e). When a user interacts with an Android application it is the various activites inside the application that is manipulated and changed (Android, 2020g). For each state transition, such as starting the application, or pausing it, an activity is affected.

Developers can use the activity lifecycle callbacks that the activities provide, to alter the application’s behavior for a certain state transition.

2.2. Kotlin for native development

Kotlin is one of Android’s natively supported languages. In 2017 the Android team announced that they would start providing support for Kotlin development, and describes it as being a modern and powerful language capable of solving several issues that previously were frequently encountered during development of Android applications (Shafirov, 2017).

Since then, the amount of developers implementing Kotlin in their Android development has been increasing steadily. The Android team emphasizes their commitment to Kotlin by making it their first priority language. They also ensure developers that they will continue to work on features which will further improve and ease Android development in Kotlin (Winer, 2019). The performance of applications written in Kotlin performs equally to Java applications and can sometimes achieve even better performance (Kotlin, 2019d). For cases where Kotlin is unable to provide sufficient functionality for certain tasks, developers are able to call and use existing Java classes directly from their Kotlin code (Kotlin, 2019b).

2.3. Honeywell Dolphin CT60

For this study, handheld computers will be represented by the Honeywell Dolphin CT60 device. The device is equipped with a laser scanner, which is a requirement for aspects which are to be investigated during this study. The Honeywell Dolphin CT60 figures as an appropriate representation of handheld computers, as it complies with the Android Enterprise Recommended requirements for a mobile device (Android, 2020d). It is

(8)

4

described as a rugged device built for work in harsh conditions such as rain and snow, and suiting enterprises seeking to achieve high productivity (Android, 2020b).

(9)

5

3. Problem

The growth in the mobile market increases the demand for more efficient ways of developing mobile applications, as the demand to support different platforms subsequently increases as well. In the pursuit of solving this problem, several cross-platform frameworks have been developed (Xanthopoulos & Xinogalos, 2013). Ideally, mobile applications would be developed using a cross-platform framework which provides the opportunity to build performant cross-platform applications with high quality, in minimal time, and at a low cost (Heitkötter, Hanschke, & Majchrak, 2013; Xanthopoulos & Xinogalos, 2013; Mercado, Munaiah, & Meneely, 2016). Currently, no approach have been established as the universally best option for fulfilling this ideal, partially due to the specific requirements and constraints that each development team has. Nevertheless, the selection of a suitable development framework is important. There are advantages and disadvantages with all cross-platform frameworks and it is therefore crucial that each development team carefully select the best option for their specific situation (Tunali & Zafer, 2015).

For companies or developers with sufficient resources, developing applications natively is common, as native development frameworks provide the developers with full access to the underlying platform and enables them to build of performant applications (Xanthopoulos &

Xinogalos, 2013). Native development frameworks, however, generally require a higher degree of specific knowledge and experience than cross-platform frameworks and might not be a reasonable option (Heitkötter, Hanschke, & Majchrak, 2013). A sufficient alternative approach could be preferable for developer teams without the knowledge required for native development.

As of 2019, Microsoft ended support for Windows 10 Mobile. The commercial customers developing toward their platform were encouraged to migrate their development to Android or iOS platforms (Microsoft, 2019a). Applications for Windows Phones are developed in C#

which is not natively supported either by Android or iOS. Migrating applications from Windows Phone to iOS or Android would therefore require for existing applications to be rewritten in another programming language which is costly (Tunali & Zafer, 2015).

3.1. Problem statement

The selection of development framework is difficult for developers. There are various aspects for companies to consider and current research is insufficient for making adequate choices (Willocx, Vossaert, & Naessens, 2016). The range of cross-platform frameworks which Biørn-Hansen, Grønli, & Ghinea (2018) presents, show that there are several to choose between, and they also encourage further research where the various frameworks are evaluated from different aspects.

(10)

6

Regarding performance, researchers frequently find cross-platform solutions to be slower than native solutions, however the performance advantage of native development is overshadowed by its drawbacks (Heitkötter, Hanschke, & Majchrak, 2013). This raises the question of whether there are viable alternatives. The native prototypes developed during the study by Heitkötter, Hanschke, & Majchrak (2013) did not perform better than the one developed using the PhoneGap cross-platform development framework. They conclude that this is most likely due to it being optimized for fast rendering of web pages, but it still shows that there are situations where native development does not necessarily provide more performant applications than other frameworks. Willocx, Vossaert, & Naessens (2015) state that all cross-platform tools come with a performance penalty, though, and therefore native development can be assumed to be advantageous in that aspect compared to cross-platform development. Performance is, however, not the only factor for determining which framework is best suited for a case. There are various other aspects which might be relevant, such as the skill of the developer, the type of application that is to be developed and whether or not there is existing code which could be reused. As no current framework is suitable for all developers, further research in existing frameworks is needed to provide better foundation for the choice of framework (Willocx, Vossaert, & Naessens, 2016).

3.2. Aim & objectives

The amount of tools and frameworks available for developing mobile applications makes the selection of a suitable solution difficult for developers and companies. Currently, none of the available options poses as the best framework for all cases. To enable better understanding of the landscape of development frameworks and to make the selection of the most suitable one for a specific situation easier, more research is required in the area (Biørn-Hansen, Grønli, & Ghinea, 2018).

The aim of this study is to select promising cross-platform frameworks and provide a comparative analysis to show if there are viable alternatives to native development when building applications for handheld computers. The objectives which need to be fulfilled in order for the aim to be reached are the following:

Obj1: Establish criteria for selecting and evaluating cross-platform frameworks Obj2: Select cross-platform frameworks consistent with the selection criteria

Obj3: Perform a comparison of the selected cross-platform frameworks and the native development framework

Obj4: Present and analyze the results

(11)

7

3.3. Research question

RQ1: Which are the most relevant criteria for evaluating alternatives to native development frameworks for developing Android applications for handheld computers?

RQ2: To which extent do the selected frameworks fulfill the criteria for being viable alternatives to native development frameworks for developing Android applications for handheld computers?

3.4. Hypothesis

Null hypothesis (H10): No criteria can be determined to be more relevant than others for evaluating alternatives to native development frameworks for developing Android applications for handheld computers.

Alternative hypothesis (H11): Performance and hardware access are the most relevant criteria for evaluating alternatives to native development frameworks for developing Android applications for handheld computers.

Null hypothesis (H20): The selected cross-platform development frameworks do not fulfill the selected criteria for being viable alternatives to native development frameworks for building handheld computer applications for Android devices.

Alternative hypothesis (H21): The selected cross-platform development frameworks sufficiently fulfill the selected criteria for being viable alternatives to native development frameworks for building handheld computer applications for Android devices.

Developing applications with cross-platform frameworks is frequently regarded as cost and time-saving despite the various drawbacks they have compared to native development (Biørn-Hansen, Grønli, & Ghinea, 2018). Research shows that cross-platform frameworks are inferior to native development frameworks in several aspects. Performance and hardware access are commonly addressed as issues for the various development strategeis (Dhillon, 2015; Willocx, Vossaert, & Naessens, 2015; Willocx, Vossaert, & Naessens, 2016;

Xanthopoulos & Xinogalos, 2013). For this reason, the expected result for RQ1 is the alternative hypothesis, H11, that these two aspects will be the most relevant aspects for evaluating alternatives to native development. Despite the fact that cross-platform development is problematic and have drawbacks compared to native development, cross- platform frameworks are widely used, indicating that the drawbacks tied to them are tolerable. The performance penalties which cross-platform frameworks induce, for instance, are generally acceptable (Willocx, Vossaert, & Naessens, 2016). This provides basis for expecting the null hypothesis H20 to be discarded by this study and for the alternative hypothesis H21 to be confirmed for RQ2.

(12)

8

4. Method

The methodologies selected to answer the research questions of this study are literature review and experiment. Literature reviews display existing knowledge about the researched area and can be used here to provide information about cross-platform development and important aspects for the comparison of frameworks. Conducting an experiment is appropriate when the research question can be answered with quantitative data. It can be used for investigating relationships between variables to provide better understanding about them. Experiments provide a high level of control over the study subject. Constant values are assigned to the variables which are not supposed to have an influence on the measurements.

Various values are carefully selected and assigned to the factors whose effects the researchers intend to study. Tests are executed for each set of values in order to provide quantitative data and thereby a way of measuring the impact of the factors. There are drawbacks with conducting an experiment. Real-world studies are costly to conduct and therefore settings are often simulated to resemble that of the real-world scenario. This introduces the risk of not producing realistic results as measurements are performed on a simulated setting.

Moreover, experiments do not provide a product which can be directly used by a company for profit (Wohlin et al., 2012).

This study will investigate whether or not the selected frameworks are sufficient for developing applications for handheld computers. The ISO 9126 standard specifies six parts which determine the quality of software (Abran, Khelifi, Suryn, & Seffah, 2003). Prior to the comparison of the frameworks, selection criteria and evaluation criteria will have to be obtained. The ISO 9126 standard figures as base for this task, as it outlines relevant aspects to regard when evaluating software. Out of the several aspects to consider for building high quality software, the most relevant factors which are feasible to investigate during this study, will be selected. Afterwards, frameworks are chosen according to the selection criteria and applications are going to be developed using these frameworks. The applications are then compared in order to find out to which extent there are cross-platform frameworks available which present a viable alternative to native development.

The frameworks are examined in the context of handheld computers. The targeted devices are equipped with laser scanners and consequently have the ability to scan barcodes. Devices of this type are aimed at replacing designated barcode scanners, commonly used in retail (Coughlan & Breslin, 2003). Barcodes are likely to be frequently used in industry over the next decade (Creusot & Munawar, 2015). The efficiency and reliability of laser scanning of barcodes have essentially made it an industry standard. Alternative solutions for reading barcodes using mobile cameras are error prone and slow. Research show that the implementation of handheld computers can help companies save considerable amounts of money, making development of applications for these devices relevant (Pihir, Pihir, &

Vidačić, 2011).

(13)

9

4.1. Literature review

Before conducting the experiment a literature review was conducted in order to compile a scientific basis for the study. Literature review is a method of indentifying, analysing and interpreting all existing evidence regarding the targeted area of research, in order to obtain a thorough understanding of it (Wohlin et al., 2012). The literature review conducted during this study is not exhaustive, and thus a complete picture of all available research available for the subject is not claimed to have been retrieved. The purpose of the literature review is twofold. The first part of it focused on answering RQ1, by finding relevant information about cross-platform development in order to establish selection criteria and evaluation criteria for the selection and evaluation of promising cross-platform development frameworks. The second part of the literature review focused on evaluating the frameworks selected according to the extracted criteria.

By searching the Google Scholar database and reviewing the relevant articles found, the literature review was aimed at providing a sufficient overview of the area. Search terms used in Google Scholar for initial papers were “mobile application development”, “android application development“, “cross-platform mobile development”, “cross-platform development”, “handheld computers”, and “barcode industry”. The functions for excluding patents and quotes from the search results were used. For each search term, the 50 first search results were regarded, although two criteria were applied to decrease the extent of the literature review. Papers published either in ACM, IEEE, Wiley or Springer were selected as these are peer reviewed publishers that are established in the area of informatics. All books were ignored during the literature review as well. Articles rendered as relevant were used to obtain further research papers through their reference lists. Not all papers referenced were reviewed, but only those deemed relevant for fulfilling the aim of this study, as well as those discussed in the related work section. Papers that did not present an abstract related to the aim were not reviewed further. Additional papers were found through unsystematic literature searches. Papers behind a paywall were not included.

4.2. Experiment threats

Experiments always include inherent risks which the researchers need to be aware of and handle. The size of an experiment determines the cost and time needed for the investigation.

Decisions have to be made about the setting to ensure that the experiment is feasible but also provides useful results. Conducting an experiment in a real-world setting allows for better generalizability but it involves a higher cost and takes more time than for a toy sized experiment. This is an external threat that needs to be carefully handled to ensure the validity of the study. As this study does not measure real-world software artifacts, the generalizability of the results is limited (Wohlin et al., 2012).

Another threat for this study is whether or not representative aspects are measured in order for the hypotheses to be confirmed or discarded. This threat to the construction validity

(14)

10

needs to be counteracted in order to draw valid and relevant conclusions from the results of the study (Wohlin et al., 2012).

4.3. Alternative methodology

Several methodologies could have been used for answering the research questions in this study. Wohlin et al. (2012) provide an overview of viable methodologies for conducting research in the area of software engineering. Besides from the methodologies selected here, the frameworks compared during this study could have been evaluated through a case study or a survey. The experimentation part could also have been replaced by extending the literature review.

Case studies are commonly used to investigate real-life projects in order to show the properties of an attribute or a relationship between attributes (Wohlin et al., 2012). The disadvantage of using case studies is that the level of control is lower than for experiments. It is also harder to ascertain that the results collected from a case study can be generalized. The use of a case study for answering the research questions asked here is reasonable, but it would require having the opportunity of studying the frameworks selected for this study, in a development project. As no suitable project was present during, experimentation was chosen as the methodology for comparing the frameworks.

The research questions could have been answered to an extent with the use of surveys.

Surveys can help with drawing general conclusions for the researched area, as it can both provide qualitative and quantitative data (Wohlin et al., 2012). It can be used to gather knowledge from people about a tool which have been in use for a while. The limitation of using surveys for this study would be that novel frameworks would be difficult to evaluate as well as recent changes in older frameworks. Finding respondants which have been using the studied subjects for a sufficient amount of time would be challenging.

An exhaustive literature review was not conducted during this study and data was merely gathered in order to provide a solid foundation for the study. Systematic literature reviews can be used for answering a research question and provide a compilation and evaluation of the scientific research available for a specific research area (Wohlin et al., 2012). As for the survey methodology, systematic literature reviews are inferior to experiments when it comes to evaluating newer artifacts, as there is an insufficient amount of publications about the study subject.

(15)

11

5. Related Work

The area of cross-platform development is widely researched and several studies have been made where various cross-platform frameworks are compared. Dhillon (2015) developed an evaluation framework which they used to compare a range of frameworks such as PhoneGap and Titanium. Jia, Ebone, & Tan (2018) compared native development with Xamarin, Cordova and Titanium with focus on performance aspects. Cross-platform approaches such as hybrid, web and cross-compiled were compared by Heitkötter, Hanschke, & Majchrzak (2013) by comparing frameworks from each category. Mercado, Munaiah, & Meneely (2016) compared different cross-platform approaches through a survey and Raj & Tolety (2012) discussed various issues and aspects to consider for each approach. In the study by Tunali &

Zafer (2015), both functional and non-functional requirements were considered during their comparison between native development, PhoneGap, Xamarin, Titanium and Smartface.

Both Willocx, Vossaert, & Naessens (2015) and Willocx, Vossaert, & Naessens (2016) are studies where performance aspects were compared between native and other frameworks such as Xamarin. Xanthopoulos & Xinogalos (2013) contrasted various cross-platform approaches and provides a comparative analysis with help of a case study. Finally, Zibula &

Majchrzak (2013) compared PhoneGap, jQuery Mobile, and HTML5 by building an application with each framework.

There are studies mapping the available research on the area and presenting general conclusions that can be drawn. Biørn-Hansen, Grønli, & Ghinea (2018) and El-Kassas et al.

(2017) are two of the most recent and relevant of these. The study by Biørn-Hansen, Grønli,

& Ghinea (2018) shows that further research studies evaluating various cross-platform frameworks, both qualitatively and quantitatively, are required. The authors list Kotlin and Flutter as interesting study subjects.

Research about the usage of handheld computers in industry has been done to a degree.

Coughlan & Breslin (2003) presented a handheld computer solution to replace proprietary barcode scanners in retail and Pihir, Pihir, & Vidačid (2011) investigated the usefulness of handheld computers with scanners in industry. Moreover, various studies have been conducted on how to improve and alter the scanning process and widening its application areas, such as Creusot & Munawar (2015), but otherwise no further research related to the area was found. Creusot & Munawar (2015) also elaborate on the usefulness of barcode scanning and the advantages of laser scanning in contrast to mobile cameras.

(16)

12

6. Results

This study aims to contribute to the general understanding of the field of cross-platform development by finding promising development frameworks and comparing them with the native development framework for Android. When selecting a development framework there are certain aspects to consider (Tunali & Zafer, 2015). A set of those aspects are evaluated in this study and specific requirements for building applications for handheld computers are also considered for the evaluation.

6.1. Comparison of frameworks

The aspects that will be studied here are whether or not cross-platform frameworks provide sufficient access to functionality, sufficient performance, and sufficient support. These aspects were obtained to accomplish Obj1. Out of the six parts of software quality that are addressed in the ISO 9126 standard, only the functionality aspect, the efficiency aspect, and the usability aspect will be considered here (Abran et al., 2003). Performance and hardware access are mentioned as issues for cross-platform frameworks in several related papers (Dhillon, 2015; Willocx, Vossaert, & Naessens, 2015; Willocx, Vossaert, & Naessens, 2016;

Xanthopoulos & Xinogalos, 2013; Biørn-Hansen, Grønli, & Ghinea, 2018). Xanthopolous &

Xingalos (2013) describe the goal of cross-platform frameworks as to be able to run on as many platforms as possible with native performance. With the current state of mobile OS usage, the former of these two goals is nowadays mostly about supporting multiple versions of iOS and Android OS (Gartner, 2018). The latter of the goals is not agreed upon throughout the literature whether or not cross-platform frameworks are currently able to fulfill (Biørn- Hansen, Grønli, & Ghinea, 2018). This makes performance highly relevant to include as one of the aspects for evaluating alternatives to native development. Although mentioned as generally important for cross-platform development, when it comes to application development for handheld computers access to hardware for manipulating the scanning functionality of the device is non-negotiable. Without the proper tools for manipulating the hardware of the device, the applications would not be able to make use the laser scanner that the device is equipped with (Creusot & Munawar, 2015). As barcode scanning using a camera is insufficient, the laser scanner is vital for efficient scanning throughout the industry.

An experiment provides the opportunity to ascertain potential differences between the frameworks which might be essential for developers when selecting strategy. From the ISO 9126 standard three aspects were deemed unfit for this study, portability, reliability, and maintainability. The usability aspect is only partially feasible to evaluate for this study. The portability aspect is not as relevant for this study as there is already an established advantage of cross-platform frameworks over native development framework in this aspect. Reliability is not as relevant for smaller projects and would therefore be more feasible to investigate in a case study but is ruled to be slightly out of scope for this study. To compare maintainability and usability, a survey would be preferable as these are subjective and requires having interacted with the framework for a longer period of time. There are usability aspects which are feasible to investigate here, though, such as the presence of an active community and

(17)

13

relevant documentation (Wheeler, 2011). When reviewing previous work and regarding the area of application development for handheld computers, the two most important aspects can be determined to be performance and hardware access. The alternative hypothesis H11

can therefore be confirmed for RQ1. Existing documentation can also be ruled to be important but not equally crucial. Nevertheless documentation will be considered during this study, as it is relevant and feasible to inspect for this study, in contrast to other, equally important aspects.

6.1.1. Available functionality

The study aims to evaluate promising cross-platform frameworks and determine whether they provide the sufficient functionality for handheld computer applications or not.

Handheld computers have multiple application areas and can be beneficial in several domains (Coughlan & Breslin, 2003; Pihir, Pihir, & Vidačić, 2011). The main difference from mobile phones is that handheld computers are equipped with laser scanners. It is therefore crucial for development frameworks to provide the necessary functionalities in order to manipulate and make use of the scanner. The focus of this study is, consequently, to ensure that cross-platform frameworks can provide comparable scanning functionalities to that of a native development framework.

6.1.2. Benchmarking

This study aims to show whether applications built with cross-platform frameworks can achieve tolerable response times in comparison to those built with a native development framework. Native applications achieve the highest performance in comparison to cross- platform applications built with various approaches (Willocx, Vossaert, & Naessens, 2016).

Therefore, the native application serves as a baseline for what can be regarded as tolerable response times. When interacting with an application, what is an acceptable response time vary depending on the requested task. Response times over 0.1 seconds can be unsatisfactory for control tasks, where the device needs to display responsiveness (Miller 1968). Previous research of response times is compared and contrasted in the study of Dabrowski & Munson (2011). The authors highlight that tasks not categorized as control tasks can have acceptable response times of up to several seconds. The response times retrieved during this study is therefore also evaluated accordingly, with regard to the context of the tasks.

The actions to be measured in this study are aimed at showing the responsiveness of cross- platform applications in comparison to native applications. Measuring all possible actions is infeasible for this study, and thus, only a selection of actions will be implemented in the applications. In the study by Willocx, Vossaert, & Naessens (2016), the launch time, the time to load a new page and the time to return to the previous page are measured to show the responsiveness during navigation in the applications. Willocx, Vossaert, & Naessens (2015) measure response time by measuring the time for starting the application, pausing the application, resuming the application and navigating to another page of the application.

The following are the actions which will be measured during this study:

(18)

14

 Launching the application:

The time between a press on the application icon in the device menu and a complete loading of the start page.

 Loading a new page:

The time between a press of a navigation button on the start page and a complete loading of the page navigated to.

 Returning to previous page:

The time between a press of the return button on the page navigated to from the start page, and to a complete reloading of the start page.

 Pausing the application:

The time for the “onPause” method to execute.

 Resuming the application:

The time for the “onResume” method to execute.

Due to the differences between the tasks measured, different thresholds are used for evaluating the applications’ abilities to sufficiently perform the tasks. For an instant response of a press of a button, which Dabrowski & Munson (2011) categorizes a control task, the applications need to present visible results that a press has been registered. Pausing and resuming the applications are the two tasks which fall into this category, as they indicate that the state of the application has changed. The other three tasks include the loading of page and are not required to be completed as quickly. The threshold for these tasks are one second, whereas the pause and resume tasks are considered sufficient if they are executed in 0.1 seconds or less (Miller, 1968).

6.1.3. Support

When selecting development framework, the level of support available is a considerable factor. Support does not necessarily only refer to a company or a branch providing help for developers. It can also include aspects such as documentation and whether there is a developer community providing help for each other or not. These are aspects which can help determine which development framework to choose. When comparing the level of support for different frameworks, existing documentation can be evaluated on certain aspects. It can be determined whether or not there is sufficient information about how to install and setup required tools, if the guides are understandable and whether or not they provide enough information for building the desired applications (Wheeler, 2011). Additionally, the level of support is determined by the degree of present code examples, links to similar problems, and user comments to help developers find solutions to their problems (Heitkötter, Hanschke, &

Majchrak, 2013).

(19)

15

6.2. Selection of frameworks

Developers require efficient ways to develop applications with sufficient performance, as the high costs of development and maintenance become problematic with native programming (Willocx, Vossaert, & Naessens, 2015). As this study is aimed at examining the capability of current cross-platform development frameworks’ ability to fulfill this demand, a selection of the most promising frameworks needs to be done, which Obj2 outlines. The cross-compiled approach is the most performant as it creates native bytecode for each platform, providing the tools to make use of platform specific properties (Xanthopoulos & Xinogalos, 2013).

Cross-compiled frameworks also exceed other strategies for accessing the hardware, which is crucial when developing for handheld computers with laser scanners (Raj & Tolety, 2012;

Creusot & Munawar, 2015).

As performance and hardware access are the two most important aspects for the evaluation of viable options to native development, frameworks from the cross-compilation category have been selected for this study. Biørn-Hansen, Grønli, & Ghinea (2018) list available frameworks and of those, Flutter and Xamarin was chosen. They suggest Flutter as an interesting research subject, due to its ability to build high-performing applications. Because of this, and the fact that no previous studies include Flutter, it was selected as one of the frameworks evaluated in this study. Xamarin was chosen due to its attractiveness for C#

programmers (Willocx, Vossaert, & Naessens, 2015). The amount of Windows Phone developers forced to migrate either to Android or iOS platforms raises the importance of sufficient research for making informed decisions (Microsoft, 2019a). Although previous studies have been made on the performance, such as that of Jia, Ebone, & Tan (2018) and the study by Willocx, Vossaert, & Naessens (2015), it is unclear whether or not Xamarin provides a complete set of native functionalities (Biørn-Hansen, Grønli, & Ghinea, 2018).

The compilation of cross-platform frameworks by Biørn-Hansen, Grønli, & Ghinea (2018) contains other cross-compiled approaches not studied here. Kotlin was selected ahead of Java as the programming language for native applications, which Biørn-Hansen, Grønli, &

Ghinea (2018) promote as an interesting subject for future research.

6.2.1. Flutter

Flutter is Google’s cross-platform development toolkit for quickly building applications with native performance. The claim that the applications developed in Flutter achieve native performance stems from the fact that native bytecode is generated separately for each platform (Flutter, 2019a). Flutter’s ability to provide separate bytecodes for each platform qualifies it as a cross-compiled development framework (Biørn-Hansen, Grønli, & Ghinea, 2018). Flutter allows developers to create their applications in the Dart programming language as it provides its own compiler to create native bytecode with AOT compilation.

Dart is presented as a modern and concise programming language which makes the system easier for developers to approach and increases the developers’ control over it. For situations where Dart is insufficient and where access to the native programming languages is required, Flutter provides mechanisms which enable the use of Java and Kotlin code in Flutter

(20)

16

applications. Currently, the supported development editors are Android Studio, IntelliJ IDEA, and VS Code (Flutter, 2019a).

6.2.2. Xamarin

Xamarin is an extension of the .NET platform which provides a framework for accessing native features on Android, iOS, and other platforms (Microsoft, 2020). Xamarin can, like Flutter, be classified as a cross-compiled development framework (Biørn-Hansen, Grønli, &

Ghinea, 2018). Whether or not Xamarin belongs to the category is not entirely agreed upon.

El-Kassas et al. (2017) classify Xamarin as an interpreted approach whereas Willocx, Vossaert, & Naessens (2015) describe it as a cross-compiling framework. Xamarin uses different strategies depending on which platform it is targeting. The performance differences between the platforms are apparent due to this (Willocx, Vossaert, & Naessens, 2016). For the Android platform applications written in C# are compiled into intermediary code which is then JIT compiled when the application is running. When a Xamarin application is launched, an instance of the Mono execution environment is launched alongside the Android Runtime instance, which enables the usage of libraries and packages available through the .NET platform (Microsoft, 2019). Xamarin suits development teams already familiar with the Microsoft .NET framework and with C# (Tunali & Zafer, 2015). This is an important advantage of Xamarin due to the amount of developers accustomed to the .NET environment and C# development. The end of support for Windows 10 Mobile forces developers to migrate their native C# applications to other platforms (Microsoft, 2019a). Xamarin is an attractive alternative for these developers, as they are not required to spend time learning about new technologies but instead are able to develop their applications instantly. The drawbacks with using Xamarin are that platform specific code is required, which decreases the inherent advantages of developing using a cross-platform framework, and also that the JIT compiling and the startup of the Mono execution environment can increase the startup time and the execution time for the applications (Willocx, Vossaert, & Naessens, 2015).

6.3. Implementation

Three identical applications were developed during this study, to allow for quantitative comparisons between the frameworks. The first application was developed natively, using Kotlin as programming language and Android Studio as development environment. The code for this application can be found in Appendix B, Appendix C, Appendix D, and Appendix E. Flutter and Xamarin were then used for building one application for each framework, which resembled the native application as closely as possible. For the native application, the Android guide for building your first application was used. This guide shows how to create the initial project using Android Studio, how to run the project and how to add another activity which can be navigated to (Android, 2019b). The application was built with an intention of only having the required functionality for enabling measurement of the actions presented in the previous section. For the start activity, a button was added to enable navigation, and a return button was added to the second activity in order to enable measurement of the navigation (Android, 2019a). In order to test the scanner functionality, a text input field was added to the application (Android, 2020c).

(21)

17

Figure 1 - Kotlin EditText control (left), Flutter TextField control (center), Xamarin EditText control (right)

6.3.1. Measuring response times

To measure the response times of the applications, timestamps were logged to files. The text input fields were removed for the applications used for testing the response times and is therefore not included in the code presented in the appendix. The implementation of the logging functionality made use of the File class and the appendText method (Kotlin, 2019a).

Permissions were added to the manifest file to enable file management. A static method was created in the application class to avoid creating a new object, using the method described in (Kotlin, 2019c). To prevent excessive time spent on file management, the files were assumed to already exist on the device. The system clock was used to retrieve the current timestamp when writing to the file (Android, 2020f). For the startup time measurements, a class overriding the “Application” class was created. Inside this class, the parent class’ onCreate method is overridden to enable logging the initiation of the startup process.

6.3.2. Flutter application

The Flutter application was developed according to the navigation example presented by Flutter (2019d). The code for this application can be found in Appendix F, Appendix G, Appendix H, and Appendix I. The pages in Flutter use a different approach than the native activities used in Kotlin and Xamarin (Flutter, 2019b). The application has no visible differences from the native application and thus the approach recommended by Flutter (2019b) was ruled to be reasonable to use for the response time comparisons. To control the scanning functionality, a TextField control was added to the start page (Flutter, 2020).

Flutter offers ways to write Kotlin code and call the created methods directly inside the Flutter project. This was used to implement the logging functionality, which allowed for the same code to be executed as in the native project, apart from the method call. Therefore, the

(22)

18

same method of accessing the system clock was used (Flutter, 2019e). The application class was overridden by creating a Kotlin class inside the project, in order to enable measuring of startup time. The manifest file was altered in order to enable file management.

6.3.3. Xamarin application

The Xamarin application, like the Flutter application, was built to resemble the Kotlin application as closely as possible, while still maintaining the development practices suggested by Xamarin, in order not to impede the performance of the application. The code for the Xamarin application can be found in Appendix J, Appendix K, Appendix L, and Appendix M. Xamarin provides the tools to create an application with similar structure to the native implementation. Activities are used in the same way as the native approach which makes the application build for this study easy to compare with the native application (Microsoft, 2018). The editText component was added to the start page in order to test the scanning functionality for the handheld computer, but was then removed for the response time tests (Microsoft, 2019b). A static method was implemented in C# for writing timestamps to a log file, as no methods for interoperability with Kotlin is inherently available through Xamarin. To enable this, the manifest file was altered. The file class was used to append text to the log file and the system clock class was used to obtain the timestamp (Microsoft, 2020a; Microsoft, 2020b). As with the other two frameworks, Xamarin provides a way to override the application class, which allows for logging the timestamp of the initialization of the startup process. The entire application, the logging functionality included, was developed using the native languages of the .NET framework.

6.3.4. Separation of actions

Instead of logging all timestamps required for the benchmarking study simultaneously, the applications were split into two separate applications for each framework, as to avoid measuring file manipulation unnecessarily. The navigation was measured in one of the applications whereas the time for pausing and resuming the application was measured in the other. As the frameworks use the resume and pause methods on slightly different occasions, these logging actions could have otherwise induced skewed conditions for measuring the navigation for the applications. This is not displayed in the appendices of this report, but rather all the logging functionality is present. The usage of the frameworks’ text input fields are not presented either.

6.4. Compiled results

The experiment is intended to show a comparison of the frameworks and their features.

Although there are aspects not investigated during this study, the data collected here can indicate whether or not the frameworks display promising results and thus can be rendered as possible alternatives to native development. The three aspects compared here are the availability of necessary functionality, the performance, and the level of support for each framework. Each aspect is considered individually for the experiment. For the functionality

(23)

19

tests, the Honeywell Dolphin CT60 was used to examine the presence of scanner support. As for the functionality tests, the benchmarking tests was executed on the Honeywell Dolphin CT60. Additionally, the same tests were executed on the Samsung Galaxy S10 to allow for more generalizable results. As the Samsung device has better performance than the Honeywell device, these additional tests indicate how the frameworks performance changes relative to the hardware. For these tests, the devices were put into flight mode and all other running applications were closed before the test executions. Each application was installed and given permission to file manipulation for the logging functionality prior to the tests.

6.4.1. Available functionality

All of the frameworks provided sufficient tools for manipulating the scanning functionality of the Honeywell Dolphin CT60 device which the experiment was conducted on. Kotlin provides the editText control which can be incorporated in activities and automatically receive and render text from a scanned barcode. In similar fashion, Flutter offers a control called TextField which provides the same opportunity for barcode scanning as the editText control of Kotlin. For Xamarin, the corresponding text input field is called Entry, but is interchangeable with the editText control which is accessible for Android development. The Honeywell Dolphin CT60 delivers data from the scanner as a keyboard input, making these results relevant for all handheld computers which use this approach for handling scanner input. In the aspect of providing necessary functionality, H21 can be confirmed for RQ2, as both Flutter and Xamarin figure as viable alternatives for this aspect.

6.4.2. Benchmarking

The results show that all of the frameworks achieve response times within the thresholds used for this study. The threshold for page navigation is one second whereas for resume and pause, the threshold is 0.1 seconds. No test on either of the devices surpassed these boundaries which figure as upper limits for tolerable response times. There are apparent similarities between the Kotlin tests and the Xamarin tests, both for the Honeywell Dolphin CT60 and the Samsung Galaxy S10 tests. The Flutter applications differ, as the samples contain outliers higher than those of both the Kotlin applications and the Xamarin applications. Below, box plots are presented for the response times achieved by the applications of each framework. Figure 2 and Figure 3 show the aggregated results of all the 50 tests for each of the five actions, thus 250 results are included in each framework’s box plot.

(24)

20

Figure 2 - Response times in seconds - Honeywell Dolphin CT60

Figure 3 - Response times in seconds - Samsung Galaxy S10

The average response time for all of the frameworks is beneath the lower of the thresholds, 0.1 seconds, even though 60% of the measurements are for page navigation. This shows both of the cross-platform frameworks are able to achieve tolerable response times that are lower than the thresholds by a margin. When comparing the frameworks with each other though,

(25)

21

there are certain performance differences. Flutter show higher response times on average than both Xamarin and Kotlin on both devices. This is due to the disctintly worse performance for startup than the other two frameworks. Even though the both quartiles as well as the median value is lower for Flutter than for Kotlin and Xamarin, the average response time is the highest for Flutter. Xamarin is less performant than Kotlin only in the tests on the Honeywell Dolphin CT60, while achieving similar performance on the Samsung Galaxy S10. Figure 4 and Figure 5 show an average of all the retrieved response times for all actions.

Figure 4 - Average response times in seconds - Honeywell Dolphin CT60

(26)

22

Figure 5 - Average response times in seconds - Samsung Galaxy S10

The differences between the samples are larger for both the Flutter and the Xamarin application when it comes to page navigation compared to pausing and resuming the applications. While Xamarin performed worse or equal to Kotlin for each of the actions, Flutter performed worse for launching, but performed better than Kotlin for the other two navigation actions. Figure 6 and Figure 7 show the frameworks’ response times for the actions included in the page navigation category, starting the application, navigation to new page, and returning to previous page. The response times are retrieved from the 50 tests executed for each action and application.