Thesis no: MGSE-2015-02
Software Quality Characteristics Tested
For
Mobile Application Development
Literature Review and Empirical Survey
WALEED ANWAR
Faculty of Computing
Blekinge Institute of Technology
SE-371 79 Karlskrona Sweden
This thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in
partial fulfillment of the requirements for the degree of
Master of Science in Software
Engineering. The thesis is equivalent to 10 weeks of full time studies.
Contact Information:
Author(s): WALEED ANWAR
E-mail: waan14@student.bth.se
University advisor:
Dr. Simon Poulding
Department of Software Engineering
Faculty of Computing
Internet :
www.bth.se
Blekinge Institute of Technology
Phone
: +46 455 38 50 00
3
A
BSTRACT
Context. Smart phones use is increasing day by day as there is large number of app users. Due to more use
of apps, the testing of mobile application should be done correctly and flawlessly to ensure the effectiveness of mobile applications.
Objectives. The objective of this research is to find out the important mobile application quality
characteristics from developer’s perspective and how developers actually test for them. Apart from that how the developers test their mobile applications are also addressed.
Methods. Two methodologies were used: the literature survey and the empirical survey. The literature
survey was used to get familiar with the most commonly known mobile application quality characteristics for which mobile applications are tested for. The empirical survey was used to get data from developers by sending an online questionnaire link to the Google Play store developers and their response was recorded and further evaluated to present results. I categorized apps as top rated apps and low rated apps.
Results. I got 43 responses as a whole from 1000 developers. I got 20 responses out of 500 from the top
rated apps developers and 23 responses out of 500 from the low rated apps developers. The results were used to compare the quality characteristics importance to testing. According to the responses from both top rated and low rated apps developer, functional testing and usability testing is considered to be the most important and are mostly tested for “Android” mobile applications. Load testing and energy testing is considered to be not that important from developer’s point of view but energy testing and memory testing is not so often tested for “Android” mobile applications by developers. When it comes to testing process, top rated and low rated apps developer perform testing by creating test cases that are predefined in scripts or documents. They prefer to execute their tests on emulator rather than on mobile and run their tests automatically via tool or service as a cloud.
Conclusion. My study reports the quality characteristics importance for which the mobile applications are
tested for from developer’s perspective, actual testing done by developers and the testing process used by them. Functional testing and usability testing are considered to be most important and mostly tested quality characteristics according to developer’s perspective. Testing is done by running their tests on emulator automatically through some tool. They prefer to create their test cases that are predefined in scripts.
4
A
CKNOWLEDGEMENT
I would like to extend my gratitude to my supervisor Simon Poulding for his vital support and
feedback till end. This thesis would not have been possible without Simon Poulding guidance
throughout the research. I really feel honored working with him.
I would like to thank my mom, siblings and friends who helped me and supported me. Finally, I
sincerely appreciate all the participants of the web questionnaire from all over the world
because without their response my thesis would not be accomplished.
5
T
ABLE OF
C
ONTENTS
ABSTRACT ... 3
ACKNOWLEDGEMENT ... 4
TABLE OF CONTENTS ... 5
LIST OF FIGURES ... 7
LIST OF TABLES ... 8
1
INTRODUCTION ... 9
1.1
AIMS AND OBJECTIVES ... 10
1.2
RELATED WORK ... 10
2
METHODOLOGY: ... 12
2.1
RESEARCH QUESTIONS ... 12
2.2
LITERATURE REVIEW ... 12
2.2.1
EXTENDED DESIGN AND DERIVING THE START SET OF PAPERS: ... 13
2.3
EMPIRICAL SURVEY ... 14
2.4
PREPARATION OF QUESTIONNAIRE ... 15
2.5
RESEARCH DESIGN AND DATA COLLECTION: ... 15
3
RESULTS ... 17
3.1
TOP RATED APPS DEVELOPER’S FEEDBACK ... 17
3.1.1
IMPORTANCE OF QUALITY CHARACTERISTICS ... 17
3.1.2
TESTING OF QUALITY CHARACTERISTICS ... 18
3.1.3
TESTING PROCESS OF TOP RATED APPLICATIONS ... 20
3.2
LOW RATED APPS DEVELOPER’S FEEDBACK ... 20
3.2.1
IMPORTANCE OF QUALITY CHARACTERISTICS ... 21
3.2.2
TESTING OF QUALITY CHARACTERISTICS ... 22
3.2.3
TESTING PROCESS OF LOW RATED APPLICATIONS ... 23
4
ANALYSIS ... 25
4.1
DATA ANALYSIS ... 25
4.2
TOP RATED APPS ... 25
4.2.1
IMPORTANCE BY TOP RATED DEVELOPERS ... 25
4.2.2
TESTING BY TOP RATED DEVELOPERS ... 27
4.2.3
TESTING PROCESS OF TOP RATED APPS ... 28
6
4.3.1
IMPORTANCE BY LOW RATED DEVELOPERS ... 29
4.3.2
TESTING LEVEL BY LOW RATED DEVELOPERS ... 30
4.3.3
TESTING PROCESS BY LOW RATED DEVELOPERS ... 32
4.4
COMPARISON OF QUALITY CHARACTERISTICS IMPORTANCE, TESTING
AND THEIR TESTING PROCESS ... 33
5
DISCUSSION ... 34
6
CONCLUSION ... 36
6.1
VALIDITY ... 36
6.2
FUTURE WORK... 37
7
REFERENCES ... 38
8
APPENDIX A: QUESTIONNAIRE... 40
9
APPENDIX B: TOP RATED DEVELOPER’S INFORMATION: ... 44
7
L
IST OF
F
IGURES
Figure 1: Comparison of important quality characteristics from top rated developer’s
perspective………..26
Figure 2: Comparison of testing level of quality characteristics from top rated developer’s
perspective. ... 28
Figure 3: Comparison of important quality characteristics from low rated developer’s
perspective……… 30
Figure 4: Comparison of testing level of quality characteristics from low rated developer’s
perspective. ... 32
8
L
IST OF
T
ABLES
Table 1: Research Questions ... 12
Table 2: Quality characteristics importance by top rated apps developers. ... 17
Table 3: Quality characteristics testing by top rated apps developers. ... 18
Table 4: Testing performed by top rated developers. ... 20
Table 5: Quality characteristics importance by low rated apps developers. ... 21
Table 6: Quality characteristics testing by low rated apps developers. ... 22
Table 7: Testing performed by low rated apps developers. ... 23
Table 8: Analysis for the importance of quality characteristics from top Rated apps developer’s
perspective. ... 25
Table 9: Analysis for the testing level of quality characteristics from top rated apps developer’s
perspective. ... 27
Table 10: Analysis for the importance of quality characteristics from low Rated apps developer’s
perspective. ... 29
Table 11: Analysis for the testing level of quality characteristics from low Rated apps developer’s
perspective. ... 31
9
1 I
NTRODUCTION
A mobile application (or mobile app) is a software application that is designed to run on
smartphones, tablet computers and on other mobile devices [1]. Mobile applications are becoming
popular and also very complex. As mobile application users expect the applications to be reliable
and fault free, so it makes the duty for application testers to ensure the quality of mobile
applications before it is released in market [2].
Testing is a technique used to identify the faults in applications and to satisfy their customers by
providing them highly reliable and error free applications [27]. Testing of mobile application is
extremely difficult because there is a lot of diversity in mobile devices, their runtime environment
and the resources needed to test a mobile application differs from mobile phone to tablet [28].
According to Mobile Application Testing Tutorial published by the IEEE Computer Society [7],
mobile application testing is different from the conventional software testing because of the
following several unique requirements.
1. Mobile applications must be able to work correctly anytime and anywhere.
2. As mobile applications are often developed for a set of targeted devices, these
applications must work properly on different platforms having different operating
systems, screen sizes, processing power, network bandwidth and battery life.
3. Mobile applications must also include multiple input channels (e.g. keyboard, voice, and
gestures), multimedia support, and other enhanced usability features in order to provide
the rich experience as expected by the mobile device users.
4. As mobile devices support a range of wireless networks (e.g. 2G, 3G, 4G, Wi-Fi,
WiMAX), mobile applications must also able to work in different network connectivity
environment.
Considering these unique requirements for mobile applications and to be fully aware of important
mobile application quality characteristics, I was interested in using the research method literature
survey and empirical survey. The literature survey was carried out to know about the well-known
quality characteristics tested for mobile applications. The quality characteristics is defined as
what characteristics should an application have for example application should function properly,
application should be easy to use for users, application should be compatible with different
devices etc.
The empirical survey was performed to find out which quality characteristics are important from
developer’s perspective and how often those quality characteristics are tested for “Android”
mobile applications by developers. Apart from that I was also interested in performing empirical
survey on the testing process used by developers for testing of “Android” mobile applications. In
empirical surveys, information is gathered by asking people questions [8]. Afterwards, those
quality characteristics were compared according to respondent’s answers which helped me in
finding out which quality characteristics were important from their point of view and how they
actually use them. Moreover, how the testing is performed was also figured out.
The empirical survey is the method used to collect data from developers
.
An online questionnaire
was made through Google Form and sent to 1000 developers and their response was recorded and
further evaluated to present results.
The apps whose rating was 4 or above was categorized as top
rated apps and the apps whose rating was below 4 came into the category of low rated apps. The
developer’s information is given in Appendix B and Appendix C. I contacted 500 developers of
top rated apps and 500 developers of low rated apps to balance my findings and it was also time
10
consuming to find developer’s information manually. The apps were chosen randomly from
different categories available on play store i.e. top grossing apps, top selling apps, new and
updated games, new and updated apps, apps, recommended for you apps, movies apps, books
apps, tools apps, communication apps, photography apps, offline apps etc.
The developers contact information is given in Appendix which can be reused when someone
wants to survey again on “Android” mobile applications. This information will help them to just
copy the email addresses and sent them their questionnaire as it will save time and effort. It took
me two weeks to get this information from random apps by checking its rating and storing it.
This thesis contributes to find out which quality characteristics are important from developer’s
perspective and how often these quality characteristics are tested for Android mobile applications.
Apart from that it also contributes to know the current testing state of “Android” mobile
applications by addressing how the developers test their “Android” mobile applications.
The
results can be used by developers to know which
quality
characteristics should be given high
priority in testing of
“Android”
mobile applications and how to actually test mobile
applications.
1.1 Aims and Objectives
This research will be organized in two surveys (i.e. a literature survey and an empirical survey).
Main aim of the literature survey is to know about well-known quality characteristics tested for
mobile application development.
Main objective of conducting the empirical survey is to find out which well-known quality
characteristics for which mobile applications are tested for is important from developer’s point of
view and how they actually test them. Moreover, the testing process used by developers to test
mobile applications is also surveyed.
1.2 Related Work
Mobile application testing is an activity which is aimed at evaluating quality of a program and
also for improving it, by identifying defects and problems [2]. In 2013, Android’s Google Play
crossed one million available applications and fifty billion downloads [3]. Therefore due to large
number of app users and their expectations from the app to be error free, it is very important to
meet the quality criteria of an application.
Mobile applications growth is increasing intensively today. According to Gartner Inc. [4], in
2013, 879.8 million mobile devices with “Android” operating system were shipped, and in 2014
the expectation of this growth might increase to over one billion. "Every month, around 20,000
new applications are released, and the current number of apps in the “Android” market is over
1,200,000 [5]". These numbers reflect that this platform is intensively increasing day by day and
that the focus of developers is to develop more and more mobile applications.
Furthermore, apps are easily available through Google Play Store so it has attracted a large
number of developers and companies to develop their apps and put it on Google Play Store [9].
Due to this ease there are millions of apps on this platform but it doesn’t ensure that apps are
error free. The apps containing errors can significantly affect the developer’s image and may
harm the users. So it is the duty of developers to test the apps before releasing to the market to
ensure the quality of apps [2]. The apps with almost no errors are most downloaded and used by
users and trust is built between user and developers [2].
11
Mobile application testing is an activity which is aimed at evaluating quality of a program and
also for improving it, by identifying defects and problems [2]. It is much more difficult than the
conventional software testing because the impact of mobile applications are smaller than the
desktop applications on a virtual machine. Mobile application testing has become a great
challenge that needs immediate action to be tackled and there is no doubt that mobile applications
need specific quality characteristics [6]. The quality characteristics are used by mobile application
developers to ensure the quality of mobile applications [2]. Google Play
framework makes it very
easy for users to search and install the app. Therefore, some developers think it is not necessary to
test the app before releasing as it requires a lot of time, so they wait for the users feedback if they
report any problem they try to fix it and release the new version of app.[9]
In my research methodology course, I contacted 200 developers to perform the empirical survey
on quality characteristics tested for “Android” mobile applications by developers. The developers
were contacted to know about the most quality characteristic that the mobile applications are
tested for. Those 200 developers are the part of these 1000 developers that are contacted. The
sample size was small and few quality characteristics were asked. But the importance, use and
testing process of “Android” apps still needs to be figured out.
There have been many empirical studies conducted on “Android” mobile application testing.
Takala applied model based user interface testing on “Android” applications and he has reported
his experience [17]. Bhattacharya et al. performed an investigation on bug reports and bug fixing
method of “Android” applications [18]. Kropp and Morales examined strengths and weaknesses
of “instrumentation framework and positron framework” for testing mobile graphical user
interface applications [19]. Syer et al. studied the 15 most common “Android” applications and
then compared it with 3 desktop applications [20]. McDonnell et al. studied the constancy and
acceptance of APIs in “Android” network [21]. Maji et al. studies and characterized the failures
in both “Android” and Symbian mobile OSes [22].
Recently there was a research done that used survey methodology, the tools were identified that
are used in testing of “Android” mobile applications by calculating their code coverage, the
presence of test case and the challenges faced by developers in testing of “Android” mobile
applications [9]. Another research was done to understand the challenges faced by
“Android”
mobile application developer’s [23].
The quality characteristics tested for windows phone has been researched recently [9]. Different
testing techniques are being introduced to overcome the challenges faced by developers but still
quality characteristics that “Android” developers think are important and are tested for “Android”
mobile applications is unknown and are being ignored. So, to overcome this gap I was interested
in performing the empirical survey on the quality characteristics importance and how often these
quality characteristics are tested for “Android” mobile applications. Apart from that I was also
interested in performing empirical survey on the testing process used by developers for testing of
“Android” mobile applications. I surveyed “Android” mobile application developers to know
about current testing state of “Android” mobile applications.
The rest of the sections are organized as follows: Section 2 presents the research questions and
the methodologies used. Section 3 presents the results of an empirical survey. Section 4 presents
the analysis of the results. Section 5 presents discussion section in which answers to the research
questions are explained briefly and connected with RQS. Section 6 describes validity threats,
conclusion and the future work needed.
12
2 M
ETHODOLOGY
:
A mixed approach: literature review and empirical survey was used in this research to carry out
the answers of research questions. The literature review was conducted to know about the
well-known quality characteristics for which mobile applications are tested for from the published
work and the empirical survey was used to know the current testing state of “Android” mobile
applications.
2.1 Research questions
The research questions considered in this work are as follows:
Table 1: Research Questions
Research Questions
Description
RQ1: What are the quality characteristics that
mobile applications are tested for?
This research question aims at finding the set
of quality characteristics used to test mobile
applications from the published work.
RQ2: Which are the most and least important
quality characteristics for which mobile
applications is tested for according to
developer’s perspective?
This research question aims to find out the
most important and least important quality
characteristics
for
which
the
mobile
applications is tested for according to
developer’s perspective.
RQ3: Which are the frequently and barely
quality characteristics for which the mobile
applications is actually tested for by
developers?
This research question aims to find out the
frequently and barely quality characteristics
for which the mobile applications is actually
tested for from the developers perspective.
RQ4: Does the importance and testing of
quality characteristics for which the mobile
application is tested for differs between top
rated and low rated apps developer?
This research question aims to find out
whether the importance and testing of quality
characteristics
for
which
the
mobile
applications are tested differs between top
rated and low rated apps developer.
RQ5: How are Android mobile applications
tested?
This research question aims to find out the
current testing process used by developers in
testing of Android mobile applications.
2.2 Literature review
A mixed approach: database search and snowballing approach was used to extract the
papers. Some papers were extracted from IEEE and ACM databases by using the
keywords that were derived from research questions. The snowballing approach is then
used which involves two steps: (1) deriving the start set of the papers and (2) performing
forward and backward snowballing. The research papers extracted from IEEE and ACM
databases were used as a start set to perform snowballing.
13
2.2.1 Extended Design and deriving the Start set of papers
:
According to Wohlen, a good start set should have the following characteristics [29].
Diversity of the papers should be targeted which should cover different publishers,
authors, years. To obtain this, it is recommended to cover this in start set.
Size of start set depends on the size of area being studied for example if you are focusing
on a specific area it requires lesser papers than focusing on a broader area.
If search string is not formulated well too many papers can be found, To mitigate it
papers which are highly cited and have more relevant references can be used by to obtain
perfect start set.
There can be a risk to miss relevant papers by using different terminology. To mitigate it,
synonyms can be used with keywords that are derived from research questions.
The following search string was used to identify the start set was the following:
(Testing types OR practices OR quality characteristics) And (Mobile applications)
I used backward snowballing which is looking at the reference of the papers by applying
inclusion and exclusion criteria. The main advantage of using snowballing is that it just focuses
on the referenced and cited papers [26]. The snowballing approach is as good as database search
and easy to perform as compared to database.
I did the literature review not a systematic literature review due to time constraint. Systematic
review is basically in depth study of the relevant topic that identifies, selects all high quality
research evidence which is relevant to that topic and to do it successfully it requires a lot of time.
Literature review is basically qualitatively reviews on a topic using general methods to collect
data. The literature review was conducted to find different quality characteristics for which the
mobile applications is tested. The literature review covers the answer of RQ1. Following are the
different testing characteristics that the mobile applications are tested for is found from the
published work.
I.
Usability Testing:
Usability testing is used for activities that assess UI content and alerts, user operation
flows and scenarios, media richness, and gesture interaction [7] [9].
II.
Performance Testing:
Performance testing is used for the activities to determine the speed or effectiveness of
computer, network, software program or device [6] [7] [9] [24].
III.
Reliability Testing:
Reliability Testing relates to testing software’s ability to function in given environmental
conditions for a particular amount of time [6] [7] [24].
IV.
Security Testing:
Security testing includes encryption/decryption techniques to ensure the data
communication, checks for multi user support, and checks within app for the access of
files [6] [7] [23] [24].
14
Memory testing is used to check how much memory the application is taking. It is also
used to check for the memory leakage, if the applications are already killed and their
active processes are still taking memory [6] [24].
VI.
Energy Testing:
Energy testing is used to ensure that the applications don’t use more energy than required
one. If more energy is taken by single application then it will reduce the battery life of
device [6] [24].
VII.
Functional Testing:
Functional testing is a quality assurance (QA) process and a type of black box testing that
is used to check each and every function works perfectly or not. Functional testing
usually describes what the system actually does. Functional testing tests the portion of the
functionality of the whole system [6] [7] [9].
VIII.
Interoperability Testing:
Interoperability testing involves testing whether a given software program or technology
is compatible with others and promotes cross-use functionality [7].
IX.
Compatibility Testing:
Compatibility testing is part of software non-functional tests, is testing conducted on the
application to evaluate the application's compatibility with the computing environment
[23] [6] [9].
X.
Connectivity Testing:
Connectivity testing is a testing activity designed to validate the continuity of network
communications [6] [7] [23] [24].
XI.
Load Testing:
Load testing is the process of putting demand on a system or device and measuring its
response to check its efficiency [9].
2.3 Empirical Survey
There are many different ways to perform a survey like street surveys, telephone survey and
electronic surveys. I used the empirical survey (online survey) to gather required data because it
is a suitable strategy to collect data from more number of people (methods, tools, developers and
companies) to gain the understanding of larger population [10]. With an increasing use of internet
users it is easy to perform surveys because you can contact the developers or companies online
(i.e. online surveys) and with the availability of this technology the interest to perform surveys is
increasing day by day this.
Kitchenham and Pfleeger’s [16] show that the interest in doing a survey is increasing in the field
of software Engineering. There is another study which is done within the German ViSEK project
depicts the growing interest of empirical survey.
My survey was based on an online questionnaire with three close ended questions and one open
ended question (optional) to collect the required information. All the three close ended questions
[11] have a fixed number of options that the respondent has to choose while in open ended
question, the respondent is required to write something as answer. The online questionnaire was
15
created on “Google Forms” [12] and the questionnaire link was provided to respondents via
email.
There are two challenges arise in collecting the online survey: (1) keeping track of response
sources and (2) preventing multiple responses from the same person [25]. I filtered responses
received based on submission time [15].Since there can be multiple submissions from one system
due to browser or internet connectivity problem so screening for time of submission (within a
2-minute interval) eliminated such duplicate responses. Ambiguous responses were not received as
our survey was limited to the “Android” developers of mobile applications in play store. Survey
responses have to be saved in the database such as Oracle or MySQL [25]. My survey was done
through “Google Forms” so responses were saved online in Google database.
2.4 Preparation of Questionnaire
Based on the results of the literature review, the questionnaire was developed .The questionnaire
was then discussed with the supervisor and some changes were made according to the research
questions. The questionnaire was then finalized and sent it to the developers for their feedback.
The questionnaire was divided into three parts: The first part was related to the quality
characteristics importance from their perspective. The second part was related to the actual testing
of quality characteristics by “Android” mobile application developers. The third part was related
to what testing process is used by developers in testing of “Android” mobile application. The
questions were close ended and every question has checkboxes provided for the predefined
answer because nowadays mostly people prefer to answer close ended questions [10].There was
also one open ended question but that was optional in case if a developer wants to add any other
information . The questionnaire is provided in the Appendix (A).
There was a sequence in the questions so that respondents get interest in filling the survey. The
questionnaire did not have a lot of questions as it is said that the more questions you ask, the more
time it will take to answer it and respondent will lose their interest in filling the questionnaire
[10].
2.5 Research Design and Data Collection:
I divided the quality characteristic importance into three levels (most important, least important
and not important) and testing of quality characteristic also in three levels (i.e. mostly used, rarely
used and never used). By doing the literature review in section 2.2, I found eleven unique quality
characteristics for which mobile applications are tested for. I included them in the online
questionnaire as close ended questions. In addition, I added an open ended question in order to
get a response about any other quality characteristic that is also uniquely used by mobile
application developers.
I decided to limit my research to only one online mobile application store. So I performed the
survey on Google Play App. store developers, because according to the Smartphone OS Market
Share analysis by IDC [13], “Android” OS has the largest OS market share as compared to other
mobile operating systems. Google Play App. Store website [14] enabled us to easily find contact
information of developers.
The questionnaire was sent on 20
thMay to 400 developers but I didn’t get enough responses. The
developers’ number was then increased from 400 to 1000. Again the email was sent to remaining
600 developer’s (300 top rated and 300 low rated) on June15. Three reminders were given to
them. First one was in the month of May, second was in the month of July and last reminder was
16
Total 43 responses were obtained out of 1000 developers contacted of the “Google Play App.
Store”.20 responses out of 500 were obtained from the top rated apps developers and 23
responses out of 500 were obtained from the low rated apps developers. All the developers
17
3 R
ESULTS
The results are categorized according to the top rated developer’s response and low rated
developer’s response separately. The summary of each quality characteristics importance, testing
and testing process is given in the tables with the help of a pie chart. There is one box on which it
is mentioned any other quality characteristics importance and its testing level. Blue color
indicates the
“Important” and “Mostly used” quality characteristic from developer’s perspective,
Red color indicates the “Less important” and “Rarely used” quality characteristic and yellow
color indicates “Not important” and “Never used” quality characteristics. This color scheme
applies to all the graphs given in the results section. The analysis on this raw results are
performed in section 4.
3.1 Top rated apps developer’s feedback
The results shown below are the feedback I got from the top rated Apps developers in terms of
importance, testing of quality characteristics and the testing process used to test “Android”
mobile applications
3.1.1 Importance of quality characteristics
The summary of each quality characteristics importance according to the top rated developer’s
perspective are given below in Table 2 with the help of pie chart.
Table 2: Quality characteristics importance by top rated apps developers.
1.1 Usability Testing
1.2 Performance Testing
1.3 Reliability Testing
Important 16 80% Important 16 80% Important 12 60%
Less Important 4 20% Less Important 3 15% Less Important 7 35%
Not Important 0 0% Not Important 1 5% Not Important 1 5%
1.4 Security Testing
1.5 Memory Testing
1.6 Energy Testing
Important 14 70% Important 13 65% Important 12 60%
Less Important 4 20% Less Important 4 20% Less Important 3 15%
18
1.7 Functional Testing
1.8
Interoperability Test
1.9 Compatibility Testing
Important 19 95% Important 11 55% Important 12 60%
Less Important 1 5% Less Important 7 35% Less Important 6 30%
Not Important 0 0% Not Important 2 10% Not Important 2 10%
1.10 Connectivity Testing
1.11
Load Testing
Any other quality
characteristic and its
importance?
None
Important 12 60% Important 10 50%
Less Important 8 40% Less Important 6 30%
Not Important 0 0% Not Important 4 20%
3.1.2 Testing of quality characteristics
The Summary of each quality characteristics testing according to the top rated developer’s
perspective are given below in Table 3 with the help of pie chart.
Table 3: Quality characteristics testing by top rated apps developers.
2.1 Usability Testing
2.2 Performance Testing
2.3 Reliability Testing
Mostly Used 14 70% Mostly Used 12 60% Mostly Used 15 75%
Rarely used 6 30% Rarely used 7 35% Rarely used 4 20%
19
2.4 Security Testing
2.5 Memory Testing
2.6 Energy Testing
Mostly Used 12 60% Mostly Used 9 45% Mostly Used 8 40%
Rarely used 5 25% Rarely used 7 35% Rarely used 4 20%
Never Used 3 15% Never Used 4 20% Never Used 8 40%
2.7 Functional Testing
2.8
Interoperability Test
2.9 Compatibility Testing
Mostly Used 17 85% Mostly Used 10 50% Mostly Used 13 65%
Rarely used 3 15% Rarely used 7 35% Rarely used 5 25%
Never Used 0 0% Never Used 3 15% Never Used 2 10%
2.10 Connectivity Testing
2.10
Load Testing
Any other quality
characteristic and its testing
level?
None
Mostly Used 9 45% Mostly Used 9 45%
Rarely used 7 35% Rarely used 5 25%
20
3.1.3 Testing process of top rated applications
Below in Table 4 it’s been summarized how do the top rated apps developer perform testing on
“Android” mobile applications.
Table 4: Testing performed by top rated developers.
3.2 Low rated apps developer’s feedback
The results shown below are the feedback I got from the low rated Apps developers in terms of
importance, testing of quality characteristics and the testing process used to test “Android”
mobile applications.
21
3.2.1 Importance of quality characteristics
The Summary of each quality characteristics importance according to the low rated developer’s
perspective are given below in Table 5 with the help of pie chart.
Table 5: Quality characteristics importance by low rated apps developers.
1.1 Usability Testing
1.2 Performance Testing
1.3 Reliability Testing
Important 21 91.3% Important 20 87% Important 11 47.8%
Less Important 2 8.7% Less Important 3 13% Less Important 12 52.2%
Not Important 0 0% Not Important 0 5% Not Important 0 0 %
1.4 Security Testing
1.5 Memory Testing
1.6 Energy Testing
Important 19 82.6% Important 12 52.2% Important 11 47.8%
Less Important 3 13% Less Important 6 26.1% Less Important 7 30.4%
Not Important 1 4.3% Not Important 5 21.7% Not Important 5 21.7%
1.7 Functional Testing
1.8 Interoperability Test
1.9 Compatibility Testing
Important 22 95.7% Important 13 56.5% Important 14 60.9%
Less Important 1 4.3% Less Important 9 39.1% Less Important 8 34.8%
22
1.10
Connectivity
1.11
Load Testing
Testing
Any other quality
characteristic and its
importance?
None
Important 10 43.5%
Important 12 52.2% Less Important 11 47.8%
Less Important 10 43.5% Not Important 2 8.7%
Not Important 1 4.3%
3.2.2 Testing of quality characteristics
The summary of each quality characteristics testing according to the low rated developer’s
perspective are given below in Table 6 with the help of pie chart.
Table 6: Quality characteristics testing by low rated apps developers.
2.1 Usability Testing
2.2 Performance Testing
2.3 Reliability Testing
Mostly Used 14 70% Mostly Used 12 60% Mostly Used 15 75%
Rarely used 6 30% Rarely used 7 35% Rarely used 4 20%
Never Used 0 0% Never Used 1 5% Never Used 1 5%
2.4 Security Testing
2.5 Memory Testing
2.6 Energy Testing
Mostly Used 12 60% Mostly Used 9 45% Mostly Used 8 40%
Rarely used 5 25% Rarely used 7 35% Rarely used 4 20%
23
2.7 Functional Testing
2.8
Interoperability Test
2.9 Compatibility Testing
Mostly Used 17 85% Mostly Used 10 50% Mostly Used 13 65%
Rarely used 3 15% Rarely used 7 35% Rarely used 5 25%
Never Used 0 0% Never Used 3 15% Never Used 2 10%
2.10 Connectivity Testing
2.10
Load Testing
Any other testing
characteristic and its testing
level?
None
Mostly Used 9 45% Mostly Used 9 45%
Rarely used 7 35% Rarely used 5 25%
Never Used 4 20% Never Used 6 30%
