An Analysis to Identify the Factors thatImpact the Performance of Real-TimeSoftware Systems: A Systematic mapping study and Case Study

(1)

Master of Science in Software Engineering February 2019

An Analysis to Identify the Factors that

Impact the Performance of Real-Time

Software Systems

A Systematic mapping study and Case Study

Sravani Bejawada

(2)

This thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in partial fulﬁlment of the requirements for the degree of Master of Science in Software Engineering. The thesis is equivalent to 20 weeks of full time studies.

The author declares that there is sole author of this thesis and not used any sources other than those listed in the bibliography and identiﬁed as references. And further declare that the author have not submitted this thesis at any other institution to obtain a degree.

Contact Information: Author: Sravani Bejawada E-mail: besr17@student.bth.se University advisor: Dr. Deepika Badampudi E-mail: deepika.badampudi@bth.se Department of Software Engineering

Industry advisor:

Iwe Lingström, Operative System Manager E-mail: iwe.lingstrom@ericsson.com

Ericsson, Sweden

Faculty of Computing Internet : www.bth.se

Blekinge Institute of Technology Phone : +46 455 38 50 00 SE–371 79 Karlskrona, Sweden Fax : +46 455 38 50 57

(3)

Abstract

Background. Many organizations lack the time, resources, or experience to derive a myriad of input factors impacting performance. Instead, developers use the trial and error approach to analyze the performance. The trial and error approach is diﬃcult and time taking process when working with complex systems. Many factors impact the performance of real-time software systems. But the most important fac-tors which impact the performance of the real-time software systems are identiﬁed in this research paper. Black box (performance) testing focuses solely on the outputs generated in response to the factors supplied, while neglecting the internal compo-nents of the software.

Objectives. The objective of this research is to identify the most important fac-tors which impact the performance of real-time software systems. Identifying these factors helps developers for improving the performance of the real-time software sys-tems. The context in which the objective is achieved is an Online charging system, which is one of the software in Business support systems. In context real-time sys-tems, the traﬃc changes in a fraction of seconds so it is important measuring the performance of these systems. Latency is also one of the major factors which impact the performance of any real-time system. Additionally, another motivation for this research is to explore a few other major factors which impact the performance. Methods. Systematic Mapping Study (SMS) and case study was conducted to identify the important factors which impact the performance of real-time software systems. Both the data collection methods, a survey and interviews were designed and executed to collect the qualitative data. Survey and interviews were conducted among 12 experienced experts who have prior knowledge regarding the performance of the system to know the most important factors that impact the performance of the online charging system. The qualitative data collected from case study are cat-egorized by using thematic analysis. From the logs i.e., quantitative data collected from industry was analyzed by using random forest feature importance algorithm to identify the factors which has highest impact on the performance of the online charging system.

Results. Systematic mapping study was conducted in the literature to review the existing literature, 22 factors are identified from 21 articles. 12 new factors are iden-tified from survey which were previously not ideniden-tified in the literature study. From the available quantitative data based on the performance impact of the factors on the system, the factors are identified. Knowing these factors helps the developers to resolve the performance issues by allocating more number of virtual machines thereby the performance of the system can be improved and also the behaviour of the system can be known. All these results are purely based on the experts opinions. Conclusions. This study identifies the most important factors that impact the per-formance of real-time software systems. The identified factors are mostly technical factors such as CPU utilization, Memory, Latency etc., The objectives are addressed by selecting suitable research methods.

(4)

Keywords: Factors, Software Performance, Analyze, Impact, Real-time system.

(5)

Acknowledgments

My journey of the master thesis is indeed an intense learning and an immensely re-markable experience. This thesis would not have been possible without the direction of the most well educated and motivating people depicted below.

First and foremost, I would like to express my warm regards! To my university supervisor Dr. Deepika Badampudi, for her continuous support, motivation, and immense knowledge throughout the learning process of this master thesis. She con-tinuously guided and encouraged me to be proﬁcient and make the best choice to be successful during my tough times. Without her tireless assistance, this thesis would not have been ﬁgured out, and I am extremely lucky and honored to have her as my supervisor.

I would also like to express a warm thanks to my industry Supervisor Iwe Lingström, for teaching me more motivational lessons during this journey and always supported me during my hard times. And my sincere thanks to Ricardo Britto, who gave me valuable support and insightful comments on this thesis. A special thanks to Sven Mikkelsen (Manager) for the gratifying experience at Ericsson. A warm thanks to all the Practitioners for participating in the interviews and also for giving valuable input, which is very important for this research.

Finally, I would like to express immense gratitude to my parents Srihari Bejawada and Suvarna Bejawada, my sister Bhavani Bejawada, brother-in-law Ganesh Marag-oni and friends for their love, help, and support. Without all these people, support, and continuous motivation, this thesis could not be achieved and dedicate my success to them.

Thank you very much, everyone!

Sravani Bejawada.

(6)

(7)

List of Tables

3.1 Data Extraction form for systematic mapping study . . . 20

3.2 List of survey and interview Participants . . . 25

3.3 Initial codes generted in thematic analysis . . . 30

3.4 Represents the themes for the initial codes . . . 31

4.1 Represents the mapping of the factors and their description . . . 40

4.2 Number of times respondents mentioned factors identiﬁed from SMS . 41 4.3 Represents the description of the factors identiﬁed in the survey . . . 43

4.4 Number of times respondents mentioned others factors identiﬁed from survey . . . 44

4.5 Excluded factors . . . 44

(10)

(11)

List of Figures

1.1 Represents Overview of the research . . . 5

3.1 Represents the database search results . . . 19

3.2 Represents the data extracted in data extraction method . . . 20

3.3 Represents the case study design and planning . . . 21

3.4 Represents the case selection and unit of analysis . . . 22

3.5 Represents Survey design . . . 23

3.6 Represents respondents for interview . . . 24

3.7 Represents Interview design . . . 26

3.8 Represents the interview questionnaire . . . 27

3.9 Represents Transcription process . . . 28

3.10 Represents the steps evolved in thematic analysis . . . 29

3.11 Represents the categorizing of initial codes to themes . . . 31

3.12 Represents all the categories . . . 32

3.13 Represents the labeled data . . . 33

3.14 Represents all the categories . . . 33

4.1 Represents the mapping of the studies with identiﬁed articles . . . 37

4.2 Represents the experience of the respondents . . . 45

4.3 Represents the total list of factors identiﬁed in this study . . . 56

4.4 Represents the most impacting metrics identiﬁed from the system . . 57

C.1 Represents Survey and interview results . . . 77

D.1 Represents survey results analysis . . . 83

(12)

(13)

Chapter 1 Introduction

“Nowadays, non-functional testing has a great impact on the real-time environment”

[1].

1.1 Overview

Performance testing is a nonfunctional black box testing. It is a type of software test-ing in which it ensures whether the software performs well under the expected load. Not only the features and functionality, even response time, reliability, resource us-age, and scalability matter. Performance testing goal is not only about finding bugs but also to eliminate the performance bottlenecks [2]. Performance testing is done to improve some common issues like long load time, poor response time, poor scalability, bottlenecks, poor usability, and inconsistency in different operating systems. Some of the common performance bottlenecks are CPU utilization, network utilization, memory utilization, operating systems limitations, and disk usage [2]. Performance testing also determines whether the software under expected workloads meets speed, scalability, and stability requirements. The software which is released with poor performance testing or poor performance metrics may lead to a bad reputation and affects the sales goals by decrementing [2].

The basic Performance testing metrics which are monitored during testing are: Processor usage, Memory use, Disk time, Bandwidth, private bytes, committed mem-ory, memory pages/second, page faults/second, CPU interrupts per second, disk queue length, network bytes total per second, response time, throughput, amount of connection pooling, maximum active sessions, hit ratios, hits per second, rollback segment, database locks, top waits, thread counts, garbage collection. For identi-fying the performance acceptance criteria, performance testing includes goals and constraints for throughput, response time, and resource allocation [2]. Only the client-server-based systems require performance testing. This doesn’t mean that any system which isn’t client-server-based does not require performance testing. For example, the Microsoft calculator can be run by multiple users, but it isn’t a client-server-based system [2].

(14)

4 Chapter 1. Introduction

In many cases, the software projects which lack performance lead to severe prob-lems and may also lead to project failure [3]. Therefore, in the early stages conducting performance testing and performance measurement are essential to know the better performance of the software [4]. While testing a web application, its performance is considered as an essential factor. The performance testing of large industrial client transaction processing software is evaluated using some measures such as resource usage, throughput, stimulus-response time, services (which have been waiting for the maximum number of tasks) and typical resources (such as network bandwidth requirements, CPU cycles, disk space, disk access operations, memory usage, and switch usage) [5].

The performance testing in traditional applications focuses on performance met-rics only on a particular workload with fixed configuration, whereas the cloud test measures the performance metrics related to workloads, which run on multiple vir-tual and real machines. For a better understanding of performance in real-time, web applications must be tested for different types of hardware, multiple operating sys-tems, and multiple browser platforms [6]. Testing performance in cloud computing with multiple operating systems, updates, multiple browser platforms, versions to understand performance in different real-time types of hardware, usually involves a large number of users. [7].

Software performance testing is used in every software vertical, including space programs, financials, digital communication, mail servers, cars, geo-positioning sys-tems, nuclear power plants, farming applications, weather forecast games, etc. When combining a high volume of data for instant processing, with low response latency and extensive logging: the need for efficient performance testing and performance modeling is important. One such vertical which includes high volume, low latency, and logging are usages and financial tracking for digital communication, which can be phrased as part of complex revenue and customer management systems using soft-ware to large Communication Service Providers (CSPs) in all geographies [8]. CSP companies use Business Support Systems (BSS) to run their business operations to-wards the customers [9]. Customers are accessed remotely in a software platform where the provider runs the web application in the cloud, and also responsible for managing and maintaining the software with Service Level Agreements (SLAs) [10]. BSS is a collection of systems and functions which are interconnected to give access to the operator. The operator collects the money from the end-users in cash management systems like bank/ATM (ATM- Automated Teller Machine) in time according to the rules of the agreement. The primary usage of the BSS, which helps to store all the information related to the customer, is to keep track of the money flow primarily usage/pricing/cost from a B2C and B2B but also revenue sharing, taxes, and financial audits [11]. While being used in the CSP, a major utilization of this system is observed in Skype - A world-renowned carrier and telecommunication medium for long-distance calls, messages, or video-conferences, which has made the cost of these services incomparable to that of the traditional CSP. The Traditional CSP is on the verge of exploring different applications in order to accommodate the newly introduced or growing portfolios such as the cloud, machine to machine

(15)

1.1. Overview 5 (M2M), and similar areas to expand the implementation of BSS [12].

Figure 1.1: Represents Overview of the research

Example: A BSS system holds all the customer’s information like business pro-cesses like postpaid/prepaid customers, the aggregated traffic/usage, and pricing. The network checks each subscriber when there is an action like data traffic, instant changes, call, message, etc., whether there is a sufficient amount, this is only in the OCS (Online Charging System) case and OCS makes that response. For offline charging, this is not checked, but depending on allowed/not allowed settings in the network provisioned from a system outside the core network (Example: via multi activation triggered by CRM (Customer Relationship Management) or data cap in the policy system) in the account to charge or applicable credit. The four primary functions of BSS are customer management, order management, product manage-ment, and revenue management [11].

The framework used in BSS is the business process framework, which is also known as the Enhanced Telecom Operations Map(eTOM) framework. This is a ref-erence framework where the operations are classiﬁed, i.e., "strategy, infrastructure

and product","operations", and "enterprise management". Activities with similar

features are grouped into one category based on their definitions [13]. For example, the activities are differentiated as organizational, functional, non-functional, and oth-ers. Within a model, all these categories are explained with the help of process flows. This framework even serves as a blueprint, which will help as a start point by setting directions for the development and integration of BSS [13].

In BSS, there is a more specific focus on the charging system, Charging (CHA) executes business rules to determine charges and updates account balances accord-ingly. To be able to rate and charge different scenarios, CHA executes rating rules, and applies prices defined in the product offering. CHA provides the single rating

(16)

function that handles administrative and recurring rating and charging interfaces but also 3GPP Diameter and RESTful API (3GPP 5GC), interfaces and calculates all charges in the system, regardless of source or type, like online, oﬄine, recurring, usage and late arrivals for retail customers [14].

The main goal of this research is to identify the factors that impact the per-formance of real-time software systems. A SMS is conducted primarily to attain the goal. Several articles related to the performance analysis in real-time software systems are taken, and various factors aﬀecting the performance of the real-time soft-ware systems are gathered. Secondly, a case study has been conducted on an online charging system. As a part of the case study, data collection methods like survey and interviews are conducted to collect the factors from practitioners knowledge in the industry. Performance metrics collected from the system are analyzed. Knowing these impacting factors, the developers can improve the performance by allocating the additional resources.

1.2 Research Problem

Cloud-based BSS, which has recently been released to the market, accounting for software scalability and existing hardware limitations, generally imposed by the cus-tomer/operator, and then it is necessary to understand how a BSS performs (e.g., CPU and RAM usage) and what factors impact its performance. Here BSS is se-lected as an example because this is one of the most prominent software which has more complexity, low latency, and high load requirements. The efficient understand-ing of how the system behaves helps for plannunderstand-ing an early dimensionunderstand-ing of needed resources, and the efficient understanding of the factors which have low significant minor impact helps to know which test cases are to be neglected and thereby reduc-ing the storage of additional data.

The goal of the study is to analyze the performance of the complex system like BSS. A myriad of factors impact performance; it is not an easy task to identify the adequate hardware for different performance requirements. Identification of factors is the path of the performance analysis, and this is one of the activities in perfor-mance analysis. Negatively impacting factors which are reducing the perforperfor-mance of the system will be identified based on their impact on the system. These factors help for the efficient understanding of how the system behaves which is essential in the industry to allocate hardware, to improve the configuration of the system and to comply with the existing constraints.

This case (online charging system) is more important because dealing with the systems which are generating revenue for companies, as there will be a loss if the system doesn’t perform as expected. The systems which are not optimized would be more expensive to deploy. Right now, the organization doesn’t have a thorough understanding of the behavior of the system and how it impacts the hardware di-mensioning, which makes it hard for the operator. Since the load in this kind of real-time system changes every millisecond for each use-case. For example, TMF

(17)

1.3. Research aim and objectives 7 SID products and also daily/weekly/monthly changes in conﬁguration can impact the performance in an unwanted way [15]. Consider, where an operator ‘x’ needs to know how much hardware is required to deal with how many subscribers and to help the operators to allocate the adequate hardware. When the organization needs more hardware resources, the company needs to invest more money. Similarly, when there is less hardware, the system doesn’t perform as good as expected. So, there is a need to understand the performance of real-time systems by measuring in a systematic way to analyze the performance.

1.3 Research aim and objectives

Aim: The overall goal of this thesis is to identify the important factors and to an-alyze the factors based on the impact to know the behavior of an existing or new software component. Therefore, the Online Charging System in the Organization is taken as an example to illustrate, which is one of the applications of BSS.

This is the following primary objective which help in achieving the overall goal of the thesis:

Objective1: To identify the most important factors that impact the performance of the real-time software system.

1.4 Structure of the thesis

Chapter1: Introduction

In this chapter, the author gives a detailed overview of the study by explaining the research problem, along with the research aims and objectives.

Chapter2: Background and Related work

This chapter provides the background work suitable for this study, and the research to this study done previously was discussed in related work.

Chapter3: Research Method

In this chapter, suitable research methods are selected, and the reasons for selecting the research methods are motivated. And also clearly explained the data collection, and data analysis methods.

Chapter4: Results and Analysis

In this chapter, all the results obtained from the various research methods are de-scribed and discussed.

Chapter5: Discussion and Threats to Validity

In this chapter, the objectives are motivated in the discussion section, and various threats mitigated to achieve the results are discussed in threats to the validity section.

(18)

Chapter6: Conclusion and future work

In this chapter, ﬁnally, the research was concluded by providing the work done to achieve the goal of the thesis, and future work was suggested for the candidates who are interested in this study.

(19)

Chapter 2 Background and Related work

2.1 Background

To gain signiﬁcant insights into the context in the current study, the primary step is to understand the online charging system and BSS. Next, to understand the factors that impact the performance of the real-time software. And to study the performance metrics and approaches from the existing literature.

2.1.1 Real-time software systems

A real-time system is defined by different authors, some of which are: “A real-time system is a type of hardware or software that operates with a time constraint” [16]. “A real-time system can be loosely defined as a system whose response time is an important determinant of correct functioning” [17]. For example, I am playing any of the games, and if the game is not sufficiently fast, then the user loses interest to play. Another example is video conferencing systems. Human interaction is included along with the transmission of images, voice, and data. If any delays take place in the transmission of images, voice, and data which degrades the system performance severely. The common key requirement is that the system has to meet deadlines within the time with high performance [16] [17].

According to Rob Williams, “Real-time systems are required to compute and de-liver correct results within a speciﬁed period of time ” [18]. To identify the real-time applications, an outline is categorized for a real-time system. “Speciﬁed limit on sys-tem response, Event-driven scheduling, low-level programming, software tightly cou-pled to special hardware, dedicated specialized function, the computer may be inside a control loop, volatile variables, multi-tasking implementation, run-time scheduling, unpredictable environment, a system intended to run continuously and life-critical applications” [18].

2.1.2 Performance Testing types

In Software Engineering, there are six types of Performance testing types, namely: load testing, stress testing, endurance testing, spike testing, volume testing, and scalability testing [2]. Load testing and stress testing are the most commonly used in performance testing. Load testing helps to determine the system performance and the behavior under normal workload conditions. Load testing is used only to simulate

(20)

10 Chapter 2. Background and Related work

under the actual workload. Load testing helps in identifying the bottlenecks, and when this testing is avoided, it may lead to a commercial loss. Stress testing is also used in identifying the system behavior and helps to know under which extreme conditions the system ails to perform. So, both load testing and stress testing are focused on this research [2], [1].

2.1.3 Performance issues

During performance testing, the most common problems observed are Speed issues, bottlenecking, poor scalability, issues in software configuration, insufficient and hard-ware resources [19]. According to Anderson, a bottleneck is defined as "the process step with the lowest production rate in the overall end to end process" [20]. Some of the major reasons for bottlenecking are lack of resources, lack of backward compati-bility, system failures, inefficient practices used in the work process. Identifying the bottlenecks and resolving them in the early stages are important. Profiling based techniques are the most common techniques used to identify performance problems [21].

2.1.4 The diﬀerent steps involved in creating test cases in

Performance testing

• Software processes that are aﬀecting the overall performance are identiﬁed. • The set of a test case, which is a manageable size, is created by determining the

input parameters, which significantly influence the performance of the system. • Using the existing data, realistic values for the parameters are determined. Using scenario which includes average and heavy workloads such be reflected by the values.

• Reasonable values should be estimated based on experience from the previous version or similar version if there are any parameters that don’t have historical usage or value [22].

2.1.5 Measurement

Measurement is the core of many systems in the software industry. Measurement is the process by which numbers or symbols are assigned to attributes of entities in the real world in such a way as to describe each of them clearly with deﬁned rules. In software engineering, the importance of measurement cannot be presented ade-quately since the software is induced in our existence [23]. Over the decades, with the expanding need for software products in the industry, the process of building the software products has taken the focus by the software researchers and practi-tioners [24]. The software process measurement is used as a crucial component in the progression of expecting high-level capacity and performance and satisﬁes the requirements, which are qualitative [25],[26]. Software measurement is given high priority as a need of components for every process improvement, and their quality results interpretation is given as measurement, an important factor for successful

(21)

2.1. Background 11 software products [27][28]. From the perception of planning, tracking, evaluating, controlling, and understanding, advantages of the software measurement are under-standing, evaluating, and controlling the products, projects, and software processes [29]. In support of decision making, measurement helps the software organizations in developing and improving the performance and quality of the software processes and contributes to accessing the software characteristics predictability [30]. Mea-surements give accurate results by stimulating and calculating the amount of eﬀort implemented for a product [31]. If there is a chance of failure in the process improve-ments, it is said that systematic and reliable measurement procedures are absent [32]. Software measurement is said to be the most systematic way for perceiving and enhancing software development projects by explaining the aﬀected areas in the quality of the system and reducing the faults in it [33].

The author DeMillo has taken a long time to explain the appropriate software measurement and software metrics. A software metric is an aggregate term used to interpret the software measurement activities in a wide range [34]. A metric is defined as “a measure or a combination of measures for quantitatively accessing, con-trolling, or improving a process, a product, a team [35]". A software metric is said to be good when it satisfies these characteristics: simple, easily definable, measurable, attainable at a justified cost for an objective [36]. These characteristics are cate-gorized into three types. They are Process Metrics, Product Metrics, and Project Metrics. The process metrics measure the software development process; product metrics measure the attributes of product in development phases, and project met-rics inspect the status and situation of the project [37]. The metmet-rics are used in large software organizations to gain the perceiving of the systems and to detect the primary design faults [38]. The primary objective of the software metric is to give data that maintain managerial decision making in the software life cycle. It gives the managers of organization support in guiding and reducing the possible risks in the product.

A performance measurement system can be deﬁned as "the set of metrics used to

quantify both the eﬃciency and eﬀectiveness of actions" [39]. A good performance

metric is one that helps to obtain quantitative results, and those performance results can be calculated with diﬀerent time periods for comparing the results. In the in-dustry, metrics are used to improve the performance and to change the behavior of the system [40]. Among diﬀerent metrics that are used for performance testing, the most often used metrics for performance testing are response time, wait time, aver-age load time, peak response time, error rate, concurrent users, requests per second, transactions passed or failed, throughput, CPU utilization, memory utilization, etc., [41][42].

2.1.6 Factors

This research is mainly focused on ﬁnding the most important factors that impact the performance of real-time software systems. In this study, among technical and non-technical factors, the goal of the research is to identify non-technical factors. Diﬀerence between technical and technical factors and motivation for not choosing

(22)

non-12 Chapter 2. Background and Related work

technical factors are discussed below. 2.1.6.1 Technical Factors

Technical factors are the factors selected from tools or methods, etc., that can influ-ence the performance of the software system. These factors can be measured using metrics. Choosing the right tools or methods can give efficient results, and a specific tool is not mentioned in this study.

2.1.6.2 Non-Technical Factors

Non-technical factors are factors that are not technical. Mostly they include man-agerial factors such as communication and coordination between the team members, product and process management, etc.,

• The reason for choosing technical over non-technical factors is because even though the technical factors can be measured, there is no study on the im-portant factors. The technical factors refer to metrics; the tools give many necessary and unnecessary metrics that impact the performance of the soft-ware. So, now it has become hard to identify important factors among those factors.

2.2 Related work

The following section gives an overview of related studies for this thesis.

2.2.1 Earlier proposed performance approaches

Earlier, many approaches are proposed for the performance analysis and testing resource allocation problems of the software. The following approaches which are considered relating to this research study are discussed below.

Software Performance Engineering (SPE) approach [3] directly fulfills the require-ments of the performance in a step by step process and the different tasks that are performed in this approach are: Identifying the qualitative data (factors affecting the performance) of the software, analyzing the performance, Performance Prediction, Performance testing, Maintenance of the software and analysis of the overall system. In the SPE approach, the factors affecting the performance are identified. But in the SPE approach, there is a need for better performance measurement tools and modeling for better results and to improve the performance of the system [3].

Saaty[43] introduced the Analytic Hierarchy Process(AHP) mathematically based, a decision making, and a multi-objective tool. In many applications like an evalua-tion of business performance, allocaevalua-tion of resources, selecevalua-tion of projects, and other areas, AHP approach is adopted. In this process, the major step is, comparisons are judged through highly sophisticated solutions from experiments or judgment [44][45]. The judgments are made by using a nine-point ratio scale when ranges for

(23)

2.2. Related work 13 two factors are equally important. However, one factor would be more important than other factors, so the decision is collected from experts to prioritize the factors. Then calculations are made by using Eigen principal vectors [46]; priorities are made in the final stage of the analysis. From different perspectives for development, AHP is an important tool for the systematic analysis of the practitioner’s opinions [44][45]. To overcome the problems in the performance measurement: a model was devel-oped by researchers named Quantitative Models for Performance Measurement Sys-tem (QMPMS) by using AHP [4]. QMPMS is used to measure the impact of factors on performance. The three most important steps in QMPMS are: Factors influencing the performance and the relationship between them are identified, structuring the factors progressively, and evaluating the impact of the factors on performance [4]. In AHP Rank reversal phenomenon is applied when there is an addition or deletion of the factors in the model [44][45].

Heiko Koziolek[47], focused on proposing new approaches for performance pre-diction and performance measurement of component-based software systems. Fac-tors influencing the component performance of software are the implementation of the component, required services, deployment platform, usage profile, and resource contention. To analyze the running system, all the factors which influence the per-formance are measured by using tools [47]. The need for improving the perper-formance testing and its automation is explained in detail by an experiment [48].

Instead of focusing on the high-level Quality of Services (QoS) like latency and throughput, the author focused on low-level network cloud infrastructure. Introduced an optical multi-layer architecture, the resource status is monitored, and network bandwidth is guaranteed at run time [49]. Worked on dynamic resource optimization for self- organized clouds, and task resource allocation problem is solved under budget instead of performance [50].

2.2.2 Research gap

Sun, Yu et al. [51] reported a QoS-oriented framework called the Resource, Op-timization, Allocation, and Recommendation system (ROAR) used for web appli-cations that automate the testing and the derivation of optimized cloud resource allocation. It even automates end to end orchestration for application deployment in the cloud platform, collection of performance metrics, and performance model tests can be analyzed. This system uses a domain-speciﬁc language called Generic Re-source Optimization for Web application Language (GROWL) [51], which generates a high level and customized resource allocation conﬁguration with optimization to meet the web application QoS goals. But with this ROAR system, the answers for best practices in load testing and resource optimization are missing. Analyzing the behavior of complex systems with model-driven input parameters is missing. To ana-lyze the behavior of complex systems, there are a large number of factors that impact the performance. So, there is a need for systematic optimization of the factors which impact the performance [51].

(24)

14 Chapter 2. Background and Related work

Compared optimal testing resource allocation with Ukimoto et al.’s model [52], which resulted in showing that Ukimoto et al.’s includes all the risks released in software fails with low reliability, as the complex systems require high reliability, which is achieved by architecture-based software model. This model did not deter-mine parameters for testing resource allocation, so; there is a need for calculating the parameters [52]. This performance analysis of the system helps or allow peo-ple to analyze and plan the resources to achieve optimum resources utilization the performance of real-time systems, more specifically, for the complex systems like BSS. From the existing literature observed that there are a large number of papers proposing new frameworks and metrics for performance [53]. Even though factors have a direct impact on the performance of the software, but there is no systematic analysis method with factors to analyze the performance of the real-time software system [5][51]. As the technology is developing, there is a need for better performance measurement to measure all the factors [3]. The main reason is many organizations don’t have time, resources, or experience to derive a myriad of factors impacting performance. Instead, developers use the trial and error approach to analyze the performance. The trial and error approach is a difficult and time taking process while working with complex systems [51]. By illustrating this analysis, performance can be improved by analyzing the behavior of the software system. Systems behavior is very important to allocate the hardware and to improve the system configuration with the existing constraints.

(25)

Chapter 3 Research Method

A well designed and planned research design is key to a research [54]. This section emphasizes the thesis plan and how it is conducted with supportive motivations and justiﬁcation. Research Question and their respective research methods are mentioned initially. The process for conducting a systematic mapping study and case study were clearly presented step by step for the objective.

In the context of software engineering, the widely used empirical methods are Experiment [55], Surveys[56], Action research [57] and Case study [57]. From empir-ical methods, Case study and survey were selected as suitable methods for this study. Action research is used for the purpose of “inﬂuence or changes the certain aspects of the subject of the research”[57]. Both action research and case study are closely related. This methodology is used to improve the situation by observing its eﬀects and take an action [58]. But there is no real-time data access to this system, so this method is no suitable.

A survey is defined as “the collection of standardized information from a spe-cific population, or some sample from one, usually but not necessarily by means of a questionnaire or interview” [57]. In this study survey was selected as an data collection method but not as empirical method. Conducted to know the important factors and performance metrics that impact the performance of the software from the experienced people working in BSS. The survey will be conducted with experts in BSS, like an architect, software developer, system managers, and practitioners. For empirical studies, a well-known strategy is performing a survey to ask people about their opinions. There are different types of surveys, like street surveys, mail surveys, or skype surveys [56]. This method is selected to know the different opinions of the experts working in the industry to capture their opinions.

An Experiment is deﬁned as "measuring the eﬀects of manipulating one variable

on another variable" and that "subjects are assigned to treatments by random" [57].

This method is not adopted because all the variables are not identiﬁed in the earlier stages. There is no full access and control over the system, but there is control on the data which has to be tested [55]. For experiment an ongoing project is more suitable for which the variables for this study should be known at the early stages.

(26)

16 Chapter 3. Research Method

3.1 Research questions

RQ1: What are the most important factors that impact the performance of the real-time software system?

From previous studies, it is identified that there are several factors that impact the performance of the real-time software systems. But these factors differ depending on the type of the system used in the organization. Answering this question helps to identify new factors which are not discussed or identified in previous studies. By finding out the important factors that impact the performance of real-time software systems helps in taking necessary steps to improve the performance. To answer this research question, the question is divided into two sub questions as follows:

RQ1.1: What are the factors that were identiﬁed in the literature? Research Method: Systematic mapping study.

RQ1.2: What factors are perceived to be important by practitioners? Research Method: Case Study

3.2 Systematic Mapping Study

A systematic mapping study is preferred for this research to study the important factors that impact the performance of a real-time software system from the exist-ing literature. The primary purpose of choosexist-ing this method is to gain knowledge about the various performance factors (that have an impact on the software) from the literature. Kai et al. deﬁned SMS as "Systematic mapping studies or scoping

studies are designed to give an overview of a research area through classiﬁcation and counting contributions in relation to the categories of that classiﬁcation" [59], [60].

A Systematic Literature Review (SLR) is avoided in this research because the study does not aim for an in-depth investigation, i.e., an SLR goes into a detailed exami-nation of synthesis results and veriﬁcation of each paper quality for evidence which is not required for this research.

3.2.1 Study Identiﬁcation

Study identification helped me in finding relevant research papers for the RQ1. The following procedure is followed for SMS. The initial step is to formulate the search string based on the keywords identified from the authors knowledge on the study. Using these keywords, database search process was conducted for finding relevant ar-ticles.Inclusion and exclusion criteria was implemented on the relevant papers before selecting the papers for data extraction.

(27)

3.2. Systematic Mapping Study 17 3.2.1.1 Database Search Process

The database selected for this research study is Google scholar as this is one of the largest databases used in the computing ﬁeld. Since all the papers published in IEEE are available in google scholar, IEEE was excluded. Scopus was excluded as there were no relevant papers in the database when the search string was used.

Keywords: The keywords are obtained from the research questions and also which are suitable to the topic.

• Factors.

• Software performance. • Real-time systems. • Impact.

By considering all the above keywords, a set of search strings are generated. • "PERFORMANCE EVALUATION OF SOFTWARE"

• "PERFORMANCE EVALUATION OF COMPONENT-BASED SOFTWARE SYSTEM"

• "IMPACTING FACTORS OF SOFTWARE PERFORMANCE (REAL-TIME SOFTWARE SYSTEMS)"

The results obtained from the database search are analyzed and the papers are excluded based upon the exclusion criteria [61]. From those papers, only the papers which are about the impacting factors of software performance are included, and others are excluded.

3.2.2 Selection Criteria

In order to select relevant papers and studies, this method is implemented as all the papers appeared in the database search are not selected as the suitable papers for this research. Inclusion and exclusion criteria are used to check whether the identified papers are suitable for this study. Defining and following inclusion criteria always guides the author into the right path without any deviations and effort wastage. Exclusion criteria validate the removal of unnecessary and not usable papers for this thesis [62].

3.2.2.1 Inclusion Criteria

The papers which are appeared in the search string and satisfy the following factors are included.

• Research papers providing information about the list of factors that impact the performance of the real-time software or application and their reasons in general.

(28)

• The research papers to which I have full-text access. • The papers are published in the English language.

• The papers which are related to performance analysis or evaluation of real-time software systems.

• The papers are limited to "COMPUTING," as this thesis is mainly concen-trated in this ﬁeld.

• As this study is not new to the software ﬁeld, there is a large collection of papers. So, the research papers are selected from the year "2001 to 2019". • Not only peer-reviewed, but even grey literature is also included in this study,

which gives complete insight for the available evidence [63]. 3.2.2.2 Exclusion Criteria

The papers which are appeared in the search string and did not satisfy the following factors are excluded.

• The research which does not relate to my study or research area is excluded. • The papers which are not published in English are excluded.

• The papers to which I do not have full access online are excluded.

• All the duplicated papers and repeated papers related to my study are excluded. Figure 3.1 presents the database search process results in which each step of the process outcome gives a set of papers. From this papers, irrelevant articles are excluded by looking into the abstract, title and sometimes the full text of the article. Inclusion/ exclusion criteria was applied in search of relevant papers, i.e., for example, some articles may be in the Swedish language instead of English literature. The outcome articles after applying the inclusion/ exclusion criteria are given full-text reading to ﬁnally ﬁnd the relevant articles that are similar to the research study.

(29)

3.2. Systematic Mapping Study 19

Figure 3.1: Represents the database search results

3.2.3 Quality Assessment

An in-depth quality assessment is not required for SMS, as the research goal is to know the factors. So, there is no need of quality assessment for each research paper. The relevant papers are identiﬁed from the Google scholar database by screening the title, abstract, full text, and by following both inclusion and exclusion criteria. All the papers which are not satisfying the inclusion and exclusion criteria are excluded.

3.2.4 Data Extraction

From the primary studies, the data is extracted by considering all the properties mentioned in the data extraction from table 3.2. Based on the research questions, the data extraction form is prepared [64].

Classiﬁcation Property RQ

Overview of the paper Title, Author and

Publication year RQ1

Research Method

Literature Reviews, Interviews, Case study, Survey, Experiment and Framework

(30)

Research Outcomes

Important Factors that impact the software performance

RQ1 Table 3.1: Data Extraction form for systematic mapping study

Figure 3.2: Represents the data extracted in data extraction method

From the formulated search string, 21 papers were selected as the related papers to this study are exported to an excel sheet from those papers by following the above guidelines papers are excluded. The relevant papers are highlighted with green color, and irrelevant papers are highlighted with the red color shown in ﬁgure 3.2.

3.3 Case study

In software engineering there are several empirical methods from those methods case study is adopted as an suitable research method for this study. A case study is deﬁned as," an empirical inquiry that investigates a contemporary phenomenon

within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident" [54]. Because this method is used primarily for

descriptive and exploratory purposes for pre and post-event studies [57]. Qualitative data is collected from surveys and interviews. Quantitative data is the data which is already existing in the industry and the data is available in the form of metrics. The available data is collected by conducting the performance testing on the online charging system is used to conduct the case study. Even though the case study is more ﬂexible, proper planning is important during data collection [55].

(31)

3.3. Case study 21

3.3.1 Case study design and planning

The main objective of this study is to identify the most important factors that impact the performance of the real-time software system. To collect the qualitative data i.e., the factors both survey and interviews are conducted as a part of case study. Quantitative data in the industry which is already available in the industry is collected to analyse. A case study is planned in an step by step process the ﬁrst step is to conduct an survey and interviews to collect the qualitative data from the practitioners. The collected data is analyzed and these results are used to analyze the quantitative data. Later data analysis is conducted on quantitative data by using statistical methods. "For analyzing the statistical data, the analysis includes analysis

of descriptive statistics, correlation analysis, development of predictive models, and hypothesis testing" [55]. To identify the most important factors from the system.

Figure 3.3: Represents the case study design and planning

3.3.1.1 Case selection and Unit of analysis

This Thesis aims at conducting a holistic case study in a well established global leading company which has its branches and roots in area of telecommunications and multimedia. Being located in a remote region Karlskrona, Sweden is one of the company site where this thesis is executed. This company has been involved in producing creative and innovative ideas and contributions for Charging systems in mobile phones and devising solutions in the Networking ﬁeld. In our case, per-formance metrics are collected during perper-formance testing conducted on the online

(32)

charging system; this is the case that would be studying. Online Charging system is an ongoing Research and development project at this site where the major task is to evaluate and make a detailed assessment on the performance of this system. A holistic case study as mentioned above is chosen for this study because the study is conducted in a single organization and in a single application. The goal of the research is to identify the most important factors(performance metrics) that impact the performance of the software so, performance metrics(factors) collected from dif-ferent test cases during performance testing is taken as a case, and as discussed earlier, in this study unit of analysis is conducted on the online charging system in the industry.

Figure 3.4: Represents the case selection and unit of analysis

3.3.2 Data Collection

Data collection consists of two means, like survey and interviews [55]. Survey is send to practitioners through company’s Microsoft outlook account and interviews refers to asking the questionnaire face-to-face to practitioners. Conducting both interviews and surveys gives eﬃcient results but depending upon the sample size. For example: When there is a small sample size conducting both data collection methods, there are no disadvantages. But when there is a large sample size conducting interviews or both interviews and surveys is costly and time taking.

3.3.2.1 Survey design

The steps evolved during the survey are: (a) First, the goal of the thesis is explained, (b) Then the questionnaire is formulated according to the goal and selection of re-spondents, (c) Check whether the survey is executable, (d) Collection and processing the data, (e) Analyzing and (f) The survey results are reported [56].

The survey was conducted in Ericsson, and the participants are selected using sampling. Sampling is a process to select the respondents in the survey. 20 respon-dents are selected, who have knowledge in this domain and working on cloud-based BSS in Ericsson [65]. The quantitative data is collected from the Ericsson experts working with BSS in diﬀerent teams.

(33)

Figure 3.5: Represents Survey design

Survey questionnaire formulation

From background knowledge from the SMS, the questionnaire is prepared in such a way that the answers from the survey objectives are obtained. The questionnaire is prepared in Google docs under the supervisor’s guidance. The survey questionnaire invitation link is sent to all the selected respondents over the company’s dedicated Microsoft Outlook account. The survey questionnaire can be seen in appendix, and the invitation letter sent to the respondents can be seen in appendix. The experts answer a ﬁxed questionnaire, and the results are collected prioritized based on their experience.

Selection of respondents

Cluster sampling was applied for the selection of respondents [65]. The main aim is to select at least one person from each role from BSS. So that all the information from diﬀerent experts will be gathered. The same participants who had participated in the

(34)

survey are chosen for interviews for an interview study. Based on the available time for conducting interviews, the experts who are available for interviews are selected. For the same role, if more than one number of participants are available, they are also considered. Overall, 20 participants were selected and approached, of which, based on their availability, 12 participants participated in surveys and interviews.

Figure 3.6: Represents respondents for interview

Overview of participants

This section presents the results acquired from 12 interviews. In order to obtain relevant qualitative data within the industry, a total of 12 interviews were conducted. A summary of the interview participants is presented in Table 3.4. I ensured that at least one person from each role was selected for an interview from the systems. In addition, the experiences of the participants ranged from one year to five years (beginner) to more than a couple of years (experienced) and more than fifteen years (highly experienced). Such diverse characteristics of the participants in terms of their experiences enabled us to capture data from varied perspectives. In this interview results, the factors are identified based on the practitioner’s opinions and experiences.

Practitioner Working group Experience Role

Practitioner 1 System Management Highly experienced Senior Solution Architect Practitioner 2 Common information

layer Highly experienced Software Architect Practitioner 3 Charging Highly experienced Chief Architect Practitioner 4 Common information

layer Highly experienced

Senior Software Developer Practitioner 5 Dimension Tool Highly experienced Senior Software

Developer Practitioner 6 Charging Experienced Junior Software

(35)

Practitioner 7 Service Delivery Highly experienced Senior Solution Architect

Practitioner 8 Testing team Beginner Tester

Practitioner 9 System Management Highly experienced Solution Architect Practitioner 10 Dimension Tool Beginner Junior Software

Developer Practitioner 11 Operational Excellence Beginner

Operational Excellence Change Leader Practitioner 12 Dimension Tool Experienced Solution Architect

Table 3.2: List of survey and interview Participants

3.3.2.2 Interviews

Interviews are conducted to capture the perceptions of the practitioners to obtain qualitative data. Interviews help to know in detail which factors had aﬀected in ear-lier projects in what way with their experience. The main objective of the interview, the researcher typically focuses on ﬁnding the relation by combining the answers from the practitioners [57].

Interview design and planning

I have adopted a semi-structured interview strategy [58] as the study is exploratory, a semi-structured interview was suitable to gather in-depth information. An invitation was sent to the practitioners over the company’s Microsoft Outlook account. The invitation included the date, time, place of the interview, along with the research goal and the purpose of the study. Based on the practitioner’s suggested timings, the available meeting room in the company were booked. All the interviews are audio recorded with the permission of the participants. The overall design and planning of the interview is depicted, as shown in the ﬁgure 3.6.

(36)

Figure 3.7: Represents Interview design Interview Questionnaire formulation

The interview questionnaire is formulated based on the SMS and survey results. All the factors which are identiﬁed are listed in the Trello board (Appendix E) to discuss and present it to the practitioner. The questionnaire is prepared manually by the author. As the author does not have prior experience in conducting the practitioners, a pilot interview was conducted to gain knowledge and experience. Semi-structured practitioners are conducted, so there is a scope of raising questions that are not formulated before. All the practitioners are audio recorded with the permission of the practitioner.

(37)

Figure 3.8: Represents the interview questionnaire Transcription

During interviews, all the interviews were recorded in an audio format with the practitioner’s permission. All the audio format files are stored and transcribed within a week. For transcribing the audio files from word to text, Express Scribe software was used. Every audio file is imported given name and serial number and later transcribed. The transcribed data is stored in a separate folder. Further analysis was done by exporting to Microsoft word.

(38)

Figure 3.9: Represents Transcription process

3.4 Data Analysis

3.4.1 Data Analysis of qualitative data

To analyze the qualitative data collected from the survey and interviews, a systematic analysis technique selected in this study is the thematic analysis, which is explained below in detail [55].

3.4.1.1 Thematic analysis

A theme is an idea behind an objective which might not be particular or specific in nature. Data Analysis plays a crucial role in this paper as the results are subjected to relevant and credential data, which are responsible for an author to reflect and conclude a few aspects of the objective. One such approach in an ocean or approaches for qualitative analysis is Thematic Analysis. Thematic analysis is one of the widely used analysis methods for qualitative data [57][66]. “Thematic analysis is a method used for identifying, analyzing, and reporting themes within data.” This analysis, has flexibility, gives a rich and detailed description of the data [67][68][69]. Due to the User-oriented and customer-centric importance of the research, other approaches have been avoided, and this method is considered in order to collect opinions and reflect on the performance-related problems using factors found from the literature as a foundation. Flexibility in analyzing the data by identifying, visualizing, and reporting is possible through this method. Utilization of the six-step thematic anal-ysis framework proposed by Braun and Clarke [69] to analyze the interview data is implemented in this study. This section presents a brief outcome and description of each step from the adopted analysis technique as follows.

(39)

3.4. Data Analysis 29

Figure 3.10: Represents the steps evolved in thematic analysis

Familiarizing with the data

Continuous Reading of the data is considered to be the first step in thematic analy-sis, like discussed by Braun and Clarke [69]. Understanding and having total control over the story or data is important for this analysis to result in effective outcomes. One must be able to track a piece of information if given a hint or statement. This helps the author to code the data under appropriate themes or codes used in the the-matic analysis to identify similar ideas and provide meaning to the data. Recordings from the Audio of interviews were transcribed and thoroughly read to reflect on the opinions and objectives of this analysis. Essential ideas and codes in this analysis were marked after transcribing all the data with the scope of further coding of data if any new idea is considered or noticed [70].

The analysis of factors impacting the software performance can be broad as large quantities of data are involved. Initially, the author became familiar with the data collected in transcribed text ﬁles. In an iterative process, going through the text ﬁle a couple of times to make sure that any important information to this study is not skipped. By following this process, the author gains more insight over the knowledge of the information extracted from practitioners.

Generating Initial Codes

Thematic analysis is a collection of unit elements known as "Codes" [69]. These codes are formulated using the raw data or information to mould and assign them to palatable and identifiable ideas or thoughts streamlined into a single code. After a few brainstorming sessions in order to decide if a manual Coding or Computer-Assisted Qualitative Data Analysis Software (CAQDAS) [71] tools like NVivo [72] and MAXQDA [73] is suitable to this paper, it is only fitting to select manual coding of data in order to attain knowledge and observe the data closely which helps to iden-tify facts and statements which are possibly not caught using the other approach. Initially, the data was continuously read, and few areas were marked as primary codes that were defined by an idea or thought to give meaning. Incrementally all the data transcripts were studied with similar objectives, and Codes were assigned, followed with Close verification.

This study aims for important factors. So, initially, codes were generated from the highlighted text in the transcribes. This is an iterative process conducted for all the practitioner transcribes. The transcribed audio ﬁles are exported to Microsoft Word and the codes were generated. The table 3.5 represents the initial codes that

(40)

were generated during this process. A total of 32 codes initially generated in this step.

S.no Codes S.no Codes

IC1 Latency IC17 Deployment platform

IC2 CPU Utilization IC18 Required services

IC3 Memory Utilization IC19 Component implementation IC4 Network Traﬃc IC20 Application complexity IC5 Network Bandwidth IC21 Security

IC6 Throughput IC22 Container threading

IC7 Load IC23 Number of subscribers

IC8 Software Architecture IC24 Disk IC9 Application Component

Cache Size IC25

Incoming traﬃc

(Transaction per second) IC10 Resource Utilization IC26 Number of contracts IC11 JVM heap size IC27 Subscriber behavior (data,

SMS and call usage) IC12 Usage Proﬁle IC28 Characteristics of

subscribers IC13 Number of clients

for each service IC29

Complex oﬀerings (Family accounts and enterprise ac-counts)

IC14 Database IC30

Average number of prod-uct oﬀerings (POs) per sub-scribers

IC15 Response time IC31 Application conﬁguration

IC16 Bottlenecks IC32 Code Quality

Table 3.3: Initial codes generted in thematic analysis

Searching for Themes

A clear picture of what, how, and where the data is being used is addressed in this phase. Potential themes are generated based on the codes formed. Various codes are compared, studied, and relations between the codes are drawn to see if there are similar codes that could be grouped together under one theme [69]. All the codes are listed, sorted, and structured for the comparison in order to group them man-ually using tables. The distinguishing process was purely based on research questions. In this step, the codes generated from the previous step are categorized. Codes were categorized because some of the factors as the same meaning, but diﬀerent words or phrases were used. So, this helps to explain the codes in an eﬃcient manner. The codes in this step are closely grouped. Initially, the codes were generated for removing the factors that are repeated and to understand each factor. The themes that are generated based on the initial codes are Incoming interfaces, Hardware, Product, Network and Software.

(41)

Theme Explanation Incoming

interfaces

All the initial codes which deﬁnes from where the traﬃc data is coming for the system are catego-rized into this theme.

Hardware All the initial codes related to physical parts of the system are categorized into this theme.

Product All the initial codes related to the product are cat-egorized into this theme.

Network All the network related codes categorized into this theme.

Software All the initial codes related to the software are categorized into this theme.

Table 3.4: Represents the themes for the initial codes

All the initial codes that are identified from the transcribes are categorized to themes and factors identified is given as the core category as shown in figure 3.10

(42)

Reviewing and Reﬁning Themes

In this step, the generated and sorted themes are verified and refined if there are any discrepancies found in the data extracts. Braun and Clarke define two broad operations that are performed to review and refine the data. First, the extracted data is Reviewed, and second, the cross-validation with themes is done. This improves the reliability and readability of the furnished analysis [69]. All the codes were revised and rechecked. All the categories are reviewed a couple of times in this process. The author cross-checked whether everything is going in the right direction, and there are no changes made in the results. In this step an new open category i.e., TPS is added as all these factors as impacts the TPS.

Figure 3.12: Represents all the categories

Deﬁning and Naming Themes

The Codes and Themes formulated are labeled, defined, and identified as a single entity where a collection of codes is present [69]. This naming procedure was per-formed under careful observations, and references from the literature and transcripts to attain appropriate labels for easy identification. Each label conveyed a story or meaning, which was the result of strong analysis of the data. All the Names drive a compelling motive towards the results of this paper. In this step, the revised categories are iteratively checked by labeling the data. The data is rechecked with transcripts to make sure that there is no manipulation in the results and also to label the codes in this study. The labeled data is depicted in figure 3.11. All the themes are again categorised under an another open category i.e., Transaction per second(TPS).

(43)

Figure 3.13: Represents the labeled data Producing The Final Themes

As a result of careful analysis and close involvement in the defining process, the themes are produced with defined codes consisting of the extracts of data from con-sidered sources. All the findings are justified and furnished with supportive argu-ments. The final data obtained from this analysis is presented as the findings in this study.

Figure 3.14: Represents all the categories

3.4.2 Triangulation

Triangulation is deﬁned as the usage of various methods in qualitative research to develop a comprehensive understanding of the research study [74]. It is also referred to as a qualitative research strategy to test validity the information gathered from

An Analysis to Identify the Factors thatImpact the Performance of Real-TimeSoftware Systems: A Systematic mapping study and Case Study