A Framework for Effective Test Charter Design for Exploratory Testing

(1)

Thesis no: MSSE-2016-22

A Framework for Effective Test Charter Design for Exploratory

Testing

RATNA PRANATHI GARIGAPATI

Faculty of Computing

Blekinge Institute of Technology SE–371 79 Karlskrona, Sweden

(2)

This thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Master of Science in Software Engineering.The thesis is equivalent to 20 weeks of full time studies.

Contact Information:

Author:

Ratna Pranathi Garigapati E-mail: raga15@student.bth.se

University advisor:

Ahmad Nauman Ghazi

Department of Software Engineering

Faculty of Computing Internet : www.bth.se

Blekinge Institute of Technology Phone : +46 455 38 50 00 SE–371 79 Karlskrona, Sweden Fax : +46 455 38 50 57

(3)

Abstract

Context. Colossal systems that are evolving are primarily system of systems (SOS). The system of systems are characteristic of functionally independent subsystems. These subsystems exhibit heterogeneity in terms of software or hardware. Each subsystem may reflect heterogeneity in dimensions such as the system complexity, system configuration, programming language and platforms, etc. Exploratory testing (ET) is perceived to be the best for testing such systems. An enhancement to exploratory testing is the session-based test management (SBTM) where several activities form a part of each session. These activities are mainly dependent on tester and the test charter of that session. There is lack of information in existing literature regarding a standard framework to design test charters for exploratory testing which forms the main area of focus of this thesis research.

Objectives. Firstly, to investigate the design of test charters in general.

Secondly, to find out the factors influencing the design of test charters.

Lastly, to develop a framework to design effective test charters in SOS context.

Methods. A mixed method approach that incorporates both qualitative and quantitative research methods is used. This research includes the quantitative leg of the online survey along with the interviews and literature review that are qualitative in nature. Literature review has been chosen to investigate the test charter design in general. Besides, interviews and online surveys have been used to research regarding the factors and test charter framework. Snowball sampling method and convenience sampling method have been used to sample the research data. Moreover, thematic analysis method is used for analyzing the qualitative data while descriptive statistics is used for quantitative data analysis.

Results. The design aspects of test charter are documented, the factors influencing test charter design and the framework for effective test charter design for exploratory testing are presented.

Conclusions. The thesis objectives are fulfilled. The findings on how the test charters are generally designed have helped in gaining insight on the primary elements that constitute a test charter design. Further, investigating the factors influencing the test charter design has helped in knowing the main elements affecting the test charter design. Finally, the main contribution of this thesis, the developed flexible test charter framework for exploratory testing encapsulates variables that should be considered, controlled or varied systematically during the course of testing. It is deemed to act as a guideline for practitioners for effective test charter design.

i

(4)

Keywords: System of Systems, Test Charter Design, Exploratory Testing, Session-Based Test Management.

ii

(5)

Acknowledgments

I thank my supervisor, Mr. Ahmad Nauman Ghazi for his undeterred supervision.

I thank him for his immense patience and support extended throughout the thesis duration. This work would not have been possible without his quality guidance and constant encouragement.

Further, I would like to extend my hearty thanks to my parents and sister for their unconditional love and support in all ways.

iii

(6)

List of Tables

4.1 Description of identified codes for general design of test charter . . 31

4.2 Codes, their inclusion criteria and themes identified for general design aspects of charter . . . 31

4.3 Answer to RQ1 through LR . . . 31

4.4 Interviewee Related Information . . . 32

4.5 Identified codes and their description for RQ2 . . . 34

4.6 Codes, their inclusion criteria and themes identified for factors influencing test charter design . . . 35

4.7 Identified codes for core contents of test charter framework and their description . . . 38

4.8 Codes, their inclusion criteria and sub-themes identified for core contents of the test charter framework . . . 41

4.9 Codes and their inclusion criteria identified for contextual questions, as part of test charter framework . . . 41

4.10 Factors influencing the design of test charters . . . 45

vi

(9)

List of Figures

1.1 Mapping research questions and the expected outcomes . . . 5

1.2 Structure of the thesis report . . . 6

2.1 Influence of context over design of test charter . . . 13

3.1 Process of mixed method approach and the research instruments used for the three research questions . . . 18

3.2 Snowballing process . . . 22

4.1 Color coding to trace the factors influencing the design of test charter 33 4.2 Color coding to trace the contents for test charter framework . . . 36

4.3 Frequency of responses for factors influencing test charter design . 42 4.4 Frequency of responses for core contents of test charter framework 43 4.5 Frequency of responses for contextual questions in test charter framework . . . 44

4.6 Modules forming the test charter framework . . . 46

4.7 Test Charter Framework . . . 47

4.8 Test charter formed using proposed framework . . . 48

vii

(10)

Chapter 1 Introduction

1.1 Problem Statement

Exploratory testing (ET) is simultaneous learning, test design and test execution as defined by James Bach [1]. Literature reflects that ET is widely used for testing complex systems as well and is perceived to be flexible in all types of test levels, activities and phases [2]. In the context of quality, ET has amassed a good amount of evidence on overall defect detection effectiveness, cost effectiveness and high performance for detecting critical defects [2] [3] [4] [5]. Exploratory testing has been found to have been addressed in the software testing textbooks since the 1970’s though the term “exploratory testing” was coined only in the year 1983 by Kaner [6].

Session-based test management (SBTM) is an enhancement to ET. It is a tool supported approach which is a way for testers to make orderly reports and organize their work without obstructing the flexibility and serendipity that makes ET more useful. SBTM incorporates planning, structuring, guiding and tracking the testing efforts with good tool support [7]. SBTM reflects the concept of sessions to track testing where a test session is a basic testing work unit and an uninterrupted block of reviewable, chartered test effort [8]. Most practitioners claim the test mission known as charter to be an SBTM element [2] [7] [8].

A test charter is a clear mission for the test session and a high level plan that determines what should be tested, how it should be tested and the deemed problems. A tester interacts with a product to accomplish a test mission or charter and further reports the results [1]. The charter does not pre-specify the detailed test cases which are executed in each session. But, a total set of charters for an entire project generally include everything that is reasonably testable. The metrics gathered during the session are used to track down the testing process more closely and to make instant reports to management [5]. Specific charters demand more effort in their design whilst providing better focus. A test session often begins with a charter which forms the first part of the scannable session sheet or the reviewable result. Normally, a test charter includes the mission

1

(11)

Chapter 1. Introduction 2 statement and the areas to be tested in its design. The test charter design may also include distinct sections. These sections reflect upon the test coverage areas or areas of the product to be tested, description about the task to be done during the test session, the goal of the test session, the approach used to testing the functionality and the test focus or the problem to look for [5] [7] [8] [9].

Colossal systems that are evolving are mainly system of systems (SOS) [10].

A system of system is defined as “set or arrangement of systems that results when independent and useful systems are integrated into a larger system that delivers unique capabilities” [10]. The characteristics of SOS are degree of cen- tralization, operational independence of participating system and independent evolution of participating system. The system of systems comprises of functionally independent subsystems that exhibit heterogeneity in terms of software or hardware. The subsystems may reflect heterogeneity in any of the dimensions such as system complexity, system configuration, programming language and platforms, type of systems or distribution of development systems in different locations [11] [12] [10] [13] [14]. Exploratory testing has been found to be most effective in manual testing of such systems [2] [15]. The activities that form a part of each session in session-based test management are mainly dependent on the tester and the charter of that session [8]. So, the design of charters has great impact on the testing process. Testing is perceived to be very challenging due to synchronization across multiple systems’ lifecycles considering the complexity of the functioning subsystems and the potential for unintended consequences [12].

Also, these subsystems may reflect heterogeneity which makes testing even more complex [10] [14]. It is important to have a clear test session goal, otherwise the results may be minimal and there could be significant loss of time and in turn the results produced by the participants may seem ineffective [6]. Also if the target software is huge and lacks a specific focus area, the efficiency of testing might get deteriorated. On the contrary, if the focus area has been tested well already, it might produce poor results. Such risks involved with testing can be mitigated using effective test charters [6]. The charter is a plan for the test which may be detailed or brief. The test session’s goal, focus area, etc. form the context variables of the test session. These context variables are the key contents or elements of the test charter [1] [5] [7] [8] [6] [16] [15]. The mission of the charter motivates the choices related to the tests, and these choices are based on the givens or the context variables. Hence, the context plays a major role in enabling or constraining the test related choices [7]. The charter plays a great role in guiding the testers through the test process [3] [14]. These context variables may overlap with the factors influencing the design of test charters. The context variables would inherently affect the design or content of test charter, because they help accomplish the mission of the test charter. This brings the significance of test charter design to limelight. Existing literature throws dim light on information about test charters that are developed for carrying out the session-based test

(12)

Chapter 1. Introduction 3 management in SOS context and there is lack of information regarding a standard framework to design test charters for exploratory testing which forms the main area of focus of this thesis research. This forms the research gap of this thesis work. It calls for the need for research related to the general design of test charter i.e. regarding the main elements that constitute a test charter. Thus, this stands to be the motivation to investigate the general design aspects of test charters, the significant factors influencing the test charter design, in order to develop a framework for effective test charter design in the context of system of systems. The framework will be the main research contribution that will form a guideline for practitioners to carry out test sessions in an efficient manner.

1.2 Research Aim and Objectives

There is a need to address the problems stated in the previous sub-section. Hence, the prime aim of this thesis is “To present a framework to design effective test charters in the SOS context”.

The following are the objectives to accomplish the aim:

• To investigate the design of test charters

• To investigate the factors influencing the design of test charters

• To develop a framework to design effective test charters in the context of system of systems

1.3 Research Questions

The following research questions are answered through this thesis work:

RQ1. How is a test charter generally designed?

Motivation: Charters play a key role in carrying out the test sessions. A test session without a charter would be an aimless mission [17]. It helps combat the risks involved in testing and has a great impact on the testing process [6]. Hence, the research intends to find out how the test charters are generally designed i.e.

the key contents that form a part of these test charters [18]. There is barely any academic work regarding the test charters in the SOS context, which pushes to knowing about the charter aspects of the test session in detail to acquire the

(13)

Chapter 1. Introduction 4 ground knowledge. So, knowing about the general design aspects of test charters would further help in producing the framework for effective test charter design in SOS context.

RQ2. What are the factors influencing the design of test charters?

Motivation: Most of the day-to-day systems are system of systems. The factors influencing the design of test charters are context dependent. It is significant to find out the factors influencing the design of test charters, since these factors may impact the context variables and in turn the test charter design. Considering the fact that SOS reflect heterogeneity in terms of hardware or software to some extent [10] [12], knowing these influential factors would help further, in producing a flexible framework for effective test charter design [3] [14].

RQ3. What is the framework to design effective test charters in the context of system of systems?

Motivation: Session-based testing is a technique that helps in managing and controlling tests that are unscripted. It sets a framework around unscripted testing and builds on its strengths such as the speed, flexibility and range. These unscripted tests can be controlled, thus they form a powerful part of the overall test strategy. SBTM echoes the actions of testers who are well experienced in testing [18]. The results of the test session cannot be guaranteed. So, a careful preparation for carrying out the tests would help in combating the risk involved [6]. Considering the fact that the test charter is a key SBTM element that incorporates the mission for testing, developing a framework for effective test charter design would be really helpful by being a great guide to the tester.

Most of the real-time systems are system of systems in nature or contain subsystems [10]. Huge target software that lacks a specific focus area can deteriorate efficiency of testing [6]. There is lack of literature on test charter frameworks and about test charter design in SOS context, wherein SOS has amassed great significance in the recent times [10]. This forms the key motivation to develop a framework for effective test charter design in SOS context.

1.4 Expected Outcomes

The objectives are mapped against the research questions and the expected outcomes are as illustrated in Figure 1.1, along with the corresponding research instrument. The expected outcomes are enlisted below:

• General design of test charter is documented

• Factors influencing the design of test charter is documented

(14)

Chapter 1. Introduction 5

• A framework for effective test charter design in the context of system of systems is documented

Figure 1.1: Mapping research questions and the expected outcomes

(15)

Chapter 1. Introduction 6

1.5 Thesis Structure

Figure 1.2: Structure of the thesis report

This thesis report predominantly comprises of six chapters in the sequence namely Introduction, Related Work, Research Methodology, Results and Analysis, Dis- cussion and Limitations, Conclusion and Future Work. The first chapter throws light on the problem statement, aims and objectives of the research study, research questions and the expected outcomes. Following it, Chapter 2 gives insight into the research topic that includes knowledge of relevant background and related work through implementation of literature review. The next chapter, Chapter 3 namely Research Methodology includes description regarding the research instruments used in carrying out the research process and also presents information on the data analysis methods implemented for this research work. Following it, Chapter 4, Results and Analysis presents the results obtained using the research instruments such as literature review and surveys. Further, the findings for all the three research questions based on the analysis of the obtained results is presented. Next, Chapter 5 namely Discussion and Limitations gives an insight over the significance of the research findings and its contribution to the existing pool of knowledge along with the threats to its validity. Finally, Chapter 6 namely, Conclusion and Future Work presents the gist of the research contribution as part of this thesis and the related potential areas of research that could possibly extend the current research.

(16)

Chapter 2 Related Work

This section is a walkthrough of the existing literature. It kick starts with exploratory testing and flows on to session-based test management and introduction to test charters. This is then followed by how test charters are generally designed i.e. the design aspects of a test charter.

2.1 Exploratory Testing

Manual testing or tester-based testing is deemed to be the best for testing potential bugs in the foreseeable future [17]. Automated testing on the other hand comparatively fails to check for all the varying bugs amidst the changing sce- narios. The recent times have got past the idea of adhoc testing and has taken the path to implementing the best technology for exploratory testing (ET) [17].

Exploratory testing is defined as simultaneous learning, test design and test execution [1]. ET is perceived to be flexible in all types of activities, test levels and phases [2]. Existing literature showcases good amount of evidence regarding the merits of ET such as its defect detection effectiveness, cost effectiveness and high performance for detecting critical defects [2] [3] [4] [5]. The factors on which the effectiveness of ET depends are software maturity, the skills of the tester, the product being tested and the time required to test the product [1].

The power of ET is dependent on the flow of information where the new tests are based on the previous test cases. ET is adaptive in nature and unveils new and unique defects [6]. Exploratory testing is more purposeful and prescriptive in nature that incorporates a combination of both careful preparation and intel- ligent decision making during testing. Testing takes the exploratory form when the rigidness of the scripts is relaxed, either partially or completely. During the exploratory testing process, the testers may interact with the application and take the information it provides to react, change course or explore the application’s functionality without any constraint [17].

7

(17)

Chapter 2. Related Work 8 ET is done in several iterations [6] :

• The tester studies about the current system which is subject to test

• Dependent on his/her knowledge, the tester speculates where the defects or the interesting behavior might occur

• The tester moves on to design a test to find the possible defects

• On executing the test, the tester checks the occurrence of the guessed defect or another, if any

• A reflection on the essence of observation is generated by the tester based on the learning just gained

The session duration generally spans from an hour to three hours. The above iterations are executed several times to assure the learning which is crucial for finding the defects.

ET is applied to any test technique and is used to design new test cases or to execute already executed test runs [6]. ET is appropriate for all development phases. Due to its nature of interactivity and feedback reliant approach, it is perceived to be more suitable for agile development [6]. Exploratory testing is suitable for the following situations [1]:

• Learning about a product which involves exploring the related concepts

• Gathering feedback on a new feature or product as ET makes testing a product easy whilst testing from scratch

• Whilst seeking variation after implementing tests that involve predefined test cases

• During the investigation and isolation of a defect

• While conducting a brief independent check of the work of another tester

• While investigating the status of a risk so as to check if scripted test cases are needed in that area

• While improvising scripted test cases or existing test cases and writing new test scripts

• While interpreting unclear test instructions involved in product analysis and test planning

(18)

Chapter 2. Related Work 9

• During the tests which are based on user manuals and checking the claims in it

• ET pushes the tester to perceive the tested features as a whole and this holistic approach in turn helps in problem recognition

Having a guide to understand the software would help the exploratory testing take a methodical form. There are 2 types of guidance for exploratory testers which helps them through the decision-making process. They are [17]:

• Exploratory testing in the small: It helps in the decision making process during the test execution process where the scope of the decision is small where the testers have to decide on petty things such as which inputs to apply, what pages or screens to visit, which menu items to select, etc.

• Exploratory testing in the large: It helps in designing the overall test plan and strategy.

The current research deals with exploratory testing in the large. A test strategy guides test case design and exploration [17]. Test planning involves the decomposition of software into smaller pieces that can be managed with more ease. This in turn reduces the effort involved in tracking testing progress and test resource assignment while it may also introduce some amount of risk [17].

Exploratory testing benefits from a mix of structure and freedom. The industry leans on testing for check and balance between innovation and dependability. The complexity in software development processes and the fallible nature of human work is susceptible to faultiness. So, testing does have some limitations. They are [17]:

• Aimlessness

• Repetitiveness

• Transiency

• Monotony

• Memorylessness

Aimlessness: Software testers may sometimes run the tests without a specific strategy or goal. The testing approach would be more effective when the following aspects are taken care of.

• Defining what needs to be tested

(19)

• Determining when to test

• Determining how to test

Repetitiveness: There is a need to have a firm clamp over what testing already took place and also about what testing is currently being done or will be carried out that would eventually add to the whole. The following aspects are concerned in this regard:

• Learning about what testing already took place

• Learning when to inject variation

Transiency: Testers can not simulate the actions of the real users in order to find all the possible bugs. It’s a wrong consideration to perceive the application release to be the end of project since the warranty period still forms a part of the testing phase.

Monotony: It is deemed necessary for the test managers to realise the significance of test strategy and tactics and invest remarkable time on the same. So, the tedious and repetitive parts of testing are automated.

Memorylessness: Most of the times, the testers are absorbed in the moment and barely spend time thinking about the future. They usually plan, execute and analyze tests but tend to forget them once they have been used. A ledger for the product would help in giving the future testers an insight over what worked well and what stood less effective.

ET can make use of any tool or technique. Exploratory testing leads testing to tale another stance which is from simple test cases to an extensive class of test strategy and technique. Software testers are often without a strategy or specific goals which remains a key area to be addressed. The test strategy and test charters help to address the problem of aimlessness by directing the testers to test using a purposeful approach and proper mission, respectively. Also exploratory testing opens way to variability in the approach to test case creation.

This helps smack the problems related to repetitiveness and monotony. Fur- ther, the exploratory nature of testing creates a platform to discuss regarding the test technique and test culture to address both transiency and memorylessness.

Though ET is free in nature, it is brought to focus through good control by the practitioners. Along with the domain expert’s support, a proper reporting tem- plate and an exceptional session preparation helps in carrying out the testing in the best manner [6].

(20)

2.2 Session-Based Test Management

Session-based test management (SBTM) is an enhancement to ET that helps in tracking the individual tester’s ET progress. In SBTM, the test results are reported in a consistent and accountable way [6]. Session-based testing is a technique that helps in managing and controlling tests that are unscripted. It sets a framework around unscripted testing and builds on its strengths such as the speed, flexibility and range. These unscripted tests can be controlled. Thus, they form a powerful part of the overall test strategy [18] , which is a set of ideas that guide the choice of test that in turn guide the test design. Also, the test strategy includes a set of ideas related to project environment, product elements, quality criteria and test techniques [7].

SBTM helps the testers in forming orderly reports and to organize the work without obstructing the flexibility and serendipity that makes ET useful. SBTM incorporates planning, structuring, guiding and tracking the testing efforts with a good tool support [19].Thus SBTM gives a framework to the exploratory testing [6]. A test log, a defect log, a session, a mission statement or charter, time-boxed nature, a reviewable result and debriefing are claimed to be SBTM elements [2]

[7] [8]. A defect log is a defect report that portrays a list of the defects found, while an ET test log refers to the record of what was tested or how the testing was carried out [2]. A test session is referred to a basic testing work unit and an uninterrupted block of reviewable, chartered test effort or a protected block of time for focused testing [8] [6]. ‘Uninterrupted’ block refers to the testing that is done without email or telephone call interruptions. ‘Reviewable’ refers to the result of the test session recorded in the form of a report, that provides all the required information to the test management. A session duration is generally long, normal or short. The time-box is the period of time between 45 minutes and 2.25 hours, while a short session is one hour, a long session is 2 hours and a normal session falls between them which is 90 minutes. So, the time boxed nature helps to make a session short enough for accurate reporting or modification in plan such as changing the charter because of a new priority, but long enough to perform good testing amidst an appropriate setup that would make debriefing efficient [7]. A debrief refers to the 5-10 minute conversation between the tester who performs the session and the test lead, wherein they discuss regarding issues that concern the testing and the test outputs [7] [8]. The reviewable result takes the form of a session sheet which follows a formal structure and the charter forms the top part of this sheet [7]. The session sheet is a page of text which is typically ASCII and its formal structure includes the following [7] [8]:

• Charter: includes the name of the charter and description of what will be explored

(21)

• Coverage areas: refers to what is tested

• Start time: refers to the start time of the test session

• Tester name: points to the current tester for the particular session

• Time breakdown: refers to time allotted for each of the tasks specified in the charter

• Data files: refers to any data that is gathered during the test session

• Test notes: refers to the step-by-step record of the exploratory test

• Bugs: refers to the bugs that are found in the exploratory test session

• Issues: points to a short description of the issues dealt and raised in the particular test session

This structure helps to provide an order and integrity for the report. Secondly, it allows a scripting tool to parse the information from the session sheets whereby the information is sent to other applications to report bugs and to gather inquiry oriented metrics.

SBTM reflects upon the concept of sessions to track testing. Each session has a mission which is mentioned and described in the test charter. The mission is nothing but the problems the testers are commissioned to clear up for their clients [7]. The charter tells about what is tested or what problems to look for. During the session, the focus of testing is specific to a feature, characteristic or a business scenario. Session setup, test design and execution, and result investigation and reporting are the three components of sessions [6]. SBTM is seen as a technique to manage and control unscripted tests, which makes unscripted testing to form a part of an overall test strategy. The technique is based on a concept which has effective limits. The test session has a limited duration during which the testers test specific part of the target system. The activities within these limits rely on the judgement of the tester [6]. These limits help to point the tester’s focus to attention, increase the reliability of metrics, repeatability of tests which in turn limits the cost of poor exploration. The activities of the tester, along with the reactions of the system, data used, conditions and ideas are recorded during SBTM [6] [18].

(22)

2.3 Test Charters

2.3.1 Context-Driven Test Planning

The context of the test session plays a great role in determining the design of test plan or the charter. The key steps involved in context driven test planning are discussed below [7]:

• Understanding the project members and the way they affect it

• Understanding and negotiating the givens so as to understand the work constraints and available resources so as to carry out the tests in an effective manner

• Negotiating the test missions concerning the project

• Making choices regarding how to test so as to exploit the givens and to achieve the intended missions

• Monitoring the project status and to accommodate the required adjust- ments to maintain coherence between the givens, choices and the missions The following figure illustrates how context influences the test charter design:

Figure 2.1: Influence of context over design of test charter

Test planning primarily concerns the design of test process elements namely test strategy, test logistics and test products to perform great within the context.

The testers and the test managers don’t have much control over the context of the work but they do have substantial control over the test process elements [7]:

(23)

• Test Strategy: It’s a set of considerations to carry out the tests in a robust manner.

• Test Logistics: Concerns how and when resources are applied to execute the test strategy which may include how people in the project are coordinated, task assignment and so forth.

• Test Products: They are the materials that are resultant of the test pro- cedures. They are visible to the clients of testing which may include test scripts, bug reports or test data to name a few.

2.3.2 Elements to Consider while Designing Test Charters

The following is a list of elements worth considering whilst designing test charters [7]:

• To find significant test related elements quicker

• To perform exhaustive quality assessment

• To standardize product quality through certification

• To minimize testing time

• To maximize testing efficiency

• To assure complete accountability of the test process

• To follow certain methods or instructions with utmost precision

• To satisfy particular stakeholders

2.3.3 Test Charter Evolvement

The sources that inspire the test charter design are listed below [20] [7] [21]:

• Identifying the major test planning challenges: It includes the search for the risks that have a major impact on time, effort or feasibility of designing an effective test plan.

• Clarifying the mission of the test: It helps in forming a clear picture of the mission elements to consider the possible work products.

(24)

• Product analysis: Product analysis involves knowing what to analyze and the ways to analyse. Analyzing the product would help in understanding the product and knowing the underlying technology and ultimately evolves most appropriate test plan for the product.

• Product risk analysis: These include analysis of the threats, vulnerabilities and failure modes of the product.

• Test strategy design: Points to the approach used to test rapidly and effec- tively based on the known product information.

• Logistics plan: It deals with the test strategy implementation during the course of which the required resources are negotiated and exploited.

• Requirements: An ideal start time for charter drafting is during discussion of requirements.

• Implicit Expectation: An implicit expectation like cross cutting quality criteria such as reliability, scalability or performance for instance, which de- serves exploration may be included in the charter.

• Stakeholder Questions: Some of the best queries pop up while examining what the given design decision implies or when mapping a connected set of features.

• Existing artifacts: Existing artifacts allied with the software such as the source code or bug database may yield interesting charter ideas.

The test charter which is an SBTM element plays a remarkable role in guiding the inexperienced testers. The charter is a test plan which is usually generated from a test strategy. The charters include ideas that guide the testers as they test. These ideas are partially documented and are subject to change as the project evolves [7]. SBTM echoes the actions of testers who are well experienced in testing and charters play a key role in guiding the inexperienced testers by providing them with details regarding the aspects and actions involved in the particular test session [8].

2.3.4 Ideal Test Charter Functions

The following form the ideal test charter functions [7] [5] [7] [8] [9]:

• Backing the progress of a quality assessment which aids in making wise decisions regarding the product

(25)

• Describes and provides justification for the test strategy

• Description and justification for special set of requirements and defining the entry and exit criteria that determines when the test project should proceed and when it should stop, respectively

• Supporting the test project activities such as delegation of responsibilities, task planning, etc.

• Project risk identification and management

• Specification of test project deliverables and delivery process

• Recording information that supports process audits and process improvement that concerns future projects

2.3.5 Test Charter Quality Criteria

The following constitute the test charter quality criteria [7]:

• Usefulness: To check if the test plan meets the intended functions

• Accuracy: Checks for accuracy with respect to any factual statements

• Efficiency: Checks if the available resources are used efficiently

• Adaptability: Checks for the immunity to change and unpredictability in the project

• Clarity: Checks for the self-consistency and unambiguity of the test plan

• Usability: Checks if the test plan document is concise, easily maintainable and well organized

• Compliance: Checks if it meets the requirements which are externally de- manded

• Foundation: Checks if it is an output of an effective test planning process

• Feasibility: Checks if it is within the reach of the organization which performs it

(26)

Chapter 2. Related Work 17 Sometimes, the product behavior may vary when it is subjected to complex input sequences that portray the extent to which it can be used or realistically misused. Testing helps gain an insight of the product behavior in different scenar- ios or contexts. It also helps in finding problems that emerge from interactions between different features and the critical flows of user behavior. Eventually, the test design requires knowledge regarding the purpose of the product and its context of usage, both of which inherently determine the mission of the test charter [7]. The analysis of this literature review to answer RQ1 is presented in Table 4.1, chapter 4.

(27)

Chapter 3 Research Methodology

The method used to conduct research to solve the mentioned research questions involves a mixed approach. A literature review (LR) is used to answer RQ1. On the other hand, RQ2 and RQ3 are answered using survey as the research method.

The mixed method approach which is adopted is characteristic of both, the qualitative and the quantitative approaches and merges the best of both to produce a comprehensive set of results [22]. This research includes the quantitative leg of the online survey as well as the interviews and literature review that are qualitative in nature. An overview of the mixed method approach is illustrated in Figure 3.1.

Figure 3.1: Process of mixed method approach and the research instruments used for the three research questions

18

(28)

Chapter 3. Research Methodology 19

3.1 Literature Review

3.1.1 Description and Motivation

“A literature review is an objective, thorough summary, and critical analysis of the relevant available research and non-research literature on the topic being studied”

[23].

Literature review has been chosen to solve research question RQ1. Evidence is described as the synthesis of best quality scientific studies pertaining to a specific topic or research question [24] [23].The literature review is the method of synthesis that helps in gathering qualitative evidence related to the topic. It is a secondary study which reviews the relevant primary studies with the objective to integrate evidence related to the specific research question [25]. It makes use of informal or subjective methods in its process to compile and interpret the literature [22].

Literature review helps the researchers glean the ideas of those interested in the particular research question. Moreover, it pushes them to read the results of studies produced by the latter [26]. Also, it can help inspire budding research innovations whilst enabling greater apprehension of the topic. A good literature review collects information from many sources towards the research topic. It has a clear selection strategy as well as good structuring which enhances the flow and readability of the review [27]. An LR for RQ1 has been chosen to conduct the research since the existing literature is sought to answer it by mustering insight over the general design aspects of the test charters.

A literature review has been opted over a systematic literature review to solve RQ1 because the latter requires considerably more effort in terms of time con- sumption than the former and is repeatable in nature to some degree [25]. Further, tertiary literature review has been deemed inappropriate for RQ1, since it deals with systematic review of systematic reviews and due to lack of such literature related to the current research topic. Moreover, systematic mapping studies have not been considered too, owing to their nature of summarizing information over synthesis of the same [24]. Since, literature review meets the current research requirements by eliminating the need for extra effort through collection of evidences based on pre-defined search strategy [25], it is perceived to be more appropriate.

3.1.2 Data Collection and Sampling

• Data collection

RQ1 is acknowledged using the literature review. Literature review usually helps in gaining an insight over the existing body of knowledge pertaining

(29)

Chapter 3. Research Methodology 20 to the chosen topic of research, which is ‘test charter design’ in the current thesis. The literature review for this thesis work has been conducted using the five-step process stated by J Rowley and F Slack in their article

“Conducting Literature Reviews” [28]:

1. Scanning Documents: It involves collection of reliable literature which is relevant to the chosen research topic. Further, this stage includes formation of keywords to carry out the search. The online database selected for the literature search in the current study is Google Scholar because Google Scholar is not restricted to specific publishers and the search covers Grey literature too. So, it has been chosen as the in- dex database for creating the start set for carrying out Snowballing or Snowball sampling (Appendix A) [29]. Moreover, it was successful in producing more search results that were apt for the topic when compared to others such as Inspec, IEEE Explore, etc. 9 search strings have been formulated to derive the desired literature contributing to the literature review. The exhaustive search process using Snowballing (refer 3.1.2), ultimately helped derive a total of 20 distinct scientific articles. The nine search strings used by Snowballing are: Exploratory testing; Efficiency of exploratory testing; Session based test management and test charter design; Test charter design in exploratory testing; Session based test management; Efficient test charter designs;

General test charter designs; Exploratory testing and general test charter designs; Test charters and software testing (Appendix A).

2. Making Notes: In this step, the literature gathered is thoroughly read and the significant points from it are jotted down. These key points form potential information which can answer the research question.

Here, the points related to test charter design in general i.e. the basic considerations and elements affecting the design of test charters are noted.

3. Structuring the Literature Review: It helps in aligning the key points along the answer line to the corresponding research question, which is being answered using literature review. With respect to the first research question, this step coordinates the various derived concepts related to charters such as exploratory testing process, session based test management, test charters and eventually their design, thereby reflecting the essence of literature review in brief.

4. Writing the Literature Review: This forms the central step towards penning the gathered and structured data into answering the research question.

5. Building the Bibliography: It involves citing and crediting the existing literature works that have been used for conducting the LR.

(30)

Chapter 3. Research Methodology 21

• Sampling Method

The method of choosing a set of respondents from a population which is under study is termed as sampling [30]. The sampling strategy used to carry out the literature review for RQ1 is Snowball Sampling. The reason for the choice of strategy is that Snowballing scales down the density of articles in its approach [31]. Also, Snowballing helps capitalize on expert wisdom and emerges to identify studies that are greatly appreciated by various stakeholders. Moreover, it assists in identifying studies outside the academic mainstream [32]. Owing to less work from academia and its apparent benefits, Snowballing has been the best fit for sampling data for the literature review.

The database Google Scholar has been pivotal in the literature search and the most relevant articles resultant from the search have been summarized to answer the research question, RQ1. Snowballing is a good alternative to database searches [33] [34]. Using Snowballing, details regarding the information rich literary sources through information explored from key informants is sought. The string of informants may initially diverge but later converge to a few sources as they get repeated over the course of search [32].

The search procedure involves browsing the source’s citation indices called forward snowball search (FSB) or searching its bibliography called backward snowball search (BSB), for the topic relevant to this research [32].The citations were retrieved using Google Scholar. The start set or initial set of 10 informants was created using 9 search strings comprising of keywords like exploratory testing, efficiency, session-based test management, test charter design and software testing. Further, using first iteration of forward and backward snowballing, 9 distinct scientific articles were collected, followed by 2 new articles in the next iteration and no new ones in the final, third iteration. Summarizing it all, a total of 21 articles were gathered for the literature review, the start set of 10 sources, 7 during the backward snowballing process and 4 from the forward process (refer Appendix A). The papers in the start set do not belong to the recent years alone. This en- sured a good chance of finding papers in BSB and FSB [29].

To complement the data set obtained from snowballing, the recommended step to derive more data sources is by contacting authors of included papers [34], which has been implemented. In this regard, the most recurring author J.Bach who is renowned in the field of exploratory testing was con- tacted. He recommended one new paper, raising the count to 22 articles at the end of Snowballing it too. Based on the title coherence and abstract relevance to the research topic, the articles were picked for inclusion, during FSB and BSB. Ultimately, all the 22 articles that were chosen passed the final inclusion criteria for literature review, once a complete reading of

(31)

Chapter 3. Research Methodology 22 the individual articles was performed. Other basic inclusion criteria for the papers have been the following [34]:

– The articles must be in English

– Articles relevant to topics namely exploratory testing, session-based test management and test charter design

– Articles with full text available online

Figure 3.2: Snowballing process

3.2 Survey

3.2.1 Description and Motivation

Survey has been chosen as the research method to solve research questions RQ2 and RQ3. Survey allows the collection of a large amount of data from sizable

(32)

Chapter 3. Research Methodology 23 population in an economical way [35]. Both, qualitative and quantitative data is collected using surveys [30]. Also, it allows the researchers to apprehend the state of practice, identify the potential for improvement and inspect the approval of a technology [30]. According to R.M. Grove [36], survey would be the most appropriate method to obtain information from professionals in various organizations.

Survey describes the characteristics of a large population. The data collected in a case study reflects the characteristics of only a single organization or a single case. This has been the reason for not opting case study as the research method.

On the other hand, an experiment involves comparison of results corresponding to two or more tools or techniques. Action research is mainly conducted to know the effect of an action. So, it is an inappropriate research method for this thesis.

Since, the current research involves investigation regarding the factors influencing test charter design and the potential constituents of the framework for test charter design in SOS context, all of the above mentioned stands to be the motivation for selecting survey from the four major empirical research methods discussed above, to carry out the research for RQ2 and RQ3.

So, survey has been opted to answer the successive research questions RQ2 and RQ3. These face-to-face interviews incorporate capturing updated information through direct interaction with the practitioners for finding the factors influencing the test charter design and thereby, produce a framework for test charter design. Finally, the information gathered through interviews is validated by a large number of respondents using online questionnaire as part of surveys.

3.2.2 Data Collection and Sampling

3.2.2.1 Data Collection

Questionnaires and interviews are the most common means of data collection in surveys [37]. The data collection methods, both face-to-face interviews and online questionnaires are used in the current research for triangulation purposes. It involves collection of different types of evidence in order to confirm the rationale that is generated [38]. Data triangulation which belongs to the between-methods triangulation category, has been adopted. This approach involves using more than one source of data to approve the formulated theory [39]. Interviews were conducted first and then the online surveys, so as to validate the interview data and to understand their coherence with the views of practitioners in the field of testing from various other organizations. During the process of literature review, it has been observed that, very few scientific articles emphasized the term

‘charter’. In this regard, interviews were carried out with the objective to amass maximum information through direct contact with experts in the field, consider-

(33)

Chapter 3. Research Methodology 24 ing the effort needed to elucidate the research topic and to take advantage of the flexible nature of semi-structured interviews [40] [41].

• Interviews

Interview Questionnaire Formation

The following three steps were followed while forming the interview questionnaire:

1. Devise questionnaire: The interview questions were formulated based on the knowledge amassed from the literature review for RQ1. It was important to devise the questions in an understandable manner so as to extract effective answers from the interviewees [42].The research questions RQ2 and RQ3 involve finding all possible factors influencing test charter design and developing a flexible framework which acts as a checklist for effective test charter design, from the viewpoint and experience of the interviewee. With regard to that, open ended questions are deemed to be the best fit for the main research questions owing to their tendency to muster as much information as possible without constricting to a set of answers like in the case of close ended questions [41]. For instance, close ended questions like those formulated to find out if the interviewee thinks test charters were effective or not, or if a framework for test charter is beneficial or otherwise, were included.

2. Review and refine questionnaire: This is an essential step following the formulation of questions. In this step, the devised questions are reviewed by the thesis supervisor which is highly beneficial considering their former experience at conducting similar research interviews and is deemed to provide a means to refine the questionnaire availing its betterment. In this aspect, suggestions were given upon including extra demographic questions like the size of the organization that the interviewee works with and also basic questions related to the research topic, which helped in strengthening the understanding over the topic. As a resultant of the review process, the final questionnaire that was formed was used to conduct the semi-structured interview,that comprised both open ended and close ended questions [40].

The semi-structured interviews are one among the three types of interviews namely structured, unstructured and semi-structured interviews [43].

3. Update questionnaire: The flexible nature of the semi-structured interview allowed it to follow an unscripted fashion whenever there was a need to extract more data or to clarify upon the interviewee’s answer [40] [41]. This ensued and greatly helped in all the 9 interviews

(34)

Chapter 3. Research Methodology 25 that were conducted for this research. Based on the previous interview data, the existing questionnaire was updated to encapsulate that question that complemented a response.

Conducting the Interview

The interviews were conducted in three stages which are as follows [42]:

1. Introduction to research and researcher: This forms the first stage where the researcher who is the interviewer gives a brief introduction to the interviewee about himself/herself like his/her name and current profession, followed by a brief description on the thesis research objectives. This helps in setting up a more familiar atmosphere ensuring the full participation of the interviewees in the interviews [42] [38].

2. Collection of general information: In this stage, the information related to the interviewee is collected . For instance, his/her name, roles and responsibility in field of software testing and years of experience in testing.

3. Collection of research related information: This is the last stage that covers major part of the interview which involves extracting maximum information from the interviewee, aiming to amass answers for the research questions.

Interview data transcription

The audio of interviews have been recorded on the researcher’s phone and using MP3 Skype recorder software in order to transcribe the interview data into text, based on face-to-face interview or Skype video call, respectively.

The data transcription of interviews was performed on the same day after completing the interview and manually without the aid of any tool. The transcribed data forms the data chunk to assist smooth flow of data analysis and also in case of any need to clarify any research related queries with the interview subjects. The audio files and the Microsoft Word documents containing the corresponding transcription were saved in the subfolders with the particular interviewee’s name. The main folder containing these subfolders formed the whole interview data, was saved in Google drive to ensure the data was intact and was also shared with the supervisor on Dropbox as part of thesis work update.

• Online Survey

The choice of electronic or online survey ensures large response rate from practitioners who are globally distributed. Moreover, it is a common ap-

(35)

Chapter 3. Research Methodology 26 proach to data collection that takes lesser time as compared to interviews and ensures a large number of respondents [44].

Design Process

Google Forms has been opted to carry out the online survey process [Ap- pendix C].It has been opted over other online tools such as SurveyMonkey, Zoomerang or SurveyGizmo for instance, owing to its advantage of being free of cost, user-friendly and ability to generate descriptive statistics im- plicitly for the research data, that further simplifies the data analysis process [45]. Thus, it has been used to fulfill the objective of the survey. The online survey process has been carried out in the following manner [30] [46]:

1. The questionnaire form for online survey is prepared using the data gathered from the interviews. The interview data is analyzed and sorted out to generate the online questionnaire form. The codes from analyzed interview data (in Tables 4.6, 4.8 and 4.9) have been rephrased (phrase expansion including the various forms/terms of the identified codes, to enable quick understanding for online survey respondents, only for survey purpose) for some of the identified codes before being forwarded to form the online questionnaire for the main research questions RQ2 and RQ3.

2. Almost the entire questionnaire form was designed with close ended questions in order to validate the potential research oriented data pro- vided by the interviewees, except for a few demographic questions which are meant to be open ended questions like the name of respondent and his/her work experience period in the field of software testing.

The online survey form for research that was created using Google Forms has two sections. Starting with a brief description of the research topic and the aim of thesis to familiarize the survey participants with the expectation of survey, the first section gathers demographic information while the second gathers information on testing and charters, mainly focusing on the influential factors and framework contents of the test charters. Many elements were considered whilst designing the form to make it effective. They are as follows:

– The size of the questionnaire form was kept optimal comprising only the crucial questions to make the survey time savvy and seek more respondents. In this regard, it has been mentioned at the beginning of survey form that it would take less than 10 minutes to fill it, in order to motivate completion of the form.

– The total of 14 questions comprised only 8 research oriented questions that formed the second section and the rest were demographic questions.

(36)

Chapter 3. Research Methodology 27 – Only the most important questions that complement the research work were marked as ‘required’ and the others were made optional.

All the questions in the second section were made compulsory while most of the demographic questions were kept optional to respect the respondent’s choice to withhold their identity from the researcher.

3. The link to the designed form is shared with the research supervisor to be reviewed before spreading it out to the respondents.

4. Thereby, the online survey form was spread across globally to experts, friends and their mutual contacts who are familiar with software testing, and was made available over a span of 45 days from May 15- June 30, 2016.

3.2.2.2 Sampling Method

Convenience sampling method has been chosen to select the set of respondents for collecting data using interviews and online survey to solve RQ2 and RQ3. It is one of the non-probabilistic sampling techniques which involves choosing the nearest and the most convenient practitioners to form the set of respondents [40].

It involves a non-systematic sampling approach wherein the entire list of target population is unavailable (’target population’ refers to the set of population to whom the survey is applicable- here, the whole body of software practitioners in the testing field, and ’sample’ refers to the set of individuals who form potential survey respondents- here, the interviewees and online survey respondents), unlike probabilistic sampling where a systematic approach is adopted to sample the population and when the list of entire population is available [30] [47] [48]. Referral sampling has not been used instead of convenience sampling because the former involves a chain process of gathering informants/references from the acquired references (who may not be most conveniently approachable) while the latter has the benefit of approaching instantly available and the nearest informants [40].

The convenience sampling method has been implemented for interviews by contacting the interview subjects who are experts in the domain of software testing with the kind assistance of my thesis supervisor, personal contacts as well as mutual contacts. The subjects have been chosen based on their experience with session-based test management and adequate knowledge regarding test charters.

Selection of interviewees who are most familiar with the research topic assures reliability of the collected interview data [49]. So, convenience sampling lead to the identification of a set of 9 participants who were competent in terms of knowledge and expertise. They helped in providing valuable inputs towards the research work through interviews that spanned between 20-80 minutes. 2 of the interviewees were chief executive officers of their own software testing consultancy

(37)

Chapter 3. Research Methodology 28 companies, one of them being the very person who coined the term ’session- based test management’ [8]. The remaining 7 interview subjects were industry practitioners holding roles such as test manager (2), test architect (2), senior systems test engineer (1),test strategist (1) and test specialist(1) from companies with high reputation. One-third of the interviews were face-to-face interviews and the rest were conducted through Skype video calls owing to the uneconomical geographical difference between the interviewer and interviewee.

Convenience sampling method for the online surveys has helped in fetching respondents with some amount of familiarity in software testing, currently holding various professional roles such as software engineers, software developers, software testers, programmer analysts and so forth.The interviewees helped in collecting responses for the online survey by sharing the survey link through e-mail with mutual contacts in the testing field. This helped in keeping track of the number of persons to whom the survey was accessible. So, approximately 50 persons had access to the form while it resulted in 40 responses on the whole. These were 40 complete responses. The interviewees were not included in this sample set.

3.3 Data Analysis

The data analysis method aims to synthesize the collected data to draw conclusions from it [24].

3.3.1 Quantitative Data

The quantitative data gathered from the online surveys will be analyzed using Descriptive Statistics. Descriptive Statistics makes use of mean values, histograms and scatter plots to reflect trends in the collected data [50]. The results from the online surveys take the form of histograms representing the frequency of factors and framework contents chosen by the survey respondents. The data points representing a minimal count of respondents (i.e. atleast one respondent) is considered and the corresponding element (i.e. influential factor or framework constituent) is included in the final results.

3.3.2 Qualitative Data

The qualitative data gathered from the literature review as well as semi-structured interviews will be analyzed using the Thematic Analysis method. This method is used to analyze, identify and report patterns or themes by synthesizing the

(38)

Chapter 3. Research Methodology 29 collected data [51]. It organizes and describes the data set in high detail in a minimal way and frequently interprets various aspects of the research question.

It can be used within different theoretical frameworks, as an essentialist or realist method that reports experience, meanings and the reality of participants [51].

Coding refers to "the analytic processes through which data are fractured, con- ceptualized and integrated to form theory" [52]. Each of the identified concepts in literature review and the interviews is converted into codes in order to repre- sent the concepts pertaining to the charter design aspects, influential factors and framework contents. The current research implements the approach to Thematic Analysis proposed by Boyatzis, where various themes are generated from coding the raw data wherein the codes are subjected to constant comparison using scoring, scaling and clustering [53]. Ultimately, the observed themes will help in outlining an effective design for test charter framework.

3.3.3 Motivation

Coding is a method which enables to organize and group similar data into categories based on common characteristics that marks the beginning of themes [4].

Dividing the literature into themes or categories is the most popular approach for framing the research review and has the potential to provide well-organized way of describing potentially diverse evidence bases in software engineering. It is deemed to allow integration of the theoretical and empirical literature [51].

Grounded Theory has not been opted over Thematic Analysis as the data analysis method for qualitative data, since the line by line coding technique involved in its approach is unlikely to be practical for large number of studies. In contrast, thematic analysis does not specify any such coding technique [51].On the other hand, narrative synthesis deals with producing a summary in a story-like fashion and hypothesis testing involves comparing the results against a hypothesis.

So, they were deemed inappropriate over thematic analysis for the current study.

Likewise, comparative analysis was not used as it involves comparing the findings and validating the findings between them [54].

(39)

Chapter 4 Results and Analysis

4.1 Data Analysis of Literature Review

It was perceived from the literature review that the charter design generally includes what the tester will work with and how he/she will get the work done.

This prompts the tester to decide upon the test story i.e. what the tester knows about the testing so far, to be manifested in the particular test session. Fur- ther, knowing the client’s needs will help apprehend the problems that need to be solved which ultimately help in determining the target or mission for the test charter. The following Table 4.1 tabulates the codes identified from the literature review to answer RQ1, "How is a test charter generally designed?".

The identified codes correspond to the key elements that form the design of test charter in general. The codes and their description is included herewith.

Codes Identified Description

Mission statement A one liner describing the mission or target of the test charter

Test priorities Tells about the order in which testing task is carried out Focus Tells about which problem to look for

Risks Areas potential to bugs

Test logistics It is about how and when resources are used to execute test strategy. For instance- how people on projects are coordinated and assigned to what tasks

Test Products They are the resultants of the tests. For instance- Anno- tation of specifications, product issue list

What to test It is the testing task i.e. it points to the specific coverage area/function of the product to be tested. For instance- test coverage areas such as test configuration, test strate- gies or system configuration parameters

Why to test Motivates the test goal

How to test Approach to testing/ test strategy i.e. how the product is covered and the problems are detected

30

(40)

Chapter 4. Results and Analysis 31

Table 4.1: Description of identified codes for general design of test charter Two themes- ‘core contents’ and ‘contextual questions’ have been associated with the identified codes as shown in Table 4.2. ‘Core contents’ refer to the contents that form the core elements of the test charters. They are prominent in almost all charters and describe the primary aspects of the tests to be carried out.

While, ‘contextual questions’ point to codes resembling queries, which address the rationale of the constituent test charter elements by supplementing the identified core contents. The code inclusion criteria has been the frequency of occurrence of the code in the literature.

Codes Identified Code Inclusion Criteria Theme

Mission statement Frequency Core content

Testing task Frequency Core content

Test priorities Frequency Core content

Focus Frequency Core content

Risks Frequency Core content

Test logistics Frequency Core content

Test Products Frequency Core content

What to test Frequency Contextual question

Why to test Frequency Contextual question

How to test Frequency Contextual question

Table 4.2: Codes, their inclusion criteria and themes identified for general design aspects of charter

General Design Aspects of Test Charters Core contents

Mission statement Test priorities

Focus Risks Test logistics Test Products Contextual questions

What to test Why to test

How to test

Table 4.3: Answer to RQ1 through LR

(41)

Chapter 4. Results and Analysis 32

4.2 Results and Data Analysis of Interviews

The interview data was a contribution from 9 interviewees, as below (Table 4.4):

Interviewee Current Designation Testing Experience (in years)

Organization Size (in persons)

INT 1 Senior Systems

Test Engineer 4 More than 500

INT 2 Test Quality Ar-

chitect 10 50-500

INT 3 Test Specialist 10 50-500

INT 4 Test Consultant 12 More than 500

INT 5 Test Strategist 3 Less than 50

INT 6 CEO, Test Con-

sultant 30 More than 500

INT 7 Test Manager 20 More than 500

INT 8 CEO, Test Lead 4 50-500

INT 9 Test Quality

Manager 13 50-500

Table 4.4: Interviewee Related Information

The recorded interview data has been transcribed manually. Further, the transcribed data was color coded i.e. associating a distinct color to the set of data addressing particular interview elements, during thematic analysis to enhance readability and understandability of collected data [55]. The different colors used for color coding are as below:

A Framework for Effective Test Charter Design for Exploratory Testing