Methodology

For the papers presented in this thesis, empirical research methods have been used. Empirical research is seeking to explore, describe and explain different phenomena through collecting and using evidence based on observations or experiences. The evidence is obtained through for example interviews, surveys or experimentation [37]. According to Seaman [35] most empirical software engineering studies combine qualitative and quantitative methods and data.

Research can according to Robson [32] have two main types of research design, fixed design or flexible design. Fixed designs, also called quantitative designs relying on quantitative data are either descriptive or experimental, and are highly pre-specified and prepared. Fixed design are often concerned with comparing two or more groups, and a theory is required in order to define what to search for [32]. Flexible designs, also called qualitative designs are concerned with studying objects in their natural setting and describe issues of the real world. The intention with the design is that it should progress based on the more knowledge the researcher gain during the study. Flexible designs are less pre-specified and rely more on qualitative data. In a qualitative study [35] where, for example, interviews are used it is important that the interview is flexible enough to allow for unforeseen information to be recorded and all information shall be regarded useful, even if the usefulness of a specific data is not known until long after it was collected.

Research methodology

A design cannot be both fixed and flexible at the same time but a design could have flexible phases followed by fixed phases; the other way around is very rare. Flexible designs can include the collection of quantitative data however, even if fixed designs can include qualitative data it rarely does [32].

The research in this thesis uses two types of research methods; surveys and experiments.

The purpose of surveys is defined by Wohlin et al [45] and the purpose is “to understand, describe, explain or explore the population”. It is difficult to give a concise definition of survey research but a survey has often three typical central features according to Robson [32]:

1. Fixed, quantitative design is used.

2. From a relatively large number of subjects a small amount of data in a standardised form is collected.

3. A representative sample of individuals from known populations is selected.

These three central features capture a large part of surveys but there are surveys where considerable amounts of data are collected from each individual but the individual do not represent themselves but rather a company or organisation.

However even if surveys often are referred to as a fixed design, Robson [32] also argue for that surveys can be based on either flexible or fixed design depending on the degree of pre-specification. In typical fixed design the data collection is made by questionnaires with closed questions and in typical flexible design the data collection is made through interviews with open-end questions. Both fixed and flexible survey designs are used in this thesis.

The research in the second part of this thesis, the part about the state of practice in the medical device domain is based on the use of surveys in two different ways. Paper I is based on fixed design and carried out through a web-based questionnaire and

Research methodology

Paper II containing in-depth interviews is based on flexible design with open-ended (semi-structured) interview questions.

The fixed design with the use of a web-based questionnaire was chosen because it is an easy way to retrieve information from a large set of people in different countries. It allows anonymity and can provide large amount of information to a low cost in a short period of time. The design was typically fixed with the closed questions, where it is possible to know that the questions mean the same to the different respondents. The design in the study described in Paper II is flexible since the number of participants was limited and it allowed interviews with open-end questions where concepts and terms could be explained to the interviewees. The open-ended questions allowed unforeseen information to be recorded and the flexibility allowed questions to be added, removed or changed. Another reason for using flexible design is that the researchers gained more and more knowledge in the area, so the used design had to allow progress based on the more gained knowledge.

A commonly used technique for preparing qualitative data to be analysed quantitatively is coding, were value for quantitative variables extracted from qualitative data in order to do some quantitative or statistical analysis. This process was used in Paper II were interviews were used to collect the qualitative data and then same statistical analysis was made. The distinction between qualitative and quantitative data is not if it subjective or objective, it is how the information is represented [35].

Quantitative data is represented as numbers or other discrete categories and qualitative data is information expressed by words or pictures. Using qualitative methods increases the amount of information contained in the collected data since qualitative data is richer that quantitative data [35].

Experiments are used as research method for Part C of this thesis. Experiments are conducted when the researcher wants control over situation with systematic manipulation of the

Research methodology

behaviour of the studied phenomena [46]. Experiments [32] are of fixed design type and are focused studies with a few variables to handle. The experimentation is a research strategy that involves manipulation of one or more independent variables by the researcher, the measurements of the effects of manipulation on one or more other dependent variable and control over all other variables.

The research in Part C of this thesis is based on two different experiments. Paper III is based on an experiment were students acted as subjects. The use of students as subjects can be questioned. Höst et al [19] conclude in a comparative study of students and professionals in lead-time impact assessment that

“there are only minor differences between the conception of students and professionals and there is no significant difference between the correctness of students and professionals”.

However it cannot be concluded with large validity that the students that participated in the experiment presented in paper III are representative of professional practitioners. So the result from this experiment should primarily serve as basis for continued experiments in the area. Another factor regarding the participants in controlled experiments is the incentives for participants in the experiment [20]. Höst et al [20] argue that the validity of a study is affected by the motivation of the participants and they introduce a way of trying to capture the motivation by looking at the experiment situation where the subjects are participants. In the experiment described in Paper III the intention was to take that into count and motivate the students as subjects, by having a seminar about risks and by designing the experiment to be representative for engineering work and linked to the project course the students attended at that time. The students being in their second year at the university are classified as “E1: Undergraduate student with less than 3 months recent industrial experience” according to Höst et

Research methodology

al [19]. Recent industrial experience means that it is experience received less that two years ago.

The second experiment is an experiment in real context also called quasi experiment. Quasi experiments are experiments when units are non-randomly assigned to experimental groups [20]. Kampenes et al [21] conclude that quasi-experimentation is useful in many settings in software engineering. Quasi experiments according to Basili [2] tend to involve qualitative analysis components and that quasi experiments easily can be done in vivo with experts and that this experiments easily deals with large projects. The subjects used in the quasi experiment described in Paper IV are three different categories of professional practitioners, software developers, medical device developers and physicians. The subjects were non-randomly selected.