Methodological approach

is valued and adopts an engineering approach to the research (Easterbrook et al. 2008).

Design science improves the environment in which it is applied and throughout the design science process knowledge is interchanged between the knowledge base and the environment (Hevner 2007; Gregor

& Hevner 2013). The design science process includes six steps; problem identification and motivation, definition of objectives for a solution, design and development, demonstration, evaluation, and the final step communication (Hevner & Chatterjee) 2010). The research within the process iterates between the design and development, the evaluation of the artefact and its refinement based on feedback (Hevner 2007; Gregor

& Hevner 2013). The research presented in this thesis includes parts of such iterations where the risk management process, RiskUse is developed and refined based on feedback but also based on knowledge seeking.

Research methodology

thesis the action researcher part has been active observations. In the performed case studies the active observations have been an integrated part. The results in this thesis are reached through the use of the research methods presented in the sections below. The design science research approach where the research iterates between design and development has influenced the design and development of RiskUse. Throughout the process, knowledge was interchanged both with the medical device organisation and the knowledge base (further presented in Section 4.3).

For each paper, research questions (RQ), type of research, research design and data collection method are presented in Figure 5. Further details on the different research studies can be found in respective paper.

Figure 5. The relationship research papers, research questions, type of research, research design and data collection method.

4.1.1 Survey

The purpose of a survey is “to understand, describe, explain or explore the population” (Wohlin et al. 2000). It is difficult to give a concise definition of survey research, but a survey has often three typical central features according to Robson (2002):

1. Fixed, quantitative design is used.

2. From a relatively large number of subjects a small amount of data in a standardised form is collected.

3. A representative sample of individuals from known populations is selected.

These three central features capture a large part of surveys, but there are surveys where considerable amounts of data are collected from each

individual, but the individual does not represent himself or herself but rather a company or organisation.

Representative samples selected from a well-defined population distinguish survey research (Easterbrook et al. 2008) and the strategy can be used to identify characteristics of a broad population of individuals (Rea & Parker 2005; Easterbrook et al. 2008). A clear research question, focusing on the nature of the target population is a precondition for conducting survey research (Easterbrook et al. 2008).

However, even if surveys often are referred to as a fixed design, Robson (2002) also argues that surveys can be based on either flexible or fixed design depending on the degree of pre-specification. In typical fixed design the data collection is made by self-administrated questionnaires with closed questions and in typical flexible design the data collection is made through interviews with open-end questions. Structured recorded reviews and structured observations are other survey instruments that can be used (Fink 2003).

The research in Paper I, about the state of practice in the medical device domain is based on a survey. The research design is fixed and carried out through a web-based questionnaire sent out to medical device companies from Europe and the US. The fixed design with the use of a web-based questionnaire was chosen because it is an easy way to retrieve information from a large set of people in different countries. It allows anonymity and can provide a large amount of information to a low effort in a short period of time. The design was typically fixed with the closed questions, where it is possible to know that the questions mean the same to the different respondents.

4.1.2 Experiment

Experiments are conducted when the researcher wants control over the situation with systematic manipulation of the behaviour of the studied phenomena (Wohlin et al. 2000), and in software engineering experiments are mostly dependent on human subjects performing some task (Easterbrook et al. 2008). Experiments (Robson 2002) are of fixed design type and the studies are focused with a few variables to handle.

Before the main data collection begins, the design details are fully pre-specified.

The experimentation is a research strategy that involves manipulation of one or more independent variables by the researcher and the effects of

Research methodology

the manipulations are measured. The measured effect is then statically analysed to confirm the significance of the effect (Wohlin et al. 2000).

Experiments reduce complexity by only allowing a few variables to vary (Easterbrook et al. 2008). Jedlitschka et al. (2008) provide a guideline on how to report software engineering experiments. The guideline unifies and extends other existing guidelines by various authors.

The experiment in Paper II is an experiment in real-life context and is a quasi experiment. Quasi experiments are experiments when units are non-randomly assigned to experimental groups (Höst et al. 2005;

Easterbrook et al. 2008). Kampenes et al. (2009) conclude that quasi-experimentation is useful in many settings in software engineering. Quasi experiments according to Basili (1996) tend to involve qualitative analysis components and they can easily be done in real-life context with experts working in large projects. The subjects used in the quasi experiment described in Paper II are subject in three different categories of professional practitioners; software developers, medical device developers and physicians. The subjects were non-randomly selected.

Paper III is based on an experiment where students acted as subjects.

The use of students as subjects can be questioned. Höst et al. (2000) conclude in a comparative study of students and professionals in lead-time impact assessment that “there are only minor differences between the conception of students and professionals and there is no significant difference between the correctness of students and professionals”. Even if the intention was that the subject in the experiment should be representative of engineers working with this type estimation in real-life experiment, it cannot be concluded with large validity that the students that participated in the experiment presented in Paper III are representative of professional practitioners.

Another factor regarding the participants in controlled experiments is the incentives for participants in the experiment. Höst et al. (2005) argue that the validity of a study is affected by the motivation of the participants and they introduce a way of trying to capture the motivation by looking at the experimental situation where the subjects are participants. In the experiment described in Paper III the intention was to take that into count and motivate the students as subjects, by having a seminar about risks and by designing the experiment to be representative for engineering work. The experiment was part of a project course the students attended at that time.

4.1.3 Case study

According to Yin (2003), ‘‘a case study is an empirical inquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between the phenomenon and context are not clearly evident’’. The study of software engineering often is of a complex nature and cannot be studied in isolation. The complexity arises from the role of human behaviour in software development (Seaman 1999) and software engineering in general, which is a complex social phenomenon. A real-life context and the study of contemporary phenomenon are conditions that are valid for many research studies in software engineering and make a case study approach feasible (Runeson et al. 2012).

The research design of a case study is flexible where the research strategy develops during the data collection and analysis (Robson 2002).

In a case study the selection of a case is rather purposive than random (Easterbrook et al. 2008) and the research questions are allowed to evolve during the study. Case study methodology typically involves multiple data collection methods such as observations, interviews, and documentary analysis (Robson 2002) and due to the flexible nature of case studies that data collection will often continue even after the analysis begun. There is a wide range of ways to analyse the data, from qualitative analysis where data from, for example, interviews are coded and categorised to the use of statistical methods (Runeson et al. 2012).

In a case study there are confounding factors, which are not entirely known and cannot be controlled, and these factors might affect the result.

Consequently, the researchers do not have the same level of control in a case study as in an experiment. The results of a case study are also more difficult to interpret and generalise than results of experiments (Wohlin et al. 2000).

The three case studies presented in this thesis have been performed in real-life settings, to gain insight in risk management in medical software development, to evaluate the contribution of usability testing to risk management and to evaluate the developed risk management process RiskUse. The objective of the first case study (Paper IV) was to collect and summarise experiences from conducting risk management with in an organisation developing medical devices and the overall objective of the second case study (Paper V) was to investigate how usability testing can contribute to a software risk management in the medical device domain.

Finally, in the last case study (Paper VI) the new software risk

Research methodology

management process was evaluated in a medical devices development project, in order to study strengths and weaknesses of the developed risk management process, RiskUse. A mixed data collection approach was used in all the three case studies, using a combination of observations and interviews with the aim to gather different kinds of data (Runeson et al. 2012). The use of more then one data collection method also allows different perspectives and enables the use of triangulation (Robson 2012).

4.1.4 Action research

In action research, there is collaboration between researchers and those who are the focus of the research (Robson 2002) and the purpose is to influence or change some aspects in the studied environment. Action research is closely related to case studies (Runeson et al. 2012). The aim, according to Robson (2002) is to improve practice, the understanding of practitioners, and the situation in which practices take place (Robson 2002). The application of action research emphasises “more on what practitioners do than on what they say they do” (Avison et al. 1999). A large part of software engineering research is using an action research strategy (Easterbrook et al. 2008). In software engineering research it is a common scenario that trying out the idea in a real industrial context develops the originally idea. The use of action research in the information systems (IS) domain has also become more frequent (Davison et al. 2004). According to Sjøberg et al. (2007) action research is regarded as “the most realistic research setting found”, because the setting of the study is the same as the setting in which the result will be applied for a given organisation, apart from the presence of the researcher. A researcher coming from outside looking at the studied phenomena with fresh eyes and different angles is beneficial. However, when implementing improvements, commitments within the organisation are needed and this is best achieved by involving people inside the organisation (Runeson et al. 2012). Since each action research project to some extent, is unique, it is difficult to draft general rules about how to carry out such projects. However there are general guidelines presented by, for example, Avison et al. (2001) and Sagor (2011).

The action research process is composed of a four-stage procedure (Robson 2002; Sagor 2011), starting out with the planning stage. The four stages are shown in Figure 6. The process is a cyclical or a spiral

process, based on iterations where the researcher in real situations wants to try out theories with practitioners, gain experience and feedback, modify the theories and try again.

Figure 6. Action research four-stage procedure.

Action research has been used in as part of the case studies in Paper IV, V and VI. It has been active observations, allowing the researchers to influence the outcome of the observed activity. In Paper IV, the aim was to observe how the activities are performed in their context, not to actually perform the activities. However, during the activities, it was natural for the researchers to give input and support to the development organisation. The aim was also to get information about aspects of the activities by asking questions and giving advice on relevant topics. In Paper V and VI the researcher had a more active role when taking part in the usability testing (e.g. as facilitator) and in the risk management process, (e.g. as participant in the risk management team), where the purposed risk management process was evaluated.