• No results found

2 Background and related work

4.4 Validity

Research methodology

risk planning. The state-of-the-art in the risk management field was also taken into account.

3. Limited candidate solution. The limited candidate solution was developed in cooperation between the researcher and the organisation, presented in Paper IV. The state-of-art were further studied and accounted for, with special focus on laws, regulations and guidelines with in the field.

4. Dynamic validation. The limited solution was validated in the two case studies presented in Paper IV and V. All the empirical data was coded and grouped (Seaman 1999), identifying patterns between these codes and incorporating these patterns in the evolving risk management process.

5. Complete candidate solution. The risk management process presented in Paper VI was developed based on the results in Paper IV and V and in depth studies of risk management standards and guidelines within the medical device domain.

6. Dynamic validation. Finally, the risk management process RiskUse has been evaluated within the organisation and the results are presented in Paper VI.

However the risk management process RiskUse will be further improved (see Section 6) and further evaluation is needed before releasing the solution. RiskUse may need to be tailored for different organisations since organisations can have different definitions, vocabulary, relative to product and development (Gorschek et al. 2006).

the rest of the papers the classification of Yin (2003). The classifications are very similar; three of four types have the same meaning and are also called the same. The fourth type denoted, conclusion validity by Wohlin et al. (2000) and reliability by Yin (2003) are both concerned with the ability to draw accurate conclusions and to replicate the study.

Validity is classified according to Yin (2003) in construct validity, internal validity, external validity and reliability:

Construct validity is affected by how well the collected operational measures represent the concept studied by the researcher. To increase the construct validity, multiple sources of evidence can be used and also establish a chain of evidence. In case studies construct validity can be increased by letting people with key information, review the preliminary case study report.

To improve construct validity in interview studies, the researchers has to ensure that the questions are interpreted in the same way by both researchers and interviewees (Runeson et al.

2012).

Internal validity is affected by factors that are outside the control of the researcher but affect the measures and reflects whether a particular treatment really has caused a certain outcome. In experiments it is critical to address internal validity so the researcher does not draw incorrect conclusions about the effect of the treatment on the outcome without knowing if there in fact is a third factor affecting the outcome (Wohlin et al.

2000). In case studies it is applicable to consider internal validity for explanatory and causal studies and not for descriptive and explanatory studies, which are not making causal claims.

However, there is a threat to internal validity in case studies due to the changes in industrial environments over time (Yin 2003).

Internal validity threats might be lowered by using pattern-matching or explanation-building (Yin 2003).

External validity concerns the problem of how general findings are with respect to the subject population and beyond the immediate study. Results obtained in a specific context or with a specific group of participants may not be representative for or transferable to other environments or settings. External validity threats can be lowered by using theory in single-case studies and

Research methodology

replication logic in multiple-case studies. In case studies focusing on understanding and explaining phenomenon within its real-life context, the aim of the study is not to generalise (Runeson et al. 2012).

Reliability concerns how reliable a study is and depends on how well the described procedures are followed and documented, so that the study can be repeated in the same way over again. “The goal of reliability is to reduce errors and biases in the study” (Yin 2003). Threats to reliability might be unclear coding of collected data or unclear questions in interviews in questionnaires (Runeson et al. 2012). The use of use case protocol is one way to lower threats to reliability (Yin 2003; Runeson et al. 2012).

There are several possible threats to validity in studies and a main concern in the work with the contributions presented in this thesis has been to identify and reduce all the validity threats as much as possible.

Threats to construct validity can be participants’ bias. There is a risk that the participants misunderstand and interpret terms, concepts or questions differently. In the studies described in this thesis, the risk of misunderstanding have been reduced by only allowing responses on such a level of detail that subjectivity from the participants and need for interpretation are minimised. During the interviews and risk meetings presented in Paper IV and Paper VI, the concepts were explained during the interviews and the definitions of terms and concepts in the risk management process were adapted to the standards in the medical device domain with the aim to avoid misunderstanding. In Paper II the instructions and risk scales were written and defined to be as clear and unambiguous as possible. Another participant bias can be that the participant gives a too positive picture of the situation. In order to lower this risk in the survey study presented in Paper I and the interview studies presented in Paper IV and Paper VI, the participants were allowed to be anonymous. However here is a risk that the participants exaggerate the negative sides in frustration over different situations and this must be taken in consideration when interpreting the results, which has also been done in the studies. To further reduce the risks of participants’ bias in the case studies (Paper IV to VI), the interviews were audio recorded and triangulation (Robson 2002; Runeson et al. 2012) was applied The interview transcripts were reviewed by other researcher,

multiple sources for the data was used and member checking of the material by the medical device organisation. Reactivity (Robson 2002) is a threat to validity in flexible design where the presence of a researcher might affect the participants and thereby influence and limit the outcome. The participants might act or respond after assumed expectations or hide facts. To reduce this threat the participants were guaranteed anonymity regarding all collected data, e.g. the survey interview and observation material was not showed or used by researchers outside the studies or by other organisations.

The threats to internal validity have been minimised in the contributions in this thesis according to Paper I, Paper IV, Paper V and Paper VI, by only analysing relations between factors and not drawing any conclusions of causal direction in the studies. In the experiment presented in Paper II there is a threat to internal validity since the participant took part in the experiment under different conditions, out of the researchers control. The threat to internal validity was lowered in the experiment presented in Paper III by letting the participants carry out the assignment at the same time. The participants analysing the given scenarios in the same order, however is a threat.

External validity primarily relates to how general the results of the study are and to what extent the findings are applicable and of interest outside the investigated cases. The evaluation of the risk management process presented in Paper VI is done with a limited set of participants from two single projects. This means that the results cannot automatically be generalised to other organisations. However, probably some of the issues in the risk management process, to some extent could be general for organisations developing software in the medical device domain. The results can help to understand phenomenon and the context and also to improve and further evaluate the risk management process. To support generalisation and allow external comparison, the context and characteristics of the projects have been presented as extensively as possible under given confidentiality constraints. To be able to show generalised results the new risk management process, RiskUse, need to be independently used in a larger amount of projects.

A threat to external validity can be that the participants are not representative of the target population. In the studies in all the papers in this thesis, professionals from the software and medical sectors have been involved but in the study in Paper III students have been used as

Research methodology

participants in the experiment. This is a threat and this experiment should be repeated again with participants working as software engineers.

Concerning replication, controlled experiments in a laboratory can be performed as direct replications and external replications can be done where the replications is conducted in, for example, different environments and then compared to the original study. In studies of flexible nature, such as case studies, the design is more difficult to recreate since the context of the study always changes. The procedures and changes in the studies presented in this thesis have been carefully monitored and documented over time to get reliable studies that can be repeated again by other researcher after minor changes and other medical devices under study. To further increase reliability several researchers were included in the studies. The reliability was also addressed by applying triangulation to the data collection and analysis procedures and final versions of Paper VI, V and VI was reviewed and approved by participants from the development organisation. Feedback was also given from the development organisation during the research process.

Regarding conclusion validity in experiments (Wohlin et al. 2000), it relates to the possibility to draw correct conclusions. A typical threat can be the use of wrong statistical tests. With this in mind the statistical tests were chosen for the studies with great care and for example in the controlled experiment presented in Paper II the analysis was also done with non-parametric test.

The ethical principles presented by Vinson and Singer (2005) has also been regarded during the research work. That is the subjects have been informed about all relevant facts concerning the studies and have decided themselves about their participation. The researchers have also undertaken an effort to maintain confidentiality of data and sensitive information and guaranteed the subjects’ anonymity.

Further details regarding the threats to validity, are discussed separately in each paper, together with applied strategies to reduce the threats.

5 Research contribution

This section presents the main results and contributions of this thesis.

The discussed results and contributions are based on the conclusions from the included papers and are summarised per respective paper and addressed research question. The relationship between the different research questions and research papers are presented in Table 4 in Section 3.1.