Paper V – Usability testing in the risk management process

Research contribution

5.5 Paper V – Usability testing in the risk management

tests were not identified as risks in the risk management process. Several of these usability problems imply risk and should be handled in the risk management process, especially the four usability problems that were found by all the users.

From the case study it can be concluded that usability tests can indicate risks that are not identified in the risk management process and give a possibility to verify if risks with high risk value actually cause the presumed problems. It is also possible to capture “problem functionality”

e.g. for functionality that is new or unknown to the user. Usability testing also catches problems that are good risk candidates, where the functionality is unclear to the users and where the developers and the users have different mental models. Timing is important when it comes to usability testing connected to the risk management process. The time must be right, so no changes are made only based on the risks, before the usability test is performed. The usability tests can, for example, verify that a risk with a high risk-value actually is a problem for the users before any changes are made. Risk values are assumptions so if they can be identified in additional ways before any action is taken, effort and time can be saved by the development organisation, due to the avoidance of unnecessary changes. Then it can be concluded that usability tests can give valuable input to the risk management process if integrated as a natural part in the development process and the development organisation can save time and effort and also receive more faultless and safer products.

Usability testing has ben integrated as a part in the new risk management process, RiskUse, and further evaluated in the research in Paper VI.

5.6 Paper VI – Evaluation of the risk management process RiskUse

RQ4: How can users be integrated in the risk management process?

RQ5: How can usability evaluation methods, especially usability testing contribute to the risk management process?

RQ6: How can a software risk management process including user perspective be designed to be appropriate for a medical device development organisation?

Research contribution

Paper VI introduces the first version of RiskUse, a medical devices software risk management process with an emphasised user perspective.

The process was used and evaluated in a case organisation developing and maintaining a medical devise. The risk management process, RiskUse was developed over time in close contact with the organisation.

The purpose of the risk management process is to provide practitioners, mainly risk managers, with a software risk management process that has a well defined user perspective, is easy to apply, and including hands-on recommendations on how to use the process. The aim is also to present a risk management process that allows the development organisation to perform adequate risk management activities that can ensure that the developed software is safe. The main goal is to integrate users and user perspective in the software risk management process and to introduce usability testing, as an integrated part in the risk management process contributing to the goal of integrating users.

RiskUse is developed based on contributions from each of the five first papers in this thesis. More specifically, RiskUse is based on empirical knowledge about the state of practice gained from the survey presented in Paper I, knowledge regarding human factors from different angels from the experiments in Paper II and III and the case study results on the risk management process in Paper IV and usability testing in Paper V, complemented with in depth studies of risk management standards, regulatory requirements and guidelines within in the medical device domain.

The evaluation was carried out using an action research approach according to the observations of risk meetings, supplemented with interviews and observations of usability testing. The goal of the case study was not only to evaluate the risk management process but also to make improvement proposals to the development organisation, based on the results from applying RiskUse in a project and to use the result to further improve RiskUse.

In conclusion, RiskUse is found to support the practitioners in their work with risks and risk management. It can also be concluded that the process has the potential to be used in a medical device organisation and bring value to the organisation. The risk management process is also found to be easy to understand and apply, according to the practitioners participating in the case study. Limitations identified through the case study (presented in Paper IV) were considered and possible solutions were implemented and evaluated. According to the results from the case

study in Paper VI it can be concluded that the solutions are worth maintaining in RiskUse and further evaluate. The findings from Paper V show that usability testing can contribute to the software risk management process was further strengthened in this case study.

Concerning further work, it includes addressing the identified improvement proposals as well as further evaluation of the risk management process. The two last phases, the monitoring phase and the completion phase and need to be evaluated and require an evaluation over time and in addition the whole risk management method. To fully understand and evaluate the possibilities and potential benefits of RiskUse, the process has to be used over time and in a complete project from the start to the end. Broad generalisations of the results can therefore not be made since this is the first evaluation of the risk management process, only involving one organisation. However the case study shows that the risk management process is applicable and the positive results provides a strong argument to continue the evaluation and to promote the risk management process, RiskUse.