Paper IV – Risk analysis and risk planning

The main objective of Paper IV was to collect and summarise experiences from conducting risk management with an organisation developing medical devices with specific focus on the first three steps of the risk management process, i.e. risk identification, risk analysis, and risk planning. Earlier versions of the study were presented in Related publication X and XI. The motivation for the study was to get experiences from having a user perspective in these three risk management steps, with the long-term objective to design an improved version of the risk management process. The main objectives were defined based on the general interests of the researchers, and the interests of the development organisation. This was defined in informal meetings between the researchers separately and between the researchers together with the development organisation. The case study was conducted at a department at a large hospital in Sweden, which develops and maintains medical devices. The development organisation has extensive experience in developing and maintaining medical devices, but not with devices including software.

RQ4: How can users be integrated in the risk management process?

RQ6: How can a software risk management process including user perspective be designed to be appropriate for a medical device development organisation?

Research contribution

The research was conducted as action research, with the aim of analysing and giving input to the organisation’s introduction of a software risk management process. The data collection was made through two different sources: interviews and observations. The defined and used process focuses on user risks, based on scenarios describing the expected use of the medical device in its target environment, in this case with a patient monitor system as the medical device. During the use of the process, different stakeholders were involved, medical physicians with competence on the monitored medical processes were involved, together with engineers with competence on the software and hardware, and personnel with competence on the required procedures in the organisation.

In the risk management process, a scenario-based identification method was used and the risks were identified through brainstorming, focusing on user interaction and user related risks. Some technical risks were also identified using the scenarios. Since technical risks are of a more general nature and not scenario-specific, there is a need for a separate risk identification regarding these risks. When using this kind of scenarios there is alsoa need for risk identification of external factors, for example, process and project risks. When the scenarios were discussed step by step, it could be noted that the user representatives, are the dominant part, since they possess domain knowledge regarding the target environment and medical issues. The developers had a more peripheral role and were consulted regarding technical aspects of the system. A tendency to discuss action proposals instead of risks during risk planning was also identified. A possible solution to the dominance factor and discussion focus could be to have very strict control of the meetings, and with the ambition to get the opinion from all the participants, for example specifically address each participant. This approach was successfully used at the risk meetings in the case study in Paper VI. The way of mixing action proposals that are implemented with proposals that are not introduces unnecessary confusion. To avoid this it is highly recommended that the risk analysis should be done prior to implementation.

From the results, it can be concluded that the system boundaries must be set carefully and not without considering dependencies between components. Before defining the system boundaries, it should be clear how components are coupled and components with strong coupling should be analysed together. The documented risk descriptions have an

impact on the risk planning process, because the descriptions influence the understanding of the risk context. In the studied process, risk descriptions typically only contain a very short summary of the nature of the risk. To lower the risk of misunderstanding and misinterpretation later in the process, the risk must be described in detail and placed in its context.

The order of estimation influenced the outcome of the risk analysis, thus the prescribed order of estimation, e.g. severity, probability, and detectability, should be strictly followed. The concept of detectability was not well understood and the provided scale did not give as much help, as was the case with probability and severity. After completing the risk analysis the development organisation decided, based on the encountered problems, to remove detectability from the process.

Although this simplifies the process, it removes potentially important information about risks. There is a need for further research on how to define and estimate detectability of identified risks, see Section 6.

It can also be concluded that the used risk process is considered effective and easy for new personnel to adapt to according to the health personnel working with risk management. The main challenge is however to find the time and right competences for the risk analysis team.

Even if it is difficult and time-consuming to produce relevant user scenarios the scenarios make the software easier to understand, which in turn improves the understanding of potential risks. With these difficulties in mind a slightly different scenario design process is used in the new risk management process, RiskUse, presented in Paper VI.

From this case study it can be concluded that the used risk management process has the potential to be used in a medical device development organisation or similar organisations. Criticism could be pronounced that, the focus might be too high on the user interface.

However, since it is well known that many risks are related to the usage of a system and the user interface, e.g. Dhillon (2008) reports that 50 % of technical medical equipment-related problems are caused by operator errors, it is important that the user interface stays in focus.

In the following work with designing the new more complete risk management process, RiskUse, covering all the steps in the process, presented and evaluated in Paper VI, great emphasis has been placed on the results and findings in Paper IV.

Research contribution

5.5 Paper V – Usability testing in the risk management