The experiment

The experiment was performed during the year of 2008, with 15 physicians, 15 developers and 6 medical device developers as participants. All the participants were presented to the same risk scenario describing the procurement of a patient monitoring system.

The involved subjects are professional developers and physicians working in Sweden. The physicians are all anesthetists employed at the same clinic in a hospital in Sweden.

As an anesthetist they alter their work between three different units, the intensive care unit, the surgical ward and the emergency care unit. Working in these three different units involves handling a lot of different medical devices. Before the risk scenario was sent out to the physicians an information meeting were held to explain the experiment and the purpose of the experiment. The risk scenario was not exposed to the physicians at this information meeting. A description of the risk scenario, a reply form and a self-addressed envelope were sent out to the 37 physicians by ordinary mail. The physicians were

Paper IV

asked to send in their answers within 2 weeks. After 3 weeks a reminder with the same content was sent out. In total 15 physicians returned their answers, i.e., a reply rate of 15/37 = 40%.

The developers were asked if they would like to participate in the experiment by e-mail. This e-mail was sent out to developers in different Swedish companies developing software and in the e-mail the experiment and the purpose of the experiment was explained. The developers were addressed directly and e-mail was sent out to 26 developers and 15 accepted to take part in the experiment. That is the reply rate of the developers was 15/26 = 58%.

The same risk scenario as the one sent to the physicians was sent to the developers with a minor difference, that some of the medical terms was explained. The risk scenario and reply form was sent by e-mail and the developer returned the reply forms after answering, by e-mail. The most difficult group to get participants from was the medical device developers. Despite 32 different information e-mails directly to individuals and different companies developing medical devices in Sweden and several telephone calls only 6 participants participated in the experiment. This means that the reply rate of this group was 6/32 = 19%. The major reason given for not participating in the experiment was lack of time. The risk scenario and reply form that was sent out to the medical device developers by e-mail were identically as the one sent to the developers. The reply forms received from the medical device developers were also returned by e-mail.

The risk scenario is around 1¼ page long and is written in Swedish describing the procurement of a patient monitoring system. When the risk scenario was written some things considered as risks by the researchers was deliberately incorporated in the scenario but no too obvious risks were chosen. The risk scenario describes the following scenario:

Paper IV

”It is the county council that have decided to buy a new patient monitoring system with the intention to have the same monitoring system at all the units at the hospital, for example at the intensive care unit, the surgical ward and the emergency care unit. The goal for the county council is to rationalise the care and reduce cost for the daily activity. The county council have requested 10 companies for tender and the tender text is presented in the scenario. The tender text is divided in four parts; Dimensioning and functionality, Location, presentation and compability, Mobile monitoring and Communication and use. The system shall, for example, have one central server, it shall be possible to monitor 22 patients through wireless communication, and the functionality that the system at least shall include is specified. It shall be possible to use the new system together with other medical devices, the new system must guarantee patient safety, it shall be possible to easy physically transport the patient together with the system, the system should be able to communicate with other wards and external partners, a user should be able to use the system after five hours of training etc. That is, the scenario includes both functional and non-functional requirements. The risk scenario ends with a description of the company the county council have decided to give the contract to. This company is a new, promising and expanding company on the medical device market and they promises to deliver the system in 6 month to the lowest cost. The majority of the staff is developers that recently have taken their degree. That is, the scenario also includes information about the development organisation and a description of the experience and competences of the developers”.

All the participant in the experiment were asked to study the risk scenario, identify risks and write down the risks in their own words (i.e. "free text form") in the reply form. The participants were also asked to estimate the probability of the

Paper IV

risk and to estimate the effect of the risk. For the probability, the participants were given a graded scale 1 – 4, where 1 represents that it is very unlikely for the risk to occur, 2 represents that it is unlikely, 3 that is likely and finally 4 represents that it is very likely the risk will occur. A scale was also given for the estimate of the effect if the risk occurs. This scale was graded 1-5 where 1 represents that the effect would be insignificant if the risk would occur 2 that the effect would be acceptable, 3 that the effect would be serious, 4 that the effect would be very serious and 5 the effect would be catastrophic if the risk would occur.

In the analysis the researcher then calculated the risk value, R, for each given risk by multiplying the given figure for probability, P, by the given figure for effect, E, as

!

R=P " E. Thus, the highest possible risk value a risk can get in this experiment is

!

R = 4 " 5 = 20.