Faculty of Technology Department of Informatics

Faculty of Technology Department of Informatics

Usability in Patient-Oriented

Drug Interaction Checkers

A Scandinavian Sampling and Heuristic Evaluation

Abstract

Drug interactions are an important source of medical error. Studies have also shown it as a topic of particular interest to patients. To allow patients to participate in decision-making regarding their own health, they need to be empowered with information. This information must be provided through usable information systems. This thesis explored availability of drug interaction checkers in Scandinavia. It also explored the prevalence and characteristics of usability issues preventing patients from benefiting from these.

Drug interaction checkers were sampled and described. Issue-based qualitative and quantitative data were gathered through heuristic evaluations. Patterns in the data were identified through descriptive statistics. Single-case and cross-case analyses explored emergent patterns in-depth. The findings were then interpreted side by side using a mixed-methods approach.

The results showed the Scandinavian public faced with drug interaction checkers addressing healthcare professionals. They also showed a multitude of usability issues in these checkers. The issues were predominantly minor, but major issues were also identified in all but one of the checkers. Catastrophic usability issues were found in two of the checkers. These had the potential to lead patients to serious medical error. Results moreover showed the checkers lacking adaptive design, patient-oriented content, and a lack of adherence to basic design principles. A positive correlation was observed between system complexity and number of usability issues. This was suggested to result from lack of user-centered design approaches, or losing track of user goals while adding features over time.

The result was a handful of generally professionally oriented drug interaction checkers known to be used by patients for their utility, but failing to accommodate them in terms of information and system quality. Empowering patients to participate in decision-making affecting their health asserts the need for developing patient-oriented information databases where these do not yet exist. These need to be presented through usable interfaces acknowledging patient behaviors.

Keywords

Abstrakt

Läkemedelsinteraktioner är en viktig orsak till medicinska misstag. Tidigare studier har även identifierat läkemedelsinteraktioner som ett område av särskilt personligt intresse för patienter. För att patienter ska kunna vara delaktiga i beslutsfattande beträffande sin egen hälsa, måste de ges tillgång till information genom användbara informationssystem. Denna uppsatsen undersökte tillgänglighet av interaktions-granskare för läkemedel för den skandinaviska publiken. Den undersökte också förekomst och karakteristiker av användbarhetsproblem som förhindrar patienter från att dra nytta av dessa interaktionsgranskarna.

Tillgängliga interaktionsgranskare listades och beskrevs. Kvalitativa och kvantitativa data samlades in genom heuristiska utvärderingar. Mönster identifierades genom beskrivande statistik, medan kvalitativa enkelfalls- och tvärfallsanalyser undersökte frambrytande mönster på djupet. Resultat från kvantitativa och kvalitativa metoder tolkades sedan sida vid sida.

Resultaten visade att skandinaviska patienter huvudsakligen möter interaktions-granskare riktade mot professionell vårdpersonal. Resultaten påvisade också en stor mängd användbarhetsproblem i dessa granskarna. Problemen var oftast av mindre vikt, men även stora problem identifierades i alla utom en av granskarna. Katastrofala användbarhetsproblem hittades i två av granskarna. Dessa hade potential att leda patienter till allvarliga medicinska misstag.

Resultaten visade dessutom att granskarna saknade adaptiv design, patientorienterat innehåll och efterlevnad av grundläggande designprinciper. En positiv korrelation observerades mellan systemkomplexitet och antal användbarhetsproblem. Detta föreslogs vara ett resultat av brist på användarcentrerad designmetod eller att man har tappat fokus på användarmål över tid medan funktioner har lagts till.

Resultatet var en handfull huvudsakligen professionellt inriktade granskare för läkemedelsinteraktioner som används av patienter för deras nyttja, men som inte lyckas med att ta hänsyn till dem när det gäller informations- och systemkvalitet. För att patienter ska få möjlighet att delta i beslutsfattande som påverkar deras hälsa, finns behov av att utveckla patientorienterade informationsdatabaser där dessa ännu inte finns. Dessa måste erbjudas genom användbara gränssnitt som tillgodoser patienters beteenden.

Nyckelord

Acknowledgments

The authors would like to thank Mexhid Ferati for his supervision and guidance in helping us make this thesis what it is. We would also like to thank My Zetterholm for suggesting the topic that was explored.

David Vingen

Kristiansand, June 15, 2020

Table of Contents

1.4 Outline of the Thesis 4

2 Literature Review 5

2.1 Patients’ Needs for Drug Information 5

2.2 Usability and Heuristic Evaluations 6

2.2.1 Measures of Usability 6

2.2.2 Methods for Evaluating Usability 7

2.2.3 Reliability of Heuristic Evaluations 8

2.3 Previous Usability Studies in mHealth 9

2.4 Heuristic Evaluations and Heuristics Applied in mHealth 10

3 Methodology 11

3.1 Study Design 11

3.2 Sampling of Drug Interaction Checkers 12

3.3 Heuristic Evaluations 12

3.4 Quantitative Analysis 15

3.5 Qualitative Analysis 16

4 Results and Analysis 17

4.1 Description of Drug Interaction Checkers 17

4.2 Quantitative Analysis 22

4.3 Cross-Case Qualitative Analysis 25

4.4 Single-Case Qualitative Analyses 28

5 Discussion 35

5.1 Availability of Drug Interaction Checkers 35

5.2 Prevalence and Characteristics of Usability Problems 36

5.3 Practical Implications 37 5.4 Method Reflection 38 6 Conclusion 42 6.1 Future Research 43 References 44

Appendices

Appendix 1: Evaluation Protocol

1 Introduction

Medical error has been identified as the eighth leading cause of death, and patients outside hospitals have been shown at particular risk of drug-related medical errors. Failure to prescribe necessary drugs, patients failing to adhere to them, or intoxication caused by interacting drugs are all examples of such errors. Many of these errors are preventable, and nearly half of all preventable adverse drug effects have been shown serious enough to cause hospitalization. The errors frequently occur in phases of prescribing or monitoring drug use. This indicates the importance of safety interventions in these stages of outpatient care (Thomsen et al. 2007).

1.1 Background

Drug interactions are when concurrent use of multiple drugs, a drug and a food, or a drug and a beverage causes changes in effects of a drug. Such changes may consist of the appearance of serious side effects, or in the suppression of desirable effects. The likelihood of drug interactions increases with the number of drugs taken. This also often correlates with age. Up to seven percent of hospitalizations have been shown attributable to drug interactions (Cascorbi 2012).

Prescription and use of drugs have important health benefits. But increased use of drugs and increasing polypharmacy, meaning concurrent use of multiple drugs, increases risk of drug interactions. Åstrand et al. (2007) did a cross-sectional study of drug prescriptions in Jämtland, Sweden, during a period of 30 years. They found a pronounced (61 %) increase in polypharmacy, and found the risk of potentially interacting drugs strongly correlated with this increase. Hovstadius et al. (2010) looked at the development of polypharmacy in a four year-period between 2005 and 2008. Using the Swedish Prescribed Drug Register, they were able to analyze the data of the entire Swedish population according to drugs prescribed per individual. They defined polypharmacy as receiving five or more prescription drugs during a three-month period, and excessive polypharmacy as receiving ten or more drugs within the same period. Using this definition, they found an 8 % increase in polypharmacy, and an increase in excessive polypharmacy of 16 %. The study thus showed a steady increase in drug use, polypharmacy and excessive polypharmacy. This was in spite of the well-known risk to patient health (Hovstadius et al. 2010).

Patient-centered care builds upon patient participation. It denotes a holistic, individualized and long-term approach to healthcare. In patient-centered care organization of care revolves around the patient through multidisciplinary and cross-institutional cooperation, rather than having patients conform to organizational structures. Patients are viewed not as generic but unique, and as experts on their own bodies, symptoms and situations. Their experiences are acknowledged through a reciprocal relationship with caregivers. This approach to care has been shown to promote better health outcomes and improved quality of care. It has also been shown to promote satisfaction and efficiency, and thus also decreased health-related costs. Patient-centered care has also been associated with better adherence to treatment, better health literacy, and healthier behaviors (Castro et al. 2016).

Patient empowerment interconnects with participation and patient-centered care, but also has political and philosophical underpinnings. It is historically rooted in social action, aiming to empower subjugated parties. It aims to raise patients from subjects of paternalistic healthcare systems, to become self-determining equals partaking in the decisions affecting their health and quality of life. This is moreover done according to what they themselves consider important. Lack of knowledge and lack of control over one’s own body are seen as the most important reasons for patients feeling powerless. Empowering patients helps them reinterpret their condition through perspectives of self-determination and enhanced control. As such, patient empowerment aims to improve patients’ quality of life beyond just health outcomes (Castro et al. 2016).

Assuming a patient-centered perspective on drug treatment suggests exploring patient needs for information related to drugs. Kusch, Haefeli and Seidling (2018) investigated patients’ personal desires related to drug information beyond the basics. They did a systematic literature review, sourcing studies on patients’ drug information needs as well as on patient inquiries to drug information services. Topics of interest were identified, and frequency of occurrence was calculated. The results consistently showed adverse drug reactions and drug-drug interactions as the most desirable topics. Discussing their findings, they indicate the need for making drug information accessible to patients. To do so, they suggest building information databases based on patient-oriented topics such as those identified in the study (Kusch, Haefeli & Seidling 2018).

In Scandinavia, a lot of work has been done to develop information systems on drug information and drug interactions. Such systems help healthcare professionals navigate large and constantly revised masses of drug information. Böttiger et al. (2009) discuss the Swedish drug interaction database SFINX, predecessor of the current Janusmed. Its database was integrated with both Swedish and Finnish healthcare, thus serving thousands of physicians and pharmacists. A subsequent study looked at the impact of SFINX in primary healthcare centers. The study compared 15 centers where the system had been implemented, with 5 centers where it had not. The results showed a significant reduction in prescriptions leading to serious drug interactions (Andersson et al. 2013).

laypersons were also using the service. Nörby et al. (2015) investigated pregnant women’s views on using the Drugs and Birth Defects section of Janusmed. They found 11 % of participants already familiar with the service in advance of their participation. They also found participants using several other information sources on drugs related to pregnancy. Some were found to become more anxious after reading the content, but the vast majority considered the information to be of great personal value and easy to understand. They also found it confirming and supporting information provided by healthcare professionals. In their discussion, Nörby et al. (ibid.) suggest that communication and patient compliance improves when patients and professionals refer to the same sources. They also note that risks of providing content to laypersons seems small, considering that most will in any case find it from other providers (Nörby et al. 2015).

In healthcare, mHealth is meanwhile emerging as a promising means of delivering patient-oriented healthcare outside hospitals. mHealth is short for mobile healthcare, meaning the delivery of healthcare services and information through mobile devices. The mobile platform allows flexible and ubiquitous access to online and offline services and information. It therefore lends itself in particular to outpatient purposes, and to individualized patient-oriented services (Reolon et al. 2016). mHealth applications have been found to predominantly be implemented through experimenting with technologies rather than through strategic planning (World Health Organization 2011). Yet these systems impact human lives at large scale, potentially reaching millions of users, and directly affecting their health. Just as it is imperative that information be accurate, it is also imperative that mHealth services not lead to user error, and thereby to adverse health effects (Reolon et al. 2016). Usability in critical systems can be a matter of life or death. Nielsen (2005) commented on Koppel et al. (2005) identifying 22 ways that automated hospital systems could result in the wrong medication being dispensed to patients. He identified most of the problems as well-known usability issues that have been understood for decades. Reolon et al. (2016) corroborated this by showing that studies on healthcare systems in general have found significant usability problems inviting a range of human errors. These also had the potential of leading to injury or even death. Within mHealth, they also found a range of usability problems, as well as a lack of user-centered design. They also argue that patient-oriented systems are in particular need of usability. This was based on the likelihood of lack of training, and a significant proportion of elderly users (Reolon et al. 2016).

1.2 Research Objective

expert-based checklist evaluation on drug-drug interaction websites. This study measured information capacity, patient usability and readability. However, no studies appear to have been done on quality of user interfaces in terms of usability for patient-oriented drug-drug interaction checkers.

The objective of this thesis is therefore to explore the prevalence and characteristics of usability issues facing patients using publicly available drug-drug interaction checkers in the Scandinavian market. This is done to facilitate development of this emerging market, and to bring awareness to usability issues facing users today. 1.2.1 Research Questions

To contribute to patient-oriented practices, healthcare efficiency and patient empowerment, two research questions will be guiding the study:

1. What services identifying drug interactions are available to the Scandinavian public today?

2. What is the prevalence and characteristics of usability problems in these services?

1.3 Delimitations

The context explored in the thesis has been delimited to the Scandinavian market. This has been done to facilitate delivery of relevant and actionable insights. It has also been delimited to services accessible on mobile platforms. This was done to provide a focus recognizing the promises of mHealth for patient-oriented and outpatient uses. It was also done to stress usability requirements, considering that mobile platforms place additional demands on usability. Functionally oriented checkers were given particular focus as opposed to handbook-style references in regard for their aptness for easy on-the-go use by patients, and also for implementation within varying use contexts and types of services.

The thesis aims to investigate usability through issue-based metrics, qualitatively describing usability problems as experienced by users. It does not aim to provide generalizable findings beyond the context of the case study, which could have been achieved through other metrics and study designs. This choice has been done in accordance with frameworks for usability study design (Tullis & Albert 2013).

1.4 Outline of the Thesis

2 Literature Review

In this section, a literature review is presented as a theoretical background contextualizing the thesis. Patients’ needs for drug information is initially explored, identifying drug-drug interactions as a particularly important topic. Usability is then defined and discussed in terms of metrics and methods of evaluation, and then in terms of reliability. Finally, previous usability studies on mHealth systems and drug-drug interaction checkers are presented, leading up to a discussion of the state of the art of heuristic evaluations in mHealth.

2.1 Patients’ Needs for Drug Information

Patient-centered healthcare aims to empower patients and involve them in decision-making. This has been shown to lead to better engagement and adherence to treatment, causing better health outcomes. Allowing patients to participate requires also informing them according to their abilities and needs. As a means of promoting patient-centered healthcare, Kusch, Haefeli and Seidling (2018) did a scoping review to investigate patients’ needs related to drug information systems. Basic drug information is often straightforward, but Kusch, Haefeli and Seidling (ibid.) aimed to investigate which drug information patients demanded beyond this, and how this information could be personalized according to individual needs.

such as those identified by the authors themselves. Too often, patients are left with leaflets presuming that one size fits all, but the authors believe a more flexible approach is necessary. Kusch, Haefeli and Seidling (2018) add that extensive research has been done on system quality of professional drug-drug interaction checkers regarding content, design and use, but that little has been done on similar services directed at patients (Kusch, Haefeli & Seidling 2018).

2.2 Usability and Heuristic Evaluations

Definitions of usability continue to vary among sources, generally referring to the ease of use of an information system. Nielsen (2012) defines usability by five components: learnability, efficiency, memorability, [absence of] errors, and satisfaction. Learnability refers to the ease with which users are able to learn the system. Efficiency denotes how little effort is needed to execute its tasks. Memorability refers to ease of regaining efficiency after non-use. [Absence of] errors refers to preventing, minimizing consequences, and facilitating recovery from errors. Satisfaction meanwhile denotes the pleasurability of its use. In defining usability, Nielsen (2012) deliberately omits utility, meaning the extent to which the system can help users achieve their goals.

Sharp, Preece and Rogers (2019) on the other hand include utility, adding also effectiveness. Effectiveness denotes the degree to which a system helps users fulfill their goals. On the other hand, they see satisfaction as a part of user experience rather than usability. Roto et al. (2011) emphasize that usability relates to objective performance as opposed to subjective interpretation and meaning. With them the subjective experience of usability is one of several components of user experience. The main industry standard of usability today is however ISO 9241-11, defining usability as:

[…] the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.

(ISO 9241-11 2018)

It emphasizes use context, reflecting a user-centered perspective, and includes aspects of both attainment of goals and objective and subjective performance. This is the definition that will be relied upon in this thesis.

2.2.1 Measures of Usability

Tullis and Albert (2013) provide a framework for quantifiable metrics for usability by identifying five categories of metrics: performance metrics, issue-based metrics, self-reported metrics, behavioral and physiological metrics, and combined and comparative metrics.

questionnaires and/or interviews, often following user testing. Behavioral and physiological metrics refer to eye tracking and physical measurements of emotion and stress through for example heart rate or skin conductance. Combined and comparative metrics refer to metrics derived from two or more metrics, or from comparing measurements to benchmark results.

In terms of issue-based metrics, a usability issue is defined as:

[…] a flaw in the design of a system that makes the attainment of a particular goal with the use of the system ineffective and/or inefficient, and thus lowers the user’s level of satisfaction with its usage.

(Law & Hvannberg 2002, p. 71)

They are qualitatively described, empirically observed usability problems such as system behaviors preventing task completion, misdirecting the user, causing frustration or causing disorientation. Issue-based metrics refer to quantitative applications of usability issues, such as frequency of unique issues, number of issues by heuristic, or number of issues by task, and so on (Tullis & Albert 2013).

2.2.2 Methods for Evaluating Usability

Sharp, Preece and Rogers (2019) categorize evaluation methods according to setting and level of control. Methods involving users in natural settings include field studies, and are particularly useful for open-ended exploration of motivations, contexts and explanations for why usability issues occur, as opposed to simply stating that they do. At the same time, they are time-consuming and generally not generalizable. Methods involving users in controlled settings include user testing and experiments. Controlling settings is a means of isolating variables to enable measurement and hypothesis-testing. This involves risk of loss of context, potentially hiding underlying reasons for the observed phenomena. Methods not involving users include different kinds of inspections, walkthroughs, modellings and analytics. These methods tend to rely on the experience and knowledge of experts in anticipating usability problems, rather than observing them empirically. Such methods are generally found particularly cost-effective (Sharp, Preece & Rogers 2019).

Usability testing is often cited as the “gold standard” of usability evaluation. It involves observation of users interacting with the system according to specified tasks while verbally explicating their thoughts. This is often accompanied by task performance measurements and post-session interviews or questionnaires elucidating self-reported data. Usability testing as such lends itself to gathering rich and complementary data on both efficiency, effectiveness and satisfaction (Sharp, Preece & Rogers 2019).

advantage of evaluator expertise is regarded as a cost-efficient means of bypassing the need for user recruitment, making heuristic evaluations a particularly cost-efficient alternative or complement to usability testing (Wilson 2014).

2.2.3 Reliability of Heuristic Evaluations

Tullis and Albert (2013) show a prevailing discussion on the number of evaluators necessary for reliable identification of most of a system’s usability issues. They describe two camps, both with compelling evidence: those who regard five evaluators sufficient, and those who believe that far more are needed.

Nielsen and Landauer (1993) in a seminal study reviewing eleven usability studies, found the probability of an issue being detected by an expert evaluator to be about 30 %. Based on this, they developed a mathematical model for the probability of an issue being detected as a function of the number of evaluators, predicting five evaluators to identify more than 80 % of usability issues. Optimal cost-benefit ratio was generally estimated at three to five evaluators. Nielsen (2012) reiterated this recommendation in 2012, refuting several critiques and presenting only a tiny correlation for number of findings with number of participants across 83 recent consulting projects performed by the Norman Nielsen Group. He emphasizes that his focus is on qualitative insights driving design improvements, rather than quantitative measurements for summative evaluation. He also emphasizes the cost-effectiveness of several studies with fewer participants, rather than fewer studies with more participants (Nielsen 2012).

On the other hand, looking back at 20 years of CUE-studies, Molich (2018) summarizes what they indicate about reliability, and their practical implications. The CUE studies have surveyed the state of the art of usability testing and investigated the reliability of findings. More than 140 professional teams participated throughout the studies. In each, multiple teams evaluated the same system using the same scenarios and objectives. They did so according to their own standard procedures, submitting usability reports anonymously. Reporting was followed by workshops discussing differences and ways to improve evaluation procedures. Molich highlights several findings, the foremost being that each system will have a number of usability issues beyond what can be identified by any single study. The CUE studies consistently found several hundreds of usability issues, a large proportion of which were only reported by single teams. The studies moreover show that even 15 teams of professionals only find a fraction of all usability issues, clearly contradicting the claim that five users will identify as many as 85 % of issues. Molich meanwhile emphasizes his support for five participants driving a useful process of iterative design. He adds that findings were rarely deemed false alarms, and that the studies moreover discredit usability testing as being a “gold standard”. CUE-4 (Molich & Dumas 2008) in particular showed that expert reviews such as heuristic evaluations produce comparable results to usability testing (Molich 2018).

encompass every issue. He also suggests practitioners consider the CUE studies to reflect upon and refine their procedures, treating usability studies as a formal science rather than an artistic activity (Molich 2018).

2.3 Previous Usability Studies in mHealth

Sharp, Preece and Rogers (2019) demonstrate the increasing recognition of usability testing in mHealth, referencing Schnall, Cho and Liu (2018) validating the Health-ITUES instrument for mobile and patient-oriented context. The Health-Health-ITUES is a post-session questionnaire following usability testing developed to enable quantitative evaluation of self-reported ease of use. Subsequent studies have implemented the Health-ITUES in combination with other methods: Stonbraker et al. (2018) evaluated an mHealth application for self-management of HIV symptoms, combining heuristic evaluation with usability testing. Heuristic evaluations were performed by observing informatics experts “thinking aloud” while performing user scenarios. Upon completion, the experts completed a heuristic evaluation checklist, rating the system as a whole according to Nielsen’s usability heuristics. Heuristic evaluations were followed by usability testing with representative users, who afterwards completed questionnaires on health literacy levels, self-reported ease of use (the Health-ITUES), and perceived satisfaction. Results showed low (good) severity ratings for each heuristic, user control and freedom being in greatest need of improvement. Representative users gave high ratings for ease of use and satisfaction, and the application was in way of conclusion deemed usable (Stonbraker et al. 2018). Beauchemin et al. (2019) also evaluated an mHealth application for self-management of HIV symptoms, performing usability testing with both representative users and informatics experts, several of which were also healthcare professionals. After completing a set of 26 tasks, both groups completed surveys on demographics, as well as the Health-ITUES. Usability testing was complemented by cognitive walkthrough with end-users testing two devices linked to the application. The results identified usability issues to address, but also confirmed usability of the application and linked devices (Beauchemin et al. 2019).

Adam and Vang (2015) did a content and patient usability evaluation on patient-oriented drug-drug interaction websites specifically. The purpose of the study was to investigate whether information on publicly available websites was valid and accurate, as well evaluating their ability to communicate information to laypersons. Information capacity, patient usability and readability were investigated using an expert based approach with grading scales.

Websites with drug-drug interaction checkers were sampled using the Google search engine, supplemented by suggestions by clinical practitioners. Sites had to be freely available to end-users without payment and had to provide drug-drug interaction checking functionality capable of identifying the drugs applied in testing. Sites in languages other than English were excluded (Adam & Vang 2015).

scale, medication pick list, and ability to store a profile account. Patient readability was evaluated using the Flesch-Kincaid grading model, as well as the Flesch Reading Ease score (Adam & Vang 2015).

Discussing the results, the authors concluded that the majority of websites was not optimally patient-oriented due to poor information capacity and limited readability. They found many features missing in many of the checkers and found information difficult to interpret. Half of the websites did not provide any severity ratings, while those that did demonstrated substantial variations in doing so (Adam & Vang 2015). The authors themselves recognized the lack of validation and consensus on the grading scales applied. They also highlighted the lack of patient involvement, pointing to practical problems of involving patients with their actual drug regimens. They also found further research necessary to improve patient-oriented drug-drug interaction information systems (Adam & Vang 2015).

2.4 Heuristic Evaluations and Heuristics Applied in mHealth

Highlighting heuristic evaluations for their cost-efficiency, Reolon et al. (2016) investigated the state of the art of heuristics-based usability studies in mHealth. They argue the need for adapting heuristics for both mobile platform and the healthcare domain, aiming to investigate to what extent such adaptations have been made. A systematic literature review was conducted, including only peer-reviewed articles published in journals or conference proceedings between 2007 and 2014. All papers mentioning heuristic evaluation of mHealth applications were included, excluding applications not geared towards healthcare, not towards mobile platform, or not applying heuristic evaluations. Applying the selection criteria on title, keywords and abstract, and subsequently through full-text analysis, pared results down to only seven articles. This showed a clear need for further research (Reolon et al. 2016).

Contrary to the authors’ expectations, the results showed the articles predominantly applying traditional heuristics such as those of Nielsen. Four out of seven applied them as-is, or only slightly adapted. Only one study applied heuristics adapted specifically for mHealth context (Monkman & Kushniruk 2013), while another two applied Nielsen’s heuristics adapted for mobile context (Bertini et al. 2009). Several did not motivate the selection of heuristics (Reolon et al. 2016).

3 Methodology

This section begins describing the study design, before proceeding to describe methodology and procedures followed in gathering and analyzing data. It describes the motivation for applying heuristic evaluations, as well as the tasks, the severity rating scale and the set of heuristics applied. It also describes means taken to ensure validity and reliability of the results.

To attain reliable and valid results, emphasis was placed on referring to frameworks for planning and conducting usability studies. Wilson (2014) suggests a framework for methodology based on four criteria: phase of product development, usability attributes sought, time and resources for training, and need for user involvement. Tullis and Albert (2013) along the same lines suggest explicating goals of the study, goals of the users, and choosing usability metrics according to a framework of prototypical usability studies. The scope of this thesis has been to investigate user interface usability as experienced by a general public. Following from the research objective, the aim has not been to assess systems according to any benchmark or predefined goal, nor to generate any kind of ranking. Rather, it has been to provide an explorative account, indicating actionable areas for improvement. Even if the services evaluated were published, the market of drug-drug interaction checkers as a whole was considered immature, and evaluations were considered formative in view of the goal of driving design improvement.

According to Tullis and Albert’s (2013) framework, the prototypical usability studies deemed most relevant for this thesis were problem discovery and comparing alternative designs. Problem discovery aims to explore usability in an open-ended manner, providing quick and actionable suggestions for design improvements based on usability issues. Comparing alternative designs likewise suggests issue-based metrics, but also performance and self-reported metrics. These would however require usability testing involving significant recruitment for reliability. The choice was therefore made to focus on usability issues and issue-based metrics exclusively.

3.1 Study Design

Figure 3.1. Model of the convergent mixed-methods study design.

The mixed-methods design was applied using heuristic evaluations for quantitative and qualitative data collection. The data collected was analyzed through descriptive statistics and single-case and cross-case qualitative analyses. The procedures applied are described in what follows.

3.2 Sampling of Drug Interaction Checkers

To answer the first research question of which interaction checkers are available to the Scandinavian public today, a sampling was performed. To identify publicly available drug information systems likely encountered by patients, the current leading health category websites were identified for each country. These were then used to find popular matches using the Similarsites.com website similarity service (SimilarGroup, Ltd. 2020). Websites providing a match of more than 80 % similarity to the sites based on category, content, word usage, and search terms were selected. The results were supplemented by websites suggested by consulting clinical practitioners in Denmark, Norway and Sweden.

Several exclusion criteria were then applied. The resources had to be freely available to the public, had to provide drug interaction functionality, and had to be aimed at Scandinavian audiences. In order to figure out which websites fit the criteria each website’s main page was examined for information about drug-drug interactions. The website’s internal search was then queried for the terms “interaction” and “drug interaction” in the local language. Search results were then reviewed for resources on drug-drug interactions.

3.3 Heuristic Evaluations

Table 3.1. Goal oriented tasks applied in heuristic evaluations.

Task 1: Find out whether drug X and drug Y produce adverse interactions Task 2: Find out which adverse reactions drug X may produce.

(I.e. with or without any other drug or substance.)

Task 3: Find out which adverse drug interactions a patient is exposed to, based on a list of multiple drugs.

Systems were evaluated by each evaluator independently in randomized order. The tasks were executed by the evaluators, who inspected the user interface step by step, supported by the heuristics along with short descriptions. Durations of sessions were limited to a maximum of four hours daily to prevent results being affected by fatigue. Usability issues were recorded in a table, along with screenshots identifying location of the issue. Upon completion of all tasks, severity ratings were given. After having evaluated each of the systems by each task, the evaluators independently reviewed the data across all systems to ensure consistency of severity ratings and mappings to applicable heuristics.

An explicit evaluation protocol (Appendix 1) was shared between the evaluators to facilitate uniform procedures. The evaluation protocol included the set of heuristics applied along with a short description, and a table for recording usability issues according to predefined variables. Usability issues identified were noted with system name, task, description, applicable heuristics, screenshot locating the issue, as well as severity assessment.

By way of researcher triangulation, the researchers met over two sessions to discuss the issues identified, consolidating duplicates and coming to agreement on mappings to heuristics, description, and severity ratings for each issue. The scheme for severity ratings followed Nielsen (1994), as described in Table 3.4.

Table 3.2. Scheme of severity rating applied.

Rating Description

0 I don't agree that this is a usability problem at all.

1 Cosmetic problem only: need not be fixed unless extra time is available. 2 Minor usability problem: fixing this should be given low priority.

Drug pairings were adapted from Adam and Vang (2015) who identify five clinically significant drug-drug interaction pairs with a broad coverage of key therapeutic areas. A medical practitioner was consulted to confirm the relevance of the drug pairings for Scandinavian contexts, as well as to provide a clinically non-significant pair to allow testing of use cases with non-interacting drugs. Task 1 was to be performed with each of the drug pairings. Task 2 was performed with only the drugs from the first column. Task 3 was performed using all drugs in combination, as well as using only Pair 6 for a no-interaction use case. The pairings are presented in table 3.2.

Table 3.3. Drug pairs applied in evaluated tasks.

Drug X Drug Y

Pair 1: Simvastatin Itraconazole

Pair 2: Warfarin Gemfibrozil

Pair 3: Levothyroxine Warfarin

Pair 4: Fluoxetine Selegiline

Pair 5: Selegiline Phenelzine

Pair 6: Cholecalciferol Paracetamol

There were thus six variations for Task 1 and 2, and two variations for Task 3, adding up to 14 tasks, albeit with considerable redundancy expected.

The mHealth heuristics developed by Monkman and Kushniruk (2013) were deemed too restrictive and too extensive for efficient application, causing the evaluators to prefer the traditional heuristics of Nielsen (Table 3.3).

Table 3.4. Nielsen’s Usability Heuristics, based on Nielsen (1994).

ID Heuristic

H1 Visibility of System Status

The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.

H2 Match Between System and the Real World

H3 User Control and Freedom

Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.

H4 Consistency and Standards

Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.

H5 Error Prevention

Even better than good error messages is a careful design which prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action.

H6 Recognition Rather than Recall

Minimize the user's memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.

H7 Flexibility and Efficiency of Use

Accelerators – unseen by the novice user – may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.

H8 Aesthetic and Minimalist Design

Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.

H9 Help Users Recognize, Diagnose, and Recover from Errors

Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.

H10 Help and Documentation

Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.

3.4 Quantitative Analysis

across all systems evaluated. Number and proportion of issues by applicable heuristic was likewise calculated single- and cross-case. Finally, the likelihood of an evaluator detecting an issue was calculated according to Tullis and Albert (2013, pp. 116–117), as a means of indicating reliability of the findings. Bar chart visualizations were used to facilitate identification of emergent patterns in the data.

3.5 Qualitative Analysis

For qualitative analysis, the data was prepared by cleaning, coding and sorting usability issues. Issues were coded by reading through all issues pertaining to each system, before going through them again to categorize them by heuristic applied, as well as inductively by theme. Themes were finally revised and merged if necessary. Emergent patterns were identified, and results were compiled as a narrative usability review for each system.

4 Results and Analysis

In this section, empirical findings and analyses are presented. Descriptive statistics are first applied to identify patterns in the data. Qualitative analysis is then applied to check and explore these at depth. A cross-case analysis is first presented to com-pare and contrast findings across services. Single-case analyses follow, highlighting patterns and themes for each individual service. The full dataset of identified usability issues is available in Appendix 2.

4.1 Description of Drug Interaction Checkers

The sampling of drug-drug interaction checkers across the three Scandinavian countries of Denmark, Norway and Sweden produced a result of six eligible services. The two Danish checkers, Medicinkombination.dk and Interaktionsdatabasen.dk were found functionally identical. To avoid redundancy in the evaluations, the latter was excluded, while Medicinkombination.dk, serving patient-oriented content, was included. The resulting services were the five presented in Table 4.1.

Table 4.1. The drug-drug interaction checkers sampled for the thesis.

Name URL Janusmed https://janusmed.sll.se/ Interaksjoner.no https://www.interaksjoner.no/ Felleskatalogen.no https://www.felleskatalogen.no/m/medisin/interaksjon Legemiddelsok.no https://www.legemiddelsok.no/sider/Interaksjoner.aspx Medicinkombination.dk http://medicinkombination.dk/

Simplified sitemaps were generated to visualize system structure as well as relation-ships to internal and external resources. Summaries are also provided, describing each service with a focus on identifying the organizations responsible for maintenance of the service, its database, its target audience, and a general description of the system and its task flow.

Janusmed

Janusmed is an integration of a number of evidence-based knowledge and decision support systems provided by Region Stockholm, the organization responsible for all publicly financed healthcare in Stockholm County. It is offered both as an integration with the Swedish electronic medical record system, and publicly available as an online checker tool associated with Janusinfo.se, the general drug information resource provided by Region Stockholm.

interactions, risk profile, effects on fetuses and effects on breastfeeding. All subsystems are accessed through the integrated search interface. Access to results for effects on fetuses and breastfeeding is openly available, whereas interaction results requires users to sign in. Accounts can however be instantly created without special authorization, but users are asked for their location as well as occupation. Risk profile is available only to users connected to Region Stockholm’s SLLnet, and is as such not available to the public.

The search interface provides feedback and result listings as dynamic sections of the page, with some subsection pages launching in new browser tabs, and some also in modal panels (Figure 4.1).

Figure 4.1. Sitemap for Janusinfo.se

Janusmed states its goal of being accessible to all users, irrespective of disabilities, impairments and aids. It recognizes shortcomings in relation to Swedish regulations demanding accessibility of digital public services, stating partial compliance with WCAG 2.1 based on internal testing. It also states ongoing work to address known accessibility issues.

Interaksjoner.no

Interaksjoner.no is today privately owned and maintained, acquired in 2015 by DIPS AS, a private company specializing in the development of eHealth applications for Norwegian hospitals and municipal healthcare providers.

The service is a standalone website, with a clear and minimalist interface. Users are provided a free text input text area and basic form controls. Results are listed on a separate page, preserving the input form for iterative searches. Result listings can be clicked, leading to a detailed description of each interaction (Figure 4.2).

Figure 4.2. Sitemap for Interaksjoner.no

Felleskatalogen.no

Felleskatalogen AS is a private company owned by the Association of the Pharma-ceutical Industry in Norway. The members of the association are Norwegian and international pharmaceutical companies developing, producing, selling or marketing pharmaceuticals for the Norwegian market.

Felleskatalogen is a reference work published by Felleskatalogen AS, aiming to provide medical practitioners with comprehensive data on commercially available drug preparations in a handbook format. It was made available for sale to public audiences in 1984 and is today primarily delivered free of charge through its open online platform.

Figure 4.3. Sitemap for Felleskatalogen.no

Legemiddelsok.no

The Norwegian Medicines Agency (NoMA) is the Norwegian national regulatory authority supervising trials, approval, production, distribution, pricing and marketing of medicines for the Norwegian market. It also has the responsibility of providing medical practitioners and the general public with information and guidance for safe use of drugs.

FEST is a drug information database provided by NoMA targeting medical practition-ers and pharmacists. It informs them with comprehensive quality controlled and continuously updated information about prescription drugs on the Norwegian market and is implemented through electronic health record systems within the Norwegian general practitioner system, municipal healthcare, and some Norwegian hospitals. Access to the database and documentation is provided by NoMA for third party implementations, and the database provides the basis for both Interaksjoner.no, Felleskatalogen.no and Legemiddelsok.no.

Figure 4.4. Sitemap for Legemiddelsok.no

Medicinkombination.dk

Medicinkombination.dk is a public drug interaction checker made available by the Danish Medicines Agency (DKMA). DKMA is an agency under the Danish Ministry of Health and Prevention, and is Denmark’s regulatory authority supervising production, trials and marketing of medicines for the Danish market.

Interaktionsdatabasen is a drug information database developed from 2001 to 2003 as a cooperative between the Association of Danish Pharmacies, the Danish Medical Association, Dansk Lægemiddel Information A/S and the Danish Institute for Rational Pharmacotherapy. DKMA took over the database in 2003. The goal of Interaktionsdatabasen is to promote quality, efficiency and safety in drug therapy by providing authoritative evidence-based information on drug interactions for healthcare professionals. Access and documentation are provided by the DKMA for third party integration, just like the Norwegian FEST database.

Medicinkombination.dk was launched as a sister site of Interaktionsdatabasen.dk in 2007, targeting the general public of laypersons. The sites are identical in design and visual identity, which has potential to confuse users. But whereas the latter targets healthcare professionals, Medicinkombination.dk serves patient-oriented content through an identical interface. The service is marketed to the public through for example Sundhed.dk, the official Danish public healthcare website.

preserved upon submission, requiring users to traverse the browser history for new searches.

Figure 4.5. Sitemap for Medicinkombination.dk

4.2 Quantitative Analysis

Having accounted for drug interaction checkers available to the Scandinavian public, descriptive statistical analysis was applied based on the quantitative data derived from heuristic evaluations. The analysis revealed a total of 165 usability issues across all services. The majority of these were classified as minor issues, but many were also classified as major, and three were even assessed as catastrophic due to their potential for harm to human health. In total, 25 cosmetic, 114 minor, 23 major and 3 catastrophic issues were detected (Figure 4.6).

Looking across the services evaluated, shows Janusmed having a total of 58 usability issues – about twice the mean of the sampled services. Interaksjoner.no was on the other hand found having only 17 usability issues, whereas the rest were closely distributed around the mean (Figure 4.7).

Figure 4.7. Distribution of usability issues across services.

The proportion of issues by severity shows high consistency across the services. Minor issues make up the majority of all issues, with most services clustering closely from 67 to 69 percent. Felleskatalogen.no is the primary exception, with very few major issues detected. Its distribution of issues even shifted all the way to cosmetic rather than minor issues, making for notably lower severity ratings overall (Figure 4.8).

Figure 4.8. Proportion of usability issues by severity across services.

real world (H2), and user control and freedom (H3) as the most frequent sources of usability issues (Figure 4.9).

Figure 4.9. Distribution of Issues by Heuristic

Breaking this up by service, again generally shows high consistency (Figure 4.10 and 4.11). Interaksjoner.no showed noticeably fewer issues related to consistency and standards (H4) and aesthetic and minimalist design (H8), but also more issues related to flexibility and efficiency of use (H7). Janusmed, Legemiddelsok.no and Medicin-kombination.dk on the other hand showed more issues regarding consistency and standards (H4) and aesthetic and minimalist design (H8).

Figure 4.11. Proportion of Usability Issues by Heuristic across Services 2.

The patterns identified through descriptive statistical analyses were supplemented through qualitative cross-case and single-case analyses.

4.3 Cross-Case Qualitative Analysis

Four major themes were identified across the services: lack of adaptive design, lack of patient-oriented content, system complexity correlating with amount of usability issues, and a lack of following conventions, design principles and patterns.

Lack of Adaptive Design

The first was a lack of adaptive design, meaning that layouts and interfaces of several of the services would not adapt to different screen sizes (Figure 4.12).

Only Interaksjoner.no and Felleskatalogen.no had well-functioning adaptive layouts. Legemiddelsok.no and Medicinkombination.dk both made no effort to accommodate mobile usage of their services, resulting in major issues pertaining to efficiency and ease of use, as well as aesthetic design. Janusmed is a particular case here, as it does in fact have an adaptive design, but still fails to avoid typical issues of non-adaptive designs, such as zoom and viewport management.

Lack of Patient-Oriented Content

The second theme was a lack of patient-oriented content. Of all the services evaluated, only Medicinkombination.dk had content clearly aimed at patient audiences. All the Norwegian services retrieved interaction data from the same FEST database but presented it somewhat differently in terms of order and priority. Its content was found appropriate in extent and complexity, but too reliant on acronyms and technical terms to allow patient accessibility. The content on Janusmed was found both too extensive and too technical for patients, even disregarding acronyms and technical terms. As for Medicinkombination.dk, the content was found accessible for patient readability, although quite short in extent. It was as such not considered as empowering for patient audiences.

More System Complexity Correlating with More Usability Issues

The third theme identified, was a positive correlation between system complexity and the number of usability issues detected. This applied in particular to the contrasting examples of Janusmed and Interaksjoner.no. Janusmed was found very complex in terms of organization, content, patterns of interaction, and layout and typography. Interaksjoner.no was quite the opposite, with simplistic interaction, little functional support, and minimalist layout and typography, to the extent of even hiding information about the database for mobile users. Usability issues for Janusmed and Interaksjoner.no correspondingly numbered 58 and 17, respectively. The three other services likewise placed themselves as moderate in both complexity and number of issues, as seen above in Figure 4.7.

Lack of Following Conventions, Design Principles and Patterns

The final theme identified, was a general lack of implementation of recognized design patterns, conventions and principles. This was manifest across all services. A successful fundamental pattern for users to interact with the systems appeared well established across the market, with some kind of free text input supported by search suggestions, followed by result listings. But in spite of this, implementations would vary, and would err on basics such as failing to provide readable type sizes, failing to provide sufficient touch target areas, using low-contrast grey for highlighting active selection, or failing to clarify relations between content and functionality through visual hierarchy and gestalt principles.

Catastrophic Usability Issues

(Figure 4.13). This indicates to users that the drug is safe from interactions with other drugs, whereas the opposite might in fact be the case.

Figure 4.13. Janusmed indicating the safety of drugs with hundreds of serious interactions.

The second issue related to the same scenario, where complete lack of emphasis and labelling on the indicator for interactions pertaining to single drug items would fail to notify users of the presence of hundreds of potentially serious interactions (Figure 4.13).

These findings showed not only a passive lack of indication, but active indication of safety, which was deemed particularly serious. They show a lack of regard for users’ mental models, answering not the question “What interactions pertain to this drug?” but rather “What interactions happen between this drug and nothing else?”, which is not the user’s intention. Contrasting this with other services, shows other services simply listing all relevant interactions even for single-drug searches.

Figure 4.14. Medicinkombination.dk providing previous results for a current search.

All of the noted catastrophic usability issues were deemed catastrophic due to their potential of leading to harm of human health.

4.4 Single-Case Qualitative Analyses

Having provided a foundation through descriptive statistics and cross-case qualitative analysis, more light will be shed on emergent themes for each of the evaluated services through single-case qualitative analyses. The full dataset for each of the services may be found in Appendix 2.

Janusmed

Janusmed distinguishing itself by a particularly advanced design pattern for its search form, putting direct manipulation to good use. Completed and recognized search terms are instantly transformed to visual objects that can be removed item by item without editing text or handling multiple input fields. Feedback is also provided immediately in place as each term is completed rather than keeping users waiting or traversing pages.

This level of technical capability is reflected more generally in the complexity of the service. Perplexingly, this is accompanied by catastrophic usability issues, potentially leading to harm to human health, as well as a severe lack of regard for fundamental design principles, such as gestalt principles, visual hierarchy, typographic hierarchy, affordances of pliability, coherent interaction idioms, and design according to users’ mental models.

the subsections, inconsistently opening content in new browser tabs with or without navigation, as well as dynamically within the subsection of the page (Figure 4.15).

Figure 4.15. Janusmed showing inconsistent layout, visual hierarchy, and patterns of interaction.

Another particularly disturbing example is that placing focus in the input field will zoom and drag the user’s viewport, hiding instructions and controls. Focus will be triggered by touching any of the label, input field, or even the checkbox below, corresponding to about 20–25 % of the screen real estate. Focus will trigger even upon initiating touchstart, for example when initiating scroll of the page over any of these elements.

Interaksjoner.no

Interaksjoner.no displayed the opposite trends of those of Janusmed, taking a minimalist approach. The interface is adaptive as well as extremely clear and structured in terms of visual and typographic hierarchy. Its interaction idioms are likewise simple, coherent and conventional, based on a non-dynamic page-by-page pattern of interaction. This provides an excellent basis of usability.

Having built a solid foundation, Interaksjoner.no fails however at providing users with necessary means of support. The task of entering long and complex names of drug substances will likely be demanding and prone to error. Meanwhile, searches on Interaksjoner.no are performed by free text entry only, providing no search suggestions or other means of help. In contrast to Janusmed, users are expected to provide exact input up-front. They are also expected to do so according to the system’s implementation model rather than having the system conform to user behavior. It will fail to recognize “vitamin d”, for example, demanding users to enter underscores (“_”) instead of spaces. It will also fail to recognize “vitamin” or “d” individually, or by means of thesaurus or similar functionality (Figure 4.16).

Figure 4.16. Interaksjoner.no with an adaptive and extremely clear design, with clear feedback, and

As such Interaksjoner.no and Janusmed provide contrasting examples: one excelling at the basics but failing to deliver the necessary advanced support; the other having disposal of every means of support but failing to deliver it according to basic principles of design.

This again reflects findings from descriptive statistics, showing Interaksjoner.no having fewer issues due to its minimalist design and success at basic usability. But it is also reflected by a larger proportion of major usability issues stemming precisely from lack of user support.

Felleskatalogen.no

Felleskatalogen.no was together with Interaksjoner.no the only two of the services evaluated to provide a well-functioning adaptive layout, resulting in fewer and less severe usability issues. Its clarity and visual hierarchy were not as strong as that of Interaksjoner.no, but still sufficient to avoid more severe issues (Figure 4.17).

Figure 4.17. Felleskatalogen.no providing a functional adaptive design, yet still breaking with many

conventions and design principles, making for an average count of usability issues.

major usability issues, as indicated also by quantitative analyses (Figure 4.8). Quantitative analyses showed however that the number of issues was nevertheless close to the mean, and not nearly as low as that of Interaksjoner.no. This reflects the pattern of correlation between number of usability issues and the complexity of the system, as identified by the cross-case analysis.

Legemiddelsok.no

Legemiddelsok.no showed a complete lack of adaptive design, utilizing a particularly horizontal layout necessitating considerable viewport management. This is reflected by the quantitative analyses showing particular prevalence of issues with aesthetic and minimalist design, and consistency and standards (Figure 4.10 and 4.11). Legemiddelsok.no also had a notable lack of basic documentation and instructions. An example of this is the complete lack of welcoming instructions, hints or prompts at the primary search form, leaving users self-helped at the entrance. Basic instructions are in fact hidden behind the information icon above the input field but pressing it will launch a modal panel sticking outside the user’s viewport due to lack of adaptive design (Figure 4.18).

Figure 4.18. Legemiddelsok.no offers no welcoming instructions or hints at how to interact with the

Another example highlighting issues relating to adaptive design and lack of efficiency and ease of use, is the case of searching for combinations of multiple drugs. Legemiddelsok.no forces users through a particularly cumbersome ordeal of adding input fields one by one while handling zoom and viewport issues (Figure 4.19).

Figure 4.19. For searches of lists of drugs, users are forced to add new input fields one by one,

zooming and panning the viewport back and forth to access form controls.

Comparing Legemiddelsok.no to other services shows an average amount of issues, and as big a proportion of major issues as those of Janusmed and Interaksjoner.no – considerably larger than that of Felleskatalogen.no.

Medicinkombination.dk

Figure 4.20. Medicinkombination.dk lacking adaptive design in spite of an explicit layperson

audience. Search interface is divided by single-drug and multi-drug query, the latter demanding excessive handling of the interface for multi-drug search.

5 Discussion

In this section, the results are discussed in terms of their characteristics, significance and practical implications. They are also compared to and contrasted with the background provided by the introduction and literature review. Reliability, validity and limitations are finally discussed in a method reflection.

Allowing patients to act as decision-makers regarding their own health requires empowering them with information. The systems conveying information must be usable for empowerment to be able to take place. This thesis explores the prevalence and characteristics of usability issues preventing patients from benefiting from drug-drug interaction checkers available to the Scandinavian audience today. The research questions applied were:

1. What services identifying drug interactions are available to the Scandinavian public today?

2. What is the prevalence and characteristics of usability problems in these services?

Issue-based qualitative and quantitative data were gathered through heuristic evaluations and analyzed through single-case and cross-case analyses to provide answers to these questions.

5.1 Availability of Drug Interaction Checkers

A handful of publicly available services were identified and described, three in Norway, two in Denmark and one in Sweden. The service providers were both national authorities, drug industry associations, as well as privately owned businesses delivering eHealth solutions for profit. All of them served content from national authoritative sources. The two Danish services served content from different databases, but through identical interfaces. Since the thesis is primarily concerned with assessing interfaces rather than content, the professionally oriented service was excluded to avoid redundancy.

shown, this does not stop patients from using such services, but has been suggested to cause anxiety for some proportion of users. It also prevents the full benefits of empowering and engaging patients in regard to their own health.

Although Medicinkombination.dk was very readable, its content was also very brief. This may represent a precautious approach to avoid causing anxiety. But it may also limit positive effects of improved communication with professionals and improved compliance as suggested by Nörby et al. (2015). Heuristic evaluations showed all services applying patterns of progressive disclosure from summarized listings to detailed descriptions. Most would also link to external resources for further reading. This might seem an appropriate pattern to preserve to accommodate the varying patient behaviors towards supplementary drug information identified by Kusch, Haefeli and Seidling (2018).

5.2 Prevalence and Characteristics of Usability Problems

Across the services evaluated, a mean of 33 usability issues were identified per service, adding up to a total of 165 issues. 3 issues were considered catastrophic, 23 major, 114 minor, and 25 were considered cosmetic issues (Figure 4.6). Although numbers of issues cannot be compared across studies, the authors suggest interpreting these as high numbers in view of the simplicity and limited extent of the tasks. Catastrophic issues were obviously notable, identifying issues unacceptable in publicly released services. But the large amount of minor issues may also indicate insufficiently systematic development and testing.