Usability evaluation of electronic dental record systems in Sweden Master thesis

(1)

Usability evaluation of electronic dental record systems in Sweden

A survey among dentists and dental hygienists

Master thesis

Author: Malaz Shelh Supervisor: Mexhid Ferati Examiner: Pauline Johansson Term: VT2020

Subject: Health informatics Level: Master’s degree Course code: 5XN01E

(2)

(3)

Abstract

Electronic Dental Records (EDR) are an important part of dental care in Sweden.

The usability of these records can affect the workflow in dental care organizations.

This study aims to measure the System usability scale (SUS) score of EDRs that are used in dental clinics in Sweden. The study will also investigate the relationship between the SUS score of EDRs and participants’ age, gender, interest in technology, number of patients per workday, professional experience, possible special training to use the EDR, and the period of the training. The study will also rank the most common usability problem in EDRs among the seven possible usability problems included in the questionnaire. The study will present how the participants describe experienced usability problems in the EDRs. The quantitative method constitutes the largest part of this study, while the open-ended questions were used to get a deeper knowledge about some of the usability problems. A digital questionnaire was used in this study to gather data from 115 dentists and 77 dental hygienists who work at various dental clinics around Sweden to get a statistical anchored description about the usability of various EDRs. SUS indicates a low usability level in the EDRs included in the study and a significant negative

correlation between the frequency of using EDRs and usability. The males showed better experience with the usability of the EDRs compared to females. The highest- ranked usability problem was the need for users to spend a long time to document patient cases. The usability problems were summarized into three categories which are: an inefficient user interface, lack of semantic interoperability, and users relying on paper.

Sammanfattning på svenska

Elektroniska journalsystem är en viktig del av tandvården i Sverige, då

användbarheten av dessa system kan påverka arbetsflödet i tandvårdsorganisationer.

Denna studie syftar till att mäta System usability scale (SUS) poäng för olika elektroniska journalsystem som används i olika tandkliniker i Sverige. Studien kommer också att undersöka sambandet mellan SUS-poäng för elektroniska journalsystem och deltagarnas ålder, kön, intresse av teknologi, antal patienter per arbetsdag, yrkeserfarenhet, möjlig specialutbildning för att använda elektroniska journalsystem och perioden för denna utbildning. Studien kommer också att rangordna det vanligaste användbarhetsproblemet i journalsystem bland de sju möjliga användbarhetsproblemen som ingår i frågeformuläret. Studien kommer att presentera hur deltagarna beskriver upplevda användbarhetsproblem i

journalsystem. Den kvantitativa metoden utgör den största delen av denna studie, medan de öppna frågorna användes för att få en djupare kunskap om några av användbarhetsproblemen. Ett digitalt frågeformulär användes i denna studie för att samla in data från 115 tandläkare och 77 tandhygienister som arbetar vid olika tandkliniker runt om i Sverige för att få en statistisk förankrad beskrivning om användbarheten av olika elektroniska journalsystem. SUS indikerar en låg användbarhetsnivå i de systemen som ingår i studien. Vi upptäckte också en signifikant negativ korrelation mellan frekvensen av att använda systemen och användbarhetsnivån. Män visade en bättre upplevelse för användbarhet av systemen

(4)

jämfört med kvinnor. Det högst rankade användbarhetsproblemet var användarnas behov av lång tid för att dokumentera patientfall. Vi sammanfattade hur deltagarna beskriver upplevda användbarhetsproblem i journalsystem under tre kategorier som är: ett ineffektivt användargränssnitt, brist på semantisk interoperabilitet och användare som skriver på en lapp.

Key words

Usability, electronic dental record, dental care, IT in dental care, health informatics, usability problems

Acknowledgments

Thanks to dentists and dental hygienists who gave their time to participate in this study.

Thanks to all managers who facilitated the gathering of data related to this study.

Thanks, Mexhid Ferati for your support during the work with this study.

Thanks, Pauline Johansson for your help to develop the study.

Thanks, Evalill Nilsson, Patrick Bergman, and Tora Hammar for your valuable comments.

(5)

Appendices

Appendix 1 (Letter to the managers of dental clinics) Appendix 2 (The questionnaire)

(7)

1 Introduction

Most healthcare professionals have dealt with the IT systems in some way, and the digital information systems have become an integral part of today’s healthcare. For example, health care professionals use IT to diagnose a disease by using the digital x-ray (1177, 2019), and to manage a disease by using decision support systems (Piovesan, Molino & Terenziani, 2014). Also, they use digital sensors to follow up on the patient's health condition (Ganchev Ganchev, Garcia, Dobre,

Mavromoustakis & Goleva, 2019), prescribe medications by e-prescription (Vårdgivarguiden, 2019), and finally document all these steps in an Electronic Health Records (her). EHRs are tasked with improving efficiency in healthcare organizations and to improve the quality of healthcare by making patient's medical history more accessible and visible to healthcare professionals (Sidek & Martins, 2017).

Dentistry is an essential part of health care, and the Electronic Dental Records (EDR), becomes a crucial part of dentistry practice in Sweden. Dental professionals use EDRs to document patient health conditions by using two different ways. The first one is a structured data entry, so dental professionals fill information about the patient in an electronic form which is a part of the EDR. The second way is to use a free text to document performed treatments, patient's desires about a treatment, information which is given to the patient, medications, etc. In addition to documentation, the EDR contains a patient’s dental X-ray archive, a decision support system to help dental professionals with diagnosis, prognosis, risk assessment, and choose the best treatment for the patient. Also, it contains the function of sending e-prescriptions and e-referrals, and finally charging the patient and insurance (Försäkringskassan) for the treatment cost (Frenda AB, n.d.; Muntra AB, n.d.).

1.1 Dentistry and eHealth in Sweden

The first law regulation for health care, including dental care in Sweden, came in 1663 and was issued by Collegium Medicum, which after several name changes is now the National Board of Health and Welfare (Socialstyrelsen in Swedish). The first modern law adjustment came in 1861. This gave the practicing dentists a formal basis for the education and described the requirements for the examination.

The law also admitted that women could be dentists because dental care was considered as a craft and not an academic profession (Lindblom, 2004). Swedish Public Dental Care (Folktandvården) started in 1938 and was the first publicly organized dental organization with the task of providing dental care to all school children in Sweden (Ordell, 2012). The public dental insurance started in 1974 and at the same time, the county councils were given the mandatory assignment to arrange free public dental care for all children and adolescents up to the calendar year they turned 19 years. The public dental insurance meant that the Swedish Social Insurance Agency paid for a part of dental treatment cost for adult patients (SOU 1979:7). A high-cost protection system is a part of the public dental insurance

(8)

for adults now, the Swedish Social Insurance Agency pays for 50% of the dental care costs between €321 to €1590 and 85% of costs exceeding €1590 (Kravitz, Bullock, Cowpe & Barnes, 2015). Ordell (2012) describes that today’s dental care in Sweden is controlled politically from two levels, the national level which sets the external borders through laws and regulations, and the locally elected county council politicians in the 21 Swedish county councils. These are responsible for the accessibility and delivery of dental and health care of the population.

The Government and Sweden's municipalities and county councils have an ambitious vision for the development of eHealth and digitalizing of healthcare in 2025, eHealth vision 2025 starts from two basic perspectives which are equality and efficiency. It means that eHealth services must support equality which is the main principle of Swedish healthcare and welfare, from another side eHealth services, must also enhance efficiency in healthcare organizations. The vision of eHealth 2025 focuses on the quality of eHealth services in which they have to increase the digital concern between the population and to gain an acceptable level of

availability, usability, as well as privacy and information security

(Socialdepartementet & SKL, 2016). Dental care is a part of healthcare which is included in eHealth vision 2025, and this study about usability in EDRs can describe our location today from eHealth vision 2025 regarding usability in the EDRs.

1.2 Documentation of dentistry in Sweden

The law of patient data has been applied in 2008 in Sweden, which replaced other laws. This law regulates various issues including documentation and confidentiality in healthcare and dental care (SFS 2008:355).

The law on patient data (SFS 2008:355) describes the obligation to carry on patient records, which means that all health professionals including dental care

professionals are responsible to document all relevant facts about patient’s health.

The purpose of driving a patient record is primarily to contribute to good and safe healthcare for patients. However, a patient record can also be a source of

information for the patient, inspection and legal requirements, follow-up and development of the healthcare organizations and research purpose (SFS 2008:355).

The law of patient data (SFS 2008:355) explains that a patient record should always contain:

- Information on the patient's identity.

- Information about why the patient seeks medical care.

- Information about a diagnosis and indications of treatment steps.

- Information about the taken and planned treatments.

- The information which has been provided to the patient or the relatives about the illness.

- Information if the patient refuses a certain type of healthcare or treatment.

- Information about the identity of the healthcare professional who writes all data.

The law describes that all this information must be written and saved in patients' journals as soon as possible on the date and time when the patient visits a healthcare

(9)

organization. Based on the requirements made by this law, the EDR must be easy to use, easy to learn, flexible, and usable (SFS 2008:355).

1.3 EDRs as a part of dental informatics

Dental informatics is the application of computer and information science to improve dental practice, research, and program administration (Sittig, Kirshner &

Maupomé, 2003). Dental informatics has developed significantly since the 1960s, when the first uses of informatics approaches to address dental issues were documented (Schleyer, 2003). In 1984, 11% of USA dentists used computers in their offices, with rates increasing to more than 85% of 166,000 dentists in the USA by 2009. A substantial growth in the adoption of clinical computing and EDRs by USA dentists has occurred, especially within the last decade. A survey of dentists conducted by the American Dental Association (ADA) from 2006 to 2007 reported further expansion of rates to 55.5% with chairside computers and 9.2% paperless clinics. A 2010 survey of California dentists showed that 23% had fully

implemented an EDR in their practice. A recent survey of the Dental Practice-based Research Network reported EDR implementation by 14.3% of solo practitioners and 15.9% of group practitioners (Acharya et. al., 2017). EDRs as a part of dental informatics must be developed to support quality improvement, the application of best evidence to patient care, decision support, and the generation of new

knowledge. To do so, we must move beyond the EDR's current primary function as a record‐keeping method that only marginally improves on the capabilities of paper (Schleyer et. al., 2012).

Charangowda (2010) describes that an EDR may consist of several different parts, which include written notes, radiographs, study models, referral letters, consultants’

reports, clinical photographs, results of special investigations, medication prescriptions, laboratory reports, patient identification information, and an exhaustive medical history. The study also describes more details about the information that must be included in the EDR:

1. Patient identification data such as name, date of birth, phone numbers, and emergency contact information.

2. Dental history

3. Clinical examination to include an accurate charting 4. Diagnosis

5. Treatment plan

6. Documentation of informed consent

7. Medical history which is taken through an investigation and includes minimally the name and phone number of a physician, dentists’ evaluation of patient’s general health and appearance, list of systemic disease, any ongoing medical treatment, any bleeding disorders, drug allergies, smoking and alcohol history, any cardiac disorders, relevant family medical history, pregnancy, and physical and emotional tolerance for procedures.

Charangowda (2010) considers that the financial information should not be kept in the dental record. This suggestion disagrees with the Swedish practice, which

(10)

contains information about payments and charging of the patient and Swedish Social Insurance Agency which has a direct connection to most EDRs.

Thyvalikakath, Dziabiak, Johnson, Torres-Urquidy, Yabes, and Schleyer (2012) investigated the protocol of using EDRs which was followed by dentists during diagnosis and treatment planning. The study finds that the dentists start by

reviewing patient demographics (birth year, gender, country, postal code, identity), and the main complaint. Then they review medical history and/or dental history, and any specific information about oral conditions included in the radiographs, the intraoral images, the hard tissue and the periodontal charts. All dentists who were involved in the study, initially checked the intraoral images and the radiographs to gather information and to make primary decisions for the cases. Afterwards, they combined this information with other information from clinical examination to put a diagnosis and develop the last decision about the treatment plan.

1.4 Functions in the Swedish EDRs

Many companies in Sweden offer various EDRs such as T4, Carita, Frenda, Muntra, Opus Dental and more, which aim to satisfy requirements of Swedish dental clinics.

Initially in this study, when reviewing EDR demo copies offered in Sweden, was found that all EDRs containing the same parts needed to manage the dental clinics.

By reviewing the EDRs websites, was found that the differences between various EDRs in the Swedish market are typically the design of user interface, usability, costs, etc. However, there is no big difference between various dental records regarding the functions and tasks that must be done by the EDRs. Frenda and Muntra are used as examples in this section because they publish clear details about the functions in their EDRs. But it could not mean that other EDRs have the same functions.

The majority of EDRs contain a journal part in the system. Frenda as an example, containing a part where the dentist or the dental hygienist registers the status of the dental health, the teeth that have an abnormal status, or the teeth that have already received some type of treatment, and any other treatment conducted by a dentist or/and a dental hygienist. Usually, the journal has a type of graphical tool, which contains a dental map. The dentist typically clicks on a certain tooth to register its status by changing the color of the tooth. Every color has a certain meaning that reflects the status of the tooth. Besides the graphical tool, the journal offers the possibility to write notes and document every visit (Frenda AB, n.d.).

The periodontal journal function is represented in Frenda on a separate part of the system to register a gingival disease and other diagnosis signs relevant to it. It is also possible to control the system by voice; the dentist and the dental hygienist can talk with the system and give the system orders to register something about gingival health. The graphical tools are used also with periodontal journal function. The system can show static graphs to depict improvements or degradations of gingival health during a given period according to the diagnostic information (Frenda AB, n.d.).

Frenda and other systems as Muntra have a function to investigate a patient’s health status by posing several questions which are important to decide a safe therapy plan

(11)

according to the patient’s case. This function in Muntra shows a warning if the patient has some health issues which need special care during treatment.

Additionally, the function has a direct connection to FASS (an online service which gives healthcare professionals information about indications, contraindications, and side effects of all types of drugs). Thus, if the patient uses any medication, it is easy to see how the medication can affect dental treatment. Muntra has also a direct connection to a medication list, which is offered by the e-health authority (Muntra AB, n.d.).

Frenda and Muntra offer appointments function, which is easy to modify and supports a digital invitation to patients by email or SMS. They also offer a possibility for patients to book their appointments online. Both EDRs offer an instrument to design a therapy plan and to get a cost suggestion with the possibility to print or send it to the patient via email. Frenda and Muntra offer support for payments and invoicing, by providing a direct connection to the Swedish Social Insurance Agency (Försäkringskassan). The amount that must be paid by the patient is immediately calculated, whereas the rest is charged to the Swedish Social

Insurance Agency. Muntra offers also statistics about the economy in the dental clinic. Frenda and Muntra offer support for e-prescriptions and e-referrals, and they can connect to most of the digital x-ray systems. It is also possible for dentists and dental hygienists to sit and examine the patient and decide the diagnosis for every tooth, which is then marked in the EDR (Frenda AB, n.d.; Muntra AB, n.d.).

1.5 Interaction between EDRs and the dental care process

Dentists first reviewed the patient’s demographics, chief complaint, medical history, and dental history to determine the general status of the patient. Subsequently, they proceeded to examine the patient’s intraoral status using radiographs, intraoral images, hard tissue, and periodontal tissue information. The results also identified dentists’ patterns of navigation through the patient’s information and additional information needs during the meeting between the dentist and the patient.

(Thyvalikakath et. al., 2012). There is a chronological arrangement of the process when offering dental care to an adult patient according to the law of patient data (SFS 2008:355) and to other real circumstances that control the way of giving dental care in Sweden. Swedish Dental Association (SDA) (2015) labels EHR as a

cornerstone in healthcare, they consider that the various treatments could not be completed without documentation. SDA (2015) applied the requirements made by the law of patient data (SFS 2008:355) on the dental care and they summarized records which must be included during documentation which are, all medical and dental history, all diagnoses made based on examinations and assessments, information about completed and planned treatment steps, information about the costs and other health instructions which are given to the patient, and treatment's steps and materials which are used during the treatments. Based on the information which must be included in EDR according to SDA (2015) and the law of patient data (SFS 2008:355), a virtual model that describes the interaction between EDR and the dental care process was developed in this study (Figure 1). As shown, every step during the process of offering a treatment to a patient, has an equivalent corresponding step which must be done on the EDR (Figure 1).

(12)

Figure 1 Interaction between EDRs and dental care process. A virtual model based on SDA (2015) and SFS (2008:355) developed by the author of present study.

The patient feels that he needs a dental care.

The patient uses an online service in electronic dental record to book an appointment. Or the patient calls the dental clinic and the receptionist books an appointment for the patient by using the appointment’s function in the electronic dental record.

The patient comes to the dental clinic.

The patient checks in by using a check-in terminal at the waiting's area. Once the patient checks in, it shows on the dentist's computer that the patient has arrived to the clinic and is ready for the meeting.

The patient is waiting the time of his appointment and the dentist prepares for the meeting.

The dentist reads in the electronic dental record about the cause of visit in the information which is given by the patient, when he books the appointment. The dentist reads about the patient's disease history from the previous visits which are registered in the electronic dental record.

The patient meets the dentist

The dentist investigates the patient about his health and about his oral health, the dentist gives some information to the patient about how to promote oral health. Both of investigation's result and information which is given to the patient must be registered by the dentist in the electronic health record.

The dentist examines the patient

Every diagnosis must be registered in the electronic dental record, x-rays are saved automatically in the electronic dental record.

The dentist suggests treatment's options

All options which are offered to the patient must be registered in the electronic dental record including patent's choice.

The dentist treats the patient

Treatment's steps and materials which are used during the treatments must be registered in the electronic dental records.

The patient pays for the treatment

According to the treatment which is registered in the electronic dental record, the system calculates the amount which must be paid by the patient, and the rest will be charged to the Swedish Social Insurance Agency.

(13)

Hence, if we have a very competent dentist who treats a very cooperative patient, we could not even then gain a good workflow if we do not have a usable EDR. The usability of an EDR can reflect usable dental care, and therefore it could be

important to investigate the usability of EDRs. This will contribute to increasing the dental care quality and to gain a better efficiency and better workflow at dental care organizations.

1.6 Theories about the usability

1.6.1 The concept of usability of an EHR.

The usability of an IT system refers to its ability to be used by a human efficiently.

The purpose behind usability is to achieve a high quality of use when it comes to the user's interaction with the system (Viitanen et al. 2011). ISO (2010) defines also usability as “the extent to which a system can be used by specific users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.” (page 3). ISO’s (2010) definition of usability supports the statement of Viitanen et. al. (2011) which emphasizes the relationship between usability and the context of use. Ottersten and Berndtsson (2002) define usability as a quality feature of interactive products, and a product has a high usability if it meets the aims of the client and target groups. Nielsen (1993) describes five attributes, namely,

learnability, efficiency, memorability, error, and satisfaction that are traditionally associated with usability.

To produce a usable product, according to Ottersten and Berndtsson (2002), the product should be designed with full consideration to:

1. The human attributes like the general attributes as how the humans see and remember and understand information, and the special attributes (like the level of knowledge, expectations, attitude, etc.) among the target groups (users).

2. The context of use like the physical, psychological, social, and organizational context.

3. The expected benefits of using the product.

Ottersten and Berndtsson (2002) describe many benefits of using an IT system which has a high level of usability:

1. Increased productivity: There is no need to spend time thinking about how the system works and understanding complicated workflows.

2. Reduced learning time: a simple system needs less time to learn.

3. Reduced costs and shorter development time: Many IT projects exceed time and cost frames, and this is often due to a lack of knowledge about users' needs.

4. Engagement with clients and users: Clients and users are involved in the development process by participating in usability works.

5. Reduced life cycle costs: Most of the costs of IT management systems are linked to the inability to meet the needs of users. Hence, by investing in usability activities, the costs of error management, poor ergonomics, and training are reduced.

(14)

6. Increased user satisfaction: An IT system that works as expected by the user creates satisfaction.

7. Strengthened brand: A usable system leaves a good experience among users. If a patient fails to book an appointment online because of poor usability in a complicated system, the patient could feel that the healthcare organization is complicated. Hence, good usability can be part of the marketing of healthcare organizations.

1.6.2 TURF an extensive perspective of the usability

Zhang (2005) moves on from usability to a larger concept which is human-centered computing. He considers that designing and implementing a health information system is not so much an IT project, but a human project about human-centered computing such as usability, workflow, organizational change, medical error, and process reengineering. Human-centered computing is based on four types of analyses: user, functional, representational, and task analyses.

TURF is an integrative conceptual framework that involves user-centered design (UCD) in the evaluation, analysis, and design of interface. The usability of an EHR system is decomposed into two components: intrinsic complexity and extrinsic difficulty (Figure 2). Intrinsic complexity reflects the complexity of the work domain and is an indication of the usefulness of the system. It also reflects the amount and complexity of work, independent of any procedures, activities, or implementations. Extrinsic difficulty reflects the difficulty when a user uses a specific representation or user interface to perform a specific task and it is an

indication of the usability of the system. Extrinsic difficulty is mainly determined by the formats of representations and the workflows of tasks. Intrinsic complexity and extrinsic difficulty together reflect the usability of the system and can be recognized by the four components of TURF: Task, User, Representation, and Function

analyses (Zhang & Walji, 2011).

User analysis is the process of identifying the characteristics of existing and

potential users. This analysis can help the developers to design systems that have the right knowledge and information structures that match those of the users. User characteristics include experience and knowledge of EHR, knowledge of computers, educational background, cognitive capacities and limitations, perceptual variations, age-related skills, cultural background, personality, etc.

Functional analysis is the process of identifying the system’s abstract structures.

The goal of this analysis is to shape an image of the ontology of a given work domain. This ontology is an obvious description of the work domain. It includes objects and their attributes, resources and their types, relations among entities and constraints on relations, operations on single or multiple objects, transformations, relations, and constraints, and workflow structures.

Representational analysis is the process of identifying an appropriate information display format for a given task performed by a specific type of users. One type of representation analysis is to compare a representation with isomorphic

representations (functionally equivalent but without using a computer) of the same structure and determine whether it is efficient for the task and the user. The work

(15)

domain ontology can be implemented in many ways. For example, for the function

‘‘order a blood test’’, it can be represented in a paper-and-pencil format, in a telephone call to the laboratory, or a task on a computer in an EHR. These different representations have various outcomes for user performance. There is no best representation of a function for all tasks for all users. However, an efficient representation or a set of efficient representations of a given function can often be identified for a specific task for a specific user under specific circumstances.

Figure 2. The TURF framework of EHR usability (Zhang & Walji, 2011).

The task analysis is the process of identifying the steps to be carried out and the information to be processed to achieve task goals by using specific representations, the relations among the steps, and the nature of each step such as time on step, task steps, and mental effort for each step (Zhang, 2005).

TURF defines usability in terms of how useful, usable, and satisfying a system is for the intended users to accomplish goals in the work domain by performing certain sequences of tasks (Zhang & Walji, 2011).

1.6.3 The theoretical reference frame.

The usability of a system is considered as an interactive property, this means that the usability of a system is determined by different properties of the context of use, and the interaction of these properties. According to Allwood (1998) four different factors together determine a system usability, adaptation, user-friendliness, user acceptance and user competence (Figure 3).

- Adaptation: The functions in the system should comply with the task which should be done by the user. A good idea is to develop a custom-designed

EHR

Extrinsic difficulty

Intrinsic complexity

Functions

Users

Representations

Tasks

System usability Useful

Usable Satisfying

(16)

system, where the development has taken place for a specific workplace with specific tasks.

- User-friendliness: It contains four different aspects, accessibility, compatibility with and support for the way of human mental function, individualization, and help resources. This means that the user must have access to the system and can trust it. The designer of the system must consider how users of the specific program work. To increase the user- friendliness, the system must meet the individual needs of different

individuals. The help function in the system must really support the users, to increase the user-friendliness.

- User acceptance: The users have a high motivation to use the system. This can be of great importance because otherwise there is a risk that users will not use the program, even though they know how it works. The users can also become unengaged and careless in their use, which results in poor work results.

- User competence: It means that the user must have sufficient knowledge to interact with the computer and the system effectively. This requires effective training (Allwood, 1998).

Figure 3. Usability aspects according to Allwood (1998).

Allwood (1998) further describes actual usability aspects for the system and gives examples of what these can be. The time perspective is an important factor to describe a system usability, by time perspective it means the time it takes to perform different tasks in the system or the time it takes to learn different parts of the

system. Other aspects can be how easy and quickly it is to use help service in the system and to find useful instructions in the help service, the users' understanding of the system and its structure, the number of errors that occur during performing a task.

Usabili ty

User competence

User-friendliness

Compatibility with function of human mental

Individualization

Help resources

Support of the function of human mental

User acceptance

Adaptation

^:

(17)

1.7 Previous studies about usability in EDRs and EHRs

The number of published studies about the usability in EDRs is limited. Hence, this is an indication that more research is needed addressing the usability of EDRs.

The healthcare professionals experience difficulties when using EHRs while performing their tasks. For instance, Viitanen et. al. (2011) have highlighted the problems and shortcomings that reduce the efficiency in healthcare organizations as the systems lack the appropriate functions to support clinical data. Those include decision making, prevention of medical errors and review of the patient's treatment schedule, or the system which requires doctors to perform the same steps and tasks with all patients regardless of the patient's case. This makes the documentation and retrieval of patient data negatively conditioned, because of the case of many patients could not need to do all steps and tasks, but the system design forced them to do these unnecessary steps and tasks. Many quality factors can affect the usability and effect of the EHRs such as the availability of templates, interface design, technical performance like speed and reliability, information quality factors included the organization, accuracy, accessibility of the patient record, service quality factors included training and technical support, system backup and unexpected downtime (Lau et al., 2012). Compatibility between EHRs and physician’s tasks, EHR support for information exchange, communication and collaboration in clinical work, interoperability and reliability are the main usability issues in many Finish EHRs described by Viitanen et al. (2011). The EHRs designed only by software architects who expect clinicians to adapt their workflows to meet the needs of the design of the EHRs, can lead to workarounds that destroy system goals and reduce the value of the EHRS. Designing effective EHRs requires the full complement of usability expertise, from the investigation of the context and ethnography required for understanding users’ needs to the development of comprehensible information and interaction designs, and finally to the evaluations necessary for understanding successes and failures in complex medical environments. This perspective of the usability of the EHRs matches with usability aspects according to Allwood (1998) and with TURF perspective of the EHRs usability that included user, function, representation and task analysis to consider user needs to achieve his work tasks in an effective way (Hochheiser & Shneiderman, 2011; Zhang, 2005).

The usability from the perspective of end-users is a common proxy used to determine the successful implementation of an EDR. The participants in the study by Sidek and Martins (2017) described many usability problems such as lack of system customization to dental department needs, time-consuming processes, excessive clicking, user unfriendliness, insufficient user adaptability period, user resistance, lack of trust in the system and data accessibility issues. The lack of usability in the EDRs leads to a need for a long time to achieve the work tasks and to a more mental effort required by the dentists and dental hygienists to do many tasks in the EDR. The usability of an EDR interface was evaluated by Walji et al.

(2013). They found that only 22–41% of users were able to successfully enter one diagnosis, while no user was able to complete a more complex task. The study identified 24 high-level usability problems reducing efficiency and causing user errors. Interface-related problems represented 38% of usability problems, which included unexpected approaches for displaying diagnosis, lack of visibility, and

(18)

inconsistent use of user interface widgets. A quarter (25%) of usability problems represented problems with diagnostic terminology related to issues that included missing and miscategorized concepts. Work domain issues involved both absent and superfluous functions constituted about 38% of usability problems. The time of a mental effort required by the participants to do many tasks in the EDR was

measured by Suebnukarn et. al. (2013). They concluded that the user interface could be improved by reducing the percentage of mental effort required for the tasks.

Consequently, the EDR is a multifunctional system used by dental professionals to perform their work tasks. According to Sidek and Martins (2017), the goal of using an electronic health record is to increase efficiency and improve quality in

healthcare organizations. But the poor usability is one of the major barriers to optimally use of the EHRs (Walji et. Al., 2013).

Lau et al. (2012) present some points which could promote the planning, designing, and implementation of a usable EHRs.

- The EHR must have robust features that support ongoing clinical use - by paying close attention to interface design and technical performance.

- Redesigning EHR to support work practices for optimal fit by reorganizing the clinical workflow to make full use of the advanced EMR features.

- Having realistic expectations on the EHR implementation effort, by setting palpable goals and putting in the time, resources, and commitment needed to achieve them.

- Involving patients in using the EHR as a communication, information and decision tool before, during and after their visit to the healthcare

organization.

While the above-mentioned studies show modest research done in the field, no study about the usability of EDRs in Sweden was discovered. This indicates the potential contribution and the new knowledge that could be generated by this study.

Usability will play a key role in the safety, efficacy, and success of medical

informatics according to Hochheiser and Shneiderman (2011). The low usability of EHRs risks patient safety and continues to be a main source of clinician frustration and increased workload. Therefore, recommendations have been made and

legislation drafted in USA to develop comparison tools that would allow purchasers to better understand the usability of EHR products prior to purchase (Ratwani, Hettinger & Fairbanks, 2016) Hence, a low usability in the EDRs is a serious problem meeting dental care professions and dental care organizations, it could give negative effects on dental professions, dental care providers, and patient safety.

Investigating the usability of EDRs is the first necessary step in the way to reform EDRs to promote a good efficiency, productivity, and work environment in dental care organizations.

(19)

2 Aims and research questions.

2.1 Aims of the study.

This study aims to measure the SUS score of EDRs that are used in dental clinics in Sweden and investigate the relationship between the SUS score of EDRs and participants’ demographics. Furthermore, the aim is to rank common usability problem in EDRs, and also present how the participants describe experienced usability problems in the EDRs.

2.2 Research questions

The following are the research questions guiding this study:

1. What is the SUS score of various EDRs used in dental clinics in Sweden?

2. What is the relationship between users’ demographics and their SUS score?

3. Which are the most common usability problems in EDRs among the seven possible usability problems included in the questionnaire?

4. How the participants describe experienced usability problems in the EDRs?

3 Methods and materials

3.1 Research design

To answer the first three research questions, quantitative methods were used, that described by Justesen and Mik-Meyer (2010) as an examination method that produces a material based on numbers or speech to enable quantifiable analysis.

Justesen and Mik-Meyer (2010) describe that quantitative methods are mean to get results by various kinds of calculations, usually of a statistical nature. A

questionnaire was used in this study to gather data from various dental clinics around Sweden to get a statistical anchored description about the usability of various EDRs. This is in accordance with the description by Justesen and Mik- Meyer (2010) who estimate that the goals of quantitative studies are whether to give a statistical anchored description, a snapshot of how certain conditions look like at that particular moment, or to explain how various factors are associated with each other.

To answer the last research question, three open-ended questions (Q14, Q16, Q23) were included in the questionnaire. In other words, these questions start with

“Why”or “Describe”, and the participants answer these three questions by providing comments. The goal behind using these questions was to gain a richer understanding of experiences and possible causes of usability problems in EDRs in Sweden.

3.2 Procedure and data collection

An information letter describing the study was sent to 17 Swedish regional public dental care providers (Folktandvården) and to a large private dental care provider.

The letter included information about the aims and the process of the study. The letter also highlighted, that by participating, the dental clinics could obtain a specific report showing the usability of the dental journal they used. This report does not

(20)

contain any personal information about people who have answered the

questionnaire, but it will be like a small study in the organization, which can give the decision-makers an overview of how the employees experience the EDR. This letter was sent in Swedish, however, both Swedish and English versions are found in Appendix 1.

Fourteen regional public dental care providers refused to participate in the study because of lack of time or because they were in the process of replacing the old dental record with a new one. Thus, it was not suitable to measure the usability of a system that is under implementation or an old system. Four regional public dental care providers accepted to participate in the study (Figure 4), given a total sample of 192 participants.

Primary data using questionnaires were collected in this study. Questionnaire questions can be seen in Appendix 2. Data was collected between the 14th of January 2020 until 28th of February 2020. The link of the digital questionnaire was shared with the managers of the four regional public dental care providers. The manager then shared the link with dentists and dental hygienists who were working in the clinics they managed.

3.3 Information to participants and the questionnaire

3.3.1 The introduction letter and approval

A digital questionnaire (Appendix 2) using Google services was used to gather data in this study. The questionnaire started with a brief introduction letter, which described the process and aims of the study and that it was optional to participate in the study. The introduction included also information about the confidential

A letter about the research was sent to 17 Swedish regional public dental care providers (Folktandvården) and to a large

private dental care provider

4 public dental care providers accepted to be a part of this study

The public dental care provider number 1 is represented with (77) participants

The public dental care provider number 2 is represented with (31) participants

The public dental care provider number 3 is represented with (11 ) participants

The public dental care provider number 4 is represented with (73) participants 14 dental care providers refused or

did not answer

Figure 4. The process of finding participants to this study.

(21)

treatment of the data. Hence, the study did not present individual data about

participants, only in aggregated form. The introduction letter to the general manager of the clinic also informed about the possibility to obtain a usability report specific to their clinic. If the managers accepted, they were sent a different copy of the questionnaire (same questionnaire, but a different link). This enabled the process to track the specific dental organization and develop a specific usability report. All four organizations included in this study made such request. All participants were also informed, in the introduction of the questionnaire, about the possibility to send a usability report to their general manager based on the data gathered from their organization. The participants were provided with contact information of the researcher if they needed any further information about the study. By the end of the letter, participants had two choices, either to agree to participate and move forward with the questionnaire, or to refuse, in which case the questionnaire closed.

3.3.2 Demographic questions

The questionnaire started with twelve demographics questions (Q1-Q12). These questions were about participants’ approval, gender, age, place of work, profession, level of interest in technologies and smart solutions in general, the number of patients they treated per workday, professional experience, which EDR was used at work, any special training they had received to use the EDR, how many hours were included in the training, and the usage frequency of EDR per workday.

3.3.3 Specific questions about the usability of EDRs within the field of dentistry

Eleven questions (Q13-Q23) developed for this study were used to obtain

participants’ experience with dental records. These questions were five-point Likert scale indicating 1: Strongly Disagree to 5: Strongly Agree. A Cronbach's Alpha test was performed on the data from questions (Q13, Q15, Q17, Q18, Q19, Q20, Q21, and Q22). The Cronbach's Alpha value of 0.726 indicated an acceptable internal consistency. Questions 14, 16, and 23 were open-ended questions that were answered by participants as comments only in cases when participants selected values 4 or 5 when they answered questions 13, 15, and 22. A total of 52 comments were given on the open-ended questions Q14, Q16, and Q23. The comments were summarized and presented under three categories in section 4.5.

3.3.4 System usability scale (SUS)

SUS is a simple, ten-item with Likert scale giving a general view of subjective assessments of usability. The statements, included in SUS, cover a variety of aspects of system usability, such as the need for support, training, and complexity. SUS is generally used after the respondent has had an opportunity to use the system being evaluated. The system usability scale (SUS) was used in the questionnaire to measure the usability of EDRs. SUS is also a five-point Likert scale. The Likert scale is simply one based on forced-choice questions, where a statement is made and the respondent then indicates the degree of agreement or disagreement with the statement on a five points scale (Brooke, 1986). SUS instrument is available only in English, and data in the present study was gathered in Swedish. SUS questions were translated from English to Swedish. The translation was reviewed, modified and

(22)

then accepted by a researcher in the eHealth institute at Linnaeus University, who has Swedish as a native language and has a good knowledge of the English language. This way in translating of quantitative instrument is called forward translation with testing method according to Maneesriwongul and Dixon (2004).

SUS contained ten questions (Q24-Q33) in the questionnaire, which is presented in Appendix 2. The calculation of the SUS score was conducted by first obtaining the user-selected values. For questions 24, 26, 28, 30, and 32 the score was the user- selected value minus 1. For questions 25, 27, 29, 31, and 33 the score was 5 minus the user-selected value. SUS outputs a single number representing a composite measure of the overall usability of the system being studied. Each item's score contribution will range from 0 to 4. For some items score contribution is the scale position minus 1. For other items, the contribution is 5 minus the scale position.

Then the sum of the scores must be multiply by 2.5 to obtain the overall value of SUS which has a range of 0 to 100 (Brooke, 1986). The final SUS value can then be transferred to the curved grading scale interpretation of SUS scores (Table 1) that have been developed by Sauro and Lewis (2016).

Table 1, Curved grading scale interpretation of SUS scores by Sauro and Lewis (2016)

SUS Score Range Grade Percentile Range

84.1 to 100 A+ 96–100

80.8 to 84.0 A 90–95

78.9 to 80.7 A − 85–89

77.2 to 78.8 B+ 80–84

74.1 to 77.1 B 70–79

72.6 to 74.0 B − 65–69

71.1 to 72.5 C+ 60–64

65.0 to 71.0 C 41–59

62.7 to 64.9 C − 35–40

51.7 to 62.6 D 15–34

0.0 to 51.6 F 0–14

3.4 Analysis

Participants’ answers to the questionnaire were accumulated into Google sheet document, which was then fed into IBM SPSS Statistics version 26.

Four statistical tests were used in this study. Cronbach alpha was used to ensure internal consistency of the data when the other tests had a direct relation to the results of the study. The selection of tests was based on the aim of the test and the type of data (nominal, ordinal, interval) and on the distribution of data between various variables whether they followed a normal distribution or not. Independent sample t-test was used to compare the mean value between two variables when the data of two variables were normally distributed (Ejlertsson, 2019). Independent sample t-test was used to compare the mean value of SUS between various professions (dentists and dental hygienists). When the assumption of normally distributed data was absent, an alternative non-parametric Mann-Whitney U-test

(23)

was used (Ejlertsson, 2019). Mann-Whitney U-test was used to compare the mean value of SUS between various genders and training groups. Spearman’s correlation coefficient was used to identify the relation between two variables when at least one of the variables was non-parametric or if the assumption of normally distributed data was absent according to Ejlertsson (2019). Therefore, in this study, Spearman’s correlation coefficient was preferred against Pearson’s test. Hence, sometimes there was an ordinal variable (Likert scale) and the relation between this variable and the other ratio variable was investigated. Or the relation between two ratio variables was investigated, although the assumption of normally distributed data was absent.

Spearman’s correlation coefficient can be used with the Likert scale variables (Murray, 2013). Spearman’s correlation coefficient was used to identify the relation between SUS and demographic data (Frequency of using EDRs, age, interesting in technology, clinical experience, number of patients served per workday, training period).

3.5 Ethical aspects

There are four principles which reflect consideration of ethical aspects in the research according to Ejlertsson (2018):

1. Information requirement: This means that all participants in the study must get proper information about the aim of the study and that it is optional to participate in the study. In this study, all this information was clearly available in the introduction of the questionnaire (see Appendix 2).

2. Approval requirement: Participants’ right to decide if they want to participate in the study or not. Therefore, the first question in the

questionnaire of this study asked about participants’ approval, and if they refused to participate in the study, the questionnaire closed immediately (see Appendix 2).

3. Confidentiality requirement: Participants in a study must have the greatest possibility of confidentiality. In this study, all results and statistics were presented in aggregated form, and no individual information was presented in the study (see Appendix 2).

4. The benefit of data: Data that is collected about individuals must only be used for purposes that participants have known and accepted. In this study, all participants were informed in the introduction of the questionnaire (see Appendix 2) about the purpose of the study and how data would be used in this study.

4 The results

4.1 Descriptive statistics

The total number of participants in this study was 192. Females constituted 76%, whereas males constituted 24%. Of the total number, 60% were dentists, whereas 40%were dental hygienists. The age distribution of participants is various, with those in range of 31 to 40 years old constituting the majority. The sample contained participants of various length of clinical experience in years, those who had more

(24)

than twenty years of clinical experience constituted the majority. A significant majority of all participants used EDR more than 15 times per workday. Most participants met between seven to 15 patients per workday. Of the total, 86% of the participants received special training to use the EDR, whereas 14% did not receive any training (Table 2). The mean value of the period of training for the 166 participants was 9.16 hours. The period of training reflects how many hours were included in the training.

Table 2. Descriptive statistics (n=192).

Gender n (%)

Male 47 24.5

Female 145 75.5

Profession

Dentists 115 59.9

Dental hygienists 77 40.1

Age

20 to 30 years old 22 11.5

31 to 40 years old 52 38.5

41 to 50 years old 45 23.4

51 to 60 years old 34 17.7

Over 60 years old 39 20.3

Professional experience

Between 1 to 5 years 24 12.5

Between 16 to 20 years 25 13

More than 20 years 80 41.7

Frequency of using EDR/workday

Less than 5 times 2 1

Between 5 to 15 times 17 8.9

Between 16 to 25 times 71 37

More than 45 times 34 17.7

Number of patients per a workday

Less than 7 patients 39 20.3

Between 7 to 15 patients 133 69.3

More than 15 patients 20 10.4

4.2 SUS score of various EDRs used in dental clinics in Sweden.

The median value of the system usability scale (SUS) was 55 for a total of 192 participants in this study, whereas it was 60 for the 47 males and 52 for the 145 females. For the 115 dentists, the mean SUS value was 57, whereas it was 52.5 for the 77 dental hygienists. Among the four organizations that participated in the study the mean and median SUS values was slightly different (Table 3).

(25)

Table 3. Mean and median values of SUS among various variables.

Md Mean n

All participants in the study 55 54.25 192

Females 52 52.72 145

Males 60 58.98 47

Dentists 57 54.91 115

Dental hygienists 52.5 53.27 77

Public dental care provider number 1,

which uses EDR (A)

47 48.5 77

Public dental care provider number 2, which uses EDR (B)

67 62 31

Public dental care provider number 3, which uses EDR (C)

65 63 11

Public dental care provider number 4, which uses EDR (C)

55 55.5 73

Details about the frequency of the SUS individual responses per question are presented below (Table 4).

Table 4. Proportions of system usability scale response

SUS Questions 1

Strongly disagree

2 3 4 5

Strongly agree 1. I think that I would like to use frequently

the journal system I currently use at work.

23 23 47 62 37

2. I find the current journal system unnecessarily complex

16 34 43 48 51

3. I thought the journal system is easy to use. 16 37 62 61 16 4. I think that I would need the support of a

technical person to be able to use this journal system.

95 62 26 7 2

5. I found the various functions in this journal system were well integrated

21 51 60 48 12

6. I thought there was too much inconsistency in this journal system

13 44 63 43 29

7. I would imagine that most people would learn to use this journal system very quickly.

28 43 62 49 10

8. I found the journal system very cumbersome to use.

37 52 47 34 22

9. I felt very confident using the journal system.

4 22 37 82 47

10. I needed to learn a lot of things before I could get going with this journal system

22 33 57 50 30

(26)

4.3 The relationship between users’ demographics and their SUS score.

In this study, I was interested to understand the relation between users’

demographics (age, gender, number of patients, etc.) and their usability experience in EDRs. Only two relations were found to be significant, SUS with gender and frequency of using the EDR (Table 5). The relationship between SUS and gender and between SUS and the frequency of using the EDR were statistically significant.

A Mann-Whitney U-test was conducted to determine if the difference between SUS and gender was significant or caused by randomness. The mean rank for males was 110.77, whereas 91.88 for females, and p < .05. Thus, the mean SUS ranks among males had larger value than among females, and this difference was statistically significant. A Spearman's rank-order correlation was conducted to determine the relationship between SUS and frequency of using the dental record system. There was a medium negative correlation between those, which was statistically

significant. This indicates that participants who used the system more frequently showed significantly lower SUS scores. There was no significant correlation in the relationship between SUS and age, profession (dentists and dental hygienists), interest in technology, clinical experience, the number of patients served per workday and training.

Table 5. Relation between users’ demographics and their usability (SUS) experience in EDRs

Demographics Type of test Significance Test value

Gender U p < .05* U = 2773

Frequency of using EDR S p < .05* r = -0.242

Age S p > .05 r = 0.02

Profession T p > .05 t = 0.551

Interesting in technology S p > .05 r = 0.114

Clinical experience S p > .05 r = - 0.025

Number of patients served per workday

S p > .05 r = - 0.001

Training group U p > .05 U = 1723

Training period S p > .05 r = - 0.092

r - Spearman’s correlation coefficient T - Independent sample t-test

U- Mann-Whitney U-test S- Spearman correlation test

* - Significant

4.4 The most common usability problems in EDRs among the seven possible usability problems included in the questionnaire.

The details which describe how the participants answered the questions about the usability problems and the number of participants who selected every choice from 1 to 5 for the respective question are presented below (Table 6).

According to the data shown in table 6, the two most common usability problems in EDRs among the seven possible usability problems included in the questionnaire

(27)

are, “The need for a long time to document patients’ cases” and “The hanging of the system".

Table 6 Frequency of usability problems response

The question 1

Strongly disagree

2 3 4 5

Strongly agree I usually write on a piece of paper and then I

move all the information to the system

144 26 15 4 3

It is difficult to find appropriate terms to describe patient cases between the terms offered in the system

57 55 57 12 11

I need a long time to document patients’ cases 7 33 51 57 44 It is difficult to send e-prescriptions and/or e-

referrals via the journal

63 42 32 29 26

I usually need a long time to reach the targeted side of the system

61 52 39 25 15

It usually happens that the system hangs 25 41 49 47 30 Schedule changes and / or other

administrative tasks are difficult

38 49 49 39 17

It is difficult to charge the patient and /or send a charge to the Social Insurance Office

69 60 41 16 6

4.5 Experienced usability problems in the EDRs - findings from open-ended questions

Inefficient user interface

1. Inability to see many pages simultaneously: It was impossible to open and see many pages simultaneously. For instance, the dentist would examine x- rays at a certain page on the system but wrote the diagnosis on a piece of paper before moving to another page to document the diagnosis.

2. Very small and densely placed buttons: The buttons on the systems are very small and too near to one another.

3. Too many clicks to reach the desired page: Many participants described that it needed too many clicks, and it was difficult to reach the desired page.

When they reached the desired page and finally could input the data, it often happened that the system hanged. This forced participants to repeat all the steps once again.

4. No voice control: Several participants especially dental hygienists considered that working could be easier if they had a voice control on the system.

5. Irrelevant shortcuts to treatments’ codes: Systems offered irrelevant shortcuts that could lead dentists to the social insurance agency treatments’

codes, which must be entered to complete a payment charge. There was no list to select, and the dentists needed to enter the shortcuts of the system first to get the social insurance agency treatment codes. One particular

(28)

dentist asked: “why there was no list which could contain the social insurance agency treatment codes, so I could select from it easily?”

6. The system is complex: Many participants commented that the dental records are very complex and non-flexible systems.

Lack of semantic interoperability

1. Old terms are still used on the system: The system uses old terms in dentistry which should be updated.

2. Too many irrelevant terms: There are too many and unusable terms on the system.

3. Lack of templates and relevant certified terms: Many participants described a lack of templates and relevant certified terms. As a workaround, they created their templates with their own terms, which could not mean the same thing to the others.

Relying on paper

Several participants described that they typically write on a piece of paper before inputting the data into the system. Reasons for such action include:

1. Hand hygiene: When the dentists or dental hygienists examined patients or performed treatments, they had dirty gloves, and they did not want to touch the keyboard or the mouse with their dirty gloves to keep a good level of hygiene.

2. Weak memory: Some of the participants considered that they have a weak memory, so they use a paper to remember information about the patients.

3. Focusing on the patient: Many participants considered that it could take their focus from the patient if they wrote directly on the system while they were treating the patient.

4. Time constraint with a patient: Some of the participants described that on some occasions they do an extensive treatment on the patient, which takes a long time. Thus, they felt they did it faster by first writing the data on a piece of paper.

5 Discussion

The results showed an average SUS score corresponds to 54.25 among 192

participants, men had better SUS scores than women and the participants who used the system more frequently showed significantly lower SUS scores. The most common usability problem was a long time needed to perform the documentation using EDRs. The participants give also many comments regarding inefficient user interface, lack of semantic interoperability, and relying on paper.

5.1 SUS score of various EDRs used in dental clinics in Sweden.

The first research question was about, what is the SUS score of various EDRs used in dental clinics in Sweden? Among participants in this study, the average SUS score was 54.25, which corresponds to one of the lowest grade D (15 to 34%) in Sauro and Lewis (2016) interpretation of SUS scores grading scale. Sauro and

Usability evaluation of electronic dental record systems in Sweden Master thesis