• No results found

A Review of Usability Methods Used in the Evaluation of Mobile Health Applications for Diabetes.

N/A
N/A
Protected

Academic year: 2021

Share "A Review of Usability Methods Used in the Evaluation of Mobile Health Applications for Diabetes."

Copied!
6
0
0

Loading.... (view fulltext now)

Full text

(1)

A Review of Usability Methods Used in the

Evaluation of Mobile Health Applications

for Diabetes

Mattias GEORGSSON 1

Department of Health Sciences, University West, Trollhättan, Sweden

Abstract. Mobile health applications for diabetes are developed like never before

and many patients use them for their personalized health needs. With increased use, an increased number of usability evaluations are performed to assure that the applications function as intended. In this review the goal was to determine what usability methods are currently used in the evaluation of mobile health applications for diabetes and how these are used. Methods: A literature review was conducted to identify applicable studies in the databases ACM Digital Library, Cinahl and Pubmed between the years 2015 and 2020. After the inclusion and exclusion criteria were applied, 32 articles remained that were included in the final review. Results: Most of the studies included one established usability engineering method such as an expert-based and/or user-based method or a validated questionnaire/instrument. Some also included a combination of these. Others used methods of their own design; commonly questionnaires and interviews either on their own or in combination. Conclusion: To achieve an adequate level of evidence and quality in the evaluation, it is important that at least one is an established usability engineering method or a validated instrument. This to assure and continue to build the evidence base in this area.

Keywords. Usability methods, evaluation, diabetes self-management, mobile health

applications

Introduction

Diabetes mellitus (DM) is a growing chronic disease, which affects an increasing number of individuals around the world. Recent figures from the WHO show 422 million people affected by it [1] and by the year 2035 it is estimated that this number will reach 592 million individuals [2]. Diabetes is complex and demanding requiring the individual patient to adhere to specific therapy requirements which often includes careful glucose monitoring and extensive life-style management. This active participation and engagement can often be quite burdensome [3].

Mobile health applications have arisen as important tools and are widely used due to that they can assist in this self-management. They have increasingly been seen as one of the most personalized digital devices to assist the individual patient in their diabetes self-management as they can be fitted to the person’s specific needs [4].

1 Corresponding Author. Mattias Georgsson, Department of Health Sciences, University West, 461 32 Trollhättan, Sweden; Email: mattias@georgsson.com

© 2020 The authors and IOS Press. This article is published online with Open Access by IOS Press and distributed under the terms of the Creative Commons Attribution Non-Commercial License 4.0 (CC BY-NC 4.0). doi:10.3233/SHTI200645

(2)

This increased use of mobile health applications also means an increased number of usability evaluations have to be performed to assure users can use them in the manner intended. Usability is essential for users to experience if they are to utilize the product. There are many definitions for usability, but one of the most common is “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” [5] which essentially means that the product has to be fit for purpose and the specific user also has to be able to use the product purposefully and in a manner suitable to the specific circumstances.

The objective of this literature review was to determine the current status of the use of usability methods in the evaluation of mobile health applications for diabetes self-management to determine what specific methods are used and how they are used.

1. Methods

A literature review was conducted to identify applicable articles in the databases ACM Digital Library, Cinahl and Pubmed between the years 2015 and 2020. Inclusion criteria included: Peer-reviewed articles or conference papers in English published between the applicable years, mobile health applications aimed at diabetes patients, empirical studies, usability evaluations/tests performed with published outcomes. Exclusion criteria consisted of omitting review articles of different kinds and study protocols. The search terms included a combination of the boolean operators AND and/or OR. Controlled vocabulary thesaurus such as MeSH terms were also used. The search term combinations used for the different databases were as follows: (Self-care OR Self-management) AND (Diabetes OR “Diabetes Mellitus” OR diabetic) AND (“Mobile Health” OR Telemedicine) AND (“Usability evaluation” OR “Usability test” OR Usability).

2. Results

131 articles and papers were detected that fit the inclusion and exclusion criteria: 62 in ACM Digital Library, 14 in Cinahl and 55 in Pubmed. After the title and abstract had been read through, 47 studies remained that were assessed in full text. Articles and papers were excluded for not being evaluation studies, method comparison studies only without an evaluation result, if usability was not specifically tested through a particular method or specific usability test. A total of 32 studies were included in the final review: 4 from ACM Digital Library, 2 from Cinahl and 26 from Pubmed.

2.1. One Usability Evaluation Method

Of the published articles and papers it was clear that a large variety of usability methods were used. For the 18 studies that focused on one usability evaluation method some focused on using one expert-based usability method only. These were for example those that used heuristic evaluation [6] and the cognitive walkthrough [7]. Some used one user-based usability evaluation method only. These were mainly those studies that included usability tests [8-10]. Other usability evaluation methods that were used on their own were those that included validated questionnaires/instruments such as the System

(3)

Usability Scale (SUS) [11], the Telehealth Usability Questionnaire (TUQ) [12] or the Questionnaire for User Interaction Satisfaction (QUIS) [13, 14]. Other than these, a large number of evaluation method studies, or 9 out of 18 studies of this method category, included own-designed usability questionnaires/surveys as the only evaluation method

[15-19] or only in-depth or semi- structured interviews [20-23]. 2.2. Multiple Usability Evaluation Methods

It was also evident that many studies used a combination of multiple methods to assess usability. Fourteen studies consisted of multiple usability evaluation methods. Ten of which involved two different methods, 3 studies three different methods and one more than three methods. Of these some used one of the expert-based methods combined with a user-based method such as the heuristic evaluation and usability test [24] or a combination of user-based methods such as think aloud with semi-structured interviews, and questionnaires [25]. Think-aloud testing and a mail panel were also used [26] as well as think-aloud testing in combination with interviews, video and face-to-face demonstrations, mock-ups, and focus groups [27]. In addition studies used a combination of usability testing, interviewing and direct observation [28]. Some also used a combination of validated questionnaires/instruments such as the System Usability Scale (SUS) and the Computer System Usability Questionnaire (CSUQ) [29] or MARS questionnaire and SUS [30]. Some studies used SUS and task completion [31, 32] or SUS and semi-structured interviews [33] for their assessment. There were also studies here that used a combination of methods of their own design, or a total of 4 out of 14 studies of this method category. These involved own- designed usability tests and interviews [34, 35] or usability tests and questionnaires [36] or usability questionnaires and interviews [37].

3. Discussion

It is clear that usability is necessary to assess for mobile health applications for diabetes, especially due to the extensive use of these in the diabetes patients’ self-management.

In this literature review it was evident that eighteen studies were performed using one usability method only for the evaluation. Half of them were performed with established evaluation methods from usability engineering such as heuristic evaluation (1 study), cognitive walkthrough (1 study), usability tests (3 studies) or through validated instruments such as SUS (1 study), TUQ (1 study), QUIS (2 studies) which was beneficial but it was also evident that in many studies, authors used one of their own-designed usability evaluation methods. In fact, the other half of these studies were methods of the authors’ own design such as usability questionnaires (5 studies) and in-depth or semi structured interviews (4 studies). In order to provide the necessary guidance for development, established usability evaluation methods should be used as they provide foundations that have an important historical use within the discipline of computer science [38]. For comparative purposes it can also be difficult when methods are used that are unique for one specific purpose and intervention. A draw-back of the one method use is that it provides only one source of information when it comes to usability.

Here it was also possible to see that fourteen studies also used a combination of methods which for ten studies included one or several established usability methods

(4)

combined with another method such as HE and usability test (1 study) and think aloud (TA) with different combinations such as TA and mail panel (1 study) or TA and semi-structured interview and questionnaires (1 study), TA and interviews, video, faced to face demonstrations, mockups, focus groups (1 study) or usability test and interviews and direct observation (1 study) or with validated instruments/questionnaires such as SUS with for example SUS and CSUQ (1 study), SUS and MARS (1 study), SUS and task completion (2 studies) and SUS and semi structured interviews (1 study). But it was also evident here that there were also own-designed assessment methods with 4 studies of this kind. Several authors point to that a combination of methods is beneficial due to that it also provides a comprehensive assessment; especially if both quantitative and qualitative methods are used to identify user concerns [39]. Usability evaluations with complementary methods are also important due to that each method has its own particular benefits and in combination therefore can triangulate problems or uncover different types of problems [40]. It is clear that such combinations provide the best evidence of the usability of the application and are therefore advisable. Validated instruments also provide an important dimension as they allow to compare different study results for interventions effectively with one another.

Overall, it is possible to see that a number of mobile health applications for diabetes self-management are developed and evaluated for usability. It is also evident that many of the methods that are used for evaluation include those of authors’ own design. To achieve an adequate level of evidence and quality in the evaluation, it is important that at least one is an established usability engineering method or a validated instrument. There are several reasons for why this is essential. Evaluations for usability are significant in determining user satisfaction, expectations and needs and to safeguard patient outcomes and quality of care [41]. Evaluations of health applications that include these kinds of assessments also contribute to strengthen the evidence base for mobile application use as well as in improving health outcomes [38].

4. Conclusions

Performing usability evaluations and also with several complementary usability methods are often important in order to determine the found usability problems in an evaluated system. In this literature review it was clear that many different kinds of usability evaluation methods are used on a large scale in evaluating mobile applications for diabetes self-management. While some studies use established usability engineering methods, such as the heuristic evaluation, cognitive walkthrough, think aloud and usability test both on their own or in combination, as well as validated instruments such as the SUS many used own-designed assessment methods only. To be able to achieve an adequate level of evidence and quality in the evaluation it is important that at least one of the evaluation methods is an established usability engineering method or validated instrument to continue to build the evidence base in this area.

References

(5)

[2] Guariguata L, Whiting DR, Hambleton I, Beagley J, Linnenkamp U, Shaw JE. Global estimates of diabetes prevalence for 2013 and projections for 2035. Diabetes Research and Clinical Practice 2014; 103: 137-149.

[3] Deakin T, McShane CE, Cade JE, Williams RD. Group based training for self-management strategies in people with type 2 diabetes mellitus. Cochrane Database Syst Rev. 2005 Apr 18; 2:CD003417. [4] Fiordelli M, Diviani N, Schulz PJ. Mapping mHealth research: a decade of evolution. J Med Internet Res

2013; 15: e95.

[5] International Organization for Standardization, Ergonomic requirements for office work with visual display terminals (VDTs) Part 11 Guidance on usability. Geneva: ISO; 1998.

[6] Georgsson M, Staggers N, Weir C. A Modified User-Oriented Heuristic Evaluation of a Mobile Health System for Diabetes Self-management Support. Comput Inform Nurs 2016; 34: 77-84.

[7] Georgsson M, Staggers N, Arsand E, Kushniruk A. Employing a user-centered cognitive walkthrough to evaluate a mHealth diabetes self-management application: A case study and beginning method validation. J Biomed Inform 2019; 91: 103110.

[8] Guo Y, Chen Y, Lane DA, Liu L, Wang Y, Lip GYH. Mobile Health Technology for Atrial Fibrillation Management Integrating Decision Support, Education, and Patient Involvement: mAF App Trial. American Journal of Medicine 2017; 130: 1388-1396.e1386.

[9] Mayberry LS, Berg CA, Harper KJ, Osborn CY. The Design, Usability, and Feasibility of a Family-Focused Diabetes Self-Care Support mHealth Intervention for Diverse, Low-Income Adults with Type 2 Diabetes. J Diabetes Res 2016; 2016: 7586385.

[10] Wickramasinghe N, John B, George J, Vogel D. Achieving Value-Based Care in Chronic Disease Management: Intervention Study. JMIR Diabetes 2019; 4: e10368.

[11] Bernier A, Fedele D, Guo Y, et al. New-Onset Diabetes Educator to Educate Children and Their Caregivers About Diabetes at the Time of Diagnosis: Usability Study. JMIR Diabetes 2018; 3: e10. [12] Janatkhah R, Tabari-Khomeiran R, Asadi-Louyeh A, Kazemnejad E. Usability of a Disease Management

Mobile Application as Perceived by Patients With Diabetes. Comput Inform Nurs 2019; 37: 413-419. [13] Pérez-Gandía C, García-Sáez G, Subías D, Rodríguez-Herrero A, Gómez EJ, Rigla M, Hernando ME.

Decision Support in Diabetes Care: The Challenge of Supporting Patients in Their Daily Living Using a Mobile Glucose Predictor. J Diabetes Sci Technol 2018; 12: 243-250.

[14] Alanzi T, Istepanian R, Philip N. Design and Usability Evaluation of Social Mobile Diabetes Management System in the Gulf Region. JMIR Res Protoc 2016; 5: e93.

[15] Kho SES, Lim SG, Hoi WH, Ng PL, Tan L, Kowitlawakul Y. The Development of a Diabetes Application for Patients With Poorly Controlled Type 2 Diabetes Mellitus. Comput Inform Nurs 2019; 37: 99-106. [16] Stein N, Brooks K. A Fully Automated Conversational Artificial Intelligence for Weight Loss:

Longitudinal Observational Study Among Overweight and Obese Adults. JMIR Diabetes 2017; 2: e28. [17] Albanese-O'Neill A, Schatz DA, Thomas N, et al. Designing Online and Mobile Diabetes Education for

Fathers of Children With Type 1 Diabetes: Mixed Methods Study. JMIR Diabetes 2019; 4: e13724. [18] Welch G, Balder A, Zagarins S. Telehealth program for type 2 diabetes: usability, satisfaction, and

clinical usefulness in an urban community health center. Telemed J E Health 2015; 21: 395-403. [19] Ding H, Fatehi F, Russell AW, Karunanithi M, Menon A, Bird D, Gray LC. User Experience of an

Innovative Mobile Health Program to Assist in Insulin Dose Adjustment: Outcomes of a Proof-Of-Concept Trial. Telemed J E Health 2018; 24: 536-543.

[20] Torbjørnsen A, Ribu L, Rønnevig M, Grøttland A, Helseth S. Users' acceptability of a mobile application for persons with type 2 diabetes: a qualitative study. BMC Health Serv Res 2019; 19: 641.

[21] Petersen M, Hempler NF. Development and testing of a mobile application to support diabetes self-management for people with newly diagnosed type 2 diabetes: a design thinking case study. BMC Med Inform Decis Mak 2017; 17: 91.

[22] Brandt LR, Hidalgo L, Diez-Canseco F, Araya R, Mohr DC, Menezes PR, Miranda JJ. Addressing Depression Comorbid With Diabetes or Hypertension in Resource-Poor Settings: A Qualitative Study About User Perception of a Nurse-Supported Smartphone App in Peru. JMIR Ment Health 2019; 6: e11701.

[23] Shin JY, Holtz BE. Towards Better Transitions for Children with Diabetes: User Experiences on a Mobile Health App. Proceedings of the 18th ACM International Conference on Interaction Design and Children, Association for Computing Machinery, Boise, ID, USA, 2019, pp. 623–628.

[24] Jeon E, Park HA. Development of the IMB Model and an Evidence-Based Diabetes Self-management Mobile Application. Healthc Inform Res 2018; 24: 125-138.

[25] Georgsson M, Staggers N. An evaluation of patients' experienced usability of a diabetes mHealth system using a multi-method approach. J Biomed Inform 2016; 59: 115-129.

[26] Castensøe-Seidenfaden P, Reventlov Husted G, Teilmann G, Hommel E, Olsen BS, Kensing F. Designing a Self-Management App for Young People With Type 1 Diabetes: Methodological Challenges, Experiences, and Recommendations. JMIR Mhealth Uhealth 2017; 5: e124.

(6)

[27] Pais S. Integrating Patient-Generated Wellness Data: A User-Centered Approach. Proceedings of the Australasian Computer Science Week Multiconference, Association for Computing Machinery, Melbourne, VIC, Australia, 2020, Article 35.

[28] Sarkar U, Gourley GI, Lyles CR, Tieu L, Clarity C, Newmark L, Singh K, Bates DW. Usability of Commercially Available Mobile Applications for Diverse Patients. J Gen Intern Med. 2016 Dec; 31,12: 1417–1426.

[29] Giordanengo A, Årsand E, Woldaregay AZ, Bradway M. et al. Design and Prestudy Assessment of a Dashboard for Presenting Self-Collected Health Data of Patients With Diabetes to Clinicians: Iterative Approach and Qualitative Case Study. JMIR Diabetes 2019; 4: e14002.

[30] Isakovic M, Sedlar U, Volk M, Bester J. Usability Pitfalls of Diabetes mHealth Apps for the Elderly. J Diabetes Res 2016; 2016: 1604609.

[31] Georgsson M, Staggers N. Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics. J Am Med Inform Assoc. 2016 Jan; 23,1: 5-11.

[32] Hartzler AL, Venkatakrishnan A, Mohan S, Silva M, et al. Acceptability of a team-based mobile health (mHealth) application for lifestyle self-management in individuals with chronic illnesses. Conf Proc IEEE Eng Med Biol Soc 2016; 2016: 3277-3281.

[33] op den Akker HJA, Klaassen R, Bul K, Kato PM, van der Burg G-J, di Bitonto P. Let them play: experiences in the wild with a gamification and coaching system for young diabetes patients. Proceedings of the 11th EAI International Conference on Pervasive Computing Technologies for Healthcare, Association for Computing Machinery, Barcelona, Spain, 2017, pp. 409–418.

[34] Otis M, Zhu J, Mustafa-Kutana SN, Bernier AV, Ma Shum J, Soros Dupre AA, Wang ML. Testing Usability and Feasibility of a Mobile Educator Tool for Pediatric Diabetes Self-Management: Mixed Methods Pilot Study. JMIR Form Res 2020; 4: e16262.

[35] Nelson LA, Mayberry LS, Wallston K, Kripalani S, Bergner EM, Osborn CY. Development and Usability of REACH: A Tailored Theory-Based Text Messaging Intervention for Disadvantaged Adults with Type 2 Diabetes. JMIR Hum Factors 2016; 3: e23.

[36] Schmocker KS, Zwahlen FS, Denecke K. Mobile App for Simplifying Life With Diabetes: Technical Description and Usability Study of GlucoMan. JMIR Diabetes 2018; 3: e6.

[37] Faiola A, Vatani H, Greenhill K, Bhuma M, Agarwal M. HYPOalert: Designing Mobile Technology for Hypoglycemic Detection and Monitoring--Based on Human Breath. Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare, Association for Computing Machinery, New York, NY, USA, 2018, pp. 402–406.

[38] Baysari MT, Westbrook JI. Mobile Applications for Patient-centered Care Coordination: A Review of Human Factors Methods Applied to their Design, Development, and Evaluation. Yearb Med Inform 2015; 10: 47-54.

[39] Lyles CR, Sarkar U, Osborn CY. Getting a technology-based diabetes intervention ready for prime time: a review of usability testing studies. Curr Diab Rep 2014; 14: 534.

[40] Walji MF, Kalenderian E, Piotrowski M, Tran D, et al. Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR. Int J Med Inform 2014; 83: 361-367.

[41] Goldwater JC. Human Factors and Usability in Mobile Health Design – Factors for Sustained Patient Engagement in Diabetes Care. Proc of the International Symposium of Human Factors and Ergonomics in Healthcare 2014; 3: 63-70.

References

Related documents

We discover documents [1, 2, 8, 10, and 13] use and modify the existing heuristic set to improve not only the artifact but also improve the heuristic set or enhance the

The included chapters in this part as: Chapter 2 - Usability and User Experience Chapter 3 - Web Usability Chapter 4 - Usability Issues Chapter 5 - Usability Evaluation Methods

LANDSTINGET BLEKINGE health portal was selected for current study as according to authors it is possible to provide the citizens with better access of... 12 health information and

The indirect metric (FA standard deviation) served as a good indication for correction performance compared to the direct metric (FA error maps) as shown in Figure

The application example is implemented in the commercial modelling tool Dymola to provide a reference for a TLM-based master simulation tool, supporting both FMI and TLM.. The

När Mattheson talar om den instrumentala musiken som ett tonrede, ett klingande tal, handlar det om att höja denna musiks status till samma nivå som textsatt musik, inte till

Criteria description: The system affords a repertoire of possible actions (functionality) to the user. The sum of these actions can be viewed as the e-service’s

Firstly, overview of usability, usability engineering, types of usability data, usability evaluation and, usability testing are detailed, which is followed by discussion of