1 Örebro University
Örebro University School of Business Course: Thesis
Supervisor: Gunnar O Klein Examiner: Shang Gao Semester/date: 2018-06-21
Usability Evaluation of a Health Web Portal: Case Study 1177.se in
Sweden
Author: Robinson Reeka robinsonreeka83@gmail.com D.O.B: 1983-01-04
2 TABLE OF CONTENTS ABSTRACT ... 3 1.0 INTRODUCTION ... 4 1.2PROBLEM STATEMENT ... 6 1.3RESEARCH QUESTIONS ... 6 2.0 LITERATURE REVIEW... 6 2.1CONCEPTUAL FRAMEWORK... 8 3.0 METHODOLOGY... 9 3.1METHOD ... 9
3.2FORMULATION OF IN TERVIEW QUESTIONS ... 10
3.3DATA COLLECTION ... 10
3.4SELECTION OF PARTICIPANTS ... 11
3.5DATA ANALYSIS ... 12
3.6ETHICAL CONSIDERATION ... 12
4.0 RESULTS... 12
4.1.EFFECTIVENESS OF THE POR TAL ... 13
4.2EFFICIENCY OF THE POR TAL ... 14
4.3SATISFACTION ... 16
4.4WHAT IS A GOOD SUSSCORE? ... 17
4.5WHY RATING LESS THAN 5 ... 17
5.0 DISCUSSION AND CONCLUSION... 18
6.0 FUTURE RESEARCH ... 19
7.0 LIMITATIONS OF THE STUDY... 19
REFERENCES ... 20
APPENDICES ... 22
APPENDIX A: ... 22
Section I: User performance evaluation task form ... 22
Section II: Personal information ... 22
APPENDIX B:USER PERFORMANCE EVALUATION TASKS FORM ... 23
APPENDIX C:SEQ(SINGLE EASE QUESTION) ... 25
APPENDIX D:SUS(SYSTEM USABILITY SCALE) ... 25
APPENDIX E ... 27
3 Abstract
The purpose of this study sought to evaluate the usability of 1177.se Portal in Sweden. The author adopted both qualitative and quantitative study approaches. Thirteen (13) participants from Örebro University were randomly sampled for the study and subjected to usability tests. The author applied two usability models (Single Ease Question and System Usability Scale) in data collection for all participants. Questionnaires with usability tests or performance tasks were distributed to students as they entered the university computer laboratory. The findings were analyzed using both MS-Excel and the Statistical Package for Social Scientists (SPSS).
The study findings revealed that the average task completion rate is 83% which is above the average, time-based efficiency of 0.703gosl/minute. Users felt highly satisfied in browsing relatively simple information in searching tasks with an average score of 6.6 out of 7.0. Most respondents who completed the tasks found the system user-friendly and able to access the information within an average of 1-3 minutes and many were satisfied with the Portal. However, users further indicated that the system is faced with some challenges including; inconsistencies of information in the different counties’ sites, congestion of words and links on the other language provisions. These according to the participants make the usability of the portal or system sometimes difficult to use.
4
1.0 Introduction
The evolution of technology has led to rapid development of medical software and systems in both developed and developing countries. These systems and software are now being used globally to help citizens’ access vital health information and services (Bangor, Kortum, & Miller, 2009). Usability helps to understand user’s success rate by addressing the following 1) Is the website difficult to use, 2) Is the website's information hard to read or doesn't answer users' key questions etc. There is a need understand better how real users interact with the product (web portal) and thus to improve the web portal by re-designing it based on the results. (Lippincott and Wilkins; 2004). The usability of health website Portals is an important factor in the health domain and has been adopted by developed countries like Denmark, USA and many others to help their citizens access health care information and services (Pagliari, Detmer, & Singleton, 2007).
The 1177.se Health Portal was launched in Sweden in 1998 (Sauro, 2011). 1177.se (Vårdguiden) originated from the telephone number 1177, used to call for health advice/services in Sweden. 1177.se web portal is a public service website where citizens can access and search for information about illnesses, symptoms, and treatments, as well as finding out about healthcare in their particular region or County (Saeed & Ullah, 2009). The Portal provides citizens with vital health information in form of texts, movies, animations for children and an anonymous inquiring service, where Doctors and Nurses answer questions. The portal has four main sections (1) facts and advice (2) Find clinic (3) Rules and rights (4) login (sign-in).
1177.se Portal is being used in all the Counties or regions of Sweden and connects through the National Health Information Exchange Platform (NHIEP) to the Electronic Health Records (HERs) systems, which was introduced to reduce the number of phone calls that were received per day and to avail health-related information to citizens regardless of the time (Sauro, 2011). The portal empowers patients to manage their own health through multiple services. Such as, Facts and advice, find care, e-Services (sign in) as shown in Figure 1. However, this study did not focus on the login (sign-in) section since it involves confidentiality and required permission from the ethics committee. The study, therefore, focused only on Facts and advice, Rules and Rights as well as Find care, where information is not confidential. The Figure 1 below shows the
5 home page of 1177.se website portal used by citizens in Sweden to access health services and advise.
Figure 1 showing the home page for 1177.se
According to Silas and Olof (2004) there have been many local and regional e-health projects and programs implemented in Sweden. Currently, a number of health portals are providing health services in different counties of Sweden. 1177.se portal health portal is one of the web portals in Sweden which provides different health care services to the citizens. Citizens can search on this portal for information about news on health care, maps and directions, county organization and activities, jobs, how to manage health, treatments and diseases, rights and obligations of the patient.
The portal is available in the Swedish language. However, it has provision for other languages. The cardinal role of developing the portal was to improve access to health information by the citizens of Sweden, though the portal seems to be experiencing usability challenges. including; navigation issues, inconsistencies of information (Lippincott and Wilkins; 2004). The purpose of
6 the study, therefore, was to investigate the usability of 1177.se portal in Sweden and how it has helped citizens’ access health services or information in an efficient and effective way.
1.2 Problem statement
The use of modern technology, especially internet as a major source of advice and information for health services has been increasing over time (Pagliari and P. Gregory; 2004). Currently, both developed and developing countries have continued to develop portals to help their citizens’ access information and other related services. Sweden with a population of approximately 10 million citizens has been facing health challenges due to increasing number of immigrants, asylums seekers, and refugees, whose accessibility to vital health services and information remains a problem (National Strategy for eHealth Sweden; 2006). In order for the Country to address the challenges, a number of strategies including developing website portals and establishing toll-free lines like 1177 among others were set up to help citizen’s access health information (Silas & Olof, 2004).
Besides, the Swedish Government developed the NHIEP intended to provide patients with a complete overview of their health-related data (Astrand, Hovstadius, Antonov, & Petersson, 2007). However, as the number of people using modern technology, especially internet for health advice increase, issues related to usage and trust, navigation, inconsistencies of information in the different counties, congestion of words and links on the other language provisions among the users emerged (Lippincott and Wilkins; 2004). This study, therefore, sought to evaluate the usability of 1177.se website portal in Sweden through the usability test with students from Örebro University.
1.3 Research Questions
The study was guided by the following research question.
▪ What are the users’ degrees of satisfaction with the 1177.se website?
2.0 Literature Review
The section presents debates of scholars on issues related to using of modern technology to improve access to health information in Sweden. The section highlights how a well-developed website can help improve access to health information in a given Country.
7 The evolution of modern technology has led to rapid development of medical software and systems in both developed and developing Countries. These systems and software are now being used globally in different languages and cultures to enable people to improve their quality of life(Bangor et al., 2009). However, users are concerned with the ease of use of the systems. According to a recent report by Group (2015)on the internet use, there are 70,000 health-related websites available worldwide being used by people to access health information, which is attributed to a number of reasons. These include among others; some people want to be well prepared and informed before meeting the Doctor, while others may be searching for support or substitute answers to their problems.
In Sweden, ICT based e-health applications have been implemented with the purpose of providing the citizens with quality health services and better access to health care (Pagliari et al., 2007).E-health portal refers to delivery of information and health services via the internet or related technologies (Kaptelinin & Nardi, 2006). E-health applications have been discovered to be useful in providing a better quality of life in a cost-effective way (Blaya, Fraser, & Holt, 2010). Counties in Sweden have their own health web portals for providing the useful health information to citizens and health professionals.
According to (Kaufman et al., 2003; Shea et al., 2006), a well-designed website portal empowers patients to manage their health and be active participants in health care decisions. For instance, the key success behind the Patient Health Care system in Sweden lies into successful adoption to website portals by their citizens. However, some healthcare professionals are concerned about the negative effects derived from patients searching their desired health-related information such as finding facts and advice about pregnancy without the support from clinicians (Schubert, Mühlstedt, & Bullinger, 2015). This raises the question of usability of health portals among patients.
A usable application system is easy to learn, even by those who lack technical expertise since they can simply navigate the use. According to (Jaspers, 2009; Zhang, Johnson, Patel, Paige, & Kubose, 2003)the usability of the system may bring up human errors and user dissatisfaction, which may influence them to abandon the system. The study was particularly concerned with the usability evaluation of 1177.se portal used by citizens of Sweden in accessing health services.
8 2.1 Conceptual framework
Linking usability of modern technology to accessibility of health services remains a great challenge due to failure by some website operators to monitor the people using their systems and parameters used to measure usability (Rosson & Carroll, 2002). On usability, the study adopted the definition by ISO 9241-11 standard, which defined usability as “the extent to which a product
can be used by specified users to achieve specified goals
with effectiveness, efficiency and satisfaction in a specified context of use”(Hussain, Mkpojiogu, & Hussain, 2015).
Other definitions which were adopted by the study according to ISO (1998) are Effectiveness defined as accuracy and completeness of user goal achievement. Effectiveness is calculated as % of users successfully achieving their goals versus the total number of users: normally, if the user successfully completes the task, it is noted by 1 (one) and if the user fails the task it is noted by a 0 (zero). Efficiency defined as “The resources expended in relation to the accuracy and completeness with which users achieve goals” and Satisfaction which was interpreted as “comfort and relevance of application” of the product.
According to Bevan (2009), Effectiveness and efficiency are Performance dimensions measured and related to business objective whether someone uses a product (web portal) and is satisfied with the results. The three parameters of efficiency, effectiveness, and satisfaction, therefore, guided the author in determining or measuring the usability of 1177.se portals in Sweden. This was measured using two usability models, that is to say, Single Ease Question and System Usability Scale.
SEQ was the best choice since it uses a great longitudinal measure from iteration to iteration, since users have a tendency to respond to what they are expecting given the device, fidelity of the interface and nature of the task they perform (Thimthong, Chintakovid, & Krootjohn, 2013). SUS was also selected as a best choice since it handles missing values: This uses a substitute method for estimating missing values especially when the study’s sample size was small (Thimthong et al., 2013).
9 Key performance metrics and satisfaction measurement instruments were used to collect data as indicated in Table 1 below.
Table 1: Performance Metrics and other measurement instruments for satisfaction
ISO/IEC 9126-4 Performance Metrics Other instruments
Effectiveness
Number of tasks completed successfully
Eff = X100% Total number of tasks undertaken
Source1
Efficiency
(source)2
Where:
N = The total number of tasks (goals) R = The number of users
nij = The result of task i by user j; if the user successfully completes the task, then nij = 1, if not, then nij = 0
tij = The time spent by user j to complete task i. If the task is not successfully completed, then time is measured till the moment the user quits the task.
Considering the author’s data got from the research, (Sauro et al, 2009)
Satisfaction -SUS: System Usability Scale.
(Brooke,1996)
The comfort and acceptability of use
SEQ: Single Ease Question
(Sauro et al, 2009)
3.0 Methodology
This elaborates the methods used in obtaining information that fulfills the study objective. The methodology comprises; method, Formulation of interview questions, data collection, selection of participants, data analysis as well as ethical considerations.
3.1 Method
This study was based on a summative approach to evaluation and measured in a variety of usability components, with both quantitative and qualitative methods applied. There was an
1 https://usabilitygeek.com/how-to-use-the-system-usability-scale-sus-to-evaluate-the-usability-of-your-website 2 https://usabilitygeek.com/how-to-use-the-system-usability-scale-sus-to-evaluate-the-usability-of-your-website
10 empirical study using fellow students as participants of the study as they (students) performed the tasks provided in appendix A, B, C, and D. The author evaluated the usability of the 1177.se portal in terms of effectiveness, efficiency and satisfaction.
The study adopted a case study strategy using both quantitative and interpretive (qualitative) approach. A case study strategy is an empirical inquiry adopted when there is a desire to investigate a contemporary phenomenon within its real-life context, especially when the boundaries between the phenomenon and context are not evidently (Yin, 2009). The case study strategy was adopted because it is suitable to investigate the "what" questions which aim at exploratory purposes.
Interpretive approach facilitates an understanding of phenomena from the point of view of actors directly, rather than using a prior construct (Cavaye, 1996; Klein & Myers, 1999), and an understanding of process whereby an information system influences or is influenced by the context (Walsham, 1993). A quantitative approach was used to generate descriptive statistics on usability which was later analyzed and discussed by the author.
According to Dey (2003), “descriptive statistics are used to present quantitative descriptions in a manageable form”. In a research study, we may have lots of measures or we may measure many people on any measure. Descriptive statistics helped us to streamline huge amounts of data in a sensible way. We have selected the quantitative approach via descriptive statistics and the interpretive/qualitative approach through thematic analysis and open coding technique as the suitable strategies to address the objective of the study. Indeed, this approach has facilitated us (i) to investigate the contemporary usability of 1177.se by direct users of the web portal and (ii) that the findings emerged from the data itself.
3.2 Formulation of interview questions
Interviews questions were formulated based on the relevance of the research question. Others question were got from standardized questionnaire that SUS method of collecting data uses, since the study adopts that method. (Brooke, 1996).
3.3 Data Collection
The author first introduced himself to the participants and later noted down their personal information (see Appendix A). Participants were later asked to complete the performance tasks
11 using the predesigned form given in Appendix B. No time limit was set on the tasks. Participants were briefed to perform tasks in an appropriate normal time in the same way they use other services and if you fail to complete a task, you still record the time of quitting the task. During the time of performing the tasks, the home page of 1177.se was made available open before starting the task.
Every participant used a checklist by ticking in the box matching to task completion or fail plus the corresponding starting and ending of the task time were noted down. Users were also asked to complete each task as accurately as they could. All participants used similar computer machines since computers in the University Laboratory are similar. Participants were asked to rate the tasks immediately after attempting each task with the use of the Single Ease Question (SEQ), a 7-point rating scale (see Appendix C) to assess how difficult the participants found the task. When a user provides a rating of less than 5 we ask them to briefly describe why they found the task difficult. The Single Ease Question approach helped the author to determine how difficult or easy participants found it to complete the tasks.
When it comes to collecting data with System Usability Scale (see appendix D), participants ranked each question from 1 to 5 (5 means they strongly agree, 1 means they strongly disagree) basing on how much they agree with the statement. System Usability Scale (SUS) was adopted by the author because it is a quick and reliable tool to measure the Effectiveness, Efficiency and Satisfaction of usability.
3.4 Selection of participants
Participants were selected by selective sample which is based on characteristics of a population and the objective of the study. University students were targeted but not as patients from healthcare data. The university students were targeted because of; i) the author could easily get students than other groups of the population, ii) they would understand him better in term research survey studies iii) Students are more computer experienced than other groups of the population. The author stood near the computer laboratory and he requested for the students who were bypassing if he could have their time for about 10 minutes for the purpose of an interview. The author targeted students and decided to be near the computer laboratory because it was flexible for the participants to get to the laboratory and do the interviews with ease.
12 After approaching several students, there were 9 students willing to participate in the study. He also invited 4 more students who were his friends, so the total participants were 13 students. This is in line with the argument of Macefield (2009), who observed that, the group size of 5-14 participants stand a sensible baseline range. Thus a user group of 13 participants was decided on a baseline and the author’s goal was to involve at least 12 participants following (Sauro & Dumas, 2009; Tullis & Stetson, 2004) with preferably English and Swedish language and experienced in computer use.
3.5 Data Analysis
The raw data was cleaned, sorted and condensed into systematically comparable data. Data analysis was done using the Statistical Package for Social Scientists (SPSS) and MS-excel, which helped to summarize the coded data and facilitate quick interpretation of the results.
For qualitative data, the investigator applied an objective coding scheme process data in the process commonly known as content analysis. Potential themes, categories, and patterns were closely examined to see how they actually emerged from the data in relation to the project objectives. The investigator utilized words to make narrative statements on how categories or themes of data were related. Once the themes, categories, and patterns were established, data was evaluated and analyzed to determine the adequacy, credibility, usefulness, and consistency of the information. Quantitative data was generated through data coding that yielded numbers; the numbers were analyzed using computer packages that yield descriptive and inferential statistics. 3.6 Ethical consideration
For purposes of autonomy, the author ensured that participants retained the right to determine what task they would or would not perform. Hence the author obtained their consent and requested permission before including the participants in the study.
To observe privacy of the participants, the author recognised the need to observe confidentiality of the information gathered from the participants as well as their identity so as not to jeopardise their security.
4.0 Results
This chapter presents the major findings of the study, interpretation, and analysis. The findings presented in this chapter were obtained through usability tests interviews by participants.
13 4.1. Effectiveness of the portal
The author measured the effectiveness of the portal by looking at the proportional time of completion of tasks by the participants. Applying the formulae in Table 1 below, it was found that on average, the accuracy & completeness with which the participants achieved specified tasks was at 83%.
According to Sauro and Lewis (2010), the average task completion rate is 78% (basing on an analysis of 1100 tasks) but one should always aim for 100% completion rate.
Figure 2: Proportion of completion of tasks
Figure 2 above indicates Proportions of successful task completion. On average 83.1% of the users, successfully completed the tasks. The graph shows that Tasks 2, 4, 5, 8, 9 and 10 were successfully completed. Tasks 1, 6 and 7 were above 50% successfully completed, and lastly, none of the users completed Task 3 successfully, thus it was a total failure (0%). Users had a usability problem with Task 3. They could not find the nearest clinic which is open on weekends and this was due to both in the inconsistencies in the information provided on different Counties’ websites and how the question was formulated by including the word (Vårdcentrals) in brackets which confused even those who understood the Swedish language.
The above findings are in line with Lippincott (2004) who postulated that although there is an increase in the number of people using web portals to access health information in Sweden with an increase in number of web portals developed for this purpose, many web portals have
0% 20% 40% 60% 80% 100%
Task 1Task 2Task 3Task 4Task 5Task 6Task 7Task 8Task 9Task 10 85% 100% 0% 100% 100% 76.90% 69.20% 100% 100% 100% P er cen ta ge Tasks
14 usability problems emerged including inconsistencies in the information provided on different Counties on health services.
4.2 Efficiency of the portal
The author measured efficiency of the system by looking at the time spent by the user to complete the task as indicated in the table below.
Table 4, showing task time (minutes) for each user
15 The author used the MS-Excel spreadsheets to sum up the results and later put them in equation. This gave the author the result for one task and therefore to get the results for the 10 tasks, he used the excel sheet to sum the result for all the tasks.
= 91.33 (goals/minute) = 0.703(goals/min)
130
Figure 3: showing average time for each task
The study findings in Figure 3 above revealed that, the average time to successfully complete the tasks was 1.66 minutes by all the participants. The author observed that users successfully completed Task 2 with the highest level of efficiency, within 1.1 minutes as compared to Tasks 1, 4, 5, 6, 7, 8, 9 and 10. Task 3 which was not completed by any of the users had the lowest efficiency since it took an average time 3.9 minutes.
When users were asked on the time spent to complete the tasks, many attributed the slow completion to poor translation as they used google-translator and they could not find the specific keyword they are searching for. Lastly, they revealed that the English site is congested with small words which would take much of their time to select the keyword on the link.
0 0.5 1 1.5 2 2.5 3 3.5 4
Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Task 9 Task 10 1.6 1.1 3.9 1.4 1.3 1.8 1.8 1.2 1.2 1.3 Ti m e in m in u tes Tasks
16 The author further noted that, counties have different information on the website which also leads to failure of completing a task if the user is not in Örebro County for which the author used to formulate the questionnaires. The author also noted that the website has provision for 15 languages. However, the information provided through it is not as rich and comprehensive as in its primary language Swedish. This is in conformity with the findings of Lippincott (2004) on the Usability of Health Web Sites for Older Adults in Sweden.
4.3 Satisfaction
The author determined the level of satisfaction by using the question in appendix C. The figure 4 below shows the level of satisfaction among the users as regards to the system in accessing health information.
Figure 4: Showing satisfaction level by task (7point scale) display
Findings in figure 4 indicate that, Users were highly satisfied in browsing through the web portal by searching Task 2, 4, 8, 9 and 10 with an average score of 6.6 out of 7.0. This therefore applies that, users felt highly satisfied with the Portal. However, users demonstrated significantly lower satisfaction level in “Finding how you can access healthcare services when you are asylum-seeker or without documents”, “Searching Information on Checklist for international travel”,
0 1 2 3 4 5 6 7 Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Task 9 Task 10 7-P o in t sc al e Tasks
17 “Find the nearest dental healthcare service” and “Finding how much it will cost you when you visit the emergency department”, especially for Tasks 7, 1, 6 and 5.
Users also felt difficulty in completing Task 3 and this was attributed to inconsistencies of the information provided on 1177.se portal. (The author noted that different Counties have different information on their sites and in Örebro county site “Find nearest clinics (Vardcentral) which are open on weekends” does not appear anywhere). Questionnaire analysis shows that most citizens are satisfied but they want more health services and features to fulfill their needs and to have the same information in all the different counties websites.
The study findings are further in line with Palmer (2002) who observed that Web site usability is affected by a number of factors including; the features of consistency, ease of reading, ease of getting the web site according to the user’s intentions, clarity of interaction, arrangement of information, speed and layout.
4.4 What is a Good SUS Score?
According to Brooke (1996), the average SUS score from all 500 studies is a 68. Therefore, a SUS score above a 68 would be considered good since it is above average and anything below 68 is not good. To interpret the SUS, scores (see table 6 in Appendix F) are done by converting it to a percentile rank through a process called normalizing, which takes raw SUS scores and generates percentile ranks. This study has given a percentile of 76 which is above the average , thus indicating a higher perceived usability.
4.5 Why rating less than 5
When a user provides a rating of less than 5 we asked them to briefly describe why they found the task difficult. In response to task3, prominent remarks were on the difficulty to find a clinic which opens on weekends (task3) e.g. user5 answered “Clinics are many which opens on weekends but since the question was specific to Vårdcentrals, I couldn’t find any and this attributed to inconsistencies of the information in the website” and readability, speed, and navigation issues, they mentioned of the site. Others mentioned that it was their first time to visit the site thus they are not familiar with it. This provides immediate diagnostics information right when the user is cognizant of what is driving the poor rating.
18
5.0 Discussion and Conclusion
The study findings discovered that majority of the citizens find the health portal user friendly in accessing health related information and services. The study further discovered that, despite a few challenges like language barrier, inconsistency in some information among other, 1177.se web portal is well designed and can be used as best practices for other public service delivery in terms of consideration for people with special needs and be providing secure, personal e-services. It has room for improvement in terms of ensuring content in other languages at par with the primary language, Swedish. However, as Sauro (2011) postulated that, although, task completion rate is the fundamental usability metric, its result alone do not represent the clear picture of usability or user experience. He thus associated task completion with the user perception and context. In this study, task completion rate is relatively above average as indicated in Table 7 below applying that 1177.se is user friendly.
Dimensions Benchmark Values Study Results
Effectiveness 78% 83%
Efficiency ?? 0.703(goals/min)
Satisfaction SEQ (average) SUS (Score)
4.8
68 6.6 76
In order for the author to determine the usability of the system in terms of efficiency, effectiveness and satisfaction, participants were given ten tasks to complete. The tasks were about searching information and how to use the 1177.se portal. Findings showed that majority of the tasks given to participants had been completed with ease apart from finding the nearest clinic (Vardcentral) which is open on a weekend; here no participant completed the task. The usability of the portal was measured in term of effectiveness, efficiency and satisfaction. The task completion rate is a fundamental usability metric, its results alone do not represent the clear picture of usability or user experience (Sauro, 2011). In this study, task completion rate was 0.703goal/min and the average time to successfully complete the tasks was 1.66 minutes. From the graph (see figure 3), it is observed that users successfully completed Task 2 with the highest level of efficiency, within 1.1 minutes as compared to other Tasks 1, 4, 5, 6, 7, 8, 9 and 10. Task 3 which was not completed by none of the users had the lowest efficiency since it took an average time 3.9minutes. However, Sauro (2011) revealed that the influence of prior experience
19 of users with a website, that users with previous experience rate the web portal 6% to 15% more usable than first-time users.
6.0 Future research
Although the findings indicated that majority of the respondents found the portal user friendly, further research need to be explored on the login section and the issues related to trust of the citizens on the quality of available health related information for their health problems.
7.0 Limitations of the study.
There are several limitations of this study ranging from finding appropriate participants, language issues since the web portal is primarily in Swedish and lastly time constraints. The university students were not the good representative of users and number participants who took part in the study was very little. This number cannot represent the large population of Sweden. Some participants who were not Swedish national used google translator and this doesn’t translate to 100% so there are some gaps in the translation. The time (1 month and 2 weeks) given was not sufficiently enough to carry out study right from planning, literature review, survey and up to the final write up of the paper.
20
References
Astrand, B., Hovstadius, B., Antonov, K., & Petersson, G. (2007). The Swedish national pharmacy register. Studies in health technology and informatics, 129(1), 345.
Bangor, A., Kortum, P., & Miller, J. (2009). Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability studies, 4(3), 114-123.
Bevan, N. (2009). International standards for usability should be more widely used. Journal of Usability studies, 4(3), 106-113.
Blaya, J. A., Fraser, H. S., & Holt, B. (2010). E-health technologies show promise in developing countries. Health Affairs, 29(2), 244-251.
Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability evaluation in industry, 189(194), 4-7. Cavaye, A. L. (1996). Case study research: a multi‐faceted research approach for IS. Information Systems
Journal, 6(3), 227-242.
Dey, I. (2003). Qualitative data analysis: A user friendly guide for social scientists: Routledge. Group, M. (2015). World Internet Users Statistics and 2015 World Population Stats. In.
Hussain, A., Mkpojiogu, E. O., & Hussain, Z. (2015). Usability evaluation of a web-based health awareness portal on Smartphone devices using ISO 9241-11 model. Jurnal Teknologi, 77(4), 1-5. ISO, S. (1998). 9241-11 (1998). Ergonomic Requirements for Office Work with Visual Display Terminals
(VDTs)–Part II Guidance on Usability.
Jaspers, M. W. (2009). A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. International journal of medical informatics, 78(5), 340-353.
Kaptelinin, V., & Nardi, B. A. (2006). Acting with technology: Activity theory and interaction design: MIT press.
Kaufman, D. R., Patel, V. L., Hilliman, C., Morin, P. C., Pevzner, J., Weinstock, R. S., . . . Starren, J. (2003). Usability in the real world: assessing medical information technologies in patients’ homes. Journal of biomedical informatics, 36(1-2), 45-60.
Klein, H. K., & Myers, M. D. (1999). A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Quarterly, 67-93.
Lippincott, W. (2004). Wilkins. 2004. Complete guide to documentation.
Macefield, R. (2009). How to specify the participant group size for usability studies: a practitioner's guide. Journal of Usability studies, 5(1), 34-45.
Pagliari, C., Detmer, D., & Singleton, P. (2007). Potential of electronic personal health records. BMJ: British medical journal, 335(7615), 330.
21 Palmer, J. W. (2002). Web site usability, design, and performance metrics. Information systems research,
13(2), 151-167.
Rosson, M. B., & Carroll, J. M. (2002). Usability engineering: scenario-based development of human-computer interaction: Morgan Kaufmann.
Saeed, M., & Ullah, S. (2009). Usability Evaluation of a Health Web Portal. In. Sauro, J. (2011). Measuring usability with the system usability scale (SUS). In.
Sauro, J., & Dumas, J. S. (2009). Comparison of three one-question, post-task usability questionnaires. Paper presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.
Sauro, J., & Lewis, J. R. (2010). Average task times in usability tests: what to report? Paper presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.
Schubert, D., Mühlstedt, J., & Bullinger, A. C. (2015). Usability Assessment of Medical Devices from a User Perspective: Results of a Focus Group Discussion. In Challenges and Opportunities in Health Care Management (pp. 223-235): Springer.
Shea, S., Weinstock, R. S., Starren, J., Teresi, J., Palmas, W., Field, L., . . . Wolff, L. T. (2006). A randomized trial comparing telemedicine case management with usual care in older, ethnically diverse, medically underserved patients with diabetes mellitus. Journal of the American Medical Informatics Association, 13(1), 40-51.
Silas, O., & Olof, J. (2004). A Sort Overview Of eHealth in Sweden. International Journal of Circumpolar Health.
Thimthong, T., Chintakovid, T., & Krootjohn, S. (2013). Evaluating design patterns of commercial web applications using net easy score. IJ Information Technology and Computer Science, 5(8). Tullis, T. S., & Stetson, J. N. (2004). A comparison of questionnaires for assessing website usability.
Paper presented at the Usability professional association conference.
Walsham, G. (1993). Interpreting information systems in organizations: John Wiley & Sons, Inc.
Yin, R. K. (2009). Case Study Research: Design and Methods. Essential guide to qualitative methods in organizational research (Vol. 5). Paper presented at the The Information Systems Research Challenge (Harvard Business School Research Colloquium). London: Sage.
Zhang, J., Johnson, T. R., Patel, V. L., Paige, D. L., & Kubose, T. (2003). Using usability heuristics to evaluate patient safety of medical devices. Journal of biomedical informatics, 36(1-2), 23-30.
22
APPENDICES
Appendix A:
Section I: User performance evaluation task form Introduction
My name is Robinson Reeka, am also a student here at Örebro University doing Master’s in informatics department. Am conducting a small study to see how users interact with 1177.se website in order to learn how to improve the design for users in the future. The study will take about 10- to -15 minutes of your time and the information I will get from you will only be used for this study. I will ask you to visit the website to find some simple information and then ask you some questions. It is not about checking your ability and there is no right or wrong. It is simply to see how it works and you can stop anytime if you don’t want to continue.
Section II: Personal information
If possible, we would like to know a little about your background • What is your official language?
• How would you rate your experience using computers?
1 (no experience) ---2---3---4---5---6---7---8---9---10 very experienced
23 Appendix B: User Performance Evaluation Tasks Form
• Participant ID: _________
Task-1 Search Information on “Checklist for international travel”.
Start Time: _______ End Time: _________
Task completed? Yes No
Task-2
Search information on “Symptoms of colds in children”.
Start Time: ________ End Time: _________
Task completed? Yes No
Task-3
Find nearest clinics (Vårdcentrals) which are open on weekends.
Start Time: ________ End Time: _________
Task completed? Yes No
Task-4
Search time for “Drop-in-time”.
Start Time: ________ End Time: _________
Task completed? Yes No
Task-5
Find the nearest dental healthcare service.
Start Time: ________ End Time: _________
24 Task-6 Find how much it will cost you when you visit the emergency
department.
Start Time: ________ End Time: _________
Task completed? Yes No
Task-7
Find how you can access healthcare services when you are asylum-seeker or without documents.
Start Time: ________ End Time: ________
Task completed? Yes No
Task-8
Search for facts and advice about “Protection against malaria”.
Start Time: ________ End Time: _________
Task completed? Yes No
Task-9
Find the nearby maternity healthcare?
Start Time: ________ End Time: _________
Task completed? Yes No
Task-10
Find the opening and closing time for the nearby health care.
Start Time: ________ End Time: _________
25 Appendix C: SEQ (Single Ease Question)
Participant ID: _________
Question:
(source)3
Appendix D: SUS (System Usability Scale) Participant ID: _________
Note: When you don’t understand clearly what to answer, check box number 3 as answer:
Strongly Disagree
Strongly Agree
1. I think I would like to use 1177.se
web portal frequently. 1 2 3 4 5
2. I found 1177.se web portal
unnecessarily complex. 1 2 3 4 5
3. I thought the 1177.seweb portal
was easy to use. 1 2 3 4 5
4. I think that I would need the support of a technical person to be
able to use 1177 Web Portal. 1 2 3 4 5
5. I found various parts of 1177.seweb
portal were well integrated. 1 2 3 4 5
26 6. I thought there was too much
inconsistency in 1177.seweb portal
functions. 1 2 3 4 5
7. I would imagine that most people would learn to use 1177.seweb
portal very quickly. 1 2 3 4 5
8. I found 1177.seweb portal very
cumbersome to use. 1 2 3 4 5
9. I felt very confident using 1177.seweb portal.
1 2 3 4 5
10. I needed to learn a lot of things before I could get going to get with
1177.seweb portal. (source)4
1 2 3 4 5
27 Appendix E
Table 3 showing the tasks and how respondent completed them
Frequency Percent
Task 1
Search Information on “Checklist for international travel
11 84.6
Task 2
Search information on “Symptoms of colds in children
13 100
Task 3
Find nearest clinic (Vardcentral) which is open on weekends
0 0
Task 4
Search time for “Drop-in-time 13 100
Task 5
Find the nearest dental healthcare service
13 100
Task 6
Find how much it will cost you when you visit the emergency department
10 76.9
Task 7
Find how you can access healthcare services when you are asylum-seeker or without documents
9 69.2
Task 8
Search for facts and advice about “Protection against malaria
13 100
Task 9
Find the nearby maternity healthcare
13 100
Task 10
Find the opening and closing time for the nearby health care.
28 Appendix F
Table 6 Showing average scores using SUS (System Usability Scale)
Questions Average
Scores Q1 I think I would like to use 1177.se web portal frequently 5
Q2 I found 1177.se web portal unnecessarily complex 5
Q3 I thought the 1177.se web portal was easy to use. 5
Q4 I think that I would need the support of a technical person to be able to use 1177 Web Portal
2
Q5 I found various parts of 1177.se web portal were well integrated 4 Q6 I thought there was too much inconsistency in 1177.se web portal
functions.
4
Q7 I would imagine that most people would learn to use 1177.se web portal very quickly.
4
Q8 I found 1177.se web portal very cumbersome to use. 2
Q9 I felt very confident using 1177.se web portal 5
Q10 I needed to learn a lot of things before I could get going to get with 1177.se web portal
