Socio-instrumental Aspects in Usability Heuristic Studies: Systematic Review

(1)

Socio-instrumental Aspects in Usability

Heuristic Studies: Systematic Review

Farrell Yodihartomo

Subject: (Information Systems)

Corresponds to: (30 hp)

Presented: 2

nd

Semester, 2020 - VT 2018

Supervisor: Franck Tetard

Examiner: Mudassir Imran Mustafa

(2)

2

Sammanfattning

Nuförtiden uppfattas användbarhet huvudsakligen som en grund för den instrumentella utvärderingen av ett informationssystem (IS) kapacitet. I synnerhet är heuristisk utvärdering (HE) of thumb-thumb användbarhet allmänt känd och används som en kostnadseffektiv och effektiv inspektionsbaserad användbarhetsutvärdering. Det är emellertid uppenbart i tidigare forskning att det finns kommunikativa åtgärder som fångas när användare interagerar med ett system. Följaktligen uppfattar socio-instrumentell (SI) perspektiv en viss händelse av användarens interaktion mot system (social handling) består av både instrumentell aspekt och social aspekt. Vår studie syftar till att förklara fullständigheten av de senaste HE-studierna, och också att beskriva underliggande samband mellan inspektionsbaserad användbarhetsutvärdering och socio-instrumentella aspekter. Vi genomförde en kvalitativ systematisk granskning genom tretton kvalificerade dokument som vi bedömde från att filtrera ut 89 dokument som består av forskningsartiklar under de senaste fem åren och befintliga professionella rapporter. På ett kvalitativt sätt har vårt studieresultat identifierat de kvalificerade dokumenten som ger en positiv grad av fullständighet. Vi har också analyserat olika användbarhetsproblem som är kopplade till specifika heuristik- och SI-aspekter. Det är tydligt i denna studie att SI-perspektivet har bidragit till att avslöja den kommunikativa grunden bakom användarens sociala handling gentemot systemet. Dessutom föreslår vi därför förbättringsförslag för ytterligare arbeten för att hantera bristerna i den aktuella studien.

Abstract

Nowadays, usability mainly perceived as a basis of the instrumental evaluation of an information system (IS) capability. Notably, rule-of-thumb usability heuristic evaluation (HE) is widely known and used as a cost-efficient and effective inspection-based usability evaluation. However, it is evident in prior research that there is communicative action captured when users interact with a system. Accordingly, socio-instrumental (SI) perspective perceives a particular event of the user’s interaction toward system (social action) is composed of both instrumental aspect and social aspect. Our study aims to explain the completeness of recent HE studies, and also to describe underlying connections between inspection-based usability evaluation and socio-instrumental aspects. We conducted a qualitative systematic review throughout thirteen qualified documents we appraised from filtering out 89 documents which consist of research articles within the last five years and existing professional reports. In a qualitative manner, our study result has identified the qualified documents present a positive degree of completeness. We have also analyzed various usability problems that are connected with specific heuristics and SI aspects. It is evident in this study that the SI perspective has contributed to revealing the communicative rationale behind the user’s social action toward the system. In addition, we, therefore, suggest improvement suggestions for further works to address the shortcomings in the present study.

(3)

3

Acknowledgement

First of all, I would express my sincere gratitude to The Lord Almighty who provides the way and blessings in my life and good health during this pandemic situation so that I have been able to finish the thesis work.

Most importantly, I would like to express my gratitude to Franck Tétard, as my supervisor, who already invested a lot of his time to guide this thesis work into a well-structured and have a distinct quality. Franck always give me a positive argumentation which makes me think deeply about the reasons behind things I included in the document. In the midst of pandemic of COVID-19 situation which forced me to shift my topic to another theme, he was able to suggest an interesting literature which inspires me to synthesize this thesis topic. He also encouraged me to have a routine progress reporting thus I can manage to make weekly progress. Without his guidance, I might not be able to finish the thesis properly. I also would like to thank my parents and my friends who always pray and give a huge support to finish this master’s degree study.

(4)

4

List of Figure

Figure 1 - Situational Usability (Hertzum, 2010) ... 13

Figure 2 - User Centered Evaluation (ISO, 2010)... 15

Figure 3 - A Communicative perspective on user interface (Sjöström and Ågerfalk, 2003)... 17

Figure 4 - Social world of SIP Ontology (Goldkuhl, 2002)... 18

Figure 5 - The Socio-instrumental Action (Petersson, 2008) ... 19

Figure 6 - Research's SLR study design... 23

Figure 7 - Document Selection Phase ... 31

Figure 8 – A Sample of Highlighted Screenshot of the dashboard (Fraser and Ball, 2017) ... 36

Figure 9 – A Sample of Raw Extraction of Interview/Questionnaire Result (Milham et al., 2003) ... 37

Figure 10 – A Sample of Verbal Narration of User's Response (Babaian et al., 2010) ... 38

Figure 11 - A sample of detailed report (Milham et al., 2003) ... 39

Figure 12 - A sample of detailed qualitative explanation in paragraph (Baguma, 2018) ... 39

Figure 13 - A sample of a document which include usability problems (Pertiwi et al., 2018) ... 40

List of Table

Table 1 - Research Objectives... 9

Table 2 - Research Questions ... 10

Table 3 - Ten Nielsen's Usability Heuristics (Nielsen, 1994a). ... 15

Table 4 - Socio-Instrumental usability (Habermas, 1984; Ågerfalk and Eriksson, 2006) ... 19

Table 5 - Search String used for Literature Review ... 25

Table 6 - Inclusion and Exclusion of Practical Screening Criteria ... 25

Table 7 - Data Extraction Features... 27

Table 8 - Completeness Criteria of Articles and Documents ... 28

Table 9 - Search Result from the Article Sources ... 30

Table 10 - Qualified Documents ... 32

Table 11 - Completeness Assessment based on Appraisal Criteria ... 34

Table 12 - Usability Problems Captured based on Socio-instrumental Factors... 41

Table 13 - Expert user participant of HE studies ... 42

Table 14 - List of Violated Heuristic in context of Desired Outcome ... 48

Table 15 - List of Violated Heuristic in context of Relative Expenditure of Resource ... 50

Table 16 - List of Violated Heuristic in context of Comfort and Positive Attitude ... 54

Table 17 - List of Violated Heuristic in context of Mutual Understanding ... 56

(7)

7

Glossary and Abbreviation

HCI – Human Computer Interaction SI – Socio-instrumental

HE – Heuristic Evaluation, sometimes also mentioned as Usability Heuristic Evaluation (UH) IS – Information Systems

IT – Information Technology AC – (Quality) Assessment Criteria

Social Actions – intended action by the actor or actors, takes account of the behavior of others and is thereby oriented in its course. Consist of Instrumental Action and Communicative Action (Ågerfalk and Eriksson, 2006).

Communicative Act – an action performed where a mutual understanding and commitment are created and fulfilled.

(8)

1. Introduction

This chapter explains the background, the motivation of this study, research questions. At the end of this chapter, it presents the research delimitations and brief outline of this research paper and what has already been discussed in this study would be shortly mentioned.

1.1. Background

Today, usability is a vital concept in the human-computer interaction (HCI) field which has been perceived as “user friendliness” or “capability of use” regarding information system (IS) (Quiñones and Rusu, 2017; Tractinsky, 2018). This concept has been emerged due to the rapid growth of technology products and the necessity of IS functionality to solve complex problems but hard to operate (Hertzum, 2010). Obviously, people will expect a product or IS to be useful, easy to learn, pleasant to use, have suitable functionalities based on their working context, thus the system should be designed to cater to these necessities (Gradišar et al., 1985). As one of the most influencing aspects in the technological area, usability can play an important role in addressing the issue of technology acceptance criteria based on personal needs or organizational needs, and its utilization can be observed through its actual use (Ågerfalk and Eriksson, 2006). Accordingly, the demand for the procedure to evaluate usability started to emerge. Usability evaluation is a widely studied area that tests and inspects the product or service based on its efficiency, effectiveness, and satisfaction which aligns with the system common ground, awareness, and coordination within an organization context (Hertzum, 2010; ISO, 2010). This supposed to have a set of appropriate methods to apply it into practice. The International Standard Organization (ISO) argued inspection-based evaluation is a cost-effective and competent method to assess system usability and complement user testing (ISO, 2010).

(9)

9 Eriksson, 2006; Rittgen, 2007). However, a systematic understanding of how the socio-instrumental approach contributes to usability is still lacking. There are previous works of literature which present the systematic literature review of usability heuristics study, but those study not concerning specifically into social-oriented factors (Jimenez, Lozada and Rosas, 2016; Von Wangenheim et al., 2016; Quiñones and Rusu, 2017). To date, the existing researches has tended to focus on the development of socio-instrumental theories rather than finding fragments of socio-socio-instrumental initiatives in different pragmatical studies and see how the established conventional ways perceived it (Goldkuhl, 2002, 2017; Sjöström and Ågerfalk, 2003; Ågerfalk and Eriksson, 2006; Rittgen, 2007; Petersson, 2008). Due to this, it is interesting to do a study regarding socio-instrumental usability in Heuristic Evaluation (HE). In addition, this research is envisaged to contribute to socio-instrumental literature and also to gain a deeper understanding of how far that socio-instrumental theory has been appraised in heuristic evaluation studies.

1.2. Motivation of The Study

This research is aiming to understand the notion of social action behind the usability heuristics development which had been established by the various researcher and experts. Accordingly, identifying the socio-instrumental aspect in various studies, including the underlying rationales, motivates the author to describe and exploit the empathetical side of human social actions toward information systems in current studies.

Subsequently, there are research objectives that become our focus in order to maintain the scope and target of this study. The objectives are listed in Table 1.

Table 1 - Research Objectives # Research Objectives

1 To present a systematic review of how the existing usability heuristics were developed and utilized

2 To identify connections between recent heuristic evaluation study with IS socio-instrumental

Referring to our research objectives, we attempt to bring out an unrevealed conceptual connection between the IS and HCI domain in terms of social action. We assumed the reader has prior experience or understand the general idea of usability and user actions. Hence, the expected readers who will be targeted as the audience of this topic there are:

(10)

10 3. The English-speaking reader who have interest with social and technology topic

1.3. Research Questions

Based on the research objectives, those bring out the main question that derives several research questions that are required to be satisfied with a thorough discussion and analysis in regard to the social and instrumental phenomenon in HE studies due to their constraints.

Therefore, Table 2 presents the research questions of this study along with the motivation of why the questions should be asked with respect to the research interest.

Table 2 - Research Questions

RQ-ID Research Questions Motivation

RQ1 What are the aims or objectives of existing heuristic evaluation studies?

Collect tacit and explicit statements about the reason for conducting a heuristic study RQ2 How are the links between socio-instrumental

aspects with the existing heuristics?

To identify the relation between underlying consideration of

socio-instrumental factor and usability problems in recent HE studies.

RQ3 How socio-instrumental aspects are being articulated in the existing usability study?

To exploit and discuss the external and internal influences that lie behind the existing studies of HE

The social communication exists between human and IT/IS system is observed as an understanding of user’s action in a context of use, and indeed it is more complex to be measured compared to the instrumental aspect (Ågerfalk and Eriksson, 2006). As one of particular concern argued by Ågerfalk and Eriksson (2006) that the view of usability on a system in earlier studies focused on Desired Outcome and abandoning social aspect of usability (ibid.). Since the research and the theory development had been conducted in 2006, there is a year gap of today’s usability studies which the orientation of study has probably shifted throughout the years. Therefore, the view of authors and information completeness regarding users’ action in recent studies within this period are necessary to be observed and reviewed in order to answer the questions.

(11)

11

1.4. Delimitation of Study

In this research, it is limited to the literature review and analysis based on reports, research articles, and journals that relevant to the field of IS study and usability heuristics evaluation. Since we concern about underlying social actions in prevalent HE research articles, it is prominent to preset a range limit of publication date. Consequently, we set the five years range of the publication year (from 2016 to 2020) as our scope of study in order to keep the focus on the latest publication and reduce the irrelevant search result. Further fieldwork or constructing a specific theory regarding this study is beyond the scope. This thesis will present the chart and reports that produced from analysis, data extraction and synthesizing perspectives. The gathering of qualitative information and its analysis are the main focus of this study. Although some quantitative findings will present in a form of tables and charts, those will help to describe the collected information without statistical analysis calculation (i.e. correlation and significance) taken into account. The reader should bear in mind that the study is based on articles that exist in the selected online journal library.

Due to the topic of usability and socio-instrumental perspective, there are limitations upon the keyword use in terms of the strategy of searching, the observed period of existing research, and article criteria which will be discussed further in Chapter 3. Overall, the study focuses to answer research questions hence another potential problem is that the scope of this thesis is too broad and may be discussed in short as additional information.

1.5. Thesis Outline

This study consists of seven chapters altogether. Firstly, this thesis work will introduce the motivation and background of reviewing usability heuristics studies in Chapter 1. It will be then continued to

Chapter 2 which describes the supporting theories of usability, user experience, socio-instrumental

usability, and the theory of systematic literature review. Chapter 3 refers to the method and study design that taken into account for this review. Chapter 4 explains the findings collected throughout the study based on the existing usability heuristics studies. Chapter 5 presents the analysis of the collected findings in order to gain a deeper understanding of the phenomenon in recent usability heuristics studies.

Chapter 6 deals with the discussions of causes, opportunities, and the qualitative result of this review.

(12)

12

2. Theory

This chapter will discuss and explain the main theoretical background of the thesis. The theory of Usability, Socio-instrumental usability and pragmatism, and the systematic review theory are discussed under this section. In addition, the links between those theories including synthesizing concepts are discussed under a sub-chapter here.

2.1. Usability

The evaluation of usability towards an implemented or a concept design has become a substantial activity in the human-interaction design process. The term of usability can be multi interpreted depending on how the usage of concept is being denoted (Hertzum, 2010). The ‘usability’ term is commonly used as “capability of being used” where it refers to the ease of use and the way users can accomplish their intended tasks using the designated tool (Quiñones and Rusu, 2019). Similarly, the definition of usability is also derived by ISO where it states that “extent to which a system, product or

service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (ISO, 2010). In a detailed manner, Nielsen (2012) defined the

word ‘usability’ is “a quality attribute that assesses how easy user interfaces are to use” and it has five quality components (Nielsen, 1994c):

1. Learnability: How easy for users to operate the system at the accomplish basic tasks the first time.

2. Efficiency: Once users have learned and familiar with the design, how fast can they accomplish the tasks.

3. Memorability: If the users have not used the interface for a long period, how fast they can recall, getting re-adapt with the system.

4. Errors: Number of errors encountered, the severity of each error, effort required to recover the error.

5. Satisfaction: Users’ feelings towards the system to accomplish tasks.

(13)

13

2.1.1. Situational Usability

Usability in this context is related to the quality-in-use of a system in a specified situation with its users, tasks, and the wider context of use. Situational Usability is understood through people, task, and other conditions that involved in developing the situation (Hertzum, 2010). By its characteristic, situational usability is determined by (ibid.):

1. Considering the whole situation, not only from the technological attribute.

2. The existence of the actual use situation becomes important to see whether the system can be useful.

In the early research of human-computer interaction (HCI), the term of ‘Usability’ in the situational context was declared by ISO 9241 where the definition is: “Extent to which a system, product or service

can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” (ISO, 2010). Based on that definition, usability term derived into facets of

effectiveness, efficiency, and satisfaction which have the description as follows (ISO, 2010): 1. Effectiveness: Accuracy and completeness with which users achieve the intended goals. 2. Efficiency: Resources expended to the accuracy and completeness with which users achieve

their goals

3. Satisfaction: Freedom from discomfort and positive attitudes towards the use of the product.

Figure 1 - Situational Usability (Hertzum, 2010)

(14)

14 users sometimes have an unclear objective when they are given a tool to be used until the context and the situation are defined. This mismatch between the tools, people and goals is tackled when the gap between facets is reduced through an actual use (i.e. training) to understand the context, for what purpose the system is used, and learn by practice (Hertzum, 2010).

In usability concept, the context of use takes several elements into consideration which construct the notion of ‘context’ once these elements constructed, such as range of user group types, characteristic and behavior of each user group, and user’s tasks and goals, and environment of the system (ISO, 2010). A better quality of system or product can be achieved if the context of use is explicated in adequate detail (ibid.).

Overall, the situational usability allows the users to explore new understanding if the system is used in the different context align with their intention and finding the new requirement they need to improve their work (ISO 9241, 1999; Hertzum, 2010). The actual situation and practice of situational usability can be experienced through conducting task analysis, prototyping, and usability evaluation (Hertzum, 2010). Apart from that, there is a risk in these practices. Situational usability practices considered less effective to reveal the potential of collaborative issues or environmental aspects since the techniques are focusing the test on one subject at a time in an artificial situation (Hertzum, 2010, 2016).

2.2. Usability Heuristic Evaluation

In 1990, Nielsen and Molich conducted the fundamental study of heuristic evaluation of user interfaces where it defines the heuristic evaluation itself as a set of activities to look at an interface and trying to construct opinions about what is good and bad about the evaluated interface (Nielsen and Molich, 1990). The aims of the evaluation are defined as those following points (ISO, 2010):

1. Collect new information about user needs,

2. provide feedback on strengths and weaknesses of the design solution from the user's perspective 3. Assess whether user requirements have been achieved

4. Establish baselines or make comparisons between designs.

(15)

15 Figure 2 - User Centered Evaluation (ISO, 2010)

It is important for the evaluators to have the general picture of the system and its features navigation in order to evaluate the usability. Moreover, the evaluators are also possible to apply a specific set of existing or customized heuristics or determine aspects of User Experience to be evaluated depending on the scope of the system (Quiñones and Rusu, 2019).

2.2.1. Usability Heuristics

Today, Usability Heuristic Evaluation (HE) is currently one of the most mentioned and applied practices of usability techniques (Hollingsed and Novick, 2007). The widely used usability heuristics is the Ten Usability Heuristics which are developed by Nielsen and Molich, who are the pioneer and one of the most influential researchers in the HE domain (Nielsen and Molich, 1990; Alonso-Ríos, Mosqueira-Rey and Moret-Bonillo, 2018). Nielsen designed a set of heuristics for conducting an inspection of usability to a system with nine heuristics at the beginning of the development, and it was improved to become the ten Nielsen’s usability heuristics (see Table 3) (Nielsen, 1994a). In his earlier research, there was a significantly increasing number of identified problems from one evaluator to five evaluators (Nielsen and Molich, 1990). It showed that the optimum number of participants for heuristic evaluation was three to five participants.

Table 3 - Ten Nielsen's Usability Heuristics (Nielsen, 1994a).

# Heuristics Description

H1 Visibility of System Status The system should always keep users informed about the ongoing events through appropriate messages in a period of time.

H2 Match between system and the real world

The system should speak the familiar users’ language which follows the real-world convention, making information appear naturally and logical.

(16)

16

# Heuristics Description

H4 Consistency and Standards Users do not have to wonder different situations or the action where lead to the same meaning.

H5 Error Prevention The system notify user which require confirmation before they commit an action which lead to the permanent change or error. H6 Recognition rather than Recall Reduce the users’ memory load to accomplish action by providing

objects, actions, and options visible and easily retrieved.

H7 Flexibility and Efficiency of use The system allows novice users to accelerate the interaction and makes the expert can tailor the action easily.

H8 Aesthetic and minimalist design Dialogues should not contain irrelevant information which diminish the relevant information visibility.

H9 Help users recognize, diagnose and recover from errors

Error message explained in a plain language, precise, and suggest a solution to the users.

H10 Help and documentation The system provides help and documentation for the users to make them easier to retrieve information and list of concrete steps to achieve goals.

Due to the uncertainty of severity range where Nielsen has only diversified the usability problems into a hard and easy problem, he came up with a rating scale for classifying the issues based on the level of severity (Nielsen and Molich, 1990; Nielsen, 1994b). The following rating scale to measure the severity of usability problems is described below (Nielsen, 1994b):

0 = I don't agree that this is a usability problem at all

1 = Cosmetic problem only: need not be fixed unless extra time is available on the project 2 = Minor usability problem: fixing this should be given low priority

3 = Major usability problem: important to fix, so should be given high priority 4 = Usability catastrophe: imperative to fix this before the product can be released

(17)

17 theoretical practice, and still relevant to date, such as (ibid.): (1) The method sometimes identifies usability problems without providing direct suggestions for how to solve them; (2) The method is biased by the current mindset of the evaluators; and (3) normally does not generate breakthroughs in the evaluated design.

2.3. Socio-Instrumental Usability

Various methods and criteria for assessing and measuring the quality of IT systems have been developed in recent years. The contemporary view of determining the requirement of the IT system is emphasizing the two types of requirements: Functional and Non-functional (Ågerfalk and Eriksson, 2006). The orientation of classifying the systems based on the function makes sense in a narrow ‘product-oriented’ view of usability (ibid.). However, As International Organization for Standardization defined usability under three criteria: effectiveness, efficiency, and satisfaction which has been discussed in the previous chapter, this has broadened the understanding of usability with respect to the behavior of users toward certain goals (Ågerfalk and Eriksson, 2006; ISO, 2010). Between the action taken and the end-goals, the action plan intervenes the reason for action based on the interpretation of social context (Habermas, 1984; Ågerfalk and Eriksson, 2006).

Following the rationale of how an IT system should be perceived, the knowledge of IS plays a role in the perspective toward user acceptance, satisfaction, playfulness, and self-efficacy of IT (Freeman, 2001). This shows communication between the user and the technology in use. Furthermore, in their study, Sjöström and Ågerfalk (2003) point out that there is a concept of pragmatic duality which describes a multifunctional phenomenon in the use of information systems: ‘when interacting with an IT-system through its user interface, people often also communicate messages to other users of the system’ (see Figure 4) (Sjöström and Ågerfalk, 2003).

(18)

18 Habermas (1984) argues that communicative action is a purposed-based language which is coordinated by shared understanding where a set of actions are comprehended and conducted along with the agreed and fulfilled commitments (Habermas, 1984). Consequently, this also means that communicative action provides instrumental orientation due to the agent’s goals, and social orientation due to the commitment and understanding (Ågerfalk and Eriksson, 2006).

The theoretical of dimensions where “social” and the system perceived as an “instrumental” are constructed under the term of “socio-instrumental” or “socio-pragmatic” concept (Goldkuhl and Ågerfalk, 2004). The socio-instrumental concept is deemed to describe the behavior of the user has a relation with the value, rationality, and norms that intervene in the user’s decision and action (Ågerfalk and Eriksson, 2006). The main feature of socio-instrumental pragmatism theory is that the concept should be effectuated in order to gain an understanding of IT instruments and the user or organization in the work field within a setting. Goldkuhl and Ågerfalk (2004) mentioned that there are multifunctional events under a single action at the same time (Goldkuhl and Ågerfalk, 2004):

a. a response to an action made by someone else (i.e. socially responsive); b. an expression of subjectivity;

c. a utilization of immaterial instruments (i.e. knowledge and language); d. an utilization of external instruments (e.g. technical artefacts); e. a compliance to organizational norms (e.g. role expectations); f. a production of action results (semiotic or material objects); g. a realization of values and intentions;

h. an attempt to influence someone else (i.e. exerting power).

(19)

19 In the literature of socio-instrumental, Ågerfalk and Goldkuhl (2004) also mentioned the way of understanding IS and organizations based on the interaction of human actions assisted with instruments and actuated within the social world (Goldkuhl and Ågerfalk, 2004). In his early research, Goldkuhl (2002) defined the fundamental idea of SIP ontology into six ontological categories (Goldkuhl, 2002):

(1) Human: main actors of the social world;

(2) Human Inner World: knowledge of the human actors;

(3) Human actions: intervention of human towards the external world; (4) Signs: a result of the communicative action which is meant to be read;

(5) Artifacts: things or object that artificially built to perform material actions; and (6) Natural Environment: objects that naturally exist.

Figure 5 - The Socio-instrumental Action (Petersson, 2008)

Socio-instrumental actions consist of two main elements: actors and action objects (Petersson, 2008). Likewise, the two dimensions of the socio-instrumental action are emphasized in the sense that there is a mutual connection between knowledge, goals, and action that leads to a holistic interpretation of this phenomenon (see Table 4). In Figure 5, the action condition affects the action result made by the actors where they do a pre-assessment of current social context or earlier actions. The IT instruments can determine this alteration from action result in a condition which enable or disable the process (ibid.). Table 4 - Socio-Instrumental usability (Habermas, 1984; Ågerfalk and Eriksson, 2006)

Socio-instrumental Dimension

Goals Knowledge Action Usability Criteria

Instrumental Instrumental Goals: Achieving given ends Technical-cognitive knowledge

(20)

20

Socio-instrumental Dimension

Goals Knowledge Action Usability Criteria

• Satisfaction: Comfort and positive attitudes

Social Social Goals: mutual understanding based on social context Practical Knowledge, Emancipatory Knowledge Communication Act: two-ways of interaction between actors-to-actors or actors-to-instrument • Effectiveness: Mutual Understanding • Efficiency: (None) • Satisfaction: Trust

(21)

21

3. Method

In this section, we will describe and motivate the methodology that is used for the study, the suggested methods, and how the methods were applied. The study design is illustrated in this section including the search strategy, how the researcher applied the practical selection, the method of data extraction, data quality appraisal, and analyzing the findings. Limitation of the study is also discussed to give an insight into what are the scope of work and the sources of information being used for analysis.

3.1. Systematic Literature Review

The Systematic Literature Review (SLR) is one of the methods in the IS domain which is widely used for supporting claims in a scholarly manner with scientific evidence (Okoli and Schabram, 2010). It is conducted for various objectives with the purpose to distill the essential factors from relevant articles (ibid.). There are multiple definitions of literature review in a practical way from different authors (Oates, 2006; Kitchenham et al., 2007; Okoli and Schabram, 2010). In a solid definition, the SLR term which has been defined by Kitchenham (2007) succinctly embodies the concept itself:

“A systematic literature review is a means of identifying, evaluating, and interpreting

all available research relevant to a particular research question, or topic area, or phenomenon of interest.” (Kitchenham et al., 2007).

The systematic literature review process, in this level of research, is expected to provide a qualitative or quantitative proof from the rigorous and structured way in order to support the author’s justification as an expert on the domain of knowledge, and also to review some articles transparently with an analytical criticism (Okoli and Schabram, 2010). Okoli (2015) introduced eight steps of conducting SLR which are briefly described as follows (Okoli, 2015):

1. Clearly identify the purpose and goals of the review.

2. Establish the protocol and train participants (reviewers) for following a detailed procedure in the execution of the review.

3. Establish explicit details of the literature search for ensuring the comprehensiveness of the search.

4. Establish criteria for including/excluding studies from the review.

5. Establish criteria for excluding studies due to their insufficient quality to be included in the review synthesis.

6. Systematically extract the relevant information from each selected study.

(22)

22 The common backgrounds to carry out SLR as an IS research project such as: to summarize the prior studies regarding empirical evidence or treatments; to identify gaps in the contemporary study with an aim to investigate the topic any further; to present the background for doing new research initiatives; to (Kitchenham et al., 2007).

The advantage of having SLR as a method is to reduce the bias of literature, provide a wide range of information collected from different settings and empirical methods to elaborate a phenomenon (Kitchenham et al., 2007). Another point is that SLR for IS research enables one to balance the qualitative and quantitative methods which particularly incorporates social science and computer science (Kitchenham et al., 2007; Okoli, 2015). Hence, the SLR method is preferred for a study where it is suitable for social and engineering topic which examines the existing implementations of a particular method. Having said that, the SLR is an evidence-based practice that was developed to synthesize the best quality scientific studies on research questions supported by accumulated of documented scientific experiments (Kitchenham et al., 2009).

“To provide the means by which current best evidence from research can be integrated

with practical experience and human values in the decision-making process regarding the development and maintenance of software” (Dybå, Kitchenham and Jørgensen,

2005).

In the context of this IS research, the SLR method is applied in order to answer the research questions so that the result of this research can suggest recommendations for future research (see Chapter 1). By following the SLR steps designed by Okoli (2015) to conduct SLR in the IS domain, the importance of a socio-instrumental perspective in current HE studies can be elaborated, analyzed, and synthesized through a structured systematic review (Okoli, 2015). The detailed explanation for each step in the study is discussed in this section’s subchapters.

3.2. Study Design

Prior to carrying out the systematic review, it is therefore fundamental to establish a study design. As we refer to Okuli’s eight steps, this study adopted the principles and adjust the variables and method used. (see Figure 6) (Okoli, 2015). The adjustment of Okoli’s process into our study design is purposed to make concise the process so it fits to help the study reach the objective and to employ the core principles of IS research. There are six main stages of the process that have to be done in chronological order according to Figure 6.

(23)

Socio-23 Instrumental perspective in HE is seen as an artifact in terms of this study design which is produced from the literature study of how important the research is. The RQs have to be answered through later processes.

Secondly, we determine the type of documents and criteria of research articles that might be the best fit according to the scope of this research topic. The strategy of this search limits the documents information contained in the articles based on the feature that relevant to the topic keyword, such as the publication years, HE practices, and more to mention in the next following subchapter. Although using a similar searching strategy, the online libraries and the professional report documents source are separated into two sources of information as illustrated in Figure 6. Furthermore, we apply the practical filter in order to sort out the number of articles and documents into a reasonable amount for the researcher to analyze. The selection criteria are separated into three layers: Preliminary, First Phase, and Second Phase (see Chapter 3.3). The selected documents will be stored in the document database management system for the next process. The document criteria and the library sources are also defined in Chapter 3.3.

Figure 6 - Research's SLR study design

Thirdly, the selected documents and articles are stored in the document reference management system. In this study, we manage the references using Mendeley as the free open-source tool for storing bibliographical information, classifying documents, and generate citations (Zaugg et al., 2011). The relevant information is uprooted from the HE documents and articles according to the Socio-Instrumental ontology and theoretical aspects as the features of data extraction (Ågerfalk and Eriksson, 2006; Rittgen, 2007).

(24)

24 information of HE and user’s interaction with the tested system (Okoli, 2015). The appraisal process is focused on qualitative facts which are relying on the Quality Checklist and Qualitative Questions. Once the extracted document or article meets the minimum QA criteria, it is included in the findings to be analysed to see if there are any inferences, suppositions, or oppositions between the theory and the findings. Furthermore, the findings and analysis phase aggregates the consensus that exists in the findings within a particular context. It is possible to synthesize an explanation by integrating the result of quantitative and qualitative evidence (Okoli, 2015). Reflecting from the earlier studies, illustrations and designs are expected and produced in this phase to help the researcher establish their ideas and relations between facts (i.e. textual rhetorical devices, visual maps, and charts) (ibid.).

Finally, the discussion in regard to the constructed analysis of HE and Socio-instrumental theory and reviewing the whole process from the beginning until the end of the study are reported at the end of the process. The conclusion of the occurred phenomenon and answered research questions are also presented in a critical way by using supporting pieces of evidence. In addition, recommendations for further research are also suggested in this stage. The detailed execution and process of this study design are explained in the next following chapters.

3.3. Documents Search Strategy and Practical Screening

Determining the rules in searching literature is a required step to reduce the number of the occurrence search result on the various sources (Okoli and Schabram, 2010). It is not realistic to have thousands of articles and analyze all of them (Okoli, 2015). The goal of setting the inclusion and exclusion rule is not only to reduce the number of feasible articles to be analyzed but also to filter the relevant and useful articles to answer the research question or hypothesis (ibid.). In order to specify the search, the searching process is performed in several main digital libraries, such as ScienceDirect, IEEE Explore, ACM Digital Library, Emerald Insight, and Web of Science. These online libraries were considered as reliable sources for research purpose and have been widely used by prior studies (Kitchenham et al., 2007; Fernandez, Insfran and Abrahão, 2011; Jimenez, Lozada and Rosas, 2016; Vieira, Silveira and Martins, 2019). The reader also should bear in mind that a number of journals and books from the online library sources are also included in this study as references for special cases.

(25)

25 Table 5 - Search String used for Literature Review

Concept Keyword Search String

Usability “Usability OR Usabi*” “Usability AND Heuristic” “Usability AND Evaluation” Heuristic “Heuristic OR Heur*”

“Heuristic”

Evaluation “Eval* OR study OR examinat* OR test* OR method* OR assess* OR design”

(*additional) Professional Report (“Company” or “Professional” or “Industry”) AND (“Report*” OR “Doc*”)

(*additional): This keyword is used additionally to search on Google Scholar in order to find the professional reports

Conducting an extensive search of the literature in the usability heuristic evaluation gives the insight to know how many studies had been published in specific areas. In order to produce the desired search result, search strings are determined based on the topic keywords which are assumed may include the specific coverage of the HE area. The search string used in the study is as described in table 5.

Table 6 - Inclusion and Exclusion of Practical Screening Criteria

Stages Inclusion Criteria Exclusion Criteria

Preliminary Search String Strategy (Table 5); Filters:

• Publication year*: 2016-2020; • Document type: Research

Articles, Professional Reports, proceeding articles, thesis documents;

• Language: English

• Publication year is older than 2016; • Non-English

• Non-usability heuristics articles; • Duplicate articles;

• Locked Access Articles;

First Stage • Usability heuristics study; • Propose or use a set of Usability

Heuristics

• Non-usability heuristics articles; • Duplicate articles;

• Q4 Articles;

• Bachelor thesis document** Second Stage

(Qualitative Screening)

• Have thorough descriptions of each usability heuristics; • Have research results of UH

implementation;

• Reports the usability problem of UH evaluation;

• Have relevancy with UH but not emphasizing on the usability heuristics application;

(26)

26

Stages Inclusion Criteria Exclusion Criteria

• Answer the research questions

*: only applies to filter the search results on the online library **: only applies to filter thesis documents and professional report

The search range for existing HE studies is scoped only for the research articles within five years, from 2016 to 2020, to see the generality of research progression. Furthermore, the reason for choosing research articles and professional reports is to provide a comparative study of presenting the HE experiments information and enrich the reporting completeness from different evaluators’ points of view. In addition, some of the thesis dissertation and master thesis is included in the document type with a purpose to bridge between the academia and professional reports completeness of the information. Despite the research limits the scope of document sources, the search result from applying mentioned keywords may produce various amount. In order to extract relevant information based on this research topic, we determine the inclusion and exclusion criteria as described in Table 6. The searching process using specific keywords and the period limit is applied to both Articles and Report sources. Only the relevant documents, that pass the preliminary criteria, are downloaded in the first place. Next, the First Stage criteria are applied to reduce the number of collected documents into a manageable number for the researcher to do further screening process. In the Second Stage screening, we process the selected documents to be sorted out based on its content. This step is important to ensure that selected documents have the minimum criteria to provide information for the quality appraisal process.

3.4. Articles Data Extraction and Quality Appraisal

(27)

27 Table 7 - Data Extraction Features

Type of Extraction

Type of Extracted Data

Features (Unit of Analysis)

Expected Research Question to be addressed Basic Articles Information

Strings, Date Date published Author Research Domain Title RQ1 Specific Articles Information

Strings Proposed Set of Heuristics Methodology of UH Development Reference of Heuristics

Artefact/System used for evaluation Number of identified problems

RQ 2 Specific Articles Information Illustrations, Narrations, Pictures, Statements

Description of Usability Heuristics Author’s Aims and Objectives Knowledge of user’s world

User’s interventions toward the system User’s Reaction or Dialogue

User’s environment world Usability Problem Listed Type of violated heuristics Lessons Learned

Future Works

Evaluation/assessment Result

RQ2, RQ 3

The data extraction procedure will be conducted independently. We follow the notion of Okoli (2005) and Kitchenham (2007) provided thorough guidelines and principles on how to conduct data extraction and coding the narratives and descriptive information (Kitchenham et al., 2007; Okoli, 2015). Despite the existence of clear guidelines, there is a possibility to have any incomplete or ambiguous data. Moreover, poorly reported documents may cause difficulties for the researcher to interpret the meaning of evidence. To address this problem, we apply a quality appraisal assessment against the documents. Kitchenham suggested quantitative and qualitative assessment for sources of information quality appraisal to apply detailed filter criteria which is helpful to reduce bias and guide the interpretation of results (Kitchenham et al., 2007).

(28)

28 based on basic elements we determine in the data extraction process (Okoli, 2015). Appraising the extracted information based on the articles and professional report documents, we adopt the assessment criteria from Roehrs (2007) and modify it to fit with the study purpose (Roehrs et al., 2017). Table 9 represents the modified assessment criteria into a new checklist for describing the completeness of quantitative measurement of quality.

Table 8 - Completeness Criteria of Articles and Documents AC-ID Checklist of Completeness

AC1 Does the article clearly show the purpose/goals of the research?

AC2 Does the article adequately present the theoretical foundation or literature review? AC3 Does the article have an experiment or research methodology described?

AC4 Does the article have detailed research results?

AC5 Does the article present a conclusion related to the research objectives? AC6 Does the article recommend future works, improvements, or further studies?

AC7 Does the article contain adequate information of detailed research process or specific action?

The Yes/No answer might not contain a firmness of argument in which context the quality is being assessed. To confirm the assessment quality, there are several open-ended questions of qualitative assessment which allows the researcher to develop a solid assessment (Kitchenham et al., 2007):

1. How well are the contexts and data sources retained and portrayed? 2. How depth, detail, and complete the data have been presented?

3. How clear is the connection between data, interpretation, and conclusion?

These questions are established with expectations that the assessment findings in regard to quality divergent between documents are able to align with the Second Stage of filtering in Table 6. The result of final document screening process will be discussed in a different chapter (see Chapter 4).

3.5. Ethical Consideration

In this SLR study, we utilized various sources of information on online libraries, books, and online portals and that information can be collected through an intensive search. However, we should consider the right of publishing and plagiarism concerns of using the obtained information in order to minimize the risk of harming other legitimate interests. To comply with the researcher ethics, we refer to the ALLEA Code of Conduct that mentioned four fundamental principles as follows (All European Academies, 2017):

(29)

29

undertaking, reviewing, reporting and communicating research in a transparent, fair, full and unbiased way. Respect for colleagues, research participants, society, ecosystems, cultural heritage and the environment. Accountability for the research from idea to publication, for its management and organization, for training, supervision and mentoring, and for its wider impacts.”

(30)

30

4. Findings

In this chapter, the findings of qualitative review from professional reports, thesis, and research articles will be discussed. The structure of this chapter will follow Figure 6 and resume the explanation in Chapter 3. Also, the result of findings is referring to the tables in Appendices section.

4.1. Documents Selection and Appraisal

The search strategy and keyword strings in Table 5 determine the result and it should be applied in the practical search. Since the search results are various from one combination to another, it is required to determine the ideal number of a search result based on a reasonable number which also aligns with the topic relevancy. We realize that each online library has divergent ability to do a deep advanced search. To produce the best possible result, different techniques are applied for each online library without compromising the usage of the keywords. The search result after the preliminary screening is summarized in Table 9.

Table 9 - Search Result from the Article Sources

Online Library Search String** Search Result*

(by June 2020)

ACM (“usability”) AND (“heuristic evaluation” OR “heuristics

evaluation”) AND (“evaluation”) AND (“study” OR “method” OR “methodology” OR “assess” OR ”assessment”)

314

IEEE Explore (“usability heuristic” OR “usability heuristics”) AND (“evaluation” OR “assessment” OR “method” OR “study” OR “assess”)

134

EmeraldInsight (“usability heuristic” OR “usability heuristics”) AND (“evaluation” OR “assessment” OR “method” OR “study” OR “assess”)

123

ScienceDirect (“usability heuristic” OR “usability heuristics”) AND (“evaluation” OR “assessment” OR “method” OR “study” OR “assess”)

Article keywords: usability heuristic, heuristic evaluation

298

Web of Science (“usability”) AND (“heuristic evaluation” OR “heuristics

evaluation”) AND (“evaluation”) AND (“study” OR “method” OR “methodology” OR “assess” OR “assessment”)

199

Google Scholar (Professional Reports & Thesis Documents)

(“usability heuristic” OR “usability heuristics”) AND (“evaluation” OR “assessment” OR “method” OR “study” OR “assess”) AND (“Company” OR “Industry” OR “Professional”) AND (“Report” or “Document”)

705

(31)

31 From the whole search result, there are 89 “research articles” and 29 documents of “professional report & thesis document” which passed the assessment based on the preliminary filter (see Table 6) inclusion criteria. Once we applied the First Stage exclusion filter, there are 53 articles candidates from the online library that satisfy the quality to be included as the material for further review. In addition, from the Google Scholar search result by using the keywords combination in Table 5 and the applying preliminary and first stage filters, we also obtained 22 candidates of usability heuristics evaluation report that considered made by professionals and thesis documents. This result is not the final inclusion/exclusion process of documents. The selected articles will be appraised by using the quality assessment criteria, the second phase of filtering (Table 6), and qualitative questions to ensure the required information exist in the documents. As the final result, we found seven research articles and six reports. In addition, the six documents from Google Scholar search have different years which represent master thesis, professional report, and professional collaboration article.

Figure 7 - Document Selection Phase

(32)

32 Table 10 - Qualified Documents

No Author Title Domain Type of Documents

[1] (Babaian et al., 2010)

Usability through System-User Collaboration? Deriving Design Principles for Greater ERP Usability Enterprise Resource Planning Professional Collaboration Article (Report) [2] (Wei and Landay, 2018) Evaluating Speech-Based Smart Devices Using New Usability Heuristics

Device and Gadgets

Professional

Collaboration Article (Report)

[3] (Milham et al., 2003)

SLEP VELCAC, IFE II BUILD - Usability Evaluation Report

Military Professional Report

[4] (Fraser and Ball, 2017)

Healthcare Application Prototype and User Evaluation Results

Healthcare Professional Report

[5] (Hämäläinen, 2008)

Usability Aspects of MP3 Player Documentation - A Case Study

Device and Gadgets Master Thesis (Report)

[6] (Wang, 2009)

Lund University Website Evaluation: Focus on homepage and English research pages

University Website Master Thesis (Report)

[7] (Pertiwi et al., 2018)

Using Heuristic Evaluation to Improve Sepsis Alert

Usability

Medical Research Article

[8] (Aabel and Abeywarna, 2018) Digital Cross-Channel Usability Heuristics: Improving the Digital Health Experience

Healthcare Research Article

[9] (Baguma, 2018) Usability Evaluation of the

e-Tax Portal for Uganda E-Government Research Article

[10]

(Bedjaoui, Elouali and Benslimane, 2018)

User Time Spent Between Persuasiveness and Usability of Social Networking Mobile Applications: A Case Study of Facebook and YouTube

Social Network Research Article

[11] (Lee and Price, 2016)

User Experience with Commercial Music Services: An Empirical Exploration

Music Research Article

[12] (Tariq et al., 2016)

Applying human factors approach to improve usability of a decision support system in tele-nursing

Medical Research Article

[13] (Väätäjä et al., 2016)

Information Visualization Heuristics in Practical Expert Evaluation

Visualization Design Research Article Green = Retrieved from Google Scholar

(33)

33 Once the screening of qualitative selection and quality appraisal, there are 13 articles from ten different domains. We found the articles and documents are diverse in terms of applying usability heuristics for evaluating a system in the different application domains. Depending on which domain HE is being implemented, we found for each qualified paper represents a different way of bringing up the evaluation report (See Table 10).

Combined from both articles and research articles, we see that Device and Gadgets, Healthcare, and Medical domain have more occurrence compared to other domains while each of them represents a unique domain. Based on the document source origin, the document number [2] represent the Device and Gadget based on its HE implementation domain on the speech-based device where they examine the voice activation and request through speech recognition devices such as Apple Siri, Amazon Echo, and Google Home while the document number [5] represents the Device and Gadget from the user documentation of iPod MP3 player device as the application domain.

Likewise, document number [7] and [12] which come from Online Libraries represent the medical domain from their HE implementation on the Sepsis Alert System and Tele-nursing Call System consequently. These systems are operated by the domain experts (Nurses and Doctors) who directly interact with patients and required to give proper decision making thus none of the safety standards are compromised. In addition, both systems presented research that utilizing default Ten Nielsen heuristics to inspect the usability of their systems (Nielsen, 1994a).

The other examples come from the Healthcare domain where this domain has representatives from both sources (Google Scholar and Online Libraries). Document number 4 (PHEME mental-health dashboard monitoring) and document number 8 (SmokeFree.gov) utilize HE to evaluate the usability of their own system which encompasses multi-channel data sources from social media, phone, and health record. In addition, both of those documents propose a custom set of heuristics to fit their system functionality and communication channels.

The rest of the articles and reports with unique topics depict each of their domain. Note that every article has a different level of completeness and details in providing information. Specific findings from these qualified documents are discussed in the following section.

4.2. Quality Appraisal Components

(34)

34 we discuss the application of the Quality Appraisal process (see 3.4) and findings obtained during the study. The findings are presented based on completeness assessment and qualitative questions guidance.

Completeness Assessment

During the appraisal process of the documents content quality from the 13 documents, we found two reports and three articles that satisfy all of the completeness components. The rest of them have different shortcomings that affect the effort of appraising the documents. The completeness assessment of the documents is presented in a checklist format in Table 11.

Based on our findings, we found all of the selected articles have a clear purpose described (AC1 – Table 11) in abstract or introduction for the research articles, and it also exists in the executive summary for the professional report. Furthermore, most of the qualified documents present theories about usability, heuristics evaluation from Nielsen or domain-specific of usability heuristics, and brief theory about the application domain (AC2 – Table 11). However, the report number [2] which discusses about Speech User Interface (SUI) did not mention a comprehensive explanation about SUI regarding its fundamental notion or basic theory. It only mentioned a sentence about Nielsen’s heuristics as the most often applied inspection method of GUI and it compensated the explanation with the recent study of SUI within 20 years. Similarly, in the report number [3] contained a brief discussion about the heuristics used, which was inspired from Nielsen and Stanney, et al. (Virtual Environment Specific Heuristics), although it did not include a detailed review of the theoretical heuristics they mentioned. In other words, the report number [2,3] contains partial information about the theory behind the experiment they executed. Table 11 - Completeness Assessment based on Appraisal Criteria

No Authors Document Type Checklist Assessment (Table 9)

AC1 AC2 AC3 AC4 AC5 AC6 AC7

[1] (Babaian et al., 2010)

Professional Collaboration Article

✓ ✓ ✓ ✓ ✓ ~ ✓

[2] (Wei and Landay, 2018) Professional Collaboration Article ✓ ~ ✓ ✓ ✓ ✓ ✓ [3] (Milham et al., 2003) Professional Report ✓ ~ ✓ ✓ ✓ ~ ✓

[4] (Fraser and Ball,

2017) Professional Report

✓ ✓ ✓ ✓ ✓ ~ ✓

[5] (Hämäläinen, 2008) Master Thesis ✓ ✓ ✓ ✓ ✓ ✓ ✓ [6] (Wang, 2009) Master Thesis ✓ ✓ ✓ ✓ ✓ ✓ ✓ [7] (Pertiwi et al., 2018) Research Article ✓ ✓ ✓ ✓ ✓ ~ ✓

[8] (Aabel and

Abeywarna, 2018) Research Article

✓ ✓ ✓

~ ✓ ✓ ~

(35)

35 No Authors Document Type Checklist Assessment (Table 9)

AC1 AC2 AC3 AC4 AC5 AC6 AC7

[10] (Bedjaoui, Elouali and Benslimane, 2018) Research Article ✓ ✓ ✓ ✓ ✓ ✓ ~

[11] (Lee and Price,

2016) Research Article

✓ ✓ ✓ ✓ ✓ ✓ ✓

[12] (Tariq et al., 2016) Research Article ✓ ✓ ✓ ✓ ✓ ~ ✓

[13] (Väätäjä et al.,

2016) Research Article

✓ ✓ ✓ ✓ ✓ _✓ ✓

Legend:

✓ (Full), ~ (Partial), X (None)

The documents present the methodology they use in their evaluation (AC3 – Table 11). Although the professional reports are proposed to give a summarized report to produce a direct improvement of the inspected system, the reports include the steps of how the companies or evaluators conduct a usability inspection procedure and how they construct a set of self-tailored heuristics for their inspection process. It is illustrated in the methodology chapter or procedures section of the observed documents.

In regard to the detail of the provided information, a thorough explanation of an article is determined based on how detail the author includes the result is (AC4 – Table 11). Despite that, not all the articles include detailed information of evaluation results like a list of violated and non-violated heuristics including its explicit rationale or usability problems in their report. As we refer to Table 11, although most of the article presents a thorough result, we found an article that contains succinct information. For instance, the article number [8] presents findings of cursory research where it only contained less than half of the information (3 of 7 heuristics) about violated heuristics. Moreover, we could not find the other four of the violated heuristic in the article whether the observed channels satisfy the heuristics or not mapped in the cursory observation.

(36)

36 In terms of information adequacy for this study purpose (AC7 – Table 11), some articles do not mention how specifically users interact with the system. For instance, the author of article number [8, and 10] did not mention how the evaluator doing specific action with the artefact or system (i.e. it could be clicking buttons, typing inputs, etc.). The information on user interaction is gathered from the usability problems noted in the articles.

To consolidate and ensure the checklist result is consistent, we extend the appraisal process by examining the papers based on the qualitative open-ended questions (see Chapter 4.3) and the findings are explained as follows.

Data Context and Portraying

Data is mostly retained in descriptive texts. Rating of the usability problems is presented in the text and the average of usability problem for each heuristic violation is also presented in paragraphs. Tables are presented containing numbers, but significance or analytical statistics formula is not performed. In some articles, for instance, article number [2, 4, 5, 6, 7, 11, and 13], a sample of problems are captured in screenshots where usability violations occurred.

Figure 8 – A Sample of Highlighted Screenshot of the dashboard (Fraser and Ball, 2017)

(37)

37 the PHEME dashboard has a usability problem and require recommendations to improve usability. This also similarly performed by the author of document [6, 7, 11, and 13] where they used the same method to capture the usability problems. Conversely, the author of document number [5] preferred to capture problems using captioned screenshots and adding paragraph explanation to point out the usability problems in the iPod user manual. Similarly, the document number [2], the authors only captured the screenshot of the tested devices and they support the images with explanations of the testing procedures on the particular section.

On the other hand, the document number [3 and 6] include the raw interview or questionnaire response which was gathered from the post-evaluation process of their study. The author of the report number [3] included the feedback from the system usability tester for technical improvement. In Figure 10, the user’s feedback from evaluating the SLEP VELCAC (Virtual Military Landing Craft Vehicle) system was recorded in a text format. Furthermore, this report also retained the percentage of user’s characteristics which contains data about the end-user’s experience with computer and their expertise background.

Figure 9 – A Sample of Raw Extraction of Interview/Questionnaire Result (Milham et al., 2003) Similarly, the author of the report number [6] captured the user’s actions toward the evaluated system through raw questionnaire results from the users and the screenshot of the system. Tagging numbers on the screenshot to note usability problems and its specific improvement in the report number [6] has a similar method with the report number [4]. In addition, this report retained the transcription of group interview sessions and open-ended online questionnaires which contain information on the user’s perspective and identified problems in the system.

(38)

38 user’s response are the document number [1, 6, 11, and 12]. These documents present verbal narration based on the distinct experience of the user through usability heuristics inspection. In the document number [1], the author quoted the response from users where the users encountered some problems regarding error messages reported by the system (see Figure 11).

Figure 10 – A Sample of Verbal Narration of User's Response (Babaian et al., 2010)

Likewise, the author from the document number [11 and 12] also presents a similar quoting style where the user’s response is recorded in a text format where it can be the qualitative evidence to support the usability problems in the system did exist.

There are also circumstances, for example in documents number [8, 9 and 10], where the HE processed data are presented qualitatively described in paragraphs (discussed in a section or in a table) as the result of the structured evaluation. The usability problems discussion is categorized based on each violated heuristic. For instance, in the document number [9] we found the author put the information about the usability problem in paragraph explanation, while the author of document number [8 and 10] present a table to detain the usability problems. From all of the 13 documents, there are descriptive statistics provided in the documents [2 and 6] where the significant calculation was performed in the document number [2] and the statistical result of the questionnaire is provided by document number [6]. Whereas, overall statistic assessment based on the rating is given in a short paragraph for other documents (i.e. mean, median, total, and percentage).

Detail and Completeness of Documents

According to our findings, the information provided by the reports and articles have a various level of detail and completeness. In this context, we make the qualitative judgment on the documents where we determine ‘detailed’ information is identified based on the author’s effort to put elements in the evaluation process, while the ‘completeness’ information based on the richness of the explanation in each element.

Socio-instrumental Aspects in Usability Heuristic Studies: Systematic Review