Bachelor thesis
The 404 error message
What type of feedback generates a good user experience?
Author: Saga Gullberg Supervisor: Romain Herault Examiner: Mattias Davidsson Semester: VT20
Subject: Media Technology Level: Bachelor
Course code: 2ME30E
Abstract
This study investigates the 404 error and in what way feedback should be given to the user in an error message to generate a good user experience. To investigate this, data was gathered from 1) a literature review looking at previous studies in User Experience Design, as well as different models to evaluate ease of use and perceived usefulness, 2) a pre-study questionnaire with nine participants who were asked questions related to error messages and feedback in general, and 3) two user tests; the first including 16 participants and the second including 46 participants. During the user tests the participants interacted with a prototype of a website that included 404 error messages. In the first user test the participants' user experience was evaluated based on the TAM model, including perceived usefulness and perceived ease of use. The second user study focused more on text and illustrations and their effects on the user experience. Three versions of texts were tested, formal, apologetic and amusing. The illustrations either included a character or an object. The result showed that, in general, the users preferred formal texts, as well as illustrations that included a character. The result also suggested that there is not one type of feedback that fits all users, in order to generate a good user experience, it is important to know the audience.
Keywords
404 error message, 404 error code, Feedback, Text, Illustration, User experience design, User interface, TAM, Perceived usefulness, Perceived ease of use, Design principles.
Acknowledgements
I want to start by thanking everyone that has been involved in this study, either by participating or helping out in some other way. I want to give a special thanks to my family and friends for supporting and pushing me when the thesis was tough to write. I would also like to thank my examiner, Mattias Davidsson, as well as my opponent, who provided me with valuable feedback to improve my thesis. Lastly, I want to give a huge thanks to my supervisor Romain Herault who has helped me and supported me tremendously throughout the work of my thesis.
Without the help from all the amazing people that have been there for me this thesis would not be completed yet.
Table of Content
1. Introduction 5
1.1 Problem description 6
1.2 Purpose 6
1.3 Delimitation 6
1.4 Research questions 6
1.5 Hypothesis 7
2. Background and related work 7
2.1 Error messages 7
2.1.1 The 404 error code 8
2.2 User experience and user interfaces 8
2.3 Design principles 9
2.3.1 Norman’s Seven Stages of Action 9
2.3.2 Benyon’s 12 Design Principles 10
2.4 Technology Acceptance Model 11
2.4.1 Perceived Usefulness 12
2.4.2 Perceived Ease of Use 12
2.4.3 Criticism on the model 12
3. Method 12
3.1 Literature review 12
3.2 Pre-study Questionnaire 13
3.3 User-centered design process 13
3.4 User testing 14
3.4.1 First user tests 14
3.4.1.1 Observations 14
3.4.1.2 Interviews 14
3.4.2 Second user tests 15
3.4.3 Pilot tests 16
4. Implementation 16
4.1 The prototype 16
4.2 The error messages 18
4.2.1 Version 1 18
4.2.2 Version 2 20
4.3 Requirements 20
4.3.1 First version of the requirements 20
4.3.1.1 Functional 20
4.3.1.2 Non-functional 21
4.3.2 Second version of the requirements 22
4.3.2.1 Functional requirements 22
4.3.2.2 Non-functional 23
5. Results 23
5.1 Pre-study Questionnaire 23
5.2 First pilot test 25
5.3 First user tests 25
5.3.1 Result from the interviews 26
5.4 Second pilot test 30
5.5 Second user tests 31
5.5.1 Part one 31
5.5.2 Part two 31
5.5.2.1 Titles 31
5.5.2.2 Illustrations 35
5.5.2.3 Texts 38
5.5.2.4 Solutions 42
5.5.3 Part three 45
6. Analysis and discussion 45
6.1 The first user tests 45
6.2 The second user tests 48
6.2.1 Evaluation of the titles 48
6.2.2 Evaluation of the illustrations 49
6.2.3 Evaluation of the texts 50
6.2.4 Evaluation of the solutions 51
7. Conclusion 51
7.1 Answers to the research questions 52
7.2 Limitations to the study 54
7.3 Future Work 54
References 55
Appendix 57
Appendix 1 57
Appendix 2 57
Appendix 3 58
Appendix 4 59
Appendix 5 60
Appendix 6 60
Appendix 7 61
Appendix 8 61
Appendix 9 61
Appendix 10 62
Appendix 11 65
1. Introduction
User interfaces are the connection between humans and computers, it is a platform that helps the user to understand the information and feedback that is given (Gong 2009). Norman (2013) writes that good communication between users and computers are important, especially when something goes wrong.
When a problem occurs, a good design should highlight the problem, help the user to understand what went wrong and guide them to a solution (pp. 8-9). One way to help users understand problems is through error messages. A user-friendly error message is one important aspect of a good, usable interface, but in order for users to achieve a specific task, the users need to understand the error message (Inal & Ozen-Cinar 2016). In this study a user interface refers to a website on the Internet and one of the more common error messages that most Internet users have encountered at some point is the 404 error message. This error signifies that the server could not find anything that matches the requested page (Sampath Kumara & Vinay Kumar 2013). In order for users to understand a problem that arises some sort of feedback is needed. Feedback refers to the information that helps the users understand what happened (Norman 2013, p.
72). When feedback is lacking and something in a system is not working, frustration arises. One of the main reasons for user frustration is errors, these errors can be irritating due to e.g. an error message lacking in terms of consistency and clearness, which leads to confusion (Joon Park, MacDonald &
Khoo 2012). By providing the right information about the impact of actions, good feedback can be given to the user (Norman 2013, p. 72) and frustration can be avoided.
When the communication is clear and understandable between human and computer a good user experience is generated. Preece, Rogers & Sharp (2019) writes that every product or system that is being used by someone has a user experience (p. 13). User experience focuses on the users feelings, opinions, physical and psychological reactions before, during and after using e.g. an interface (Nationalencyklopedin, n.d.-b). A person's view and reactions, how the user experiences a product, that is what user experience is all about (Nationalencyklopedin, n.d.-b). For a user’s experience of a product to be good, it is required to meet the user’s needs regarding the context of the situation the user is using the product (Interaction Design Foundation, n.d.). The process of designing for a good user experience is referred to as User Experience Design (UXD), the focus of this process lies on the quality of the user experience rather than on different design methods (Preece, Rogers & Sharp 2019).
According to Fethi & Fethi (2004) a good user interface is the key to a good user experience and that it is based on the user’s perceived usefulness and perceived ease of use of the interface. Perceived usefulness indicates if and to what degree the user thinks that the product will help the user to perform their tasks better.
Perceived ease of use indicates if the user thinks that the product can be used without effort, if the user will be able to use the product or is it too complicated (Davis 1989). The Technology Acceptance Model (TAM) uses these two key factors to measure user's acceptance of technology (Davis 1989), e.g. user interfaces.
1.1 Problem description
Error messages can often lead to frustration, when something is not working as it should, irritation arises. The users need to be kept in the loop of what is going on, and if something is not working as it should, they should receive the right type of feedback to guide them to a solution. The 404 error message is one of the more common error messages that users come across on the Internet. Therefore, the 404 error message is the focus of this study and ways to generate a good user experience for users when they encounter a 404 error message on the Internet was investigated.
1.2 Purpose
The purpose of this study is to investigate the 404 error message and explore what type of feedback can help generate a good user experience. The different types of feedback that are included are; text and illustrations. The text was presented in three different ways; formal, apologetic, and amusing. The illustrations included two types of content; characters and objects. The user experience was partly investigated through the TAM model which focuses on perceived usefulness and perceived ease of use, and partly by evaluating the users' opinions regarding the types of feedback.
1.3 Delimitation
This study only investigated the error message 404, which occurs on websites when a specific page could not be loaded for some reason. Furthermore, the tests that were made investigated this error message on computers only, and another study should be made for mobile devices.
1.4 Research questions
RQ 1. What type of feedback should be included in 404 error messages to improve a user’s perceived usefulness and perceived ease of use, when using a website?
RQ 1.1 How should text-based information be presented and formulated in 404 error messages on websites in order to improve a user’s perceived usefulness and perceived ease of use?
RQ 1.2 In what way can illustrations help improve a user’s perceived usefulness and perceived ease of use when encountering 404 error messages on websites?
RQ 1.3 In what way can illustrations that include a character affect a user's experience when receiving a 404 error message on a website?
1.5 Hypothesis
A hypothesis for this study is that a 404 error message should include text that is presented in a user friendly way and include some type of illustration to generate a good user experience.
2. Background and related work
To find related work and information to the thesis subject, a literature review was conducted, this section presents the information that was found. The subjects that have been investigated are Error messages in general, as well as specific information about the 404 error message. But also, User experience, User interfaces, Design principles, and the Technology Acceptance Model (TAM), including Perceived usefulness and Perceived ease of use. The methods used to conduct the literature review can be found in the methodology section (see section 3.1 Literature review).
2.1 Error messages
When something in a system is not working, frustration arises. One of the main reasons for user frustration is errors. These errors can be irritating due to an error message lacking in terms of consistency and clearness, which leads to confusion (Joon Park, MacDonald & Khoo 2012). When it comes to online forms, a user-friendly error message is one important aspect of a good, usable interface.
In order for users to achieve a specific task, the users need to understand the error message (Inal & Ozen-Cinar 2016). According to Inal (2016) one way to measure the degree of user-friendliness of an error message design is its ability to easily catch the users’ attention and help the user to correct the error.
Most software companies are concerned with the integration of usability in the software development and develop more user-friendly interfaces (Inal &
Ozen-Cinar 2016). Inal & Ozen-Cinar (2016) states that developers know the importance of involving usability in the development process, but existing software still lacks elements that make interfaces user-friendly, meaning that the users’ needs and expectations are not met. This can be due to developer’s unawareness of design principles, or having limited or no knowledge about usability activities (Inal & Ozen-Cinar 2016). The result of a study conducted by Inal & Ozen-Cinar (2016) showed that in the process of designing error messages, the developers include information that is useful both to the users and the developers themselves by embedding error codes in the messages. Even so, the developers that took part in the study stated that it is inappropriate to emphasize the error code (Inal & Ozen-Cinar 2016).
According to Preece, Rogers & Sharp (2019) computers should be polite, when this is the case users tend to be more forgiving and understanding if an error occurs (p. 177). It is more common today that a computer apologizes and explains after making a mistake rather than presenting more technical error messages such as the classic "404 error” message (Preece, Rogers & Sharp 2019, p. 177). Displaying apologetic messages can be a way to influence the user experience and how the user feels about a system (Joon Park, MacDonald &
Khoo 2012). Apologetic messages affect emotion, which plays a big part in aspects of attitude, behaviour, and perception. Furthermore, this type of message can help bridge the gap between user and computer when errors appear in the system (Joon Park, MacDonald & Khoo 2012). A study conducted by Joon Park, MacDonald & Khoo (2012) evaluated three different types of systems; Neutral, Apologetic and Non-apologetic. The result showed that users tended to
appreciate the aesthetics and usability of the apologetic system rather than the non-apologetic and neutral systems. Furthermore, the user frustration was lower when using the apologetic system (Joon Park, MacDonald & Khoo 2012). The result of this study can be directly connected to one of Benyon's (2019) design principles, Conviviality (see section 2.2.2 Benyon’s 12 Design Principles).
Kadekar, Sohoni & Craig (2018) carried out a study that investigated students’
ability to understand and fix programming errors based on error messages. They developed three different types of error messages, Default; that provided a single line of information, Link; that provided a more detailed description of the error and provided a hyperlink for further information, and Example; that provided detailed information with a relevant example (Kadekar, Sohoni & Craig 2018).
The result of the study showed that the Link-error message helped the students to understand and solve the problem in the least amount of time (Kadekar, Sohoni
& Craig 2018).
2.1.1 The 404 error code
Sampath Kumara & Vinay Kumar (2013) writes that the 404 error code signifies that the server could not find anything that matches the requested page. This code also indicates that there is no information regarding if the condition is temporary or permanent. According to Preece, Rogers & Sharp (2019) the 404 code comes from the HTML language, and it indicates that there was a problem when the user tried to access a certain webpage. The three numbers included stands for different things. The first 4 implies that there was a client error, this means that the server is informing the user that something was conducted in the wrong way. This could be a mistake such as the user trying to reach a page that does not exist, or the error could have occurred due to the user misspelling the URL. The 0 refers to a general syntax error, this could also be because of a spelling mistake. The second 4 represents the specific reason that the error occurred. In a study conducted by Sampath Kumara & Vinay Kumar (2013) they write that most experienced Internet users have at some point encountered the 404 error message.
2.2 User experience and user interfaces
According to Preece, Rogers & Sharp (2019) user experience (UX) focuses on the users of a product or software. It refers to users’ feelings and what they are feeling when using a product or software. It is about the users’ general impression, down to the details and how the user feels about them, it is all the aspects of the feel of a design (p. 14). Preece, Rogers & Sharp (2019) points out that a user experience cannot be designed, but one can design to achieve a good user experience, the experience itself cannot be designed, but the factors that will generate the experience can (p. 14). User experience design is about designing products that generate a good user experience by understanding the users’ needs.
One size does not fit all is something that Preece, Rogers & Sharp (2019) points out and they write that it is important to understand each different user group (pp.15-16). By learning more about the users, their needs and capabilities, incorrect assumptions that the designer might have is often revealed (Preece, Rogers & Sharp 2019, p. 16). This stresses the importance to know who the
design is being made for and how the design can be created to generate a good user experience.
Wildner, Kittering-Rosanelli & Bosenick (2015) write that the user experience (UX) has become more important than ever before, this is due to the mundane rate that products are becoming more digital or completely digital. Usability is a central concept and what is being measured to evaluate user experience, according to Wildner, Kittering-Rosanelli & Bosenick (2015) UX stretches beyond the ease of use. UX is formed by different qualities and dimensions that are connected and depend on each other. There are three dimensions, task-oriented qualities, self-oriented qualities and the aesthetic qualities. The task-oriented qualities involve learnability and usability, the self-oriented qualities involve user needs and inspiration, and the aesthetic qualities involve the look and feel of the product (Wildner, Kittering-Rosanelli & Bosenick 2015).
User interfaces are the connection between humans and computers, it is a platform that helps the user understand the information and feedback that is given (Gong 2009). A good user interface can help the user on several levels, the communication, the ease of use, and guidance so that the user can avoid making mistakes (Gong 2009). Gong (2009) points out the importance of user interfaces and that they should meet different kinds of users’ needs. User satisfaction is the key to the success and usability of an interface. Users generally want the software that they are using to be easy to manage, they do not want to work with difficult interfaces (Fethi & Fethi, 2004). According to Fethi & Fethi (2004) a good user interface is the key to user satisfaction and this depends on the users perceived usefulness and perceived ease of use (see section 2.6 Technology Acceptance Model (TAM)) while using an interface. When it comes to user interfaces there are many components that are important to help the users with performance, their effectiveness, efficiency and satisfaction (Inal & Ozen-Cinar 2016). Inal & Ozen-Cinar (2016) points out that aesthetics and the usability of an interface to be two essential factors.
2.3 Design principles
2.3.1 Norman’s Seven Stages of Action
Norman's Seven Stages of Action (Norman 2013, pp. 71-73) can be a valuable design tool as it contributes to a basic checklist of questions that can be asked when developing a product. Each step thus contributes to a question:
1. What do I want to accomplish?
2. What are the alternative action sequences?
3. What actions can I do now?
4. How do I do this?
5. What happened?
6. What does it mean?
7. Is this okay? Have I accomplished my goal?
Anyone who is using a product should always be able to answer the seven questions, meaning that the design should be easy to understand and should not raise any questions that the product cannot answer itself. Norman (2013) points
out that it is up to the designer to make sure that the product provides the right information to make the questions answerable (pp. 71-72).
To answer these questions, the user needs to get the right kind of information when using an interface. This information is generated using seven design principles, each question can be linked to one of its design principles:
Discoverability; helps the user understand what actions are possible and the current state of the device (Norman 2013, p. 72). Feedback; there is constant information about actions’ results and the products state, it should be easy for the user to understand what is going on (Norman 2013, p. 72). Conceptual models; a sort of mental model for how something works (Norman 2013, pp. 25-30), a good design provides the information necessary for users to create a good conceptual model of the system. Affordance; the relationship between an object and a person, it refers to the object’s properties and the users’ capability to understand how they can use it, essentially how something can be used, based on its appearance (Norman 2013, p. 11). Signifiers; signifiers and affordance are related to each other, affordance indicates what can be done and signifiers communicate where the possible actions should be done (Norman 2013, p. 14).
Mapping; this refers to the design and layout of controls and displays (Norman 2013, p. 21), meaning that information needs to be provided for users to understand which controls that controls what. Finally, Constraints; refers to actual constraints, guiding the user to what is possible to do and what is not (Norman 2013, p. 73).
There are two important notions related to the Seven Stages of Action; the first is feedforward, which refers to the information that helps the users to understand how things are executed, and the second notion is feedback, which refers to the information that helps the users understand what happened (Norman 2013, p.
72). It is the process of guiding the users through what can be done, what is happening and what happened. According to Norman (2013) feedforward is accomplished by the proper use and combination of signifiers, constraints, and mapping. Conceptual models have an important part to play here as well. Good feedback is provided by giving the right information about the impact of actions.
The feedforward and the feedback need to match the users’ needs, and more importantly the users must understand the information (Norman 2013, p. 72).
2.3.2 Benyon’s 12 Design Principles
Benyon (2019) writes about twelve basic principles that guide designers during the design process, they can both help to evaluate and critique prototype design ideas. These design principles are based on other design principles such as Norman’s design principles (see section 2.3.1 Norman’s Seven Stages of Action). The principles interact, complement, and sometimes conflict with each other, but essentially, they assist the designer in terms of finding key features for a good design (p. 116).
Visibility; make sure important elements are visible and that the user understands what is going on. Consistency; it is important to design consistently regarding language and design features, both within the design itself and similar systems.
Familiarity; use elements that the users are familiar with.Affordance; make sure that elements are designed in a way that is easy to understand and what the
elements are made for, what can be done. Navigation; provide the users with means to be able to move around the system in an easy way. Control; it should be clear who or what is in control of what and allow the users to take control.
Feedback; give the users constant information about what effect their actions have to make sure the user knows what is going on. Recovery; make sure users have a way to recover from actions, especially errors and mistakes. Constraints; provide constraints so that users cannot actively make mistakes or make sure they do not try to complete an action that is not possible. Flexibility; make it possible to make the same action in different ways, this helps to include all users when they vary in levels of experience and interest in the system. Style; the design should be appealing and stylish. Conviviality; design for the users to have a good experience, make sure that the design is polite, friendly and generally pleasant to use (Benyon 2019, pp. 117-118).
2.4 Technology Acceptance Model
The Technology Acceptance Model (TAM) is one of the most used models when it comes to evaluating if technologies will be accepted by users (Hamid, Razak, Bakar & Abdullah, 2016; Benyon 2019, p. 112). Davis (1989) writes that there are many variables that can influence the users’ acceptance of information technology, but there are two main variables that are the key variables. The first, perceived usefulness, indicates if and to what degree the user thinks that the product will help the user to perform their job better. The second, perceived ease of use, indicates if the user thinks that the product can be used without effort, if the user will be able to use the product, or it is too hard or complicated (Davis 1989). This model is used within many subjects and can be adapted to each study it is being used for (Benyon 2019, p. 112).
There are many different versions of this model but the most common one is based on perceived usefulness (PU) and perceived ease of use (PEOU) (Moores, 2012). TAM focuses on users’ attitudes towards using a system that, according to TAM, is based on the users’ beliefs that a system can be useful to them or that it is easy for them to use (Moores, 2012). According to Moores (2012) it is theorized that the perceived usefulness is influenced by the perceived ease of use, if a system is easy to use users are more likely to find it useful. According to Fethi & Fethi (2004) they found a correlation between PU and PEOU, that supports the theories regarding the correlation. If the user does not find the software to be easy to use, they will most likely not find the software to be very useful (Fethi & Fethi 2004). Furthermore, PU and PEOU is related to user satisfaction and user experience (Fethi & Fethi 2004), a product that is easy to use and useful corresponds to a good user experience. The results from a study conducted by Fethi & Fethi (2004) found that PU was a better way to predict the end-user satisfaction and PEOU had a stronger impact on user guidance.
2.4.1 Perceived Usefulness
Davis (1989) defines the perceived usefulness as ”the degree to which a person believes that using a particular system would enhance his or her job performance”. A product that is considered to have a high PU is one that the user believes to be useful for the users and that it might improve their performance (Davis, 1989). Perceived usefulness refers to the extent of which a user believes
that a product will be useful to them and even enhance their performance (Hamid, Razak, Bakar & Abdullah, 2016).
2.4.2 Perceived Ease of Use
Davis (1989) defines the perceived ease of use as ”the degree to which a person believes that using a particular system would be free of effort”. A product that is considered to be easy to use or even easier to use than other products are considered to be more likely used and accepted by users (Davis, 1989). PEOU refers to what degree the user believes that a product will be effortless and easy to use (Hamid, Razak, Bakar & Abdullah, 2016).
2.4.3 Criticism on the model
Moores (2012) writes that there exists criticism towards TAM and that there are three main ones: First, system usage is too narrow to define the acceptance, this definition needs to be broadened. Second, the research existing regarding design and implementation of PU and PEOU are lacking and as a result of this there exists many different versions of TAM. Third, the model does not explain behavioural and performance-based consequences of acceptance. The confusion regarding TAM can be traced to the lack of definition of the origin of the terms PU and PEOU, instead of explaining, further research has added to the existing knowledge of PU and PEOU (Moores 2012).
3. Method
This section will present the methodology that has been used to conduct the study. The goal of this section is to explain how the study was conducted in order to answer the presented research questions. Figure 1 shows the different steps taken by the study.
Figure 1: The methodological steps of the study.
3.1 Literature review
The first step was to gather information about the subject and what type of methods that exist for this type of work. The information gathered was found by using academic search engines to find articles related to the subject. The search engines that were used are OneSearch, Google Scholar and IEEE Xplore. When using the academic search engines, a set of keywords were used in different combinations to find articles related to the thesis subject. To find relevant articles, the search was divided into three phases. The first phase consisted of reading titles, keywords, and abstracts of the returned articles and selecting relevant ones. Then, the articles that were picked out in phase one were read more thoroughly. In this phase the articles' introductions and conclusions were read to narrow down what articles were relevant for the thesis and these articles were saved for the last phase. In the third phase, the whole articles were read and
paragraphs about the relevant parts were written down to be included in the study. Moreover, some of the selected articles were discovered by checking the references of other articles. When a relevant article was found, the reference list of that article was reviewed to investigate if any of the article's references could benefit the literature review. All articles that were found are peer reviewed. The articles that were included in this study can be found in section 2. Background and related work.
3.2 Pre-study Questionnaire
A questionnaire was created to gather information about feedback and error messages (see Appendix 2). This questionnaire was based on a software called Omnitrack that is owned by the company Omnifinity. This is due to the fact that this study was supposed to be completed in a collaboration with this company.
However, due to the COVID-19 pandemic, the study was later conducted separately from the company. Due to the original intent of the study the questionnaire that was used to gather information regarding feedback and error messages were answered by customers of Omnifinity. Since this questionnaire was based on the company's software only parts of the questionnaire regarding computer use, feedback and error messages were used for this study. Omnifinity works within VR and how the VR experience can be enhanced by allowing free movement. They are the producers of the Omnideck, which is a motorized treadmill that creates a 360° walking area.
The data gathered from the literature review and the questionnaire was converted to requirements for the prototype that was used for this study, see section 4.3 Requirements.
3.3 User-centered design process
The prototype that was used was improved by using an iterative user-centered design. When using a user-centered design process the intended users are involved in the process (Preece, Rogers & Sharp 2015, p. 38). When a design and development process is conducted in an iterative way it means that the design is tested with the users to find potential problems and fix them. The user gives feedback to the developer, this feedback is then processed and the potential problems are dealt with. The design is then tested with users again to evaluate the effects of the fixed problems. This is a circular way of working, and this process can be repeated as many times as necessary (Preece, Rogers & Sharp 2015, p. 48). The prototype was tested and evaluated by users in order to receive feedback that could improve the prototype. This feedback was then analysed and made into requirements that were implemented to create an improved version of the prototype. Figure 2 illustrates the iterative design process that will be used.
Figure 2: The user-centered process that was used to improve the prototype.
3.4 User testing
3.4.1 First user tests
The users participating in these tests were given realistic tasks to complete based on a scenario that was presented to them before the tests began (see Appendix 4 for the tasks and see Appendix 5 for the scenario). During the tests, the users were introduced to a prototype of a webpage called Recepthittaren (The Recipe Finder) and in this prototype, the users were presented with four different versions of the error 404: The page could not be found. During the test, the users were presented with the different error messages in different situations, and at the end of the test, the users were presented with all the error messages at the same time, as a sort of recapitulation. When testing different versions, there is always a risk of unfair advantages for the last version that is tested due to that the user learns to perform tasks while testing the first version (Rubin & Chisnell 2008, pp. 76-77). This means that the result may be biased. To avoid this, the users were presented with the different versions in a random order, a total of 8 random orders were generated and each order was evaluated by two different users. By taking these precautions the result was counterbalanced and bias was avoided to a certain degree (Rubin & Chisnell 2008, pp. 76-77).
The majority of the tests were completed in Swedish, all except one, that test was conducted in English.
3.4.1.1 Observations
According to Preece, Rogers & Sharp (2019) observations can be conducted by observing users as they are performing specific tasks within a controlled environment (p. 287) Observations were made during each test session, which was done both in person in a controlled environment and via the video communication tool Zoom. The test sessions were, after the consent of the tester, recorded. Meaning that the recordings were further analysed to make sure no valuable information was lost. The users’ actions were observed to gather information about how they reacted to the events that occurred in the user test session. Preece, Rogers & Sharp (2015) writes that participants of user tests might get asked to think aloud as they are performing the given tasks. This helps the researcher to understand what the users are thinking and planning to do during the tests. Therefore, the users were asked to think out loud during the test.
3.4.1.2 Interviews
Once the tasks were completed and the users had completed the test, some follow-up questions were asked to gather information about the users’ opinions and their user experience. The technique used for these questions was semi-structured since this technique is optimal for the type of information that was wanted. This method keeps the interviewer on a certain topic but at the same time it gives room for follow-up questions. The interview can be structured to fit the situation and this type of interview can be modified and changed depending on the answers that are given during the interview (Robson & McCartan, 2016, pp. 290-291). Semi-structured interviews are optimal to use when the interviewer is well-versed regarding what is being researched. It is considered to be a positive factor if the one that creates and manages the interviews, is also the
researcher (Robson & McCartan, 2016, pp. 290-291). With this information at hand, it seemed like a good idea to conduct semi-structured interviews. The questions were based on the TAM model (see section 2.4 Technology Acceptance Model (TAM)) by asking questions that investigated how the user experienced the prototype and the error messages based on perceived usefulness and ease of use. Other questions regarding the general user experience were used as well.
The information that was gathered during the interviews were analysed and generated a second set of requirements that was implemented in a new prototype.
3.4.2 Second user tests
To further investigate the 404 error message, a survey was sent out to gather people's opinions related to the subject. The survey was distributed throughout different platforms such as Facebook, LinkedIn, Discord and Slack. This survey was divided into three parts. In the first part, some demographic data were collected, and the participants were asked about their computer habits and computer skills. The second part was based on evaluation and the participant’s opinions regarding the 404 error message. Based on the error messages used in the first user test, the error message was divided into four parts, Title, Illustration, Text, and Solution (see Figure 3). The participants were presented with five to seven different versions of the four parts and were asked to pick the option they preferred the most and the least, and motivate their choice. The third and last part gave the participants the chance to create their ideal 404 error message. The respondents were provided with a link to a prototype that allowed them to choose freely among the options that they had evaluated in the previous part of the survey (see Appendix 8). They could create their error message based on the four parts; Title, Illustration, Text, and Solution. Once the participants had composed an error message that they were satisfied with, a code that corresponded to their specific choices was presented and they entered this code as the last part of the survey.
Figure 3: Here are the four parts that the error messages were divided into;
Title, Illustration, Text, and Solution.
3.4.3 Pilot tests
According to (Rubin & Chisnell 2008, p. 133) pilot tests are a method that helps the researcher to evaluate a test to make sure that it is clear and understandable, regarding design, tasks, hardware, software, and documentation. It is a way to
”test the test” before presenting it to the intended audience. Therefore, prior to the user tests, a set of pilot tests were carried out to evaluate if the tests were ready to be conducted with the intended users. This was done to gather feedback regarding the way the tests were conducted and the content of the tests. This made it possible to make changes to improve the intended user tests before involving the actual users. Pilot tests were conducted for both user tests. During the pilot tests for the first version of the prototype, the flow of the test was evaluated, including the way the information was presented, the interview technique, as well as the actual design. During the pilot test of the second version, the method was evaluated - the technique that was used for the questionnaire as well as the questions that were asked.
4. Implementation
In this section the process of developing and iterating the prototype that was used during this study.
4.1 The prototype
The prototype that was used during the user tests was a prototype that had been created before this study was conducted. The prototype is a website that helps the users to find recipes (see Figure 4, and see Appendix 1 for a link to the first version of the prototype). In the first version of the prototype a total of four different error messages were implemented, all four were error messages that informed the user about the error 404: The page could not be found. The prototype itself (excluding the error messages) was based on existing recipe websites and design principles similar to the principles that can be found in Norman’s Seven Stages of Action and Benyon’s 12 Design Principles (See section 2.3 Design principles). The main design principles that were used are Visibility, Feedback, Consistency and Affordance. Visibility was incorporated through elements such as buttons, text fields, and drop-down-lists that indicates what actions are possible. Feedback was included mainly by visualising the users’ actions and their result, when the users clicked an element the prototype gave response to the action by signifying what happened. In terms of Consistency the prototype had a consistent theme, and the same type of element had the same design throughout the prototype. Regarding Affordance the user received feedback when hovering over buttons and other elements, the feedback included colour change or information popping up. The text fields include explanatory text, that signifies their functionality and purpose. Other principles can be connected to the prototype as well such as Discoverability, Mapping, Familiarity, Consistency, Navigation, Flexibility, and Style (See section 2.3 Design principles), but the main principles that have influenced the prototype are the ones discussed above. The UX design program Axure RP 8 was used to create and edit the prototype. This means that the prototype simulates the functionalities that are available on the website. The website itself is not implemented.
Figure 4: This is the starting page of Recepthittaren, the prototype that was used to conduct the first set of user tests.
In the second version of the prototype there were less parts included, the first part included the page ”Veckans tips”, this part was iterated based on new requirements that was gathered from the interviews (see section 4.3.2 Second version of the requirements). The other part was interactive, where the users created their own 404 message by using arrows to choose from the different options that they had evaluated during the survey (see Figure 5, and see Appendix 8 for a link to the second version of the prototype).
Figure 5: The users created their own 404 error message by using the arrows.
4.2 The error messages
The 404 error messages that have been included in this study are mainly based on what a 404 error is, and the requirements (see section 4.3 Requirements). But they have also been inspired by existing 404 error messages found via Pinterest.
Several of Benyon’s Design Principles can be connected to the error messages, the most obvious ones being Feedback and Recovery, but also Visibility, Consistency, Familiarity, Affordance, Flexibility, Style, and Conviviality. The design for the error messages were created with the design program Adobe Illustrator and put together in the UX design program Axure RP 8. The first version of the prototype consisted of four error messages, based on user testing and pilot testing, this number later expanded, and several additional versions were created to complement the given feedback. More specific information about the different versions can be read in the two following sections.
4.2.1 Version 1
The first version of the prototype included four 404 error messages that presented different ways that feedback can be introduced to the users. Two of the messages were purely text-based and gave the user information regarding the error, what could have caused it and what the user can do to recover from the error (see Figures 6 and 7). The other two were text and illustration-based that informed the user about an error and what they could do to recover from the error (see Figures 8 and 9). Two of the messages (one text-based and one text and illustration-based) were formulated in a more formal way and the other two were formulated in a way that was more apologetic. One of the illustrations depicted a character, and the other illustration did not. All the messages included a hyperlink as a solution to the problem and a way for the users to recover from the error. The prototype and the error messages were in Swedish to facilitate the testing process and allow users to focus on the tool rather than the language.
Figure 6: The neutral error messages based on text. The text says; 404: The page could not be found. An error has occurred. The page could for the moment not be loaded, it could have been removed or is for the moment not available. Go back to the previous page and try again or try to choose another page.
Figure 7: The apologetic error messages based on text. The text says; Oh no, something went wrong... Sorry, we could not find the page you are looking for, it could have been removed or is for the moment not available. You can go back to the previous page and try to choose something else.
Figure 8: The neutral error messages based on text and illustration. The text says; 404: The page could not be found. An error has occurred, the page could for the moment not be loaded. Go back to the previous page and try again or try to choose another page.
Figure 9: The apologetic error messages based on text and illustration. The text says; Oh no, something went wrong... Sorry, we could not find the page you are looking for, but do not worry! You can go back to the previous page and try to choose something else.
4.2.2 Version 2
The second version of the prototype included more variations of titles, text, illustrations, and solutions. User testing generated feedback that suggested a third type of formulation of title and text; amusing. The user tests also generated a lot of opinions regarding the elements of the error messages. Therefore, different versions of titles, texts, illustrations and solutions were created and tested in the second user test. Two amusing titles, and texts were created, and four new illustrations were designed (two including a character, and two not including a character). Besides this the solution was expanded to include buttons as an alternative to hyperlinks. A problem was discovered with the prototype’s typeface during the first user tests, in the first version, the typeface changed from the intended sans serif typeface to a serif typeface, this depended on what type of computer system the user was using. This flaw was fixed in the second iteration of the prototype to make sure that all the testers received the same experience.
4.3 Requirements
The requirements were generated from the data gathered from the literature review (see section 3.1 Literature review) and parts regarding feedback and error messages from the pre-study questionnaire (see section 3.2 Pre-study questionnaire), as well as the feedback that was gathered during the first user test (see 5.3 First user tests). Based on this data, a list of both functional and non-functional requirements was created, these requirements were the base for the prototypes and their content. This section will present the requirements that were used to create the prototypes that were evaluated in this study. A total of two sets of requirements were generated, the second set of requirements were iterated based on feedback from users.
4.3.1 First version of the requirements
The first set of requirements were generated by analysing the literature review and the relevant data gathered from the pre-study questionnaire. These requirements were used as a base for the prototype that was used for the first user tests.
4.3.1.1 Functional General
● Different versions of feedback should be tested
● The feedback will be tested on a computer
● Text and illustrations should be tested The design
● There should be an easy way to recover from errors
● The language should not be too technical
● There should not be too many elements in the design
● The text should be easy to read
These functional requirements regarding the design were gathered from the literature review and the pre-study questionnaire. One of Benyon’s design principles (2019) is Recovery, which states that users need to have a way to recover from their actions, especially errors and mistakes. One of the participants in the pre-study questionnaire stated that they did not want the language to be too technical, and another participant thought that a User Interface should be clear and not include too many elements, especially if something goes wrong.
According to Gong (2009) User interfaces are a platform that helps the user understand information and feedback that is given.
Error messages
● Needs to be consistent and clear
● To solve the problem the users should be presented with a link
These functional requirements regarding error messages were gathered from the literature review. According to Joon Park, MacDonald & Khoo (2012) error messages can be irritating and confusing due to the lack of consistency and clearness. Therefore, in terms of consistency the error messages follow the
prototype’s theme by incorporating the same colours and general style in the error message. The reason that a hyperlink was chosen as the given solution to the error was due to the study conducted by Kadekar, Sohoni & Craig (2018), their result showed that links helped users to understand and solve a problem in the least amount of time.
4.3.1.2 Non-functional The design
● The communication between the user and the computer needs to be clear
● The feedback needs to be easy to understand
● The user needs to be kept in the loop of what is going on
● It is important that the user understands what can be done, what is happening and what happened
● The design should be nice, polite and apologetic
● The design needs to be perceived to be easy to use and be useful
These non-functional requirements regarding the design were gathered from the literature review and the pre-study questionnaire. One of the participants in the pre-study questionnaire stated that the communication between the user and the computer needs to be clear, and another once stated that the feedback needed to be clear, and as stated before Gong (2009) writes that User Interfaces helps the users understand. Several participants thought that users need to know what is going on, if there is an error there needs to be an explanation, and that feedback to the user is important. These comments stress the importance of users being kept in the loop. Information gathered from the different design principles indicates that it is important that the user understands what can be done, what is happening and what happened. Preece, Rogers & Sharp (2019) states that computers should be polite, one of Benyon’s design principles (2019) indicates that the design should be polite, and Joon Park, MacDonald & Khoo (2012) states that apologetic systems are appreciated more by users. This was the reason that different formulations of the text were investigated, one formulation being;
apologetic, to find out if this was the case for 404 error messages as well. Based on TAM the requirement regarding perceived ease of use and perceived usefulness was included.
Error messages
● The user needs to understand the error messages
● Needs to catch the user’s attention
● Error codes should be avoided
Inal & Ozen-Cinar (2016) writes that in order for users to achieve specific tasks, the users need to understand the error message. The hyperlink that is presented as a solution in the error messages has a different colour than the rest of the text.
This decision was based on Inal's (2016) statement regarding error messages being user-friendly if they catch the user’s attention and help the user to correct the error. By giving the hyperlink a different and brighter colour, it stands out and attracts the users attention. Inal & Ozen-Cinar (2016) stated that developers include information that is useful both to themselves and the users, this is one of the reasons why the 404 code was included in two of the error messages, even though they also stated that error codes should not be emphasized.
4.3.2 Second version of the requirements
The second set of requirements were generated by analysing the data gathered from the first user tests. These requirements were used to iterate the prototype that was used for the second user tests. The changes and new requirements have been marked in bold to show the way the requirements have been iterated.
4.3.2.1 Functional requirements General
● Different versions of feedback should be evaluated
● The feedback will be tested on a computer
● Text and illustrations should be evaluated
● Different ways to recover from the problem will be evaluated
The term ”evaluated” was added to the second version of the requirements due to the nature of the second user test (see section 3.4.2 Second user tests). In the second user test the feedback was evaluated, rather than tested. Based on comments from the participants from the first user test regarding the possibility to recover from the error, more than one way to solve the problem was added for evaluation. Buttons were suggested as a solution, and the thought regarding the amount of information in the hyperlink was divided. Some suggested that the information should be shorter.
The design
● There should be an easy way to recover from errors
● The language should not be too technical
● There should not be too many elements in the design
● The text should be easy to read Error messages
● Needs to be consistent and clear
● To solve the problem the user should be presented with a link 4.3.2.2 Non-functional
The design
● The communication between the user and the computer needs to be clear
● The feedback needs to be easy to understand
● The user needs to be kept in the loop of what is going on
● It is important that the user understands what can be done, what is happening and what happened
● The design should be nice, polite and apologetic
● The design needs to be perceived to be easy to use and be useful
● Text that is purely information-based should not be clickable
● Lorem ipsum text should not be used to avoid confusion
The two new requirements were based on feedback from participants from the first user test. One user pointed out that text that is purely information-based should not be clickable because it could cause problems if a user wants to highlight that text. Some of the participants thought that the placeholder text, or
the ”Lorem ipsum text”, was distracting and confusing. Therefore, this type of text was excluded from the second version of the prototype.
Error messages
● The user needs to understand the error messages
● Needs to catch the user's attention
● Error codes should be avoided
● The text should be spelled correctly
● The error messages could be more fun and funky
More than one participant from the first user test noted the spelling in some of the error messages. To these users correct grammar was important and therefore became distracted by this fact. An example of what was distracting to them are the titles that included the phrase ”Åhnej”, they wanted to split up the words in order for the phrase to be grammatically correct. Therefore, this was fixed in the second version of the prototype. Some users pointed out that they wanted error messages to be fun and ”funky”. Therefore, this was added to the new requirements.
5. Results
This section will present the results from the different parts of the data collection, including the first questionnaire and interviews from the user tests.
5.1 Pre-study Questionnaire
This questionnaire was created using Google Forms and was conducted online.
A total of nine people participated. This questionnaire was built in collaboration with Omnifinity and had two different goals, one for Omnifinity, and one for this study. After the thesis changed direction (due to COVID-19 as discussed before), the section that was made for the study was included in the study’s new direction. The section of the questionnaire that was used in this thesis, gathered the participants opinions regarding feedback and error messages for interfaces, as well as their computer user.
The participants indicated that their computer use ranges from 4-9 hours per day, the majority ticked the box for 7-9 hours per day (see Figure 10). All of the users indicated that they were comfortable when using a computer. Most of the participants declared that they use a computer for work, and some indicated that they use one for entertainment as well. All of the participants except for one, considered themselves to be good at solving computer related problems.
Figure 10: The amount of time the users indicated that they spend by a computer per day.
The information generated from the questionnaire regarding feedback and error messages indicated that the users need to be kept in the loop of what is going on.
When something is not working as it should, there needs to be clear communication between the user and the computer. If the system is not responding for some reason, there needs to be an explanation and the feedback needs to be clear. User interfaces should be clear and there should not be too many elements, especially if something goes wrong. Feedback to the users is important, the language should not be too technical, the message needs to be clear and there should be an easy way to solve the problem. Finally, participants indicated that error messages could be useful in every situation. These comments influenced the requirements for the prototypes heavily regarding the importance of clear and understandable feedback to the user, as well as guidelines regarding design.
The answers regarding what type of feedback that would be relevant for the software that was being evaluated were very split. The options were; text, images icons, animations, and sound. The user also had the possibility to write their own proposals. Text got two votes, images got two votes, icons got three votes, animations got four votes and sound got three votes. Some users answered, a combination of text and descriptive images. This study focuses on text and illustrations; therefore, animations were not further included, even if animations received the most votes.
5.2 First pilot test
Two people were involved in the pilot tests. The tests followed through as the tests were planned to be conducted with the users. The testers were firstly given an introduction to the study and the test that was about to be conducted. Then, the test started by giving the participants a scenario which they had in mind when conducting the test. Once the test was completed the testers were interviewed where they were asked questions related to their experience of the prototype.
The feedback that was given during the pilot tests regarded the introduction that was given before the test itself started. One participant pointed out the
importance of encouraging the user to read the text that is included in the different error messages. Due to that the tester did not remember the different texts and that the text seemed similar, which means that, as a user, you start to think that the texts are the same even though they are not. This feedback led to a change in the information that the participant is given before the session starts:
the user will be encouraged to read all the text that is presented and that it is important that they understand it.
At first, the participants were only presented with the different error messages during the test, but one of the testers indicated that they had a hard time recalling the different error messages. At the end of the test the participants were asked questions related to the error messages they were presented with during the test.
It was suggested that all error messages are presented to the user at the end, this would give them a chance to have a second look at the error messages and prevent the users from not being able to answer because they do not remember.
The text in the error messages were fairly similar at first, the apologetic messages included slightly more information than the neutral information. The two versions of the apologetic messages included the same information, meaning that the difference was that one of them included only text and the other one included text and an illustration. This was the case for the neutral messages as well, they included the same text, but one of them included an illustration as well. After some discussions the text in the messages were changed. The text-based messages instead include more information and the illustration-based messages instead include less text. This decision was made because the messages should be different to investigate what type of information the user prefers and in what amount.
The last thing that was changed based on the feedback from the pilot tests are the interview questions. The questions were formulated in a way that makes more sense to the participants, and the questions needed to be more specific.
5.3 First user tests
The interviews that were conducted during the first set of user tests generated a lot of different opinions regarding the prototype and the error messages. The tests and interviews were both completed in person and online. A total of 16 people was involved in the first tests, which lasted from 6 to 30 minutes, with the majority being about 15-20 minutes. During the sessions, users were given an introduction, told what they could expect, and that they would remain anonymous (see Appendix 3).
The test revealed that the majority of the participants liked the prototype itself.
They thought that it was easy to use, logical and appealing aesthetically. But a few issues were discovered. The main issues that also were relevant for the iterative prototyping process were mainly two. The first version of the prototype included Lorem Ipsum text, which is a type of text that is used as an example within design to illustrate what the finished result could look like. Meaning that there is no real information that can be read in the text, it is meant as a placeholder and example. The Lorem Ipsum caused several users to become
confused, as they did not understand the text and therefore, they noticed the Lorem Ipsum text more than they should have and it was a bit distracting to them. Another aspect that was pointed out by some participants was the importance of spelling and that it should be grammatically correct. They noted that the title said ”Åhnej” instead of ”Åh nej”. The information text (when choosing a recipe from Veckans tips) was in the first version of the prototype clickable, but one user did not think that it was optimal. The argument was that a user might want to highlight part of the text and when the text is clickable this is not possible without difficulties. The suggestion was that the title should be clickable, and the text of the information should not be. The feedback that was given was turned into new requirements and used when implementing the second version of the prototype.
During the interviews, two users indicated that they would have liked an error message that was more fun and ”funky”. This feedback was evaluated, and a third type of message was added to the second user test, the amusing type of 404 error message.
Some of the answers that were given during the interviews were followed up because the answer needed to be verified. This resulted in some of the original answers being changed.
5.3.1 Result from the interviews
Next, the result from the interviews will be presented. During the interviews the users were not given any sort of limitations, they were free to discuss. As a result of this the participants could choose more than one alternative as the answer to the questions that were asked. Due to the open-ended nature of the interviews, there were more answers than the number of participants. This means that some of the participants chose more than one error message or wanted to combine two error messages. The participants' choices are described below as votes.
Error message 3 was perceived to be the easiest to understand by the participants. Error message 1 received two votes, error message 2 got three votes, error message 3 got 12 votes, and error message 4 received five votes.
One user would have liked a combination of error message 1 and 3, the user wanted the text from number 1 and the illustration from number 3. See Figure 11 for a column diagram of the result.
Figure 11: The result from the interviews regarding the error message that was perceived to be the easiest to understand. There were a total of 23 votes.
Error message 3 got the most votes for being the error message that the users perceived to be the easiest to use, see Figure 12 for a column diagram of the result. Error message 1 got seven votes, error message 2 received six votes, error message 3 got 11 votes and error message 4 got eight votes. The reason that this question got the most answers, and therefore generated the most votes, was because five of the participants voted for all of the error messages.
Figure 12: The result from the interviews regarding the error message that was perceived to be the easiest to use. There were a total of 32 votes.
Regarding the error message the users perceived to be the most useful information wise, the votes were evenly divided between error message 1 and 3, both received six votes each. Error message 2 received five votes, error message 4 got two votes and one user would have liked a combination of error message 1 and 3. See Figure 13 for a column diagram that shows the result.
Figure 13: The result from the interviews regarding the error message that was perceived to be most useful, information wise. There were a total of 20 votes.
The error message that received the most votes for being most useful design wise, was error message 4 who received eight votes. Error message 1 got two votes, error message 2 received one vote, error message 3 got seven votes. Three of the users would have liked a combination of two different error messages, regarding the text and the illustration. One user wanted to combine error message 3 and 2, and two users would have liked a combination of error message 1 and 3. See Figure 14 for a column diagram that shows the result.
Figure 14: The result from the interviews regarding the error message that was perceived to be the most useful, design wise. There were a total of 21 votes.
The error message that received the most votes for being the most appreciated error message was number 3, who received eight votes. Error message 1 got one vote, error message 2 received two votes, and error message 4 got three votes.
Three users wanted to combine two different error messages, two users wanted a combination of number 1 and 3, and one user wanted a combination of number 2 and 4. Figure 15 shows the result in a column diagram.
Figure 15: The result from the interviews regarding the error message that the users liked the most. There were a total of 17 votes.
The error message that received the most votes for being the least appreciated error message was error message 1, with a total of nine votes. Error message 2 got six votes, error message 3 received one vote, and error message 4 received zero votes. See Figure 16 for a column diagram that shows the result.
Figure 16: The result from the interviews regarding the error message that the users liked the least. There were a total of 16 votes.
