Smart users, smart homes, and the search for a smarter application : A qualitative study of the usability in a smart home application and how it can be improved

(1)

Linköpings universitet SE-581 83 Linköping 013-28 10 00, www.liu.se

Smart users, smart homes, and the

search for a smarter application

- A qualitative study of the usability in a smart

home application and how it can be improved

Alicia Hagelberg

Employer: Indentive AB Advisor: Björn Lyxell Examiner: Rachel Ellis

(2)

ii Upphovsrätt

Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare – under 25 år från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida:

http://www.ep.liu.se/.

Copyright

The publishers will keep this document online on the Internet – or its possible replacement – for a period of 25 years starting from the date of publication barring exceptional circumstances. The online availability of the document implies permanent permission for anyone to read, to download, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page:

http://www.ep.liu.se/.

(3)

iii

Abstract

Smart home technology and IoT is rapidly gaining popularity, and many systems use mobile applications to control the hardware in the homes of the owners. Connective created by Indentive is one such system, which allows users to control things like the temperature in their homes via their smartphones. However, the usability of applications such as this one can be limited, which leads to frustration and even loss of interest for the users. This can result in critical implications for the company producing such a system – both economically and in regards to popularity. It is therefore important for companies to detect issues with their systems early on in order to address these issues and prevent negative implications. Connective was thus tested in a qualitative study using contextual inquiries with 7 participants. The data from the inquiries was then structured through thematic analysis, and the usability of the application was also tested using SUS-evaluation. Wireframe suggestions for changes to the application were then brainstormed with the thematic analysis and the result of the SUS in mind. Major issues that were addressed were access to help, information, the layout, the handling of errors, and feedback. Future studies should include further, more in-depth qualitative studies, perhaps in combination with quantitative data, to further improve and develop the wireframe-suggestions into working prototypes that can then be user tested.

Keywords: Smart homes, internet of things, application, usability, design, qualitative

(4)

iv

Acknowledgements

To my contacts at Indentive, for all your helpful comments and suggestions, and for allowing me to work with you on this project; Carl von Koch, Christian Rydberg & Johan Gustafsson. To my advisor Björn Lyxell, and my examiner Rachel Ellis, for helpful insights, input and constructive criticism. To all the participants for being so generous with your time. To my fantastic classmates and to my friends who have acted as sound boards for my thoughts and ideas, and provided some well-needed laughs and great support: Sofia Rönnberg, Hanna Johansson, Cornelia Böhm, Madelene Wikström, Johanna Jonasson and Ellen Vestermark. And finally, to my family and to Gonzalo Rubio, for supporting me and helping out in every way possible throughout these last 3 years.

Linköping, June 2018

(5)

v

Introduction

...

1

1.2 Purpose & research questions ... 1

1.3 Delimitations ... 2

2. Background... 3

2.1 Smart Homes and the Internet of Things ... 3

2.1.1 Wireless communication ... 5

2.2 Connective... 5

2.3 Wireframes ... 7

2.4 Usability ... 7

2.5 User centered design... 8

2.5.1 User experience ... 10

2.6 Methods to conduct user centered design ... 11

2.6.1 Contextual inquiries ... 11

2.6.2 Semi-structured interviews ... 12

2.6.3 Thematic analysis ... 13

2.6.4 System Usability Scale... 14

2.7 Selection of methods ... 15

3. Method ... 17

3.1 Participants ... 17

3.2 Ethics ... 18

3.3 Data collection and analysis ... 19

4. Results ... 21

4.1 Results of the System Usability Scale ... 21

4.2 Thematic analysis... 22

4.3 Wireframes ... 27

4.3.1 Instructions and information (walkthrough) ... 27

4.3.2 Instructions and information (wizard) ... 30

4.3.3 Help ... 33

4.3.4 Errors and Feedback ... 34

(6)

vi

5.1 Method... 35

5.2 Result ... 37

5.2.1 Practical use of results & the future ... 37

6. Conclusion ... 39

(7)

1

1. Introduction

Various smart home solutions and devices is something that has gained a lot of popularity throughout the last couple of years. For instance, these devices can be used to turn off the lights in your house via your smartphone while you are at work, or heat up your car before you have even left the house. Indentive, with their headquarters situated in Linköping, Sweden, have developed a cloud-based IoT-platform called Connective, which connects different devices and collects data from them, and also offers a wide range of different services to give their customers the opportunity to create a smarter home. This is mainly done via different types of sensors that might look different depending on which service provider the customer chooses to use. These sensors can for instance control the temperature indoors, measure levels of moisture etc. However, Indentive has realized that their customers, primarily the ones with less technical experience, often get confused when they receive an installation package and are not quite sure what to do to install the product correctly and how to use it together with their application - something the company, of course, wants to avoid.

Thus, this project will be carried out in collaboration with Indentive, who have expressed a need to change and improve the installation process and the application to make it more intuitive to avoid unnecessary mistakes by the user, while at the same time not over-simplifying it too much which might lead to exclusion of the customers who have a keen interest in technology and might end up feel like they are missing out on something.

1.2 Purpose & research questions

The purpose of this project is to explore how Indentive could improve and change the installation process via their platform that is in use today and to make it more user-friendly, by compiling and mapping out user needs through observations and interviews.

Research questions: How do the users experience the installation process in use today?

What parts of the application need to be changed, either through removal or improvement, and why? How can this be done?

(8)

2

1.3 Delimitations

This project will only focus on the platform Connective created by Indentive, and the sensors and similar equipment offered by the service providers connected to it. This means that any suggestions for changes to the installation process and the application will only be applicable to the customers of Indentive and the equipment they will be using. Furthermore, the results will only focus on the application, since Indentive has no influence over the hardware provided by third parties. The observations and interviews will only be carried out with a limited number of individuals (7) to make it possible to deliver concrete and useful suggestions for this group within the set time frame.

(9)

3

2. Background

This section presents theories, concepts and methods that are considered relevant to this study. Smart homes, the Internet of Things and wireless communication will be presented in section 2.1 and 2.1.1 respectively, as to give the reader a better understanding of these terms and what they entail. In section 2.2 the application used in this study, Connective, is presented in more detail to give the reader an understanding of how this application works and to give the basis for the answers to the second research question. Medium fidelity wireframes are presented in section 2.3 as this was the method used to answer the third research question. In section 2.4 and 2.5 respectively, the terms usability and user

centered design are clarified, as to give the reader a better understanding of what separates

these terms, and what design process was used for this study. Section 2.6 describes the various methods used in this study. Finally, section 2.7 motivates the choice of methods in relationship to the research questions.

2.1 Smart Homes and the Internet of Things

Smart homes, and what is known as The Internet of Things (IoT), has exploded in popularity during the past couple of years – primarily due to technological progress that has made it possible. Smart homes however, is a very broad term, and different people define it very differently. One early definition was made by Lutolf in 1991, when he defined smart homes as;

“The smart home concept is the integration of different services within a home by using a common communication system. It assures an economic, secure, and comfortable

operation of the home and includes a high degree of intelligent functionality and flexibility.”

(Lutolf, R. 1991, 277)

Intertek (2003) via Alam et al. (2012) instead present a more recent definition where they defined smart homes as a communication network which connects different appliances and/or services and makes it possible to access, monitor or control them remotely. They also suggest three things that a home needs to contain to be considered “smart”:

(10)

4

- Some type of intelligent control (A gateway to manage the system & to connect the appliances)

- Automation (Meaning that products in the home are linked to services and systems outside of it)

Alam et al. (2012) also created their own definition, since they considered other definitions to lack various aspects:

“A smart home is an application of ubiquitous computing in which the home environment is monitored by ambient intelligence to provide context-aware services and

facilitate remote home control.”

(Alam et al. 2012, 1190)

For this study, the abovementioned definition made by Alam et al. in 2012 will be used when mentioning smart homes hereafter. Internet of things, or IoT, is just like smart homes a very broad term that envelops several different definitions. Shemshadi et al. (2017) and Saariko et al. (2017) all agree on a general explanation for it: a paradigm or concept that aims to connect various objects to the Internet, in one way or the other. Many consider this to be one of the main technologies that will change the way humans lead their lives in the coming years, and right now IoT is gaining the same reverence as the World Wide Web had back in the 1990’s with an estimated 20,8 billion devices connected to the internet by the year 2020. (Weber, 2016). Furthermore, a study conducted by Mindshare & Huddle (2016) showed that there is great interest among consumers to get their household objects connected to the internet – 64% of the participants were interested in the technology and what it could provide for them. This further indicates that the general population has positive feelings towards IoT and the services it can provide.

So, the aim of IoT is to connect common, everyday objects to the Internet. That means it gives you the ability to set your lights to turn on every day at 5.30pm when you get home from work, or to control the temperature in your house while you are at the office. Through various wearable devices, you can send information about your current medical state to your dedicated physician, without having to leave the house (Weber, 2016). Through digital assistants like Alexa or Siri built into devices like the Amazon Echo or the Apple iPhone you can play music, order food, ask about the weather, send messages,

(11)

5

search the web, get directions, lock your door, turn on your TV or call someone. (Amazon, 2018). Teslas’ driverless cars are another example of something being made possible through IoT. To summarize, the possibilities of IoT are near endless, and it is a field that is continuously developing at a rapid pace. (Weber, 2016).

2.1.1 Wireless communication

To understand how the sensors communicate, it is essential to know what kind of communication protocol is being used. There are, as stated, multiple ways for smart appliances to communicate. Samuel (2016) defines several of these - WiFi, Z-wave and Bluetooth. WiFi is probably the communication method most of us are familiar with, and it is the most widely used one since it is capable of large amounts of information transfer. However, it does not have a very large range, which makes it less optimal for smart-home connections for people with large homes. Z-wave on the other hand is a low-power communications protocol specifically used to remote control things, and it is found in a variety of appliances all over the world. It is not capable of the same amounts of data transfer as WiFi, but it is a perfect fit for smaller appliances such as door sensors or automated meter readings. Bluetooth is another way of communication that many of us are familiar with. It is designed to be cheap and secure way of communication, and it saves a lot of energy. It offers a direct connection between appliances such as smartphones which means they can be used to remotely control things – however, its range is very short and it is thus not optimal for remote control at larger distances. (Samuel, 2016).

Finally there is Zigbee, which is one of the most popular protocols for wireless communication in smart home-applications due to the fact that it is low cost, has a low power consumption, is readily available and has a very long range. Zigbee is primarily used for monitoring and to control different sensor applications, like lightbulbs etc. (Ronen et al. 2018). Zigbee is also the communication protocol used in the sensors for this study. (Indentive, 2018).

2.2 Connective

Connective is an open, cloud based platform for IoT that has been developed by Indentive AB. It collects data and connects devices and services with one another. It is built to be able to handle large amounts of data, be automatic and also enable facial recognition and

(12)

6

artificial intelligence to enhance the experience for frequent users. Add in customizable visuals and analytics, and there you have Connective (Indentive, 2018).

There are a few things that are required for a user to be able to use Connective. First and foremost, you need to download the actual application Connective to your smartphone, and set up an account. After that, you need various kinds of sensors – which ones you need depend on what you want to get out of Connective. These sensors all have different tasks, for example to monitor energy levels, measure humidity, and recognize movement or the like. Then you also need a gateway which is the unit that communicates with the sensors and which looks like a router that is commonly used in many households today. The application then communicates with the gateway, and thus also indirectly with the sensors that are installed. This means you can use the application to control or get updates from sensors, straight to your smartphone (Indentive, 2018).

Connective comes with a set of basic services, all designed for a specific purpose. There is Connective Secure which makes it possible to set up various alarms of different kids that send notifications straight to your smartphone, and gives you the possibility to add neighbors or other family members for quick access. There is also Connective Home which you can use to turn on/off lights or control the temperature in your home, or create scenarios that carry out several instructions at once – for example, to turn on all the lights at 7am. Furthermore, there is Connective Sustainable, which keeps an eye on the use of energy and water in your house, and can give you tips on how to improve this or send you an alert if it reaches critical levels. It also gives you the option to set up goals for saving energy, and gives you updates on the process. Finally there is the Connective Store which is – just like it sounds – a store where you can get what you need to expand your smart home, like extra sensors or services from a third party (Indentive, 2018).

Connective also has a set of basic sensors that can be used in the applications. These consists of door and window sensors, which are used to detect when doors and/or windows are opened in order to alert the user of possible break-ins etc., multi-sensors that can detect movement, measure UV-radiation, moisture levels, light etc., and smart plugs – electrical outlets that can be used to turn lights on and off or measure energy levels. These basic sensors are also the ones that have been used in this study (Indentive, 2018).

(13)

7

2.3 Wireframes

Arvola (2016) describes wireframes as a part of the design process where suggestions for designs are sketched out in low fidelity, meaning they are not highly detailed but rather focus on the functionality and the layout of the product and its design - in this case a smartphone application. These wireframes can then act as the basis for further development of more high fidelity prototypes which can then be user tested. The wireframes are also usually accompanied by some comments, questions or evaluations to give the designer an easier overview of the pros and cons of a certain design. All of this means that wireframes are a very basic way of visualizing design ideas, and it is only one of the first steps in the design process.

Previous studies have shown that usability data from low fidelity and high fidelity prototypes are roughly the same. This means that there is no need for designers to create intricate and complex design suggestions right off the bat unless they want to – instead, low fidelity prototypes such as wireframes are a low-cost way to explore potential problems with a design early on. Since you do not have to spend countless hours on perfecting a fully-functional, high fidelity, digital prototype that can act as a fully developed product, you can instead put the money and effort into exploring the basic functionality of your product, and see where it might need improvement or changes (Maryam et al. 2012).

2.4 Usability

Usability is a broad term which has changed definitions many times throughout the years. For instance, Shackle (1986) via Arvola (2014) coined what was to be known as the LEAF-definition. For a system to be considered usable the following criteria has to be achieved:

 Learnability – it should be easy to learn how to use the system.  Effectiveness – the system should be efficient to use.

 Attitude – the attitude of the user towards the system should be positive.

 Flexibility – the system needs to be flexible, so that it can be used in various ways depending on the context.

(14)

8

Nowadays, the industrial standard for usability is ISO 9241-11, which defines usability as:

“The extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use”

(Arvola, 2014, p. 32. Freely translated from Swedish)

Tidwell (2011) indicates that there are several ways to increase the usability of a system through its design – far too many to create an exhaustive list, considering the many different contexts and purposes of various systems. Tidwell (2011) also emphasizes that each user is unique – what one person considers to be useable is not necessarily useable to someone else, for instance due to functional variations or the like. However, Tidwell (2011) suggests these various features (amongst others) to implement in a system to generally make it more useable:

 Wizards – A feature that leads the user through the interface of the system step-by-step to complete tasks in a set order. It is very specific and focused, but also does the trick very quickly of teaching the user how the system works. Novice users usually welcome this, since making decisions is an unwelcome burden, while expert users might find it limiting. It is therefore suggested to implement the option to cancel the wizard. The wizard can also be replaced by a walkthrough, which achieves a similar goal.

 Same-page error message – An error message that enhances the usability of the system, since it instantly appears on the same page when the user makes a mistake, and indicates what is wrong and how they should fix it.

 Uniformity – A system needs to be uniform to be useable. If the different pages or functions of a system constantly change, it will become very hard to use and will thus not live up to the ISO 9241-11 standard.

2.5 User centered design

To create good usability, you also need to achieve what is known as “user centered design”, which is a design process in which a product or a system is designed with the

(15)

9

end-user in the center of the process at all times. By designing something that meets the demands and needs of the actual user, the usability of the product increases – and so does the chances of the product actually being used (Arvola, 2014). Kraft (2012) further explains how, more often than not, the reason why products fail on the market can often be traced back to one common denominator; the failure to identify the actual needs of the users and failing to keep them at the center of the design process. This leads to the creation of products which do not actually respond to the needs of the users, and thus the users simply stop using the product, or refrain from buying it in the first place. A study conducted by Mindshare & Huddle (2016) concluded that:

“The innovations that will drive the IoT forward and begin to weave it into our everyday lives are those that demonstrate real value to the consumer with minimal

effort on their part.”

Since the end-users of a product essentially create the success or the failure of said product, keeping the user in mind and involving them when designing anything is paramount (Arvola, 2014). Additionally, as shown by Magnusson (2003), actual users involved in the design process contribute, on average, ideas that are more original and useful for the project than only the members of the project group themselves would contribute with. This means that the benefits of involving and collaborating with users throughout the design process leads to reduced costs for development and shorter time spent on the design process, which is essential when it comes to marketing and launching a product – the sooner you can get it out on the market, the bigger the return on investment (providing the product is successful).

There are several different ways to conduct a user centered design process, and one definition is made through ISO 9241-210 (2010) in Arvola (2014), which defines user centered design as an iterative process, meaning it uses different perspectives to create an understanding of the user, and that it is possible to “redo” earlier steps of the process through findings in later steps. The ISO-standard emphasizes the importance of involving both the users and the stakeholders throughout the entire process, to make sure all needs are met. Usually, the user centered design process follows three basic phases with variations to them depending on the project. First, there is the concept phase in which needs and demands of the stakeholders and users are explored through collection of data.

(16)

10

This then leads to the processing phase, in which the analyzed data is used to create basic sketches and prototypes. This then leads to the detailing phase, in which the prototype is refined and then delivered to the client together with its specifications.

2.5.1 User experience

Kraft (2012) suggests that user centered design and user experience are two terms that are sometimes used interchangeably. However, user centered design is, as discussed above, a process that keeps the user in mind when designing something. User experience on the other hand, is what this process leads to – the actual experience of a product or artefact. User centered design is thus a way to achieve a good user experience. When someone uses a product, it is easy to imagine the various feelings the user might go through in the process. Positive feelings naturally mean the user experience is increased, while negative feelings have the opposite effect. If there are too many negative feelings associated with a product or artefact, you will lose the interest of the user and through that possibly lose a customer for all your other products Kraft (2012). Tidwell (2011) argues that instant

gratification is one way to dramatically increase the positive user experience of a system.

People like to see instant results of their actions, and if the user feels successful within the first couple of seconds of using a system, they will automatically feel more confident and positive towards it. However, if the user has to wait to see results, the negative feelings will instead take over. Thus, as said, the user experience can be a mix of both positive and negative emotions. Kraft (2012) visualizes the user experience of using a certain product in Figure 1 below, which makes it clear that user experience can involve a variety of emotions:

(17)

11

Kraft (2012) also discusses how different people react in different ways and have different emotions in different situations – even the same individual may experience different emotions in the same situation, all depending on context. Stress is for instance a huge factor in how some people react to incidents, and thus a stressed person might drop more rapidly on the user experience scale than he or she would when feeling calmer. Some things may even cause users to go from happiness to rage in a matter of seconds, such as a program crashing when you have spent an hour compiling an important e-mail. Furthermore, a major source of frustration can reveal itself when the user in question knows that there are better solutions to his or her problem out there, but cannot use them. Thus, of course, the goal of any product design should be to create the maximum amount of positive feelings for the users. Kraft (2012) also suggests a final, important element of any design – to eliminate the worst of the negative feelings when using your product. This is because one negative user experience might need ten or more positive experiences to weigh up for it, which puts more strain on the remaining design elements of the product.

2.6 Methods to conduct user centered design

Keeping the user at the center of the design process and aiming to create a product with good usability and a positive user experience is paramount, according to Arvola (2014). There are several methods to conduct a user centered design process, amongst them contextual inquiries, semi-structured interviews, thematic analysis and SUS (Arvola, 2014). All of the aforementioned are explained in greater detail in the following sections.

2.6.1 Contextual inquiries

Arvola (2014) describes a contextual inquiry as combining interviews and observations to learn more about how a user actually uses a product. Interviews are a basic method when it comes to user research, but it is important to remember that people do not always act as they say they do. Therefore, it is also a good idea to conduct observations to complement information gathered through interviews. Crandall, Klein & Hoffman (2006) suggest several things to keep an eye out for during an observation, for example:

 What information is being used, and how is it presented?  In what order do the users carry out actions?

(18)

12

 What errors are possible to make?

 What kind of support or help could be useful in various situations?

As the name implies, a contextual inquiry takes place in the actual context where the activity takes place – such as an office if you are studying the usage of an intranet, or the like. Instead of generally looking at how a task is carried out, it can also be beneficial to set up a specific task that is to be completed. During the task, the user is then asked to explain their actions and thoughts – what is known as a think aloud-protocol. The designer also asks questions based on what he or she observes to better understand what drives the user and what their goals and needs are – called verbal probing. Thus, the designer and the user mutually create an understanding for the system in use through the observation. The think-aloud protocol and the replies to the verbal probing are then usually recorded and transcribed, and based on these protocols it is then possible to analyze how the user thinks in certain situations, and what can be improved within a system. (Arvola, 2014).

2.6.2 Semi-structured interviews

Arvola (2016) suggests that it is often hard to follow a strict interview protocol when carrying out a contextual inquiry, as the situation may unfold in many different ways, and people usually do things in very different ways as well. Instead, what is known as a semi-structured interview protocol can be used, in which a set of standard questions are created and then used when (or if) needed. This makes it possible for the interviewer to have some kind of structure to the interview, for instance to make sure that the most important questions are answered and that the conversation stays on topic. (Arvola, 2014). Semi-structured interviews are thus used when there is some knowledge about the topic or issue in question, but you want to find out more in-depth information and more details. (Wilson, 2016). But, it also enables the interviewer to explore concepts and ideas that perhaps previously had not been considered, but that would be of great interest to the study. They also provide the interviewer with a tool to help redirect an interview that might stray off-topic. Semi-structured interviews thus mean that the interviewer keeps some of the structure of a normal interview, but also has the possibility to gain knew knowledge in other areas. (Arvola, 2014). Wilson (2014) suggests the following steps to conduct a successful, semi-structured interview:

(19)

13

 An introduction to the purpose and topic of the interview

 A list of topics and questions to ask about each topic

 Suggested probes and prompts

 Closing comments.

This structure is somewhat followed in the semi-structured interview protocol that was used for this study – an introduction to the topic before the contextual inquiry, a list of questions relating to major topics of interest, suggested questions to encourage the participant to develop their answers and closing remarks to make sure the information gathered was correct.

2.6.3 Thematic analysis

Thematic analysis is a fairly generic, straight forward and easily-understood method of analysis for textual data. It is primarily useful to analyze data in circumstances where the data collection is finished and complete, the data consists of textual information such as interviews or focus groups, and the data is rich in detail. In short, thematic analysis is a method which is used to categorize data into major themes or descriptive categories to gain an overview of the most important features of the data (Howitt, 2010).

Howitt (2010) describes thematic analysis as a way to analyze what is being said by participants in a study, rather than how it is said. This requires that the researcher in question identifies a handful of themes that adequately describes what is actually happening in the collected, textual data – such as interviews or observations. Furthermore, this requires that the researcher is very familiar with the data being analyzed, which means that they themselves have to collect it, read it, and then re-read it to gain as much understanding of the data as possible.

The central processes in thematic analysis are, according to Howitt & Cramer (2011) via Howitt (2010), transcription, analytic effort and theme identification. The process does not follow this strictly though, but rather the steps overlap and intertwine as the process goes on. This means that thematic analysis is a highly iterative process where you can go back and redo steps as you go along. Furthermore, Howitt (2010) presents the following steps to conduct thematic analysis:

(20)

14

1. Data familiarization 2. Initial coding

3. Searching for themes 4. Review themes 5. Defining themes 6. Writing the report

Following these steps, the researcher first familiarizes him or herself with the data, as previously described. The data is then coded into small segments of perhaps only a couple of lines at a time to indicate the contents of the data – these codings can be considered brief descriptions of the content at hand. There are no exact rules for how to execute this part of the process, but generally more conceptual codings are considered better.

From these codings the researcher can then develop and identify themes that describe the major features of the data – for instance, if every participant in a study has complains regarding a certain feature, that issue can be considered a major theme. Each theme then needs to be clearly defined, so that it does not intertwine with other themes. The themes are then presented in a report or the like, and it is suggested that the presentation includes some excerpts from the data. This is usually in the form of quotes from the participants, in order to illustrate particular features of the themes more clearly.

2.6.4 System Usability Scale

In 1996, John Brooke developed what is known as The System Usability Scale (SUS) which is a low-cost, reliable and easy way to measure the usability of various systems. Of course, there is no one right or “ultimate” way to measure usability, since it differs between contexts, but the SUS offers a simple way to do this once you have defined your users and the context the system will be used in. The questions of the questionnaire (see appendix 3) cover various aspects of usability, from training to complexity, and it is therefore a measurement for usability that can be applied to basically any system. The questionnaire should be filled out by the participant immediately after using the system that is being tested, before any discussion takes place, in order to avoid external forces affecting their answers. The participants shall also be instructed to answer the questions instinctively, without thinking about them for too long.

(21)

15

The scale has 10 questions in total, with 5 step scales (1-5), where half of the questions are negative (for instance: I thought the system was easy to use) and half of them are positive (for instance: I thought the system was unnecessarily complex). Answers to separate questions in the questionnaire mean nothing on their own – instead, a total SUS-score is calculated by summarizing what is called the “SUS-score contributions” from each question. For every positive question you take the scale position minus 1, and for the negative questions you subtract the scale position from 5. The score contributions are then added up and multiplied by 2.5 to get the final SUS score, which can range from 0-100, with 100 being the best it can be. Sauro & Lewis (2016) present a suggested interpretation of these scores with American school grades:

Table 1: SUS-scores & respective grades (Sauro & Lewis, 2016)

2.7 Selection of methods

To address the first research question, contextual inquiries will be used in the participants’ homes as to gain a deeper understanding of how the user experiences the installation process and the application, and what he or she thinks about it. The contextual inquiry will use a mix of think-aloud protocols and verbal probing for added depth, and will also be followed up by use of a semi-structured interview to gain more understanding regarding areas that might be related but unexplored. To answer the second research question, thematic analysis will be used to structure the collected data and gain an overview of the most pressing issues with the application, and to come up with suggestions for how it can be improved. Finally, wireframe suggestions will be developed

SUS Score Range Grade

84.1–100 A+ 80.8–84.0 A 78.9–80.7 A− 77.2–78.8 B+ 74.1–77.1 B 72.6–74.0 B− 71.1–72.5 C+ 65.0 – 71.0 C 62.7–64.9 C− 51.7–62.6 D 0.0–51.6 F

(22)

16

to answer research question three, and give examples for how the suggestions developed during the thematic analysis can be implemented into the application.

(23)

17

3. Method

This is a qualitative study where the data was collected through observations and interviews with potential users of Connective. The choice to conduct a qualitative study instead of a quantitative study was simply because qualitative studies focus on how people experience things – what they feel, think and perceive. As the final product of this study is based on what the users think and how they experience the installation process, it is paramount to get a profound understanding of this, rather than to collect a large amount of shallow numerical data. (Arvola, 2014). The aim of this qualitative approach is thus to gain a deeper understanding of how the user experiences the installation process, and where something might go wrong so that it can be improved upon. Suggestions for improvement was then created on the basis of this data, and compiled into several, medium fidelity wireframes to visualize these improvements. The wireframes were created in a program named proto.io, which is an online prototyping tool.

A pilot study was conducted to determine the best way to carry out the observations, for instance whether to use think-aloud protocols or verbal probing, and in what order to ask the participants to complete tasks. The pilot study was also conducted to get an estimate of how long the whole process would take, and to make sure that all the components worked as intended. The pilot study did not result in any major changes to the contextual inquiries or the interviews, as everything worked out according to plan, and a mix of think-aloud protocols and verbal probing were used.

The contextual inquiries and interviews were also recorded as to not lose any important information along the way. This was of course done with the participants’ informed consent (see appendix 1).

3.1 Participants

The participants of this study were recruited through a combination of convenience sampling and snowball sampling due to time and resource restrictions. Secondary contacts were contacted via email where they were asked to provide further contacts that could participate in the study. In total, 35 individuals were contacted via email and telephone to see if they would be willing to participate in the study. 8 individuals accepted. According to Nielsen (2012) 5 participants is enough to conduct relevant

(24)

18

usability testing and identify the major issues with a system, as the researcher will discover reoccurring problems just through tests with these 5 participants. A slightly larger population will only confirm already found issues. This study thus covers the minimum number of users, and the extra participants will contribute to a deeper understanding of the possible issues with the system being tested.

The study consisted of 7 participants (8 originally, but one opted out of the study), where the age varied between 19 and 47 years with a median age of 23 and a mean of 26.8 years. The participants were all from different socio-economical and educational background, and lived in various parts of Sweden. Furthermore, the participants were asked a predetermined set of questions regarding their perceived technical skill level – for instance, if they considered themselves to know their way around technology, how often they use technology (like a computer or a smartphone) in their daily lives and so forth (see appendix 2). This then made up the basis of analysis later on, and made it easy to see whether there were any common denominators for a certain issue within the application, or the like.

The criteria for being considered technologically skilled was if the participant answered 4 or above in questions 6-10 (see appendix 2), and conversely if they answered 3 or below they were considered to be less technologically skilled. 4 participants (3 female and 1 male) were thus considered to be technologically skilled based on the self-evaluating form that was filled out before the observation started. This means these participants considered themselves to use technology a lot, were confident in their abilities and said they had a keen interest in the area. 3 participants (2 male and 1 female) were considered to be less technologically skilled as they judged themselves to have little technical experience, doubted their own abilities and had little or no interest in the area. 3 of the participants were male, and 4 of the participants were female.

3.2 Ethics

When collecting data and conducting research that includes individuals, there are several ethical aspects to consider. The Swedish Research Council (2002) presents four requirements that need to be met in order for research to be conducted in an ethical way. These are the information requirement, consent requirement, confidentiality requirement

(25)

19

and the fact that gathered data can only be shared for research purposes. The participants in this study were presented with a written form informing them about the observation and what the results were going to be used for, as well as the fact that they were free to stop at any point and that their information will not be used other than for research purposes. They were also informed that the results are confidential. Thus, this study fulfils all of the requirements to be considered ethical by the Swedish Research Council (2002).

3.3 Data collection and analysis

Data was collected through contextual inquiries, where the participants were observed while installing the product and were also asked semi-structured interview/probing questions throughout the process. This combined approach was used since it is important to actually see what the participant does (what he/she picks up, where he/she puts it etc.), but also to understand why they do it and what their thought process looks like (Arvola, 2014).

Indentive provided an installation package containing a door sensor, a multi-sensor and two smart plugs, a gateway and other equipment (third-party instruction booklets for the sensors) which was then used during the contextual inquiries. The inquiries took place in the participants’ home, as to create a natural environment that would be as similar to a real-world situation as possible. The participants were presented with the consent form, which they all signed, and were then presented with the installation package as it would look upon delivery, and were then simply asked to install it using the hardware and the Connective-application that provides instructions. The participants were asked to complete 5 tasks:

1. Install all of the units. You are free to choose in what order to install them. 2. Make sure the units are connected & work.

3. Create a routine – for instance, at 5pm the light should turn off. 4. Create a switch-panel and an extinguisher.

5. Remove the units from the application.

These tasks were chosen to be performed as they cover the basic functionality of the application, and include functions that the average user is very likely to use in their

(26)

day-20

to-day usage. Units refers to the gateway and the sensors. A routine allows the user to create an “if this, then that”-function, for instance “when the sun goes down, turn on all the lights”. A switch-panel is a function that allows you to group up several sensors and turn them on or off all at once, with a single press of a button, while an extinguisher does the same for lights. Tasks 3 and 4 switched places during half of the inquiries, as to eliminate any possible learning bias in the results. Pre-determined questions were also asked throughout this process (see appendix 4), as well as unplanned questions, if needed, to get clarification on certain things

These sessions were recorded as to not lose any important information along the way, but major points of interest were also noted in writing during the sessions. The recorded data was then analyzed further after the observation sessions, and important phenomena, opinions and thoughts were categorized into different themes according to the thematic analysis defined in Howitt (2010) to create an overview of what areas need improving, and how this can actually be done.

(27)

21

4. Results

In this section, the results of the study will be presented. The results of the SUS-evaluation are presented in section 4.1, and in section 4.2 the results of the thematic analysis are presented. Both these sections answer the first and second research question. Finally, the wireframe suggestions are presented in section 4.3, as an answer to the third research question.

4.1 Results of the System Usability Scale

The results of the SUS-questionnaire filled out by the participants after the contextual inquiry are presented below. Table 2 shows the participants, their assessed skill level when it comes to technology (based on the self-assessment questionnaire) and the total SUS-score they gave the application. Table 3 presents the summarized SUS-value along with the mean and standard deviation.

Table 2: Participants & scores

Participant Technologically Skilled (more/less) SUS-score 1 More 65 2 Less 62,5 3 Less 37,5 4 More 67,5 5 More 65 6 Less 30 7 More 37,5

As presented in Table 2, there seems to be some correspondence between the level of technological skill and the SUS-rating that participant gave the application. 3 out of 4 participants with a higher level of technological skill gave the application a SUS-rating of 65-67,5 which corresponds to grade C as shown in Table 1 of section 2.6.4. Respectively, 2 out of 3 participants with a lower level of technological skill gave the application a SUS-rating of 30-37,5 which corresponds to grade F as shown in the

(28)

22

aforementioned table. This might indicate that people with a higher level of technological skill find the application easier to use, due to their previous knowledge in the area, while people with a lower level of technological skill find the application harder to use. However, there is still 1 individual in each group that gave the system a SUS-rating that does not correspond with their technological skill level. Reasons for this will be discussed later on in this study.

Table 3: SUS-scores

Questionnaire Total score M SD

SUS 365 52,14 16,29

Note: Total score = Total SUS-score from figure 1. M = Mean. SD = Standard Deviation

As Table 3 shows, the mean SUS-score for the application was 52,14, which means that the usability of the application corresponds to grade D, as shown in Table 1 of section 2.6.4. This is a very low score and indicates various usability issues within the application.

4.2 Thematic analysis

The results of the contextual inquiry and the subsequent thematic analysis are presented below. For each theme, a goal that could solve the issues is presented. The main problem with the system as a whole for the participants was the hardware and the accompanying instructions. However, this is not something that Indentive has any real control over, and thus it will not be discussed further in this report – but it is something to keep in mind. The discovered themes are presented in Table 2 to give a brief overview, but are also explained further below.

Table 2: Identified themes & brief explanations

Theme Explanation

Instructions & information

Subtheme: Layout & intuitivity

Lack of information and instructions leads to frustration. The layout of the application is good, but lacks intuitivity and needs clearer instructions.

Help No simple ways to get help when needed.

Errors & feedback Very little feedback in general, especially when errors occur.

(29)

23 Instructions and information

The lack of instructions in the application was one of the most prominent issues that emerged during the contextual inquiries. The hardware comes with physical manuals, but these are long, complicated and not very straightforward, and the application itself does not provide any information on how to actually start the installation process:

“I don’t think I can do this, there’s just too much information in these manuals… To be

honest I feel a bit defeated, how am I supposed to know what to start with?”

(Participant 3, freely translated from Swedish)

“How do I even add things? It’s not via my profile, or in the overview or units tab. I thought I got it right when I pressed “units”? How do I proceed from there?

What the different sensors are used for was not clear for all participants either. Some took great care in reading all of them from front to back, while others did not read the physical manuals at all, since they were too long and with too much information:

“Oh, the outlet plug is a sensor? I thought you were supposed to plug in the gateway to it! They should tell you that in the app, not in the middle of the manual.”

The issue with information and instructions was one that kept re-appearing, with two participants even giving up and admitting defeat because they did not know how to proceed. Once they managed to connect one or two sensors however, things started to run more smoothly due to the fact that the participants now knew how to navigate the application. That was not clear from the start though, which it should have been:

“Well, you want it to be quick and easy and clear as soon as you open up the package

and start the app, which it’s not at the moment. Some kind of ”quick-start guide” for both the application and the sensors would be very helpful”

Suggestion: Include a walkthrough when the user first opens the application. A

(30)

24

for instance which buttons do what. The walkthrough can be opted out of, and can also be found again in the help-section should the user need it later on. The walkthrough can either be combined or replaced with a startup wizard instead, which is a short tutorial to help guide the user through the initial steps of using an application.

Subtheme: Layout & intuitivity

The participants were generally positive to the design and layout of the application at a first glance. Once they were instructed to do things however, the layout of the application was not as intuitive as some would have wished:

“The switch panel is under the overview-tab? Why is it there? That makes no sense. There should be another tab for setting up things like this, just like there is for

routines.”

The steps that proved to be the easiest were to create a switch panel and an extinguisher. The participants were asked to do this alternately, some starting with an extinguisher and some with a switch panel – none of the participants found this task particularly challenging, and proceeded to complete them both very quickly in comparison to the other tasks. This was most likely due to the fact that they had already clicked around the application several times during the initial installation process, and knew how to find their way around it. However, some participants commented on this fact:

“I don’t think it would have been as easy if I hadn’t done all the other things first. This time I knew where to go since I had seen it in the beginning.”

Several of the participants struggled with the fact that the content and function of the menu-button changed depending on what tab you currently had active. For instance when one participant first opened the app to install and connect the gateway, and the application automatically opens the tab Överblick (overview). The participant clicked on the menu, then “add”, they got the option to add switch panels, dimmers and the like. This was not

(31)

25

what the participant wanted to do, so they continued to the tabs for routines and units, but did not understand that the contents of the menu changes according to the active tab.

Suggestion: Implement a more intuitive layout with tab-names that actually corresponds

to their content, or clearer information on where to find things (just like in the theme for information presented above). The functions need to be uniform throughout the application, so clearer information is needed on where to find things.

Help

The users sometimes need help when using the application, and one major issue that arose was the lack of it within the application. Not merely via the lack of instructions for how to use the application and how to set up the hardware, but also because there was no easy way of finding instructions or getting help with how to do anything. Many of the participants ran into trouble several times and had to trial-and-error their way out of it or to make something work, which was a major source of frustration:

“I’d like to get some help from the app here, but there doesn’t seem to be any… There

should be a FAQ or something like that, perhaps a chat or a phone number. I don’t want to contact the company to get help with something this trivial.”

“I don’t think I can do this, is there some sort of help I could get? I can’t find anything

on here, the only way is to send an email to the company and that doesn’t feel right!”

(Participant 7, freely translated from Swedish) Help-pages are such an integrated part of most technologies today that all of the participants took it for granted that they would be able to get some sort of help from the application itself. This is a great feature to have, as it provides the user with more in-depth instructions without them having to contact the company in question directly.

Suggestion: Include a FAQ (frequently asked questions), a help-page, pop-up suggestions

(32)

26 Errors & feedback

The actual process of pairing the hardware with the application proved difficult as well. The participants did not find it clear or simple at all to understand what to do or when to do it. It was particularly hard to understand that you had to go back in the application and restart the pairing itself if it failed – a majority of the participants simply tried pressing the buttons on the hardware again, resulting in a longer installation time and more frustration. Some participants even found the process stressful, since they had to try and pair the hardware before the timeout-message appeared. This process needs to be clarified in steps, and with feedback and information on what to do if the pairing fails.

“Why does it timeout after a certain time? How am I supposed to have time to do this before it stops working, when it’s not working properly?”

However, a vast majority of the participants found that the little sound the gateway makes when you manage to pair hardware successfully was incredibly helpful and nice. Some did not understand at first what it was or what it meant, but in combination with the message on the smartphone application it became clear. Several of the participants expressed feelings of pride and self-confidence upon hearing this sound:

“Wait, what was that noise? Did I do it? I did it! I’m not useless at this! ://That little noise is a very nice touch, it’s good to get feedback when you get things right and not

only when you’re making mistakes.”

“Oh, that’s a fun sound! Does that mean I did it right?”

Suggestion: Implement clearer feedback when something goes wrong, like a

warning-message or explanation of what to do if something goes wrong, to guide the user further in how to handle errors.

(33)

27

4.3 Wireframes

In this section, the wireframes suggestions for improvement of the application are presented. The wireframes are presented graphically, followed by a motivation and explanation for the choices made. The sections follow the naming of the themes from the previous chapter. As the theme instructions and information has a subtheme regarding the layout and intuitivity of the application, the suggestions for the subtheme are intertwined with the suggestions for the main theme

.

4.3.1 Instructions and information (walkthrough)

Several of the participants expressed confusion and frustration with the user interface of the application, since they had no idea what part of the application was used to actually connect the hardware to the application – how to “install” it, so that it could be controlled via the smartphone. When done correctly, this is done by opening the application, navigating to the page Enheter (units), and then pressing the menu-button consisting of three dots in the upper right-hand corner (see appendix 5). However, this was very unclear since the application has no instructions regarding this – the tab for Rutiner (routines) and

Överblick (overview) have some brief explanations, but the tab Enheter (units) has

nothing. Therefore, a majority of the participants proceeded to click around the application at random to see what would happen. Three participants did not understand it at all, and asked for help on how to accomplish the task (which was provided, considering they had tried for several minutes without success). Several of the participants expressed a wish for a walkthrough when you open the application for the first time. This would give the user a brief overview of the main uses of the application, which would facilitate a faster understanding of the application as well as instant gratification which is considered positive for the usability according to Tidwell (2011). Suggestions for such a walkthrough (or quick-start guide) are presented below.

(34)

28

Figure 1: The first pop-up after logging

into the app – explains that the tabs have different functions. Progress-bar indicates the existence of more pages.

Figure 2: The second page of the

pop-up. Briefly explains the tab

(35)

29

Figure 3: The third pop-up page.

Briefly explains the tab Rutiner (routines).

Figure 4: The fourth pop-up page.

Briefly explains the tab Enheter (units) and that you add your gateway and sensors through it.

(36)

30 4.3.2 Instructions and information (wizard)

As stated in the previous section, a majority of the participants experienced frustration and anger when they were not able to find the information they were looking for to get started on using the application and their hardware. A suggested solution for this is to implement what is known as a wizard (Tidwell, 2011) which – in its broad sense – is a tutorial that guides the user through the application in a step-by-step manner the first time they use it. Ideally, it should be possible to restart the wizard even after completing it the first time. Wireframe suggestions for such a wizard are presented below.

Figure 5: The user is greeted with a

message the first time they open the application, explaining how they do not yet have any units connected and that they should press Enheter (units) to get started.

Figure 6: After navigating to the

units-tab, the text prompts the user to press the menu-button in the top right corner to connect their gateway, and later other sensors. The user can also press the help-button for extra assistance.

(37)

31

Figure 7: After adding at least one

gateway and unit, the user is greeted with this text informing them to proceed to the tab Rutiner (routines) to make it possible to automatically control their home. It should also be possible for the user to opt out of creating a routine, and instead skip to the page presented in Figure 9.

Figure 8: After navigating to the

routines-tab, the text informs the user on how to create a new routine – by pressing the menu-button in the top-right corner. They can also get extra assistance through the help-button if needed.

(38)

32 Figure 9: Finally, after creating a routine (or opting

not to create one), the user is greeted with an informative text on the tab Överblick (overview), for instance that they can create groups to extinguish all lights at once. It is also possible to get extra assistance through the help-button if needed.

(39)

33 4.3.3 Help

Another issue that arose was the lack of help within the application. Not only via the lack of instructions, but also due to the fact that there was no simple way of finding instructions or the like for how to actually do things. A help-page or a FAQ (frequently asked questions) would be beneficial in this case, and a suggestion for such a page is presented below. It is recommended that a similar feature is adapted and available for Överblick (overview) and Rutiner (routines) as well.

Figure 10: The first page on the tab Enheter (units). Explains that you

press the menu-button to add hardware, and provides a help-button for more detailed instructions.

Figure 11: The help-page for Enheter

(units). Provides more detailed instructions on the steps needed to install the gateway and the sensors.

(40)

34 Figure 12: Added information on the

pairing page instructing the user what to do if something goes wrong.

4.3.4 Errors and Feedback

There was also a lack of feedback and information when errors were made, which led to frustrations among the participating users – particularly in the stage where the user connects the sensors. If the sensor does not connect properly, the application says to simply “try again”, but the user does not understand that you have to go back a step and restart that step of the installation process – instead, users just try and re-connect their sensors instantly, which does not work. Clearer instructions, information and same-page error messages as suggested by Tidwell (2011) would be beneficial to avoid unnecessary steps and irritation, and a suggestion for this is presented below.

(41)

35

5. Discussion

This final part of the thesis will discuss various aspects of the current study. In section 5.1 the choice of methods will be discussed, and in section 5.2 the results and their implications are discussed – as well as suggestions for future studies.

5.1 Method

This study used a combination of convenience sampling and snowball sampling, which could be considered problematic since there is generally a risk of bias – the sample all derive from the same original people, which means that there is a risk of community bias. However, none of the participants had partaken in studies before, knew the end-goal of the study or had ever been in contact with either Connective or Indentive – in fact, none of the participants had ever even heard of the company. Then there is of course also the fact that the sampling methods are not random, but this would have been very hard to accomplish within the limited time frame. Finding participants for a study like this can be rather challenging, and thus a convenience- and snowball sampling was preferred to conduct the study within the set time frame.

Additionally, the sample was rather small (again a result of it being challenging to find participants) and a larger sample would perhaps give better insights into the issue at hand. This does not have to be the case though, as Nielsen (2012) argues. In the case of this study, major issues with the application were detected already after just 3 sessions of data collection, and the remaining sessions only further confirmed this. This corresponds to the findings of Nielsen (2012), however, with a larger sample size it would perhaps be possible to generalize the results in a better way than possible today.

One of the advantages of using qualitative methods is that you need fewer participants than you would need when using a quantitative method, as you are interested in peoples’ feelings and thoughts rather than numbers (Arvola, 2014 & Howitt, 2010). That being said, it would be interesting to take a quantitative stance on this study as well, just to get a different angle on the results and perhaps more “substantial” results – quantitative data can usually be made more visually accessible than quantitative data, which might be appealing to some.

Smart users, smart homes, and the search for a smarter application : A qualitative study of the usability in a smart home application and how it can be improved