IN
DEGREE PROJECT COMPUTER SCIENCE AND ENGINEERING, SECOND CYCLE, 30 CREDITS
STOCKHOLM SWEDEN 2019,
How In-House Usability Testing Increases Product Development
JONATAN SEGER
KTH ROYAL INSTITUTE OF TECHNOLOGY
SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE
ABSTRACT
Can the Product Development Process at Tcom be improved by moving usability testing in-house?
The interest in this question lies in the agile transformation Tcom initiated in early 2018. Once agile, the company’s interests are efficiency and increasing quality. One way of increasing the quality of their digital products is to improve the process behind it, where involving users in usability testing should be central in an iterative development process. A qualitatively driven mixed methods research approach facilitated the acquisition of the empirical data necessary to successfully answer the research question. The culture of Tcom’s usability testing was collected via a survey. The survey was
complemented with the current testing process through four interviews and two observations of usability tests, one internal and one external. Parameters between the two tests were compared and weighted against each other. The main findings reveal that the internal process shows promising characteristics. It is, up to, 1/10 as expensive per participant, has a quicker call-to-test-time, facilitates transparency, and allows refinement of the process. With the main findings in hand, the conclusion points to the fact that usability testing should be composed and conducted by Tcom itself. The greatest challenge to such a change is how a usability perspective can be integrated into the Scrum
methodology.
SAMMANFATTNING
Kan produktutvecklingsprocessen på Tcom förbättras genom att flytta användbarhetstester internt?
Intresset i denna fråga ligger i den agila transformation som företaget påbörjade tidigt 2018.
Effektivitet och kvalitet är centrala förbättringspunkter med det agila arbetssättet. Ett sätt att öka kvaliteten på deras digitala produkter är att förbättra processen bakom, där involveringen av användare i användbarhetstester bör vara centralt i ett iterativt arbetsförlopp. En kvalitativt driven mixed methods research metod tillämpades för att förvärva den empiriska data som krävdes för att framgångsrikt besvara frågeställningen. Tcoms användbarhetstestkultur undersöktes via en enkät. Denna metod kompletterades med 4 intervjuer och 2 observationer av användbarhetstest, ett externt och ett internt.
Parametrar från de två observationerna ställdes mot varandra. De centrala upptäckterna avslöjar att den interna processen uppvisar lovande egenskaper. Den är upp till en tiondel så kostsam per deltagare, har snabbare beställning-till-test-tid, tillåter transparens, och möjliggör förbättring av själva testprocessen.
Med hänsyn till huvudfynden kan vi konkludera att användbarhetstest bör utformas och utföras av Tcom själva. Den största utmaningen till en sådan förändring ligger i hur ett sådant
användbarhetsperspektiv ska implementeras i det agila ramverket Scrum.
How In-House Usability Testing Increases Product Development
Jonatan Seger
KTH, Royal Institute of Technology Stockholm, Sweden
jseger@kth.se
ABSTRACT
Can the Product Development Process at Tcom be improved by moving usability testing in-house? The interest in this question lies in the agile transformation Tcom initiated in early 2018. Once agile, the company’s interests are efficiency and increasing quality. One way of increasing the quality of their digital products is to improve the process behind it, where involving users in usability testing should be central in an iterative development process. A qualitatively driven mixed methods research approach facilitated the acquisition of the empirical data necessary to successfully answer the research question. The culture of Tcom’s usability testing was collected via a survey. The survey was complemented with the current testing process through four interviews and two observations of usability tests, one internal and one external. Parameters between the two tests were compared and weighted against each other. The main findings reveal that the internal process shows promising characteristics. It is, up to, 1/10 as expensive per participant, has a quicker call-to-test-time, facilitates transparency, and allows refinement of the process. With the main findings in hand, the conclusion points to the fact that usability testing should be composed and conducted by Tcom itself. The greatest challenge to such a change is how a usability perspective can be integrated into the Scrum methodology.
Author Keywords
Product Development Process; User-Centered Design;
Usability Testing; Agile Framework.
ACM Classification Keywords
H.5.m. Information interfaces and presentation (Human- Computer Interaction).
INTRODUCTION
Product validation at any company is an important component [20] in the Product Development Process (PDP). Uncertainties and irregularities affecting this process have a tendency of decreasing consistency and
usability throughout a product, most notably in cases where a product is continuously iterated upon. There are various types of methods to assess quality in products.
A shift towards an agile working process is under implementation at the company. Although many aspects of an agile environment are in place, there are areas that need further attention. Such as the lack of usability attention when developing products. Where an insufficient product validation, in other words testing, is what this report aims to investigate. Testing with users, which according to User- Centered Design (UCD) methodology [26] concludes an iteration as well as providing insights for the following iteration. This perspective puts the users in center by involving them in the process. UCD methodology with a focus on usability is applied in this report since the author perceives it can enhance product quality and due to a trend of usability as sales arguments [4]. UCD is not yet fully established in the PDP at Tcom and we asked ourselves how difficult it would be to integrate usability testing with their existing processes and whether the end result would be better if the testing was done in-house as compared to being done by others afterwards. In particular, we wanted to hear the attitudes of the employees towards such a change. Thus, part of the study presented here aims at investigating the company culture and responses towards integrating such a testing phase. Interviews were conducted with project managers in order to understand the current testing process.
This method was complimented by a survey, where Tcom employees’ perception of testing and the testing process were in focus. The results indicated the prerequisites to scale up the testing process, which is the next step for Tcom in their pursuit of improving their PDP.
Two usability tests were studied. The former being a first- ever in-house usability test executed at Tcom. The purpose of this test was to investigate both a newly developed purchase flow and the effectiveness of having this form of testing in-house. Both tests will be described in greater detail later in the report. The second usability test was carried out by an external agency called Usbil, which is a market research company that operates business-to- business. They conducted a usability test ordered by an employer organization called Mploy, where the purpose was to test a set of scenarios in their new website. Usbil is a Tcom partner, where they perform various types of testing for the company. I took part in both respective testing processes, acting as an active observer. As an active observer I focused mainly on the process itself, while also contributing with comments and insights to the test results.
Paste the appropriate copyright/license statement here. ACM now supports three different publication options:
• ACM copyright: ACM holds the copyright on the work. This is the historical approach.
• License: The author(s) retain copyright, but ACM receives an exclusive publication license.
• Open Access: The author(s) wish to pay for the work to be open access. The additional fee must be paid to ACM.
This text field is large enough to hold the appropriate release statement assuming it is single-spaced in Times New Roman 8-point font. Please do not change or modify the size of this text box.
Each submission will be assigned a DOI string to be included here.
Qualitative and quantitative data [13] was obtained through the three earlier mentioned methods: online survey, interview, and active observation. The questionnaire was distributed in the company through two channels, for the sole reason to increase the number of responses. The interviews were held with four employees which resulted in qualitative data regarding the current PDP. The two active observations were included in order to procure hands-on insight into the execution and post-process of usability testing. In combination, these provided adequate data to answer the research question:
- Can the PDP be improved by moving usability testing in-house?
BACKGROUND
The background section presents the basics of UCD as well as how UCD methods are applied in industry. Benefits and different usability testing methods are furthermore introduced. The section ends with further attention to the two agile frameworks Scrum and U-Scrum.
UCD & frameworks
The UCD process has a strong focus on involving the human perspective in all its stages during its iterative development cycle [26]. Which is essential for a computer- based system to be used sufficiently by its users [10]. It puts a central focus on insights guiding iterations until a desired solution has been established. Insights are most often generated from the likes of testing. UCD principles revolve around: early focus on users and their tasks, evaluation and measurement of product usage, and iterated design [26].
They can be decomposed further, but for the sake of this report the key takeaway is that UCD is an ongoing process with respect to users. Different methodologies are applied by companies working with software development.
Whether it is Scrum [17], Kanban [12], RABBIT [26], U- Scrum [22], or any other agile PDP the iterative design process is to a great extent similar. All of them advocates testing with users. But the testing method and execution time of which differs. A plethora of testing standards can be applied to accomplish the last stage of each iteration.
However, not all are suitable for every occasion and some of the agile frameworks have not successfully integrated HCD into the process [25]. Various parameters must be considered in order to execute the most appropriate form of testing.
Usability testing methods
When collecting input from (representative) users there is a great number of methodologies to be used, to name a few:
A/B-testing [21], guerilla testing [16], usability testing [14], and various statistical tools to map user behavior. Each of these methods is acknowledged for its strengths and how they suit different occasions, all will not be covered since this work revolves around the process of usability testing.
Usability assessment, as defined by Murillo and colleagues [14], is another way to distinguish usability testing from other forms of testing. The assessment consists of two types
of methods: inspection methods and test methods. The executor of the assessment decides which of the methods is used. Inspection methods are performed by inspectors, most often experts in the subject. Test methods, on the other hand, engage representative users of the targeted customer segment in a controlled environment, where usability testing is a subcategory to these test methods. Depending on which form of usability testing the observation differs, both in terms of technology as well as the applied strategy of the observers. Online or remote, usability testing utilizes the opportunity of recording screen, video, and/or audio while the user performs a test [1]. Face-to-face usability testing has the same possibilities as well as the advantage of interviewing the participant. A think-aloud method can complement both types of usability testing [19].
Benefits of usability testing
What makes the testing phase so important is the opportunity to discover and map out errors and problems perceived by the test participants [3]. Their input results in insights which can pivot the following iteration, advocating improvement. All errors cannot be acted upon, rather it mainly depends on the magnitude and worth of fixing the error, as Cockton argues in his research on Worth Centered Design [2]. Nonetheless, usability testing provides procedures to unveil problems that could harm products.
Usability testing is advised to execute in different stages with different focus during the PDP [27] to ensure a high- quality product. Most of the studied companies in [5]
perform usability testing in the stage before launch, to make sure there are no vital problems. This approach may seem as cost-efficient since the company only carries out one test.
However, since changes late in the PDP are costly, the strategy is flawed. Last-minute fixes are also not a guarantee that the product did in fact improve. What has been suggested by the Nielsen Norman Group in their paper [16], is how light-versions of usability testing can be applied and generate important feedback with time- and cost-efficient orientation. The main argument is that usability testing can provide valuable insights without it being conducted as a full-scale procedure.
Scrum
The agile working method, mentioned earlier, is furthermore implemented at Tcom utilizing the Scrum methodology [6]. This framework is currently popular in the software development world, where it has its advantages of establishing a flexible and time-efficient development process. The latter aspects of Scrum have however not been fulfilled by a set of companies studied in [5]. The problematics occur when the testing phase is executed in the next coming sprint, and not in the current one. This can result in insights from the test becoming expired, and thus, are not relevant. The little attention to usability in Scrum methodology is not surprising due to its software development foci. The organization of Scrum defines three roles: Product Owner (PO), Scrum team, and Scrum master [4]. The PO is responsible for the product
backlog and deliverables. The development team can include workers from many disciplines, they produce deliverables through working in sprints. The Scrum master makes sure that the development team can complete its deliverables. This is a brief general view on Scrum organization.
U-Scrum
Traditional Scrum fails to adequately address the usability needs of the user [23]. Mona Singh applies a usability perspective on Scrum, where she argues how Scrum lacks the focus and resources to develop usable software products. U-Scrum is suggested to enhance usability as it applies such expertise on a high level in the Scrum team organization through having two POs. The traditional, single PO is complemented with another PO focused on usability and User Experience (UX). This extension of Scrum has not been adequately studied in current research, but its takeaways can yield value for future agile development teams.
METHOD
Mixed Methods Research (MMR) approach has evolved among researchers in recent years [9]. This way of combining multiple forms of data in one research project provides a deep understanding of context, experience, and complexity [7]. A qualitatively driven approach was applied for this report in order to cope with the complex nature of usability testing at Tcom. Where qualitative data helped shape the online survey as well as complementing the quantitative data for a deeper understanding. This research method enables comprehensive data collection of both qualitative and quantitative characteristics [8]. Whereas using only one method would have either supported a detailed understanding of a rather niched aspect, or a wide perception on a general level [8]. Therefore, this method illustrates how accountability, budget, and opinions influence the company culture in terms of in-house usability testing. The empirical data was furthermore used to detect any deviation of an objective reality. Each method has its strengths and weaknesses, which is why results of the survey and the interviews were put in relation to one another, i.e. two methods validated each other’s truthfulness. To deal with systematic errors during face-to- face interviews [22] for example, the result of the interviews is compared with results from other methods to ensure its veracity. The three following methods were utilized: online survey, observation, and interview. The headquarter was the appointed office for this study since it runs Tcom’s digital products.
Survey
An online survey allows for fast distribution to a wide audience, which made it the best possible tool for the quantitative data collection since the headquarters of Tcom holds 2200 employees and consultants ranging across multiple departments distributed throughout an 8-story building. The survey had a two-sided purpose in this data collection: provide insight into employees’ attitudes
towards usability testing and ensuring interviews are in line with the objective reality. Attitudes serve as a fundamental argument for changing the current usability testing process since employees are the ones utilizing processes and tools in the work their opinion matter, which is similar to the fundamentals of UCD.
However, the questionnaire designed for this thesis had to take a wide set of respondents into account which affected the technical terms and structure used. Survey participants’
answers will constantly be influenced by the phrasing of questions and associated answer alternatives [22]. This influence is possible to avoid up to a certain extent by carefully structuring the order of questions. Where the most important questions are advised to ask in the middle of the survey, since respondents need priming before encountering more cognitive demanding questions. While having important questions late in the structure are likely to be influenced by respondent mood, such as tiredness. Unlike an interview, which can be slightly modified toward the respondent in question [18], a survey must be the same for every response.
Abstract concepts and detailed niche questions have a tendency of confusing the respondent making her unable to comprehend what is asked for [18]. This, in turn, results in answers that do not reflect the question and increasing the measurement error. Non-response is also a consequence due to complex questionnaires. That is why an easily grasped design and user-friendly approach was applied. Complex subjects should be addressed by decomposing questions into multiple elements, this has been explored in the MultiTrait-MultiMethod (MTMM) [22]. This methodology was appointed in the survey used for this report. Concept- by-postulation is defined, by Willem E. Saris and Irmtraud N. Gallhoferas, as the complex subject being investigated [22]. This is achieved through exploring concept-by- intuition, which is the subject in question decomposed to a number of sub-subjects. These sub-subjects are additionally converted into multiple measurable questions, defined as request-for-answer. Request-for-answer will furthermore be the terminology used throughout this report.
The survey generated both qualitative and quantitative data, with more focus on the quantitative part. It included 31 requests-for-an-answer with the following structure: 3 straight open-ended requests, 4 multiple choice closed- ended requests with an open-ended ‘other’ option, 12 single choice closed-ended requests with an open-ended ‘other’
option, 2 multiple choice closed-ended requests, 10 multiple choice closed-ended requests. Both nominal and ordinal [22] response scales were utilized. Where 4- and 6- point symmetrical ordinal scales were applied when requesting ratings. Saris et. al. suggests odd-number scales to provide a neutral option for the respondents. However, this was not implemented because of a will to encourage respondents to take a stand and thus activating the participants’ system 2 [11]. The nominal scales allowed
respondents to provide neutral answers, which were used in the rest of the closed-ended requests.
The survey consisted of two separate flows: usability testing and user testing. The requests were distributed over 6 sections, in the following order: a) respondent background, b) experience with user/usability testing, c) benefits of user/usability testing, d) user/usability testing and Tcom, e) user/usability testing statements, f) Tcom in- house user/usability testing. Section b) separated respondents by knowledge to assign them into their appropriate flow. Respondents familiar with usability testing faced the term usability and were assigned to group 1, whereas the rest met the term user and were assigned to group 2 or group 3. Group 2 had knowledge of user testing but not of usability testing, and group 3 had knowledge of neither. The data from the respective groups will be analyzed and interpreted with respect to their size and knowledge, i.e. group 1 is perceived as more significant than group 3.
Together the 6 concepts-by-intuition generated company attitudes towards testing, i.e. concept-by-postulation. The online survey was distributed in two channels: Tcom intranet and 90 posters put around the headquarter. The posters were integrated with a link and a QR-code. The intranet article generated 242 views whereas the Bitly-link via the posters had 17 clicks. The posters around the headquarters were not used for the response rate since one could not identify how many of the roughly 2200 people in the building noticed it. The QR-code had no tracker and thus could not be used in calculations. Together the intranet article and the Bitly-link served as the denominator for the response rate of 29,3%. Although this response rate is viewed as representative for the audience that accessed it.
An alternative response rate with responses / total number of employees yields a less representative rate of 3,1%. Since the survey targets opinions, the results can serve as a guideline for Tcom employee’s attitudes, but not as decisive.
Active observation of usability tests
The two observations enabled an understanding of how usability tests ordered by Tcom functions and how Tcom could execute tests themselves. The two usability tests were observed with a focus on the process. The following aspects were studied: agenda, test environment, staff behavior, participants, post-test effort, time, and expenses. Agenda included features such as welcoming participants, transcript, and test scenarios. Technology, furnishing, and room layout were studied to understand which facilities were available. The staff’s behavior was studied to understand how the personnel acted throughout each test.
Whereas the participant gathering process revolved around how attendants were chosen. The post-test effort category focused on the efforts put in after the usability tests had been conducted, where report compilation was of major interest. Expenses for parts of the process and the whole
process formed the economical perspective. A time perspective was used in three ways: time per test, time to execute a usability test, and time until the report is finalized.
Unstructured interviews
Four unstructured interviews were conducted at the headquarters with managers from different departments to map the organization and the Tcom PDP. The first of which was held with Product in order to understand how the PDP works in practice, from idea to launch. Secondly, an interview was conducted with Human Resources with the aim of comprehending how the departments are distributed and function alongside one another. The two described interviews were conducted before the survey since that qualitative data was a prerequisite for section a) and d) in the survey. The third interview was held with an employee from a department responsible for ordering usability tests among other responsibilities. This interview provided an understanding of how the testing process functions. Lastly, the head of UX was interviewed to interpret how the testing process functions. Additional interviews were not conducted due to adequate data from the four interviews as well as the restricted timeframe.
Except for having a main topic, unstructured interviews do not follow a certain pattern [28]. Where the purpose is to get as much qualitative data as possible through a conversation with a user or stakeholder. Despite the number of flaws with an unstructured interview [28] they serve as an honest and investigative tool that results in an understanding of opinions and experiences. A semi- structured interview format would have resulted in more structured in-depth data [15]. However, since it requires a lot more effort to analyze the long transcripts it was not assessed to fit the timeframe of the project.
RESULTS & ANALYSIS
Data gathered with the three methods are organized throughout the following section. Where it initiates with presenting the survey respondent distribution followed by the current usability testing process at Tcom. The latter is complemented with relevant findings from the Usbil observation. Whereas the Tcom in-house test presents how a usability test could be executed. The section ends with opinions towards usability testing and preferences regarding a desired testing process.
Respondents
76 Tcom employees responded to the survey out of which 69 were located at the office of interest. The product development is performed at this office and that is why the 7 employees from other offices were excluded. 47, or 68,1% of the respondents had previous experience from usability testing and were assigned to group 1. Group 2 made up 13,0% of the respondents who had experience of user testing but not of usability testing. Group 3 (18,8%) had no experience with any testing methodology. The most represented departments: IT Development (14%), IT User
Experience (14%), Network (13%), Product (12%), and IT System Management (9%). Group and department distribution are visualized in figure 1. Abbreviations used for the departments are listed in table 1.
Figure 1. Department distribution of respondents.
Department Abbreviation
Information Technology IT
IT - Development IT D
IT - QA IT QA
IT - System Management IT SM
IT - Architecture IT A
IT - UX IT UX
Customer Service CS
Quality and Processes for CS CS Q
Customer Relationship Management CRM
Sales Sa
Product Pr
Communication Comm
Business Intelligence BI
Insight In
Commercial Cml
Marketing Ma
Finance Fi
Legal/Security L/S
Business Transformation and Integration BT&I
Human Resources HR
Internet of Things IoT
Technology Te
Network Ne
Customer Implementation CI
Project Management PM
Table 1. Abbreviations of Tcom departments.
Current process Interview
The Tcom department structure is complex since it utilizes the agile framework Scrum in parallel with a traditional
structure. The UX department is a good example of a
‘floating’ department since it functions dynamically in the sense that its resources are applied in teams when necessary. It is difficult to categorize all 2200 employee’s positions in the building because not all departments work agile. It also depends on how one defines the departments.
Which is further complicated by a recent company reorganization. Scrum explains how the development teams are organized but not the more traditional departments.
Although there is not one coherent way to successfully divide all departments, the attempt used showed positive results in the analysis of the quantitative data.
Currently, Tcom orders between two and three usability tests each year from Usbil. The order-procedure is hugely dependent on which employee(s) are involved in the project. Hence, there is no established process for how to request a usability test. The agile transformation which Tcom initiated in early 2018 has been carried out in less than one and a half years. The transformation has been distressed by a reorganization in the company. The agile environment is not completely fulfilled, which is highlighted by gaps in the PDP. How to order a usability test is one process that has been affected by this. The process is described as ‘unclear’ and serves a lot of competition from other processes in terms of budget and attention. The result of this flaw is that usability testing occurs rarely and late in the PDP.
When Tcom orders a usability test from Usbil, which is their only partner in terms of testing, they regularly need two weeks of preparation until the test can be conducted.
However, their fastest order-to-test saw a usability test executed within a week from the date of the request. The post-test effort from Usbil consists of a report that usually is completed a week after the test. The total cost for these tests is between 60-80 KSEK, most commonly 80 KSEK. The expenses are bound to number of participants, participant characteristics, post-test effort inter alia.
The absence of in-house usability testing at Tcom is mainly due to budget restrictions according to the head of UX. The resources are not enough at the moment to conduct more usability testing. Tcom utilizes the following procedures of UCD: personas (epics in Scrum), user journeys, expert review, sporadic guerilla testing, and rare usability testing.
Guerilla testing and expert review are perceived as more time- and cost-efficient than of usability testing.
Survey
An average of 3,07 (out of 6) reveal that employees partly disagrees with Tcom performing testing properly (Fig. 2).
26% of group 1 and 22% of the total respondents disagrees (2) with the testing process at Tcom being performed properly. Only 26% agrees (4 to 6) to some extent the testing process is properly executed, whereas 74% rates the process on the disagreeing part of the scale (1 to 3).
Figure 2. Tcom performs testing properly.
25% of the total respondents believe Tcom performs their own testing. As mentioned above, this is not the case as current external partner Usbil conducts testing for Tcom.
16% got the correct answer “no”, whereas 43% had no opinion and 16% answered “I don't know”.
54% assess prototyping as the most important step in the PDP (Fig. 3). 10 respondents had the opinion that testing should be performed during each step in the PDP.
Figure 3. Which stage Tcom should test during the PDP.
Usbil observation
The usability test, ordered by Mploy and held by Usbil, was split into two days. Day one focused on the testing itself whereas the second day worked through the notes during a workshop. Usbil aimed for 5 tests having 7 participants scheduled. One participant per hour was the predefined testing rate, where each test was scheduled for the duration of 45 minutes. 15 minutes in between the tests was reserved for discussion within the crew regarding the participant, the results, and any other comments.
The test room in which the moderator and test subject were located in were equipped with microphones, video cameras, a 4*2-meter glass-mirror, and an iPhone 6 used to run the prototype. The smartphone-screen was live-streamed to a monitor in the room behind the mirror where the rest of the crew were located. The crew consisted of 4 Mploy employees, me, and an Usbil employee. The moderator was furthermore also employed by Usbil.
Each participant was welcomed, and a small chit-chat took place. The participants were encouraged to present themselves with name, age, work, and work role. Their general knowledge of the topic in question was discussed followed by specific Mploy questions. Each participant was instructed to perform 4 scenarios. 5 minutes were reserved at the end of the test for any further questions from the crew. The test ended with the participants receiving their compensation.
The crew had responsibility to catch everything of interest on post-it-notes. They were furthermore put under either of the four following categories: pain points, gain points, about the participant, and other. The post-it notes were re- categorized the next morning by the Usbil crew before the workshop officially started. Sub-categories filled with unique post-it notes were the major changes made.
The 3-hour workshop was structured as follows: refine problem definition, construct a parking lot (thoughts that might be paid attention), list quick-fixes, discuss actions (main part of the workshop), and prioritize the actions. The remaining effort of assembling a report is made by Usbil.
The average time for the report to arrive is 5 business days.
Mploy had stated certain requirements for the participants that were recruited by Usbil. The more preferences, the longer time Usbil need to acquire a minimum number of participants.
Total cost for the 2-day usability testing session was 80 KSEK. Divided on 5 participants, the cost ended up being 16 KSEK per participant.
Tcom in-house usability test observation
The first ever in-house usability test at Tcom was scheduled during a full day, where a maximum of 10 participants would attend, 9 showed up. Each test had a duration of 30 minutes with an additional 15 minutes reserved in between tests. When recruiting participants, Tcom reasoned that they aimed for at least 6 tests in total and thus booking more than needed.
The testing took place at the company headquarters. A laptop with the desktop adapted prototype was located on a table, with a Tobii eye tracker mounted below the screen.
Video, audio, and the eye tracking movement were all live streamed to another room, where the crew was located to observe and analyze.
The moderator initiated the interview with a quick session of friendly conversation before presenting the agenda of their upcoming 30 minutes. Once the participant had agreed to be recorded, the moderator lead the participant into the subject by asking a few test-oriented questions about the participants’ situation. The moderator put emphasis on the fact that the participant is not being tested, it is the prototype. The test consisted of three scenarios.
The crew in the observer-room thoroughly studied the eye- movements, comments, and body language of the
participants on a 42-inch screen. The crew consisted of a floating number of employees, at least three employees were present for each of the tests. Notes were generated based on anything which the crew felt appropriate. Which was shared and discussed in the room after each test, with the moderator present. Thoughts, tips, and manuscript comments were continuously shared to iterate and improve the test.
Once the 9 tests were completed, the following procedure took place: combine and summarize notes, draw conclusions, create a report. Dropbox Paper was utilized throughout this process, where we were 4 persons in total that contributed to the post-test effort. The report was compiled after three business days.
Tcom appointed Rcru to recruit participants for the usability test. Specific requirements can be accomplished by the recruiting company. In this case, Tcom desired participants based on two requirements. The more requirements, the longer time Rcru need.
Total expenses for the Tcom usability test was 15,5 KSEK.
The expenses include recruitment costs and compensation.
Cost per participant ends in 1’722 SEK.
Tcom employees’ opinions towards testing
98,6% of the respondents believe testing is beneficial/partly beneficial on a scale from 1 (not beneficial) to 4 (beneficial). Where 65,2% rated it as beneficial. The three groups agree to a large extent about testing being beneficial, where the exception is group 3 in which 46% consider testing to be partly beneficial and 46% beneficial.
“Doing the right things” was a common response in the open-ended feedback. Additional comments: “finding errors”, “mitigate risk”, “to see if you are on track”,
“understand how the product is used”, “gives insights that are impossible to predict”.
The primary benefit of testing stated by group 1 and group 2 is to study customer behavior, whereas group 3 ranks error discovery as the primary asset of testing. Validating assumptions is considered as the second most beneficial attribute from the respondents familiar with usability testing (group 1). This is however not shared by the other two groups, where study customer behavior and error discovery are those group’s top choices.
88,4% believes testing contributes to the quality of products when ranked from 1 (nothing) to 6 (essential contribution), shown in figure 4. Group 3 is slightly skeptical towards rating the contribution higher than 4 (moderate contribution). In average, group 2 were most optimistic about testing contributing to product quality with an average score of 5,1. Followed by group 1 with an average of 4,9, whereas group 3 indicated more skepticism in relation to the other groups with an average of 4,5.
Figure 4. Testing contributes to product quality.
Request-for-answers about continuous testing during PDP resolved in an average score of 5,39. The scale in figure 5 was as follows: 1 (strongly disagree) to 6 (strongly agree).
Where 59% of the respondents strongly agreed continuous testing should be part of the PDP. 5 (agree) and 4 (partly agree) were the second and third most common answers with 25% and 12% respectively. Group 3 returned the most pessimistic result with an average score of 4,69.
Figure 5. Testing should be continuous during PDP.
Two-thirds of the respondents strongly agreed with testing helps increasing product quality. An average of 5,51 revealed almost no negative opinions towards the relation of testing and product quality. Group 1 indicated the highest average of 5,62, whereas group 3 showed a more discouraging relation with an average of 5,08.
The importance of testing saw respondents across the three groups agree, indicated by an average of 5,41. 62% strongly agreed (6), 20% agreed (5), and 13% partly agreed (4) with testing being important.
Tcom employees testing process preferences
There is nothing stopping Tcom from introducing a test-day every other week. Where a strong will amongst the employees is to move usability testing in-house. There are however a few unresolved aspects that obstruct in-house testing: ownership, budget, time, participant gathering, compensation, and administration. Many of which are admittedly plausible to resolve, where the most significant facet, ownership, is where the many discussions have come to a halt. Expertise is not considered as an issue since UX and Insight both obtain the skills needed.
There are aspirations of having an in-house testing process at Tcom where an employee responsible for the whole testing process is suggested. Consistency is a concern if an in-house process would be realized.
Tcom was rated as the number one company to perform testing for its products followed by Tcom’s current external partner Usbil. Which received nearly half the votes in relation to Tcom. Other external partner showed a similar result as Usbil. 62% of the respondents belonging to group 3 opted no opinion, which is higher than for the other two groups.
One day is considered as 8 hours. The average testing time per week lies somewhere between ½ day and 1 day (Fig. 6).
4 respondents stated that the testing time per week
“depends on the product” and 46% had no opinion in the subject. Where no opinion distribution was as follows: 36%
of group 1, 78% of group 2, and 62% of group 3. A majority stated that the time needed depends on the complexity and the magnitude of the product in question.
Some of the respondents indicated that the time needed should make up 20-25% in relation to the total time spent on the product. “More tests - better products” was also mentioned.
Figure 6. Average testing time per week.
61% of the respondents believe Tcom should perform their own testing. Whereas 16% are against that idea and 23%
had no opinion. 5 answers were excluded due to inconclusive format “Both”. 70% of the most knowledgeable respondents (group 1) are of the opinion that Tcom should perform testing themselves. Respondents who answered yes (39) could answer the two follow-up questions regarding who should perform and who should be responsible for in-house testing at Tcom.
UX (31%) and Product (13%) are thought to be the two departments that should be conducting in-house testing.
10% of the respondents had no opinion in the matter.
Figure 7. Which Tcom department to conduct in-house testing.
36% of Tcom employees believe Product should have the responsibility of in-house testing. Insight (15%) and UX (13%) are the two following options stated by the employees.
Figure 8. Which Tcom department to be responsible for in- house testing.
The primary benefit of in-house testing is not mutual between the three groups. Opportunity to refine the process was ranked as group 1’s first choice of benefit with 30%.
Whereas group 2 assessed more frequent testing and control in the process as the two most important benefits with 33%
of the votes respectively. Group 3 considered cost- efficiency as the greatest impact of in-house testing. 38% of group 3 had no opinion in the subject. Three benefits stand out when the primary benefit was requested: opportunity to refine the process (1), control in the process (2), and cost- efficiency (3) (Fig. 9).
Figure 9. Top 5 benefits with in-house testing at Tcom.
All groups agreed bias may influence the results is the main drawback of in-house testing (42%) if we ignore the 46% of group 3 which had no opinion. “Demands too much time”
is the second rated primary drawback with a total of 28%.
Only a few (6%) assessed no drawbacks of in-house testing.
Figure 10. Top 5 drawbacks with in-house testing at Tcom.
DISCUSSION
The discussion revolves around the findings in result &
analysis and is structured in a similar way. It initiates with the current testing process, followed by Tcom employees’
opinions towards usability testing. Then follows a comparison between the two observed usability tests. The section ends with employees’ preferences on the process of usability testing.
Current testing process
Ownership, budget, time, and administration are the main factors restricting an in-house usability testing process.
Although most of the employees believe an in-house process is a more efficient solution, the process falls in between departments as it is structured now. Insight lacks the resources to be responsible and to carry out the testing.
Although UX employees have the mandate to request usability tests, the current go-to for executing a usability test demands a lot from the requester as they are required to lead the whole process which results in few tests.
A clear majority of the respondents (74%) believes Tcom’s testing is inadequately performed. The few numbers of annual usability tests, as well as a vague order-process, have raised these opinions around the agile departments.
The prototyping stage is considered as the most vital stage to test. When testing occurs at Tcom, it is during this stage.
It can be dangerous to stall tests until the prototyping stage since decisions on the fundamental aspects of the project have been made without solid validation. Hence, expensive adjustments may have to be carried out in order to make sure products are of the desired quality. Scrum methodology is implemented to avoid these incidents, but since Scrum mostly concerns the software development rather than usability these incidents can still occur.
Tcom employees’ opinions towards usability testing Respondents in all three groups consider testing as important, beneficial, and that it increases product quality.
Testing should be conducted between four and eight hours per week according to employees. The answers indicate differences in their perception of testing showing a pattern descending with experience of user- and usability testing, i.e. employees with much knowledge rates testing as more beneficial and vice versa. Respondents in group 3 consider a transparent process as the most beneficial aspect with in- house testing, which could originate from a lack of insight into Scrum teams progress.
Bias and expertise in an in-house process is a concern among respondents. The expertise exists as observed in the active observation of Tcom’s in-house usability test and stated during two of the interviews. Skewed results from in- house testing are however not believed to be a concern according to an Insight manager. Some of the respondents furthermore considered “the ones closest to the project” as best suited to conduct testing. Insight can assist with moderators for tests with no insight into the PDP and thus providing an objective approach.
Comparison between in-house and Usbil test
Tcom in-house tests show promising costs of 1/10 per participant in relation to Usbil. Although this ratio does not include expenses for Tcom employees time and efforts, the price tag is still favorable. The execution time is not advantageous compared to Usbil unless the process is changed. Where a standing test day every other week could be the solution if time-efficiency is to be enhanced in terms of call-to-test-time. There is another aspect of time- efficiency which revolves around the post-test effort and go-to steps based on insights during the tests. Usbil requires one week to compile a report, which is inefficient compared to Tcom’s three business days. The report is furthermore not as important as one might expect, the go-to steps are the reason for usability tests to be conducted in the first place.
Which can be extracted directly from the tests and avoiding a report, and thus reducing the time until changes in the product are applied. Usbil’s usability test procedure is however more organized than Tcom’s, as it follows a certain framework, which is likely to generate more on- point insights.
Tcom employees testing preferences
Employees have expressed their willingness to have a greater insight into the advancement of projects. Which can
be realized via inviting squad members to observe tests as well as through the possibility to stream in-house tests.
The benefits and gains that can be achieved when testing early in the process cannot be pursued with the current testing process due to scarce recognition of usability. An in- house testing process is believed to proclaim more testing during different stages in the PDP. Expenses can furthermore be reduced if the project validates its advancement from the get-go. Although a range of limitations are holding in-house usability testing back, employees have expressed its potential of improving the PDP. Continuous usability testing has been requested for a few years but has served resistance from the current framework Scrum.
Nature of usability testing
The usability testing methodology studied in this work is one of the many formats’ customer input can be obtained.
Various methodologies and technologies can be applied for the sake of validating and studying prototypes and products.
The in-house testing process shows the most prominent characteristics of the two, but there are further alternatives who can bear an equal or perhaps better process. A compromise between an in-house and external process is one example, where the test is conducted through a near collaboration with external and internal experts at an off- site lab. Another procedure could be how a company utilizes external experts in an in-house process. The main argument here is that there are several ways in which usability testing can be organized and that this work is just one of the many ways’ usability testing can be conducted.
Like the data collection method used in this work (MMR), companies do not solely rely on one product validation tool.
Usability tests are usually complemented with quantitative statistics such as heat maps or drop-out rates. They can be further complemented with A/B-testing, where two versions of a product/prototype are released and compared against each other. Customer feedback during the customer journey is a voluntary instrument which can help companies in understanding their products. Multiple tools will strengthen the empirical data and thus increasing the accuracy of the analysis.
U-Scrum
The usability focus of U-Scrum is thought to provide fundamental aspects in the product lifecycle, foremost during the early stages in order to understand the users and their needs. Once the business case is mapped the need for two POs could thus decrease and the organization is back to having solely one PO. One PO would resolve in distinct leadership whereas two of them could perhaps trigger a stalemate situation where focus would turn to who is the
“true” leader instead of the methodology’s main purpose.
Such a situation is likely to harm the PDP. In contrast, two POs have the opportunity to focus on different aspects throughout the process resulting in efficient management.
The main concern against two POs is when they both are
involved in the same aspect with different opinions, who will obey? This question is left unanswered in today's research and will need further attention to be dissolved.
CONCLUSION
Tcom initiated their implementation of Scrum in early 2018, where the agile working method is applied but has an acknowledged flaw in terms of usability attention.
Employees perceive usability testing as scarcely performed.
Budget motives and ownership restrains testing due to other processes being prioritized. As a result, usability testing occurs twice a year on average. Tcom has little control in the current testing process since Usbil takes care of the process once the order is made, which is expressed as opaque by some employees in the Scrum teams.
- Can the PDP be improved by moving usability testing in-house?
In-house usability testing could conduce to more testing since it is far cheaper. The expense per participant from the in-house pilot test was roughly 1,7 KSEK, that is a close to 1/10 of Usbil’s cost per participant. Tcom will also have total control in the process which enables iterative improvement of the process itself. Where time-efficiency shows great potential, both call-to-test-time and time spent on post-test effort can serve enhancement. The ability to provide a more transparent process is another advantage with usability testing conducted in-house. Although bias and expertise were stated as concerns of in-house testing, Tcom currently holds the expertise to manage those concerns. As a result, improving the usability testing process will give greater insight into products and thus advocating improvement of the PDP.
Another finding during this study was how the agile framework Scrum deals with the concept of UCD and usability. Personas, user journeys, and expert reviews are utilized in Tcom’s PDP. Conclusions from those methods are not validated through testing with users, where one must question if Scrum is the most appropriate framework from a usability perspective.
FUTURE WORK
In order to introduce an in-house usability testing at Tcom there must be an executive decision made regarding which department(s) should be involved in the process as well as setting aside a budget. Exploring U-Scrum methodology is possibly one solution for this issue. Once the responsibility is resolved Tcom can initiate structuring the process and investigating which partners to work with. The integration of a usability testing process will presumably see a trial period before it is fully implemented.
ACKNOWLEDGMENTS
I would like to thank Tcom for taking me in and giving me the opportunity to be a part of their team and allowing me to explore their PDP. It has been a pleasure to have worked side-by-side with such an enthusiastic and welcoming set of
individuals. A special thanks go to my supervisor and my stand-in-supervisor at Tcom.
REFERENCES
1. Albert, B., Tullis, T., & Tedesco, D. 2010. Beyond the usability lab : conducting large-scale user experience studies. Beyond the Usability Lab.
2. Cockton, G. 2006. Designing worth is worth designing.
In Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles (pp. 165- 174). ACM.
3. Davis, K., & Horst, D. 2012. The note-taker’s perspective during usability testing: A hands-on approach to recognizing what’s important, what isn’t.
In Proceedings of the Human Factors and Ergonomics Society. pp. 2412–2416.
4. Düchting M., Zimmermann D., Nebe K., 2007.
Incorporating User Centered Requirement Engineering into Agile Software Development. In: Jacko J.A. (eds) Human-Computer Interaction. Interaction Design and Usability. HCI 2007. Lecture Notes in Computer Science, vol 4550. Springer, Berlin, Heidelberg.
5. Eloranta, V.-P. et al. 2013. Scrum Anti-Patterns -- An Empirical Study. 2013 20th Asia-Pacific Software Engineering Conference (APSEC), 1, pp.503–510.
6. Gonçalves, L. 2018. Scrum. Controlling &
Management Review, 62(4), pp.40–42.
7. Halcomb, E. & Hickman, L. 2015. Mixed methods research. Nursing standard (Royal College of Nursing (Great Britain): 1987), 29(32), pp.41–47.
8. Hall, Jori N. & Ryan, Katherine E. 2011. Educational Accountability: A Qualitatively Driven Mixed- Methods Approach. Qualitative Inquiry, 17(1), pp.105–
115.
9. Hunter, B. 2009. Advances in Mixed Methods Research: Theories and Applications. Drug and Alcohol Review, 28(5), p.575.
10. Issa T., Isaias P. 2015. Usability and Human Computer Interaction (HCI). In: Sustainable Design. Springer, London.
11. Kahneman, D. 2011. Thinking, fast and slow 1. ed., New York: London: Farrar, Straus and Giroux; Allen Lane.
12. Leopold, K., & Kaltenecker, S. 2015. Kanban Change Leadership: Creating a Culture of Continuous Improvement. Somerset: Wiley.
13. McLeod, S. A. Qualitative vs. quantitative research.
2017. Retrieved May 21, 2019 from https://www.simplypsychology.org/qualitative-
quantitative.html
14. Murillo B., Vargas S., Moquillaza A., Fernández L., Paz F. 2017. Usability Testing as a Complement of Heuristic Evaluation: A Case Study. In: Marcus A.,
Wang W. (eds) Design, User Experience, and Usability: Theory, Methodology, and Management.
DUXU 2017. Lecture Notes in Computer Science, vol 10288. Springer, Cham.
15. Newcomer, K.E. 2015. Conducting Semi‐Structured Interviews. In Handbook of Practical Program Evaluation. Hoboken, NJ, USA: John Wiley & Sons, Inc., pp. 492–505.
16. Nielsen, J. 1994. Guerrilla HCI: using discount usability engineering to penetrate the intimidation barrier. In Cost-justifying usability, Randolph G. Bias and Deborah J. Mayhew (Eds.). Academic Press, Inc., Orlando, FL, USA 245-272.
17. Opelt, A. 2013. Agile contracts creating and managing successful projects with Scrum, Hoboken, N.J.: Wiley.
18. Ornstein, M. 2013. A Companion to Survey Research, London: SAGE Publications Ltd.
19. Paz, F. & Pow-Sang, J.A. 2015. Usability Evaluation Methods for Software Development: A Systematic Mapping Review. 2015 8th International Conference on Advanced Software Engineering & Its Applications (ASEA), pp.1–4.
20. Quiñonesa, D., Rusua, D. & Rusub, V. 2018. A methodology to develop usability/user experience heuristics.
21. Rubin, J., Chisnell, D., & Dawsonera. 2008. Handbook of usability testing how to plan, design, and conduct effective tests (2nd ed.). Indianapolis, IN: Wiley Pub.
22. Saris, W.E. & Gallhofer, I.N. 2014. Design, Evaluation, and Analysis of Questionnaires for Survey Research, 2nd Edition, John Wiley & Sons.
23. Singh, M. 2008. U-SCRUM: An agile methodology for promoting usability. In Agile 2008 Conference (pp.
555-560). IEEE.
24. Siroker, D., Koomen, P., & Harshman, C. 2013. A/B testing: The most powerful way to turn clicks into customers.
25. Sohaib, O., & Khan, K. 2010. Integrating usability engineering and agile software development: A literature review. In 2010 international conference on Computer design and applications (Vol. 2, pp. V2-32).
IEEE.
26. Still, B., & Crane, K. 2017. Fundamentals of User‐
Centered Design: A Practical Approach. Boca Raton, FL: CRC Press.
27. Sy, D. 2007. Adapting usability investigations for agile user-centered design. Journal of usability Studies, 2(3), 112-132.
28. Wilson, C. 2014. Interview techniques for UX practitioners: A user-centered design method.
www.kth.se
TRITA-EECS-EX-2019:561
