How do UX Professionals Apply UX Methods and Practice Lifelong Learning?
Johannes L. Geiser Uppsala University
Uppsala, Sweden
johannes.geiser.3194@student.uu.se
ABSTRACT
Due to fast-paced technological disruptions and diversifying users, user experience (UX) professionals are experiencing a flood of new UX methods and a need for continuous learning. Literature has shown that with a lack of understanding, UX practice research has designed too abstract UX methods making them hard to understand and to apply. With a thematic analysis of an interview with 13 UX professionals, this study presents results on how UX professionals choose UX methods and insights into their lifelong learning. The results from the thematic analysis agree that UX methods are hard to integrate into Agile, too complicated, take too much time to learn, and colleagues have shown to be an essential component for learning. These findings indicate that UX methods might work better if they are designed less complicated and deliver results quicker following the design of Scrum. Also, companies could use novel ideas to ease the access to users and to learn, e.g., lunch lectures.
CSS CONCEPTS
• Social and professional topics~Management of computing and information systems • Social and professional topics~Computing and business • Human-centered computing~HCI theory, concepts and models
Keywords
User Experience; Agile development; UX practice; UX methods; Lifelong Learning.
INTRODUCTION
Over the last years, professionals focusing on User Experience (UX) in their work have had various challenges [29,37]. The introduction of new technology in short periods and the constant change in user needs and demands have brought in new complications [29]. In response, researchers have introduced many new UX methods. These have caused a lifelong learning journey for UX professionals [29,46]. The main challenges for the UX professionals are that they need to acquire new knowledge about changes in technology,
adapt to changes in social structures, learn or adapt to new methods, and continuously recognize and analyze current trends [21]. What started as usability testing of one user in front of a computer has developed into a diffuse problem, bringing challenges to UX professionals to find out “what to study, what to design for, and which methods to use” [37].
These diffuse problems are accompanied by barriers such as lack of UX awareness in the company, organizational inconveniences, and limiting time constraints [6,29,34]. In addition, there is a lack of knowledge of UX practice in research [17,18].
This study uses the definition of UX by ISO 9241-210: “a person's perceptions … from the use of a product, system or service” [47]. UX methods are in this context defined as methods used to conduct user research to improve the UX for products and services [37]. Professionals working with UX methods can have various titles (e.g., UX Designer, UI Developer, Usability Designer), but in this context, they are referred to as UX professionals [8]. Lifelong learning is defined as the continuous learning of skills, acquiring knowledge, a need for growth, and applying knowledge creatively in different environments [21,33]. Traditionally, learning is performed formally (e.g., training, lectures) or informally (e.g., self-study, watching videos) [20,23].
For three decades, the most favored software development strategy has been Agile [43]. The most popular type of Agile strategy is Scrum, which is organized in Sprints (often 1-2 weeks) [2]. Agile development processes gained momentum due to their open and flexible approaches towards requirements from customers and users, in contrast to traditional, more linear processes such as the Waterfall [3,11]. Traditionally, flexibility and speed, the core components of Agile, make the processes perfectly suitable for changing conditions (e.g., technology disruptions, societal changes) [48].
Few researchers have explored UX practice and the use of methods in Agile [8]. Research has shown that current solutions (e.g., Sprint-0) for integrating UX methods into Agile are still unsatisfactory for the UX professionals [8,30].
Even the Agile Manifesto itself does not include UX professionals [48]. Furthermore, most research has focused on lifelong learning for other professions, but there is a lack of research on UX professionals. Notably, studies are lacking
This work was submitted in partial fulfillment for the Master of Science degree in Human-Computer Interaction at Uppsala University, Sweden. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.
Copyrights for components of this work owned by others than the author(s) must be honored.
© 2020-06-01 Copyright is held by the author(s).
how UX professionals learn about methods and how Agile affects it.
The neglect of the understanding of UX practice in research motivates this study. The aim in this study is to investigate how UX professionals apply UX methods. Mainly, what the barriers and enablers are for using UX methods. Concerning lifelong learning, this study aims to widen the current knowledge. The gained knowledge from this study could support researchers in developing UX methods that are more suitable for UX professionals.
Therefore, an interview study was conducted to answer the following two research question:
1. What aspects do UX professionals consider when choosing UX methods in agile development?
a. What methods do they use today?
b. What barriers and enablers affect their choice?
2. How do UX professionals working in agile practice lifelong learning?
a. Where do they find information on the UX methods?
b. How do they learn about UX methods?
BACKGROUND
UX Research and Practice
Researchers have introduced many new UX methods and studied how effective these methods are for the UX professionals [36,37,41,42]. These studies show that UX professionals experience these new UX methods as too abstract, making them hard to understand and to apply, which leads to distrusting UX research on new methods. UX methods are impractical and tough to use and sometimes not used at all because the methods do not fit well into UX practice [6,37]. UX methods often require much time to learn, which the UX professionals do not have [36,37].
Furthermore, David Roedl and Erik Stolterman [36] have analyzed that UX methods from research tend to generalize situations making it impractical to use in UX practice.
Despite the efforts made in trying to discover the needs of UX professionals, there is still information missing on how they apply and learn to use UX methods [8,24]. There is a gap between academic research and design practice [18].
Challenges Integrating UX in Agile
Agile software development has been prevalently used in companies. This has happened often without integrating UX methods well enough in the processes [9]. Consequently, research has introduced solutions like “Sprint-0” to better integrate UX practice into Agile development [5,8]. A Sprint-0 period is typically a phase in which the team, including the UX professionals, develop the backlog of requirements described with a list of user stories that determines the goals of a project. However, it has also been
shown that UX professionals were rarely involved in developing the first backlog and were rarely a part of the initial stages of a project [30]. Further, when it comes to aspects of integrating UX methods in Agile, companies focused more on customers rather than users. In contrast, research has developed UX methods with a focus on users [8]. One reason for this contrast is that companies' access to users is limited, compared to researchers' access. It takes too long to involve the users and costs too much, or is impossible due to policies and restrictions [8]. Additionally, research has shown the UX professionals only have one Sprint to finish their work, which can cause much stress [5,30].
Barriers of UX Work
Previous research has identified four significant aspects that influence UX work and the application of UX methods negatively. First, organizational structures in the company affect UX work by decreasing productivity, satisfaction, and well-being [6]. Second, several studies have demonstrated that communication is a major issue in the companies and prevented UX professionals from carrying out their work [8,29,36]. These studies have shown that misunderstanding between UX professionals and other stakeholders in the projects resulted in stress, losing time, and misunderstanding of why UX is necessary. A recent review of the subject [28]
has found that raising awareness and fighting for UX acceptance is still a challenge in many companies. Third, the project design affects the work of UX professionals by deadlines controlling the work [6]. Fourth, even though many studies have declared the involvement of users as essential [22,34,44,45], involving a user has been a complicated process and is not always possible [6].
However, a study about user-involvement and UX [6]shows that involving the user does not guarantee improved user experience [6].
Lifelong Learning
Lifelong learning has become essential for every UX professional because of the never-ending change of technology and industry [23,39]. Literature about software engineers’ lifelong learning [16] shows that their knowledge has a half-life of 2.5 years, which means that most of their knowledge is outdated after 2.5 years. Another study about lifelong learning [39] has shown that the constant pressure of changing could make professionals uncomfortable and result in stress. It also shows that the only way for engineers is to
“constantly learn, constantly upgrade their skills, and constantly adapt to new situations” [39]. Two different articles [15,23] highlight that the company could incentivize its employees by offering formal learning opportunities (e.g., training, access to conferences), or building a thriving informal learning setting. Guest [23] predicted that this would contribute to professional and personal development alike. He also concluded that learning will be lifelong, mostly self-directed, and supported by mentors and colleagues. He also added that professionals will borrow and exchange
knowledge and solutions with other fields. In his view, it would be harder and irrelevant to distinguish between professional and personal development.
Formal and Informal Learning
In a study about what affects lifelong learning for scientists [20], the authors have found that the firm’s innovation was positively related to the learning of the employees. The same study shows that there is a trend that employees take less part in training. The most common way participants practice learning was learning by doing, meaning professionals learned by applying the method or the skill [20]. Recently, companies have integrated novel ideas such as Hackathons as part of learning according to research [14]. Novel ideas and approaches have also come with research, taking into account communities and social support [27].
Competence
In order to achieve lifelong learning, it is required by professionals to improve their competence [21]. According to the literature [16,25], UX professionals need to improve their knowledge about frameworks, UX methods, and tools.
Research also shows that design competence includes
“holistic competencies around generic capacities,” such as experiences, attitudes, and capabilities [26]. Another research study has extended the definition of design competence with non-technical skills or usage of methods and tools such as communication with people [1,19].
Self-Directed Learning
As part of lifelong learning, studies have shown that self- directed learning is a primary indicator of the company’s growth [16,23]. According to another study [46], Google is the primary tool being used to find learning sources. The study [46] has found four important aspects of self-directed learning: “where to look, what kind of queries to write, when to ask, and whom to ask.” Additionally, the research has shown that self-directed learning could cause stress, commonly when employees believe they have not learned enough to complete the task. The study also shows that learning often happens in the employees’ spare time, which can lead to burnout and stress [46].
Summary
Previous research has shown little results on barriers and enablers of UX methods in Agile. In addition, Agile development comes with benefits and drawbacks. Research has also shown that access to users is often limited, involving them takes too much time, or it is impossible due to policies and restrictions. Research on lifelong learning has shown that due to ever-changing conditions at work or technology, professionals are required to keep learning about new methods, new skills, and current trends consistently.
Research has not investigated the specific situation of UX professionals and Agile software development, especially focusing on the lifelong learning of UX methods like it is done in this study.
METHODOLOGY
This work used a qualitative approach to gather insightful data and ensure an in-depth understanding of UX practice [12,31]. The data was collected in an interview study and was part of a Master thesis over a period of four months in spring 2020. The interview questions were created deductively by using previous research. The thematic analysis was mixed, where some themes were deductively pre-defined from the interview questions, and inductive in that new themes were identified in the data [4,13].
Data Collection
This research used a purposive sampling strategy where the purpose was to recruit UX professionals working in Agile [4]. Besides, effort was made to vary the years of working experience, the companies, and the industries. To improve the quality of a thematic analysis and increase the likelihood of patterns, at least two participants belonged to the same company [4]. Due to time-constraints and COVID-19 restrictions, the research needed to find participants conveniently [4,8]. Therefore, the authors recruited UX professionals on their professional network or LinkedIn.
Most participants were known to the author by previous professional network events, and some were recommended by former colleagues or participants. This resulted in a convenient selection of participants from two countries:
Germany and Sweden. Last, an effort was made to pick participants from companies with different sizes and different industries and no other specific reason.
The method of choice was semi-structured interviews, with a fixed set of questions prepared from previous literature on lifelong learning and UX methods. The conversation was open, and the interviewer followed the conversation of the participants with corresponding questions. To reduce the efforts of travelling to different countries and fulfill the regulations of COVID-19, the interviews were conducted with consent remotely using a video call software as a recording and calling software. The average length of the interviews was 45 mins.
Participants Responsibilities
1,2,3,4 Software company creating digital products for businesses and consumers.
5,11 Design agency or consultants working for clients in various industries.
12,13 Responsible for the e-commerce platform of the company.
6,7,8 Design agency with smaller clients around the world.
9,10 Long-term partners with a company and responsible for UX of their product.
Table 1. Participants working in different types of companies.
With this strategy in mind, 13 participants were recruited, seven females and six males, with an average age of 35. The participants worked at four companies in Sweden and two companies in Germany. The six companies in which the participants worked in ranged from a small-size company with less than 50 employees to a global company with over 200,000 employees. In Table 1, an overview of which participant worked in which type of company. Most participants were part of a company in which they were responsible for their own software sold to clients. Two UX professionals worked as consultants. The participants had an average of six years of experience in UX, and they had worked across many industries (e.g., healthcare, IT, aerospace, automotive).
To fulfill the guidelines of GDPR [40], an effort was made to anonymize and store the data securely. Overall, there was no harm involved, no invasion of privacy, no deception, the participant was given a consent form, and they could withdraw at any time. Before the recording started, the participants were asked for their agreement to the consent form, which was presented visually and verbally over screen sharing in the video call software.
Data Analysis
After transcribing the interviews from the recording, a thematic analysis was conducted according to Braun and Clark [7]: reading the transcription for familiarization, creating codes, and lastly, categorizing the codes into themes. A spreadsheet and a whiteboard were used for the codes and the themes. Last, the themes were reviewed and used to answer the research questions.
Ethics and Limitations
An ethical problem was that the sample size within one company was small, which could make it possible for colleagues to identify each other [4]. To counter, an effort was made to not directly quote, and to make the participants aware of that before they gave consent.
Further, the sampling strategy is prone to researcher bias and a homogenous sample. The latest was countered by providing as detailed information as possible without violating privacy concerns to make the results more transferable [12]. Bias was countered in practicing an objective analysis following the process of Braun and Clark [7], and by following a semi-structured interview with little interference from the interviewer as possible.
Conducting remote interviews had the advantage of following the regulations of COVID-19. However, it made it harder to react to social clues such as body language and create a pleasant atmosphere [38].
RESULTS UX Methods
When asked about what types of UX methods they most frequently use, the participants mentioned (1) prototyping, (2) user testing, (3) user journeys, and (4) workshops.
(1) Prototyping: According to the participants, prototyping was used for different purposes and situations. All participants, except three, used prototypes from low fidelity to high-fidelity to confirm ideas and figure out if something is missing. The remaining three worked in a small agency, which required to constantly look at the price and time, so they mostly used low-fidelity prototypes for quick feedback and high-fidelity prototypes as the final product. When discussing the importance of this method, most participants indicated that using prototypes was valuable to them. It helps them evaluating if a project is steering in the right direction.
It also showed that prototyping comes in variations of fidelity, which can be used flexibly and making it easier to use in different time-constraints.
Besides, Participant 1 and 2, working in a large global company, expressed the importance that methods can be used both for professional and personal cause. They mentioned “Dark Horse Prototyping” and “Science-Fiction Prototyping” (SFP). Dark-Horse prototyping was about implementing “crazy ideas” no one thought about, which might have the highest impact, and Science-Fiction prototyping was described as utilizing futuristic technology and social structures. When asked about how they used these methods, Participant 2 explained that she used SFP to figure out how AI can be implemented in software and how users perceive it. She also used it to personally evaluate how to position herself ethically towards AI replacing work for users. The participant also indicated that testing “crazy ideas” brings more fun to do prototyping and opens one’s own to new perspectives.
(2) User Testing: All participants agreed that user testing is highly valuable. However, even though some indicated an increasing demand from the clients, they have to fight to get access to users. One participant said: “I think the demand for user testing is remarkably high. Clients start to understand the value of user testing”. As indicated by the data, she seemed to be surprised by the high demand and explains it with that clients understanding the value more and more.
This view was supported by data from participants working in smaller companies; they reported to struggle a lot to convince clients of the importance of UX and time-intensive methods such as user testing. Participants from larger firms had, on the other hand, organizational and compliance issues when it came to applying user testing. Overall, it seemed like the participants needed to validate their actions and protect their role in the development process regularly.
(3) User Journeys: This method was mostly used by participants in larger companies to build the requirements in the early stages of the project. Most participants used it to understand the process of the user and the product. However, one participant noted a problem; she stated that user journeys are often used less strictly and based on the professional’s
“gut feeling”. Her response indicated that she does not feel safe using the results to make decisions; she said the outcome feels “less justified.” Whether this is an inherent problem to the method or external problems influencing the correct use remains unclear. Despite this problem, the participant noted that using a data-driven version of journey mapping would make her feel more confident.
(4) Workshops: All participants used workshops in every phase of the development. Many participants used this method for creating design thinking sessions for innovating new ideas inside a company or understanding the customer’s needs. Participants who were interviewed after the COVID- 19 mass outbreak indicated problems about organizing workshops remotely. Participants from larger companies indicated that workshops are an essential method for the project development. The participants made clear that this is urgent. It showed that the companies need to resolve organizational barriers, and the researchers need to redesign workshops to work remotely.
Barriers to Using UX Methods
By far, the most common problem towards the choice of a method was time. Either a method costs too much time or there is not enough time to gather users. For two participants working in a smaller company, time was limited, resulting in the non-use of high-fidelity prototypes. To solve the involvement of users, one participant mentioned that the company is using user representatives instead of real users.
Another participant explained an interesting way for companies to decrease the time and effort to recruit users and to conduct user testing, where the company recruits participants that matches the Swedish population: “[…]
since I started, we had this process that every week we have a company recruiting users who are kind of matching our product […] matches the Swedish population. They are coming each Wednesday, and then each of the projects is free to sign up if there are free spots. The idea with this is that we just want to make the barrier for usability testing as low as possible.”
The second most mentioned barrier was the
“understanding” of UX. Participants who had issues with UX awareness among stakeholders around them mentioned the problems about making clients or stakeholders understand what UX means, how it works, and what it costs.
These participants explained that convincing stakeholders about the importance of UX was essential and necessary for them to use the UX methods and be part of projects. Most participants solved this by integrating stakeholders into the
feedback of the product. However, the same participants recognized that involving the client deeply into the process resulted in a less user-centered and more into a customer- centered approach.
The participants also shared some problems with teamwork;
the participants who had many restrictions concerning software mentioned problems of working in-sync (having a common understanding and goal) and communication. One participant emphasized that this is vital for the success of a product. He further explained that if changes on the product are not communicated well to developers, it could result in results in losing time and money.
Participants in larger companies reported they had issues resulting from company regulations and restrictions. This made it impossible or harder to use software for easing communication or using methods. The reasons behind this were complicated license regulations and privacy restrictions. One participant mentioned that this could make it impossible to use a type of a method (e.g., high-fidelity prototyping) in a project. It also made it harder for them to argue for accessing the user. This indicated that many organizational barriers prevent the use of a method. It also showed that the limited access to software and, therefore, methods make the participants not try out new things making them less innovative on new technology.
Organizational issues and the nature of methods also made it hard for some users to use methods such as workshops remotely. In the best case, a workshop would require having access to users face-to-face. However, as one participant indicated, COVID-19 regulations and trends before show a move towards remote. One participant explained: “[…]
remote working has always been a big challenge, especially in times of this virus. You are always limited in the way [how]
to conduct them, meaning remote working, asynchronous working, and since the tools and techniques are not designed for that […].” As indicated by the participant, workshops always come with challenges. She also pointed out that due to the restrictions that many methods cannot be used because of the given remote tools and organizational obstacles (e.g., compliance and company restrictions).
Enablers to Using Methods
One strongly expressed enabler by all participants was a well-functioning team. Most participants stated that excellent communication is essential for spreading UX awareness, which is important when arguing for certain methods to use. Some participants also indicated that collaboration makes it easier to work more efficiently, to feel safer in making decisions together, and to integrate stakeholders into the process. Responses from participants working in companies with easy access to new software indicated having fewer problems with communication and collaboration.
The participants with fewer years of experience noted that colleagues enable them to use methods. They expressed that they work as mentors, showing them which and how to use a method when they are too complicated to understand. One participant explained how a colleague supported her in teaching how to prototype with Sketch and Axure effectively. The responses of the participants with fewer experiences indicated that they are scared of failing to conduct a method in time and that knowledge about methods is missing, which scares them. This makes them wish for guidance.
Overall, the participants mentioned that methods should be more flexible, quick, and adaptable. One participant said:
“reacting ad-hoc on the needs of the teams and the customers, you cannot always plan everything, you have to stay flexible.” This participant also referred to the current situation around the COVID-19 and the different restrictions, which requires her to make a workshop work remotely. Some participants also indicated that development or the management is interfering, which requires them to make changes in the method while running them. They further stated that they have problems with methods such as user testing, which do not fit into one Sprint and cannot be carried to the next. These responses indicated that the structure of the development process might be a barrier to the methods or that methods like workshops inherently work bad remotely.
Some participants mentioned that a method needed to have the ability to work at low cost, and the data should be reusable and transferable to other situations. They indicated that results from a method in one situation could be transferred to another situation, which would make it time- efficient and, therefore, more comfortable to argue for.
Lifelong Learning
One of the most frequently mentioned sources for learning new things, or learning best practices, was learning from colleagues. The participants explained that colleagues inform them of the latest trends, and they work as a source of inspiration to try out new things. One participant explained that a colleague taught her how to use Sketch to create a high-fidelity prototype, which then would help her to conduct user testing. She also indicated that this collaborative learning and working supported her in working more coordinately. The response also indicated that the colleague was building trust in the participant resulting in dividing work: “And eventually, she could drop off these projects, and I could start doing it all by myself.”
Connected to the colleagues was the learning culture overall.
The participants defined the learning culture as a tight collaboration between colleagues, founded on sharing information, benefitting from each other’s skills, and incentives from the employer. Some participant even mentioned that the learning culture and the culture around colleagues was the main reason for them to switch jobs. They
stated that the culture gives them energy and enjoyment to learn new things and be open to new methods. This shows how important and vital a learning culture is for lifelong learning.
Most participants followed “experts” or “mentors” on social media (e.g., Twitter, Instagram) to get inspired and learn about the latest methods, trends, UX skills, and personal skills. One participant said: “[…] it is about having someone to guide you and answer questions.“ This indicated that the participant might be overwhelmed by all the changes and all the available methods and needs someone to break them down. Also, someone who guides them on what to do and what to learn.
Many also used podcasts, websites, blogs, and books to listen to experts. The most mentioned website for inspiration was Medium, which offered a short version of how to use a method. Basically, a short but scientifically grounded theory on methods that could be consumed quickly. One participant especially wished for this format that is quick to read and easy to understand.
Two participants mentioned that they had lunch lectures which were organized by the company. In those, experts with diverse backgrounds presented topics such as accessibility.
The other participants visited conferences and individual training but were limited in how often they could visit a conference to a few days every year. The employer for eight participants incentivized learning by giving these participants a week off every year. Some used this week to learn off-topics, whereas others used the time to visit conferences related to their work. The responses indicated that the UX professionals need to argue for time, and learning itself takes time and money from the employer and the professionals. The participants’ employers decreased this effort and reducing the cost by allowing access to education during lunch.
Even though a few participants restricted their learning to the working hours, most agreed that they often learn during their spare time, which was mostly voluntary. These participants used social media platforms to stay informed on the latest trends, learn about new methods, and getting entertained.
However, some participants mentioned that they felt a constant pressure to learn things without having much time at work. This forced them to move learning into the hours outside of work, which they perceived as highly negative.
These responses show the positive and negative effects of learning for work in spare time. Potentially, it indicated that companies could offer incentives to UX professionals in learning outside of work to save money while reducing pressure for its employers.
Learning Barriers and Enablers
Four participants, with the most years of experience, expressed that there is no need, or they do not want to learn
new things. One participant explained: “So that over the years […] you have filled your tool case.” These responses indicated that some participants work in an environment where they are not required to learn new things constantly.
In contrast, two participants who had less than two years of experience expressed more need to learn and understand everything related to UX. It seemed the participants who work in smaller companies feel a higher need for staying innovative and learn new things because of the dependency to clients.
Some participants also stated that restrictions by the employer made it impossible to find time to learn. They also indicated the need to argue for learning time. Some participants mentioned that they were only allowed to learn things that were relevant and wanted by the customer or the employer because of costs. This showed that some employers had not seen the benefit of supporting UX professionals in their lifelong journey.
Some participants mentioned that when using Scrum, a set of concepts like the Daily Scrum meetings, the Sprint reviews, and the Retrospectives meetings support the UX professionals in reflecting and learning from their work. The response indicated that these meetings amplified their motivation for learning and even for developing personally.
DISCUSSION
By interviewing 13 UX professionals, this study aimed to understand how UX professionals apply UX methods and to widen the knowledge of lifelong learning of UX professionals. This research was able to gather interesting insights on this population. Some findings confirmed previous research, others have changed, and some new have emerged.
The Usage of UX Methods Customer-Centered Approach
The most controversial aspect which influences the choice of methods was whether the approach of the project should be customer or user-centered. A shift towards customer- centered has been seen in previous literature [8]. A customer for the participants was the client or the product owner.
However, several participants had a conflict on which approach to follow. They mentioned that they would like to follow a user-centered but also the customer-centered approach. This conflict was often influenced by external factors. One external aspect was the structure of a project.
Some participants could not be part of the planning to argue for their approach and, therefore, the methods they wanted to use. Another problem that the participants confirmed is that UX awareness is still a problem [6]. The data suggests that the lack of UX awareness limited access to users and no freedom to choose a UX method. Having to struggle with validating their work constantly, made the UX professionals follow the clients' wishes rather than pursuing their own and,
therefore, the users. This struggle raises the question of whether research should develop methods that follow a more customer-centered approach to ease the life of a UX professional. However, the participants seem keen and want to use a more user-centered approach, which suggests that this more of an issue that needs to be discussed in companies.
Nonetheless, this problem of UX awareness has existed and reported for so many years [6]. Researchers and practitioners could discuss both approaches and develop a compromise for the time being to ease the life of the UX professionals.
Some methods also come with a very time-intensive nature making it hard for the participants to argue for and to apply in their process. Some participants use a lite version of a method such as low-fidelity prototyping rather than high- fidelity to overcome this. This suggests that a more efficient version of a method could increase the usage and ease the application. This could either mean that there is a problem with the development process where a method cannot be fitted into a Sprint, but it can also mean that methods might need to increase their efficiency or can be transformed to a lite version. A recommendation could be to have methods like prototyping come in different forms. Each form is having its own fidelity, efforts, and qualities. The UX professional could then decide which form is possible to integrate and how much is needed in terms of quality. This could be discussed in future research to explore how to transform methods into different versions.
Another influence of the choice of methods is the clients of the UX professionals. Depending on whether the client sees value in an approach, she determines, for example, access to users. The participants in this study indicate that this results in whether or not to use methods like user testing. In consequence, the participants use a user representative or a fixed weekly session. This can be a person from the user group or a manager of the group. This workaround needs to be looked at highly critically because the results could be semi-representative and could cause much diffusion on the actual user experience of the product. Companies that restrict access to the user need to be aware of the consequences and the resulting quality of results.
Agile Structure
The integration of UX and the structure of Agile has a major influence on the choice of methods. This is pointed out by the response of the participants and from previous literature [9]. The participants have to follow the outline and the structure of the Sprints. For the participants, the end of a Sprint is basically just a deadline for results which the developers depend on. As a consequence, the participants cannot freely pick a method and determine the duration and the number of iterations. The responses also indicate that this causes a stressful situation for the participants [30]. It can also directly affect the quality of a product. These problems have affected UX practice and the choice of methods. First,
the interviewees use a “quick and dirty” version of a method or, if possible, one with less quality, which still delivers reasonable results. Alarmingly, the results show that some participants try to predict the outcome of using a UX method and try to match it to the desired results, which could potentially harm the quality of UX.
These results suggest that Agile has some issues in its application or by nature. Sprints might not be the best environment for UX methods and might need to be changed.
A solution could be to separate the design phase and the development phase. This, however, needs further research, and deeper insights since it might differ between companies.
On the other side, methods could be redesigned to match the structure of Sprints (e.g., 1-2 weeks). However, as indicated by the participants, there are other external factors such as organizational issues that influence the application of methods. And not to forget, the results show that the participants are working in an intense situation and under high constraints which need to be addressed and discussed.
Integration of UX into Agile
Earlier research has shown ways (e.g., Sprint-0) of improving the integration of UX professionals into Agile [5,8]. This concept was supposed to integrate the UX professionals into the early stages of development, making it possible to argue for methods early on. However, as the participates in this study state, those solutions are not used, or sometimes they are not involved in it. This makes it hard for them to spread UX awareness, argue for the method of choice, and plan the use of methods ahead. Planning is important for the participants to determine whether or not access to users is needed, what software is needed, and how the process will look like. This could be one reason for why some participants have to fight for access to users or have limited access to users if they were not part of the planning.
This reveals some serious issues which needs to be tackled in the companies. Having UX professionals in the team but limiting the influence of them might be problematic first for the product and second for the joy of the UX professionals.
The Influence of Tools
As the study population pointed out, the availability and the kind of software highly influenced the choice of a method.
The availability depended on the companies’ policies and restrictions. The participants also picked software depending on available functions such as collaboration, sharing, familiarity, and visual aesthetics. This means that when designing UX methods, it must be considered what tools people are using, why they are using them (e.g., collaboration), and whether those tools are easy to use within organizations (e.g., compliance). The importance and influence of tools emphasized by the participants have not been pointed out enough in previous research. Future research needs to take software tools into account when designing UX methods. The data suggests that even a perfect
UX method would not be used if it cannot be carried out on software that is currently used. However, it must also be discussed whether that influence is positive or not.
Method Design
Due to the events around COVID-19, the participants emphasized the issue of remote working. Reflecting on UX methods, they argued that these issues have existed but became more prominent and unavoidable with new restrictions regarding social distancing. Problems stated were time-zone issues, no on-site and face-to-face contact with users, and the limitation of remote tools in terms of interactions. Possible reasons why the participants had issues could be organizational restrictions from the company. They also indicate that many software tools do not have the functions they need. This suggests that methods and software could be designed together where the software brings the functions needed to carry out the method. Otherwise, when designing methods, the researcher could take into account UX professionals and what software they are using to design them together.
Previous research [37] and partially, the participants of this study point out that new UX methods from HCI are hard to understand and hard to apply. The participants argue that the presentation of new methods is often hiding behind a complicated and long scientific paper. This is problematic for the UX professionals because it costs them much time and requires guidance. The barrier of understanding the UX methods should be lowered if researchers want their UX methods to be used. The participants wish for UX methods to be presented in a condensed and more visual way. They mentioned a format similar to a Medium article or an Instagram post: Five pictures breaking down the basics of a method. Thus, researchers could try to reach out with UX methods and findings on platforms where UX professionals are primary looking to learn; which in this study population was mainly social media and Medium. This could perhaps be one of the main issues; researchers and UX professionals are not communicating in the same platforms or in similar ways, making a gap between them.
Team Setup
The team setup has been shown in literature and in the results of this study to be a large factor influencing the choice of UX methods [5]. Factors such as co-location and remote working have been reported by previous research to influence teamwork negatively [5,11]. The participants from this study partly agree. Some participants indicated communication problems with developers and other stakeholders. This could have caused the lack of UX awareness reported by the participants. A possible explanation for the problems could also be the access to communication software.
Time
According to the results of this study, time has been the most influential factor on the UX professionals’ practice and the
choice of method. For the participants, time dictates which method can be used and limits the access to the user. One explanation could be that the structure of Agile simply does not allow a method to take the time it needs. This means that the problem with time could be resolved by having yet another form of integration of UX in Agile. Even though previous research [5] has proposed concepts like Sprint-0, the participants in this study exposed that some companies do not use it or do not use it in the right way. Meaning that UX professionals are not part of Sprint-0. Another solution could be to separate design work from the Agile process and let it work before the development starts. This approach was mentioned by one participant, but he said that this is uncommon because it often takes more time to finish the whole development of a product making it less viable for the employer.
Having time issues could also mean that the method itself is not efficient and might need a redesign. For example, the participants used prototyping in different forms. They used low-fidelity prototyping if there was not much time, and they used high-fidelity prototyping when enough time was available. This is similar to a concept called minimal-viable product [32], which is the product that carries only the necessary features. This lean approach has the goal of making it easier to deliver a workable product in a shorter time. For future research, it might be interesting to adapt UX methods such as workshops to this lean approach, which delivers results quicker and cheaper.
Getting no time for methods can also be come from organizational issue. As mentioned earlier, UX awareness is still a problem. As indicated by the participants, they constantly need to validate their work and fight for the use of a method. They need to argue for why UX work is valuable.
An aspect which influences the time and the choice of a method is the access to users, which was pointed out by many participants. One participant mentioned, however, a way of easing the access to users. In his company, a fixed group was responsible for recruiting users regularly. Everyone in the company then had the chance to book sessions, which was an easy and cheap way for UX professionals to get feedback from users about the product quickly.
Lifelong Learning
The Importance of Colleagues
Colleagues have been the most important source for learning, which is shown by the results of this study and is indicated by earlier research [18]. Literature, however, did not express how immense the importance is. For the participants, colleagues work almost as teachers explaining what UX methods to choose and how to apply them. There are concepts from research such as pair designing, which embrace this collaborative form of working [10]. The results in this study also show that colleagues are not only a source,
but also enabler and accelerator to learning. This might even influence the willingness to learn new UX methods. This suggests than when researching the topic of lifelong learning, one must look deeper into the interaction and the social structure around colleagues. Also, colleagues and the company form together a culture. The data indicated that this culture is vital for lifelong learning and even for attracting talent. This suggests that when implementing and improving lifelong learning, it is important to keep an eye on the culture.
Mentors
The participants also learn from “mentors.” The need for someone who can break down information and present it in an easily digestible way can show two things: first, as indicated by research, there are many new methods out there, leaving the participants with many open questions and desire for guidance [39]. Second, the way information is presented, mostly in the form of scientific papers, is not the form the participants want to learn. This is supported by other findings that participants prefer sources of learning on social media platforms in a condensed form. These results show an opportunity for where to put learning material and how to present it. New UX methods can be presented to attract the audience of social media platforms. They can also be advertised on Twitter channels. Moreover, they can also be made easier and faster to grasp with less information being displayed. These are possibilities that need to be assessed or tried out, but the participants from this study confidently indicate much interest in that.
Learning in the Spare Time
As previously mentioned, many UX professionals are forced to or voluntarily learn in their spare time. Not being able to distinguish between professional and personal learning has been predicted in previous research [23]. For some participants, this was not a problem at all. Others, however, felt stressed and did not want to use time outside of work.
This study suggests that the company could incentivize learning in their spare time more. In other words, every stakeholder should find a way to embrace this possibility. It would highly benefit the employers in having UX professionals learning without extra time costs, and it could also benefit the UX professionals for personal development.
This is an interesting phenomenon and has much potential, and it should be researched more thoroughly in the future.
Barriers and enablers
A large barrier for learning is that learning at work costs time. Therefore, some participants were only allowed to learn if it brings value to the business. Based on the responses of the participants, this study suggests that companies could lower the barrier, e.g., lunch lectures. Besides, elements of Scrum, such as Retrospective, Daily Scrum, appear to amplify and support learning. Together with the reluctance to support from the employer, this suggests an opportunity to integrate learning into work cheaply.
Future Work
Future research needs to discuss the customer-focused approach critically in practice when designing UX methods.
Future studies must evaluate an iterative and lean method structure better to understand the implications of the results of this study. Besides, during the process of this study, it has been confronted with unpredictable situations due to COVID-19. The results concerning the regulations of COVID-19 forced UX professionals to work remotely from home. This has existed before, according to the participants, and therefore is not an entirely new situation. Nonetheless, it opens an entire new problem space about conducting methods such as interviews as used in this study or workshops indicated by the participants. There have been recent attempts by NVIDIA and Google Meet to remove issues such as background noise [35]. However, there are many other issues that need to be looked at.
Overall, this first attempt to approach the lifelong learning of UX professionals has shown some interesting insights.
However, there is yet much to discover. Culture and colleagues need to be researched more thoroughly to understand structures and dynamics. A contextual inquiry over a long time might bring interesting insights.
Limitations
The present study has only explored the current stage of UX practice in a focused domain. The fact that it changes quickly has been shown by the effects of COVID-19. This research has not evaluated the effectiveness and efficiency of the UX methods.
By providing detailed information on the participants and the companies, keeping privacy in mind, this research attempted to make the results more transferable. Overall, the broad sampling helped to explore and to get different perspectives from different years of experience and companies. As the study has shown there are many results which overlap between the participants and the different companies.
CONCLUSION
In this study, we aimed to investigate how UX professionals apply UX methods, what barriers and enablers there are for using UX methods, and to widen the current knowledge on lifelong learning among UX professionals.
Based on qualitative data from an interview study, this research has brought some insightful details in UX practice.
UX professionals struggle with barriers to apply methods such as UX awareness and company’s mindset in approaching development more customer-centered rather than user-centered. We also discovered that Agile highly influences UX practice and makes UX professionals wish for better integration of UX methods. Our results imply that methods could integrate a lean approach with an iterative structure, which would make them more suitable for Agile projects and the UX practice by being quick, flexible, used
in pairs, and iterative. The study has also demonstrated that access and recruitment of users for testing is a significant factor influencing the choice of a UX method. Besides, the participants wish for a less complicated and condensed way to present and teach UX methods. The findings also emphasize the importance of improving how UX methods can work remotely.
Concerning lifelong learning, the results from this study agree with earlier predictions that the UX professionals do not separate between learning for professional and personal development. The investigations in this area show that colleagues are an essential element to the practice of lifelong learning. They function as a source of inspiration, teaching, and mentoring. This paper underlines the importance of a learning culture to support UX professionals in the journey of lifelong learning. The findings could encourage companies to benefit from the UX professionals’
professional and personal development by incentivizing training and support self-directed learning. In this matter, the findings from the participants of this study can help to understand what future research needs to look at. It should also make companies aware of the importance of learning settings and the possibility of how to contribute to that.
The highlighted findings of this study should encourage both research and employers in the industry. Improved UX methods and understanding lifelong learning could ease the work of UX professionals, reduce stress, lower the barriers to involve users, and contribute to a future where UX professionals deliver better products to society. A future in which UX professionals can work more independently from wherever they want without any limitations and still deliver gifting services to everyone.
ACKNOWLEDGMENTS
First, I would like to express my sincere gratitude to my supervisors Åsa Cajander and Marta Lárusdóttir, for all the continuous and helpful support in all stages of publishing this thesis. Furthermore, I would like to thank the thesis organizers and all fellow students for all the time and helpful directions. Lastly, I sincerely thank all the participants helping in exploring the topic and answering my research questions and all my friends and colleagues, which supported me and reviewed my writing.
REFERENCES
[1] Henrik Artman and Mattias Arvola. 2008. Studio life:
The construction of digital design competence. Nordic Journal of Digital Literacy 3, 02 (2008), 78–96.
[2] Atlassian. Sprints. Atlassian. Retrieved May 19, 2020 from https://www.atlassian.com/agile/scrum/sprints [3] Kent Beck. 2000. Extreme Programming Explained:
Embrace Change. Addison-Wesley Professional.
[4] Emma Bell, Alan Bryman, and Bill Harley. 2018.
Business research methods. Oxford university press.
[5] Hugh Beyer. 2010. User-centered agile methods.
Synthesis lectures on human-centered informatics 3, 1 (2010), 1–71.
[6] Inger Boivie. 2005. A fine balance: Addressing usability and users’ needs in the development of IT systems for the workplace. Acta Universitatis Upsaliensis.
[7] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (January 2006), 77–101.
DOI:https://doi.org/10.1191/1478088706qp063oa [8] Anders Bruun, Marta Kristin Larusdottir, Lene
Nielsen, Peter Axel Nielsen, and John Stouby Persson.
2018. The role of UX professionals in agile development: a case study from industry. In Proceedings of the 10th Nordic Conference on Human-Computer Interaction (NordiCHI ’18), Association for Computing Machinery, Oslo, Norway, 352–363.
DOI:https://doi.org/10.1145/3240167.3240213 [9] Åsa Cajander, Marta Larusdottir, and Jan Gulliksen.
2013. Existing but Not Explicit - The User Perspective in Scrum Projects in Practice. In Human-Computer Interaction – INTERACT 2013 (Lecture Notes in Computer Science), Springer, Berlin, Heidelberg, 762–779. DOI:https://doi.org/10.1007/978-3-642- 40477-1_52
[10] Gerardo Canfora, Aniello Cimitile, Felix Garcia, Mario Piattini, and Corrado Aaron Visaggio. 2007.
Evaluating performances of pair designing in industry.
Journal of Systems and Software 80, 8 (August 2007), 1317–1327.
DOI:https://doi.org/10.1016/j.jss.2006.11.004 [11] Kieran Conboy, Sharon Coyle, Xiaofeng Wang, and
Minna Pikkarainen. 2011. People over Process: Key Challenges in Agile Development. IEEE Softw. 28, 4
(July 2011), 48–57.
DOI:https://doi.org/10.1109/MS.2010.132
[12] Martyn Denscombe. 2014. The good research guide:
for small-scale social research projects. McGraw-Hill Education (UK).
[13] Casey Fiesler, Jed R. Brubaker, Andrea Forte, Shion Guha, Nora McDonald, and Michael Muller. 2019.
Qualitative Methods for CSCW: Challenges and Opportunities. In Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing (CSCW ’19), Association for Computing Machinery, Austin, TX, USA, 455–460.
DOI:https://doi.org/10.1145/3311957.3359428 [14] Kiev Gama, Breno Alencar Gonçalves, and Pedro
Alessio. 2018. Hackathons in the formal learning process. In Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE 2018), Association for Computing Machinery, Larnaca,
Cyprus, 248–253.
DOI:https://doi.org/10.1145/3197091.3197138 [15] Francisco J. García-Peñalvo and David Griffiths.
2014. Transferring knowledge and experiences from informal to formal learning contexts. In Proceedings of the Second International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM ’14), Association for Computing Machinery, Salamanca, Spain, 569–572.
DOI:https://doi.org/10.1145/2669711.2669957 [16] Colin M. Gray. 2014. Evolution of design competence
in UX practice. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14), Association for Computing Machinery, Toronto, Ontario, Canada, 1645–1654.
DOI:https://doi.org/10.1145/2556288.2557264 [17] Colin M. Gray. 2016. “It’s More of a Mindset Than a
Method”: UX Practitioners’ Conception of Design Methods. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16), Association for Computing Machinery, San Jose,
California, USA, 4044–4055.
DOI:https://doi.org/10.1145/2858036.2858410 [18] Colin M. Gray, Erik Stolterman, and Martin A. Siegel.
2014. Reprioritizing the relationship between HCI research and practice: bubble-up and trickle-down effects. In Proceedings of the 2014 conference on Designing interactive systems (DIS ’14), Association for Computing Machinery, Vancouver, BC, Canada, 725–734.
DOI:https://doi.org/10.1145/2598510.2598595 [19] Colin M. Gray, Austin L. Toombs, and Shad Gross.
2015. Flow of Competence in UX Design Practice. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ’15), Association for Computing Machinery, Seoul,
Republic of Korea, 3285–3294.
DOI:https://doi.org/10.1145/2702123.2702579 [20] Andries de Grip and Wendy Smits. 2012. What affects
lifelong learning of scientists and engineers?
International Journal of Manpower (2012).
[21] Julia Gross. 2012. 6 - Lifelong learning and your career. In Building Your Library Career with Web 2.0, Julia Gross (ed.). Chandos Publishing, 99–117.
DOI:https://doi.org/10.1016/B978-1-84334-651- 7.50006-6
[22] J. Grudin. 1991. Interactive systems: bridging the gaps between developers and users. Computer 24, 4 (April 1991), 59–69. DOI:https://doi.org/10.1109/2.76263 [23] Graham Guest. 2006. Lifelong learning for engineers:
a global perspective. European Journal of Engineering Education 31, 3 (June 2006), 273–281.
DOI:https://doi.org/10.1080/03043790600644396 [24] Jan Gulliksen, Inger Boivie, and Bengt Göransson.
2006. Usability professionals—current practices and
future development. Interacting with computers 18, 4 (2006), 568–600.
[25] Stefan Holmlid and Mattias Arvola. 2007. Developing a thematic design curriculum as a Bologna master.
(January 2007).
[26] Imre Horváth. 2006. Design competence development in an academic virtual enterprise. In Proceedings of IDETC/CIE, 10–13.
[27] Julie S. Hui and Elizabeth M. Gerber. 2017.
Developing Makerspaces as Sites of Entrepreneurship.
In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW ’17), Association for Computing Machinery, Portland, Oregon, USA, 2023–2038.
DOI:https://doi.org/10.1145/2998181.2998264 [28] Pariya Kashfi, Agneta Nilsson, and Robert Feldt.
2017. Integrating User eXperience practices into software development processes: implications of the UX characteristics. PeerJ Comput. Sci. 3, (October 2017), e130. DOI:https://doi.org/10.7717/peerj-cs.130 [29] Yubo Kou and Colin M. Gray. 2019. A Practice-Led Account of the Conceptual Evolution of UX Knowledge. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), Association for Computing Machinery,
Glasgow, Scotland Uk, 1–13.
DOI:https://doi.org/10.1145/3290605.3300279 [30] Kati Kuusinen, Tommi Mikkonen, and Santtu
Pakarinen. 2012. Agile User Experience Development in a Large Software Organization: Good Expertise but Limited Impact. In Human-Centered Software Engineering (Lecture Notes in Computer Science), Springer, Berlin, Heidelberg, 94–111.
DOI:https://doi.org/10.1007/978-3-642-34347-6_6 [31] Jonathan Lazar, Jinjuan Heidi Feng, and Harry
Hochheiser. 2017. Research methods in human- computer interaction. Morgan Kaufmann.
[32] Valentina Lenarduzzi and Davide Taibi. 2016. MVP Explained: A Systematic Mapping Study on the Definitions of Minimal Viable Product. In 2016 42th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 112–119.
DOI:https://doi.org/10.1109/SEAA.2016.56
[33] Norman Longworth and W. Keith Davies. 1996.
Lifelong Learning: New Vision, New Implications, New Roles for People, Organizations, Nations and Communities in the 21st Century. ERIC.
[34] Steven E. Poltrock and Jonathan Grudin. 1994.
Organizational obstacles to interface design and development: two participant-observer studies. ACM Trans. Comput.-Hum. Interact. 1, 1 (March 1994), 52–
80. DOI:https://doi.org/10.1145/174630.174633 [35] Jon Porter. 2020. How to use your Nvidia graphics
card to improve the quality of your calls. The Verge.
Retrieved June 14, 2020 from
https://www.theverge.com/2020/4/21/21229508/nvidi a-rtx-voice-background-noise-reduce-graphics-card- improve-call-quality
[36] David J. Roedl and Erik Stolterman. 2013. Design research at CHI and its applicability to design practice.
In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13), Association for Computing Machinery, Paris, France, 1951–1954.
DOI:https://doi.org/10.1145/2470654.2466257 [37] Yvonne Rogers. 2004. New theoretical approaches for
human‐computer interaction. Annual review of information science and technology 38, 1 (2004), 87–
143.
[38] Clive Seale, Giampietro Gobo, Jaber F. Gubrium, and David Silverman. 2004. Qualitative Research Practice. SAGE.
[39] Ernest T. Smerdon. 1996. Lifelong learning for engineers: Riding the whirlwind. American Ceramic Society bulletin 75, 12 (1996), 51–52.
[40] John Smith. 2018. GDPR: Guidelines, Recommendations, Best Practices. European Data Protection Board - European Data Protection Board.
Retrieved April 29, 2020 from
https://edpb.europa.eu/our-work-tools/general- guidance/gdpr-guidelines-recommendations-best- practices_en
[41] Erik Stolterman. 2008. The nature of design practice and implications for interaction design research.
International Journal of Design 2, 1 (2008).
[42] Alistair Sutcliffe. 2000. On the effective use and reuse of HCI knowledge. ACM Trans. Comput.-Hum.
Interact. 7, 2 (June 2000), 197–221.
DOI:https://doi.org/10.1145/353485.353488
[43] CollabNet VersionOne. 13th Annual State of Agile report (2018).
[44] Stephanie Wilson, Mathilde Bekker, Hilary Johnson, and Peter Johnson. 1996. Costs and Benefits of User Involvement in Design: Practitioners’ Views. In People and Computers XI, Springer, London, 221–
240. DOI:https://doi.org/10.1007/978-1-4471-3588- 3_15
[45] Stephanie Wilson, Mathilde Bekker, Peter Johnson, and Hilary Johnson. 1997. Helping and hindering user involvement — a tale of everyday design. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems (CHI ’97), Association for Computing Machinery, Atlanta,
Georgia, USA, 178–185.
DOI:https://doi.org/10.1145/258549.258699
[46] Carol Zander, Jonas Boustedt, Anna Eckerdal, Robert McCartney, Kate Sanders, Jan Erik Moström, and Lynda Thomas. 2012. Self-directed learning: stories from industry. In Proceedings of the 12th Koli Calling International Conference on Computing Education Research (Koli Calling ’12), Association for
Computing Machinery, Koli, Finland, 111–117.
DOI:https://doi.org/10.1145/2401796.2401810 [47] ISO 9241-210:2019(en), Ergonomics of human-
system interaction — Part 210: Human-centred design for interactive systems. Retrieved January 30, 2020 from https://www.iso.org/obp/ui/#iso:std:iso:9241:- 210:ed-2:v1:en
[48] Manifesto for Agile Software Development. Retrieved December 14, 2019 from https://agilemanifesto.org/
