Postal address: Visit address: Telephone:
Usability Evaluation of a Production System
Development Framework
A Meta-‐Study Performed on the Use of a Production System
Development Framework in the Development of a New
Production System at Xylem
Fredrik Arnesson
Johan Bengtsson
THESIS WORK 2012
PRODUCTION SYSTEM
Postal address: Visit address: Telephone:
Usability Evaluation of a Production System
Development Framework
A Meta-‐Study Performed on the Use of a Production System
Development Framework in the Development of a New
Production System at Xylem
Fredrik Arnesson
Johan Bengtsson
This thesis work has been carried out at the School of Engineering at Jönköping University in the subject area production development and leadership. The the-‐ sis work is a part of the Master of Science program Production System.
The students take full responsibility for opinions, conclusions, and findings pre-‐ sented.
Examiner: Glenn Johansson Supervisor: Kristina Säfsten Scope: 30 credits (D-‐level) Date: June 2012
Acknowledgements
We would like to take this opportunity to thank our supervisor Kristina Säfsten for her guidance and help during the process of writing this thesis. We would also like to thank our supervisor at Xylem Roger Sandström for providing us as-‐ sistance throughout the performed work at the company. Finally, we would like to express our gratitude to all involved employees at Xylem for their contribu-‐ tions and support.
____________________________________ ____________________________________ Fredrik Arnesson Johan Bengtsson
Summary
Today’s competitive global market has placed companies under great pressure and the focus on production systems has been more prominent. Although there are several claimed benefits with using frameworks in the development of pro-‐ duction systems, companies are reluctant to use these. Consequently, a relevant question formulation is: Are frameworks in the development of production sys-‐ tems usable?
The purpose with this thesis work was therefore to evaluate the usability of pro-‐ duction system development frameworks (PSDFs) in practice. In order to achieve this purpose, two research questions were established:
RQ1. How can usability of frameworks be evaluated?
RQ2. How does the use of a framework contribute to the development of a new
production system?
In order to answer the posed research questions, Bellgran and Säfsten’s PSDF was used in the production system development (PSD) process of a new produc-‐ tion system at Xylem. Based on the PSD process, a meta-‐study was performed to evaluate the practical usability of the PSDF. Usability was defined and evaluated based on the five usability terms learnability, memorability, efficiency, effective-‐ ness, and satisfaction.
The result showed that all the five usability terms contribute to the usability evaluation of PSDFs. However, memorability was considered difficult to use on only one study since the user has to think a step further and make a qualified guess to answer if it is possible to memorize a framework. Therefore, it was con-‐ sidered memorability is only appropriate to use in a multiple study.
The results also showed that Bellgran and Säfsten’s PSDF contributed most in the beginning of the PSD process by putting emphasis on the planning phase and providing a structure to follow. Due to the nature of a framework (i.e., to serve as a guide for structures to follow), this was not unexpected. However, the contri-‐ butions from a structure or plan are hard to exactly distinguish. Since companies most often want tangible and accurate evidences, frameworks’ vague contribu-‐ tions are considered to be a major reason to why companies do not use frame-‐ works more frequently.
Keywords
Production system development, Framework, Usability, Evaluation, Meta-‐Study
Table of Contents
1. INTRODUCTION ... 1
1.1 BACKGROUND ... 1
1.2 PURPOSE AND RESEARCH QUESTIONS ... 2
1.3 DELIMITATIONS ... 2
1.4 OUTLINE ... 2
2. THEORETICAL BACKGROUND ... 4
2.1 PRODUCTION SYSTEM ... 4
2.1.1 Production System Development ... 5
2.1.2 Academic versus Industrial Perspective in Production System Development 6 2.1.3 Production System Development Frameworks ... 7
2.2 USABILITY ... 11
2.2.1 Definition and Description of Usability ... 12
2.2.2 Measurement of Usability ... 13
2.2.3 Evaluation of Usability ... 14
2.2.4 Usability of Frameworks ... 14
3. METHOD ... 16 3.1 RESEARCH PROCESS ... 16 3.2 RESEARCH METHOD ... 17 3.2.1 Case Study ... 17 3.2.2 Meta-‐Study ... 17 3.3 DATA COLLECTION ... 18
3.3.1 Primary Data Collection ... 18
3.3.2 Secondary Data Collection ... 19
3.4 DATA ANALYSIS ... 20
3.5 QUALITY OF DATA ... 20
3.5.1 Validity of Data ... 20
3.5.2 Reliability of Data ... 20
4. CASE DESCRIPTION ... 22
4.1 PRODUCTION SYSTEM DEVELOPMENT PROCESS ... 22
4.1.1 PLANNING PHASE ... 22
4.1.2 PREPARATION PHASE ... 23
4.1.3 DESIGN AND EVALUATION PHASE ... 24
5. USABILITY ANALYSIS OF PRODUCTION SYSTEM DEVELOPMENT FRAMEWORK 27 5.1 GUIDELINE FOR USABILITY ANALYSIS ... 27
5.2 USABILITY ANALYSIS OF BELLGRAN AND SÄFSTEN’S PRODUCTION SYSTEM DEVELOPMENT FRAMEWORK ... 27
6. DISCUSSION AND CONCLUSION ... 33
6.1 RQ1. HOW CAN USABILITY OF FRAMEWORKS BE EVALUATED? ... 33
6.2 RQ2. HOW DOES THE USE OF A FRAMEWORK CONTRIBUTE TO THE DEVELOPMENT OF A NEW PRODUCTION SYSTEM? ... 34
6.3 THREATS TO VALIDITY AND RELIABILITY ... 36
6.4 SUGGESTIONS FOR FURTHER RESEARCH ... 36
7. REFERENCES ... 38
APPENDIX A: PLAN FOR PRODUCTION SYSTEM DEVELOPMENT ... I § 1 INVESTMENT PLAN ... I
§ 2 PROJECT PLAN ... I
§ 3 GUIDELINE FOR REQUIREMENT SPECIFICATION AND SYSTEM SOLUTION ... II
§ 5 ADAPT THE WAY OF WORKING TO THE SPECIFIC SITUATION ... IV
APPENDIX B: REQUIREMENT SPECIFICATION ... V § 1 GENERAL OVERVIEW ... V
§ 2 SYSTEM OVERVIEW ... V
§ 3 REQUIREMENTS AND OBJECTIVES ... X
§ 4 INTERFACES ... X
§ 5 CHECKLIST ... XI
APPENDIX C: SYSTEM SOLUTION ... XII § 1 GENERAL OVERVIEW ... XII
§ 2 CONCEPTUAL SYSTEM SOLUTION ... XII
§ 3 DETAILED SYSTEM SOLUTION ... XII
§ 4 CONTINUATION OF PRODUCTION SYSTEM DEVELOPMENT ... XX
1. Introduction
This chapter presents the background, purpose, and delimitations of the thesis work. An outline is also provided in order to give a general overview of the report.
1.1 Background
Today’s competitive global market has placed companies under great pressure and the focus on production systems has been more prominent (Neumann et al., 2002). Along with the increased competitive industrial situation, it is clear that increased levels of output, efficiency, effectiveness, and quality can only be achieved by developing new and better production systems (Bennett, 1986; Shang and Sueyoshi, 1995).
Production system development (PSD) can take different amount of time, all from a couple of weeks to several years. When something is done infrequently, like developing a new production system, it can aggravate the use of previous experience and knowledge. However, by using structured work methods and trying to use others’ experiences, one can overcome these barriers. A structure makes it possible to focus on the important parts, such as preparing and creating new and accurate production systems (Bellgran and Säfsten, 2010).
Nevertheless the claimed benefits with a structured approach, there are many companies that tend to deal with activities in development projects in an ad hoc manner (Bellgran and Säfsten, 2010). Chryssolouris (1992) means the industrial practice is more of a trial-‐and-‐error approach:
1. Guess a suitable production system; and
2. Evaluate the performance of the system. If it satisfactory, then the design process stops, otherwise return to step 1.
According to Bellgran and Säfsten (2010), there are different reasons to why companies do not use a structured approach when developing new production systems. Some of the reasons are for example time pressure, low priority, fear of low flexibility, difficulties to access information with high quality, and a lack of existing methods on the market (Bellgran and Säfsten, 2010).
In two company studies performed in Sweden, which focused on PSD, the intro-‐ duction of new products or product models was the main reason to why the in-‐ vestigated companies were developing their production systems. At the same time, the companies saw a potential to increase ergonomics and work environ-‐ ment, automatize, get better workflow, and increase the volume capacity (Bell-‐ gran and Öhrström, 1995; Bellgran 1998; Säfsten and Aresu, 2000; Säfsten, 2002, see Bellgran and Säfsten, 2010, pp. 110-‐111).
Based on empirical and theoretical data, Bellgran and Säfsten (2010) have devel-‐ oped a framework including a structured way of working in order to assist the PSD process. The usefulness of this production system development framework (PSDF) has however not been tested and therefore it is interesting to investigate how usable PSDFs really are.
First however, in order to know if something is usable or not, it is important to define what usability really is. In the context of usability, there are different defi-‐ nitions of the term and considerable confusion exists over the term’s meaning
on the ISO 9241-‐11 standard and the usability consultant Jakob Nielsen’s usabil-‐ ity goals: effectiveness, efficiency, satisfaction, learnability, and memorability (Nielsen, 1993; ISO 9241-‐11, 1998).
An extensive literature review shows that there is little research made within the production development area in terms of measuring and evaluating usability of frameworks. This showed to be true even in the closely related product devel-‐ opment area; an area that is generally more emphasized, both in industry as well in theory (Bellgran and Säfsten, 2010). In terms of usability in the product and production development areas, it is found that the final results (i.e., the final products, services etc.) achieved from when using frameworks are more in focus. In an industrial environment, it is therefore believed to be relevant to investigate how usable PSDFs are.
To be able to evaluate the usability of a PSDF, a practical situation is required, and an opportunity to do this arose when the manufacturing company Xylem proposed a PSD project in their winding shop.
Xylem is a world-‐leading provider of fluid technology and equipment solutions for water-‐related issues (Xylem Inc., 2012). Currently the company is in the planning phase of developing a new production system, and as a response to cope with the dynamic competitive environment, the company at the same time wants to improve the productivity, reduce the scrap rate, and increase the ergo-‐ nomics in this new production system.
1.2 Purpose and Research Questions
The purpose with the thesis work is to evaluate the usability of PSDFs in prac-‐ tice. In order to achieve this purpose, two research questions have been estab-‐ lished:
RQ1. How can usability of frameworks be evaluated?
RQ2. How does the use of a framework contribute to the development of a new
production system?
1.3 Delimitations
Since the development of a new production system is a long process and the timeframe for this thesis work is limited to 30 credits (i.e., 20 weeks full-‐time study), the thesis work will not cover the actual production system realization. In order to evaluate the usability of frameworks, the students will use Bellgran and Säfsten’s PSDF. Moreover, in this thesis work the framework will only been used and tested at one company.
Finally, the thesis work is limited to the development of three production lines and to the requirements set by the company.
1.4 Outline
Chapter 2 – Theoretical Background
This chapter mainly presents two central theoretical aspects connected to the thesis work. The first aspect is production system, where emphasis is put on the development through the use of frameworks. The second aspect is usability and how it can be used when evaluating frameworks.
Chapter 3 – Method
This chapter presents the methods used in the thesis work. A thorough explana-‐ tion of all used data collection techniques is given together with an evaluation of data quality.
Chapter 4 – Case Description
This chapter presents the results from the PSD process, where a production sys-‐ tem solution was developed at Xylem by the use of Bellgran and Säfsten´s PSDF.
Chapter 5 – Usability Analysis of Production System Development Framework
This chapter presents the result from the meta-‐study performed on the PSD pro-‐ cess. The definition of usability, together with the estimation levels of usability, is introduced. Based on this, the actual usability analysis is presented and compiled in a table at the end of the chapter.
Chapter 6 – Discussion and Conclusion
In this chapter, a discussion is held based on the analysis. Through the discus-‐ sion, conclusions are drawn, which answer to the thesis work’s established re-‐ search questions. As a final discussion, critique to the thesis work’s chosen methods is given and suggestions for future research are presented.
Appendix A-‐C
The appendices are available for the reader who wants in-‐depth information about the results from the PSD process. The appendices are documents created to assist the PSD process and are the result of following Bellgran and Säfsten’s PSDF. The attached appendices are Plan for production system development (A),
2. Theoretical Background
This chapter mainly presents two central theoretical aspects connected to the thesis work. The first aspect is production system, where emphasis is put on the development through the use of frameworks. The second aspect is us-‐ ability and how it can be used when evaluating frameworks.
2.1 Production System
A production system is a system where a product is produced physically (Bell-‐ gran and Säfsten, 2010), and the function of a production system can generally be regarded as a transformation process where input is transformed into output (Olhager, 2000). Input can consist of material, labor, and capital, while output can be a product or service. It is during the transformation process that value is created for the customer. Value is the amount a customer is prepared to pay for a product or service (Porter, 1985, see Olhager, 2000, p. 17). In order to get the most value of a resource as possible, one needs to design production processes that facilitate efficient and effective production systems, but one also needs to manage the operations so they produce products that are economically advanta-‐ geous in a competitive environment (Arnold et al., 2008).
According to Bellgran and Säfsten (2010), a holistic perspective is important when considering a production system, and this is since a production system is an open system and constantly affected by external factors. A holistic perspective can be achieved by adapting a system perspective, which is a way of focusing on all factors affecting the system in order to understand the system as a whole (Bellgran and Säfsten, 2010). Chryssolouris (1992) has a similar view as he de-‐ scribes production as a system based on equipment and humans bound by com-‐ mon material and information flow.
A production system can be classified in several different ways depending on perspective. An example of production system classification is the hierarchical perspective (Seliger et al., 1987, see Bellgran and Säfsten, 2010, p. 41); the pro-‐ duction system is viewed as a part of a manufacturing system, and consists of assembly system (or line) and parts production system, see figure 2.1.
Figure 2.1 A hierarchical perspective on production system (Bellgran and Säfsten, 2010)
A production system is thus linked to other systems, and can be affected by fac-‐ tors from both inside and outside the system. Bellgran and Säfsten (2010) have identified three general factors that affect a production system:
1. External influences: history, trends, globalization, and company struc-‐ tures;
2. Actual options: technology, work environment and organization, and planning and control; and
3. Strategies and fundamental attitudes: management strategies, production philosophies, and company culture.
2.1.1 Production System Development
In order to be able to classify reasons for changes in an organization, Porras and Robertson (1992) have developed a model where various degrees of organiza-‐ tional change are related to internal and external reasons, see table 2.1. If a change is triggered by reasons from inside the company, it is most likely a planned change, and consequently, if a change is triggered by reasons from out-‐ side the company, it is most likely an unplanned change. To which degree the change is made is also separated; if the change is minor, compared to current conditions, it is called a first-‐order change, and if it is more radical it is called a second-‐order change (Porras and Robertson, 1992).
Table 2.1 Different categories of change (modified from Porras and Robertson, 1992)
Planned change Unplanned change First order change Developmental Evolutionary
Second order change Transformational Revolutionary
The reasons for companies to develop new production systems can emerge from different perspectives:
• Logistic perspective: companies competing worldwide, transportation and movement of material are cheaper, more effective, and faster (Arnold et
al., 2008);
• Product perspective: shortened product lifecycles forces companies to fo-‐ cus on product development (Almgren 1999; Surbier et al., 2009); and • Work environment and ergonomic perspective: stricter legislations force
companies to think about their employees’ health and wellbeing (Christ-‐ mansson et al., 2000; Jensen, 2002).
All of these reasons for developing production systems are based on the same basic idea: To increase the profitability for the company. This is, however, the primary objective for any profit-‐driven company (Olhager, 2000).
According to Slack and Lewis (2008), a company requires a strategy to achieve its goals, and one crucial aspect in a strategy is to reconcile the market require-‐ ments with the company’s resources. The reconciliation can also be described, as how well the company manages to meet its customer demands (Slack and Lewis, 2008).
How well a company is performing is to a large extent depending on how well it aligns its performance objectives with its customer demands (Slack and Lewis, 2008). These performance objectives can to some degree vary from different authors. Ferdows and De Meyer (1990) illustrate the performance objectives in a sand cone model including quality, dependability, speed, and cost efficiency, see figure 2.2. The basic idea behind this model is to show the performance objec-‐ tives to be cumulative. Slack and Lewis (2008), on the other hand, view perfor-‐ mance objectives as generic, changeable factors. As a response to the dynamic
market, Slack and Lewis have also added flexibility as a fifth performance objec-‐ tive.
Figure 2.2 Sand cone model illustrating the cumulativeness between quality, dependability, speed, and cost efficiency (Ferdows and De Meyer, 1990)
Another way of improving a company’s performance is to make use of the poten-‐ tial in ergonomics (Eklund, 2003) since a lack of focus on ergonomics can affect a company’s competitiveness severely (Neumann et al., 2002).
International Ergonomics Association (2000) defines ergonomics, or “human factors”, as “the scientific discipline concerned with the understanding of the interactions among humans and other elements of a system, and the profession that applies theoretical principles, data and methods to design in order to opti-‐ mize human wellbeing and overall system”. Although the definition clearly states that ergonomics has a direct impact on a system’s efficiency, ergonomics is commonly interpreted in a more narrow sense with focus on only human factors (Eklund, 2003).
Helander and Burri (1995) argue the increased need for ergonomic design in today´s manufacturing environment is a result of the increased technological complexity. For instance, an operator that is working with an automated ma-‐ chine must both take a supervising role and repair the machine in case of mal-‐ function, as well as a be a back-‐up to produce manually if the machine breaks down.
2.1.2 Academic versus Industrial Perspective in Production System Development
The view of PSD tends to vary between theory and practice, and a distinction between the two perspectives can be made (Bellgran and Säfsten, 2010). Chryssolouris (1992) argues the most common industrial approach is trial-‐and-‐ error. However, Bellgran and Säfsten (2010) mean this description is a large ex-‐ aggeration, although they agree that the use of a systematic approach in industry is limited due to several reasons.
Duda (2000) means that many industrial companies follow some kind of “design philosophy”, which guides them in their PSD process. Being a philosophy is nei-‐ ther defined explicitly, nor includes a structured methodology. However, a de-‐ sign philosophy can have profound impact on an organization and its develop-‐ ment of production systems (Duda, 2000). The most known example of a design philosophy today is Toyota Production System (e.g., Womack et al., 1991; Ohno,
1995; Monden, 1998; Liker, 2003); despite never being documented into any formal written specification, Toyota´s huge success has made competitors and researchers aware of their approaches towards PSD, they have analyzed them, and they have also imitated them (Duda, 2000).
Christmansson and Rönnäng (2003) have investigated the industrial PSD pro-‐ cess in their study of eleven Swedish companies. There they identified several deficiencies in how companies developing new production systems:
• Lack of structured methods;
• No evaluation of either the finished production system or the PSD pro-‐ cess;
• Lack of routines for knowledge transfer; and • Used methods were performing unsatisfactory.
However, these results were unexpected for the above, mentioned authors since a lot of resources had been invested in the investigated companies’ PSD projects. A conclusion to this might be that the resources were not spent on the “right things”. More resources should instead have been devoted to project time, knowledge, and tools (Christmansson and Rönnäng, 2003).
2.1.3 Production System Development Frameworks
A framework can be defined as “a real or conceptual structure intended to serve as a support or guide for the building of something that expands the structure into something useful” (Whatis, 2008). Khademhosseinieh and Seigerroth (2011) describe a framework as a structured methodology consisting of several closely linked method components. Frameworks are generally more prescriptive than a structure and thus they provide directions for structures to follow (Whatis, 2008).
In order to have a structured work method, one must ensure sufficient time is spent on the planning stage in the beginning of the project (Johansson, 2008). According to Bellgran and Säfsten (2010), a structured work method:
• reduces the time spent on structuring work procedures;
• facilitates the coordination and management of the project; and • provides opportunities for well-‐thought out system solutions.
A thoroughly planned project will make the realization phase smoother, faster, cheaper, and less troublesome. However, the use of accurately planned projects is usually lacking in industry, where it instead is more common to use an ad hoc approach (Bellgran and Säfsten, 2010).
According to Johansson (2008), the area of industrial PSD is both large and com-‐ plex, and this has made many researchers reluctant to develop their own frame-‐ works covering the whole PSD process. According to Bellgran and Säfsten (2010), frameworks are lacking when it comes to present concrete methods for optimal solutions, and they are not leading to a detailed PSD. The lack of frame-‐ works, containing the whole PSD process, has inspired many companies to de-‐ velop their own methods for how their concepts and ideas about how production systems should be designed (Duda, 2000). Bellgran and Säfsten (2010) mean that there are four integrated theories that handle the production system design:
• Design frameworks and strategies: involve manufacturing strategy, but lacks a methodology for choosing between design alternatives and thus it does not lead to detailed designs;
• Philosophies with sets of techniques and methods: certain techniques and methods supporting different philosophies (e.g., JIT, TPM, and Kaizen); • Design by philosophy: integrated approach to production system design
and is based on what constitutes a good production system (e.g., TPS); and
• System engineering: top-‐down design approach aimed at creating prod-‐ ucts, systems, and structures that are competitive.
Even if the focus on the interrelationship between products and production is important, focus within the industry is mostly directed to the product develop-‐ ment area (Bellgran and Säfsten, 2010). Bellgran and Säfsten (2010) summarize the distinctions between the development processes within the production and product development areas:
The process of developing production systems has not been, and is not, focused in the same way as the process of developing products, neither in academia nor in industry. (p. 5)
This gives a notion that the product development area focuses more on the pro-‐ cess than the production development area does. There are a lot of different frameworks that focus on the product development process:
• A framework for the product development process (Ulrich and Eppinger, 2008);
• An empirically-‐based framework for analyzing product development time (Adler et al., 1995);
• A model-‐based framework to overlap product development activities (Krishnan et al., 1997); and
• A framework to increase innovation and flexibility within the product de-‐ velopment process (Malhotra et al., 1996).
Within the product and production development areas, there are parts that are similar and thus some theoretical aspects are applicable in both fields. However, when regarding the PSD process as a whole, there are some differences, and therefore some developed frameworks might not be applicable for the other re-‐ spectively field (Bellgran and Säfsten, 2010).
Bellgran and Säfsten´s Production System Development Framework
Bellgran and Säfsten´s PSDF is built upon both theoretical and empirical data. The framework puts a lot of focus on the double task, which means the planning phase is separated from the actual accomplishment phase (Bellgran and Säfsten, 2010).
Johansson (2008) describes Bellgran and Säfsten’s framework as “almost com-‐ pletely comprehensive” and “very ambitious” compared to what is used in the industry. The framework also includes several practical aspects, rarely men-‐ tioned in the literature. One example is how the investment request needs to be considered in the early stages of the development process. Bellgran and Säfsten
also emphasize the necessity of the framework to adapt to each separate compa-‐ ny requirement in order to be as beneficial as possible.
The framework consists of three main blocks, each containing two elements, see figure 2.3.
Figure 2.3 Bellgran and Säfsten’s production system development framework (Bellgran and Säfsten, 2010)
To assist the PSDF, Bellgran and Säfsten have developed a Structured way of
working, which provides a detailed map for the PSD process, see figure 2.4. The
structure ensures that all aspects within a phase are being considered and doc-‐ umented at the end of every element.
Plan
The Plan block consists of the elements Structured way of working and Manage-‐
ment and control. The first two phases result in a document called Plan for pro-‐ duction system development, which provides the input data for the next phase,
the Preparatory design phase. It is important to remember that data in this doc-‐ ument are not static but may change as the project proceeds. However, Bellgran and Säfsten stress the importance of having some kind of control and quality as-‐ surance document as a starting point.
Design and Evaluate
The empirical studies made by Bellgran and Säfsten showed the importance of separating the Preparatory design from the Design specification. In the Preparato-‐
ry design one evaluates the conditions for developing a new or existing produc-‐
tion system. This is done by first reviewing current production systems, both internally and externally, in a Background study. After this, a Pre-‐study is per-‐ formed where focus is on the future state of the production system. It can for example be about the company strategies and which demands the production system might face in the future. The results from the Background study and Pre-‐
study are finally summarized in a Requirement specification.
The next element, the Design specification, consists of three phases. Design of
conceptual production systems is the process of generating different system solu-‐
tion systems. The solution deemed most appropriate according to the previously
established requirements is then developed and designed in detail in the De-‐
tailed design of chosen production system. Implement
Input to the next phase Build production system is the System solution. It is in this phase the realization of the production system begins. Plan start-‐up is carried out in parallel to the when the System solution is being established. The success of the Start-‐up is a direct result of how well the PSD project has managed to meet the requirements set on the production system, as well as the quality of the Plan
start-‐up phase. The final phase of this structured work method is the Evaluate the result and the way of working phase.
Context and Performance
The part Context and performance is of a different nature than the other blocks since it affects all the other three blocks throughout the whole PSD process. Empirical studies show the contextual aspects to be of such dignity, as they affect both the planning process as well as the actual development process, and this is why the part is placed separately in Bellgran and Säfsten’s PSDF.
Figure 2.4 Bellgran and Säfsten’s Structured way of working with production system development (Bellgran and Säfsten, 2010)
2.2 Usability
“Usability refers to the measure of success of the product – whether it be soft-‐ ware, computer systems or a product.” (Faulkner, 2000, p. 12) Usability is mainly mentioned within the human-‐computer interaction (HCI) area, which is a part of the product development area (Zimmerman et al., 2007). Usability consists of two usability dimensions (Han et al., 2000; Hornbæk, 2006):
1. Subjective usability measures: measures based on users’ perception of or attitudes towards the system, interaction or outcome; and
2. Objective usability measures: measures that can be obtained, discussed, and validated in ways not possible with subjective measures.
When designing and evaluating systems, these two usability dimensions are con-‐ sidered equally important (Han et al., 2000) since the overall acceptability of a system is the combination of both its social and practical acceptability (Nielsen, 1993). Furthermore, Hornbæk (2006) states that objective and subjective usabil-‐ ity measures can lead to different conclusions regarding system usability, and using both usability measures gives a more “complete picture” of the term usa-‐ bility.
Shackel (1991) means that usability is mainly mentioned as one attribute when talking about acceptable systems, as he means an acceptable system also has to be:
• functional;
• suitable for the user; and
• balanced in a trade-‐off against cost.
According to Shackel (1991), the degree of usability is directly linked to the level of system understandability. Hence, the term usability has big impact on whether a system is acceptable or not, and therefore it is essential to focus on it as a cor-‐ nerstone to system success. However, a big problem arises when trying to design for usability as usability requires skills in human factors, and it is difficult to in-‐ tegrate usability with other existing design processes (Bevan and Macleod, 1994).
Usability is context dependent (Newman and Taylor, 1999) and shaped by the interaction between tools, problems, and people (Shackel, 1991; Naur, 1992). But it is difficult to explicitly describe what features and attributes that shape usability, since features and attributes depend on the context of the system in use (Bevan and Macleod, 1994).
Users want user-‐friendly systems and it is up to the developers to produce them; if a system is difficult to use, it wastes the user’s time, causes frustration and dis-‐ comfort, and discourages further use of the system (Bevan and Macleod, 1994). However, usability is neither one-‐dimensional nor user characteristic, and that are factors that make usability particularly difficult to measure and even harder to truthfully evaluate (Bevan et al., 1991; Shackel, 1991, Nielsen, 1993; Bevan and Macleod, 1994).
According to Bevan and Macleod (1994), usability can be viewed in different ways, for different purposes, and focus on one or more of the three following corresponding views:
1. The product-‐centered view: the usability of a product is the attributes of the product, which contribute towards the quality of use;
2. The context of use view: usability depends on the nature of the user, prod-‐ uct, task, and environment; and
3. The quality of use view: usability is the outcome of interaction and can be measured by the effectiveness, efficiency, and satisfaction with which specified users achieve specified goals in particular environments.
2.2.1 Definition and Description of Usability
According to Jokela et al. (2003), the “main reference” of usability is the ISO 9241-‐11, while the best-‐known definition is the one defined by Jakob Nielsen. The ISO 9241-‐11’s definition is one part of the ISO 9241 standard that is about ergonomic requirements for office work with visual display terminals, whereas Nielsen’s definition is developed as a means within the HCI area. Thus, both the definitions are based on the development of HCI systems.
The ISO 9241-‐11 (1998) standard defines usability as: “The extent to which a product can be used by specified users to achieve specified goals with effective-‐ ness, efficiency and satisfaction in a specified context of use” (p. 2). The terms in the definition are further defined as (ISO 9241-‐11, 1998):
• Effectiveness: accuracy and completeness with which users achieve speci-‐ fied goals;
• Efficiency: resources expended in relation to the accuracy and complete-‐ ness with which users achieve goals;
• Satisfaction: freedom from discomfort, and positive attitude to the use of the product; and
• Context of use: characteristics of the users, tasks and the organizational and physical environments.
Nielsen (1993) defines usability in a more “ambiguous” way (Jokela et al., 2003) when he defines the term with five different usability attributes:
1. Learnability: the system should be easy to learn so that the user can rap-‐ idly start getting some work done with the system;
2. Efficiency: the system should be efficient to use, so that once the user has learned the system, a high level of productivity is possible;
3. Memorability: the system should be easy to remember, so that the casual user is able to return to the system after some period of not having used it, without having to learn everything all over again;
4. Errors: the system should have a low error rate, so that users make few errors during the use of the system, and so that if they do make errors they can easily recover from them. Further, catastrophic errors must not occur; and
5. Satisfaction: the system should be pleasant to use, so that users are sub-‐ jectively satisfied when using it; they like it.
Together with the ISO 9241-‐11 and Nielsen’s definitions and descriptions of usa-‐ bility, there are also other explanations (also based on the development of HCI systems) of what attributes usability consists of, see table 2.2.
Table 2.2 Usability attributes of various standards or models (modified from Seffah et al., 2006)
Shackel
(1991) Schneiderman (1992) Nielsen (1993) Preece et al. (1994) ISO 9241-‐11 (1998) Constantine and Lockwood (1999)
Effectiveness
(speed) Speed of per-‐formance Efficiency of use Throughput Efficiency Efficiency in use Learnability
(time to learn) Time to learn Learnability (ease of learn-‐ ing)
Learnability (ease of learn-‐ ing)
Learnability
Learnability
(retention) Retention over time Memorability Rememberability Effectiveness
(errors) Rate of errors by users Errors/safety Effectiveness Reliability in use Attitude Subjective
satisfaction Satisfaction Attitude Satisfaction (comfort and acceptability of use
User satisfaction
Jokela et al. (2003) state that a lot of usability efforts and challenges are directed to how to define and determinate usability requirements, rather than how to measure, evaluate, and test usability. However, they mean this to be a good de-‐ velopment since it is “through measurable usability requirements the usability work of the projects becomes more recognized and goal-‐driven” (Jokela et al., 2003, p. 59).
2.2.2 Measurement of Usability
Measurement of usability attends to make the term usability more concrete and easier to evaluate (Hornbæk, 2006). When one wants to measure usability, it is important to measure aspects that are relevant for the use of the system. Bevan and Macleod (1994) describe this as: “context of measurement must match con-‐ text of use” (p. 7). The challenge here is to “develop subjective measures for as-‐ pects of quality-‐in-‐use that are currently mainly measured by objective measures, and vice versa, and evaluate their relation” (Hornbæk, 2006, p. 92). According to Hornbæk (2006), measurement of usability has three motivations:
1. The meaning of the term usability is to a large extent determined by how one measure it;
2. Usability cannot be directly measured and uncover validity problems in how usability is operationalized and reasoned about; and
3. The majority of approaches to user-‐centered design depend critically on measures of the quality of interactive systems.
Nielsen (1993) argues the most traditional way to measure usability is to use a number of test users that fill in their answers with help of various grading scales (e.g., Likert scale). Thus, the grading scales make the answers quantifiable. Based on the result one normally takes the mean value of each of the attributes and combine these into a single usability factor (Nielsen, 1993).
The most common measures of usability are effectiveness, efficiency, and satis-‐ faction (Bevan and Macleod, 1994; Hornbæk, 2006), but there also exist other, broader measures, for example, flexibility and low rate of errors. However, Bev-‐ an and Macleod (1994) mean measuring effectiveness, efficiency, and satisfac-‐ tion across a range of contexts makes it possible to assess these broader measures, which flexibility, low rate of errors etc. are.
2.2.3 Evaluation of Usability
According to Rosson and Carroll (2002), “usability evaluation is any analysis or empirical study of the usability of a prototype or system” (p. 227). Evaluation of usability is an essential procedure in system development (Kwahk and Han, 2002), and consists of iterative cycles of designing, prototyping, and evaluating (Nielsen, 1993). Most usability evaluation methods tend to be quantitative, but in order to evaluate a system thoroughly, it is necessary to gain and evaluate quali-‐ tative measures as well (Faulkner, 2000).
There are three common activities when evaluating usability (Ivory and Hearst, 2001):
1. Capture: measuring and collecting usability data;
2. Analysis: interpreting usability data to identify usability problems in the interface; and
3. Critique: suggesting solutions or improvements to mitigate problems. Since evaluation is based on measures, usability evaluation also has to take place in an appropriate context of use (Shackel, 1991; Nielsen, 1993). In order to have a good alignment between measures and evaluation of usability, Bevan and Mac-‐ leod (1994) argue a description of the context of measurement is crucial for an evaluation report.
Wixon (2003) propose three important aspects to focus on when trying to evalu-‐ ate usability as accurately as possible:
1. Learn all from own practice;
2. Evaluate methods by applying them to real products embedded in real engineering, corporate, and political environments and not on simulated systems or hypothetical models; and
3. Adopt a case study rather than an experimental approach.
Findings from usability evaluation tend to be unsystematic and unpredictable (Ivory and Hearst, 2001). For instance, different usability evaluators studying the same user interface can come up with widely spread usability findings (Nielsen, 1993). To uncover unsystematic and unpredictable usability evaluations, one solution is to increase the number of evaluators and study participants (Ivory and Hearst, 2001).
2.2.4 Usability of Frameworks
As stated earlier, the theory of usability is mainly covered within the HCI area. However, there is nothing that says usability is important only for HCI systems. In fact, usability is important to consider for any product, system, or service (Chincholle et al., 2002).
In a way, frameworks and HCI systems are similar as they both aim to guide a user to accomplish a task. However, a HCI system is a coherent collection of ob-‐
jects (Carzaniga et al., 1998), while a framework is a prescriptive structure that guides objects in a system to perform something useful for the user (Whatis, 2008). Therefore, it is believed the use of HCI systems is more straightforward and clear compared to the use of frameworks.
Due to the strong alignment between definition, measurement, and evaluation of usability (i.e., they are all context dependent), combined with the similar charac-‐ teristic between frameworks and HCI systems, the evaluation methods used for evaluating usability in systems within the HCI area (see section 2.2.3) are be-‐ lieved to be applicable also for the evaluation of frameworks. However, to be able to evaluate usability of a framework, one needs to define what usability of frameworks means.
Within the HCI area, researchers (e.g., Shackel, 1991; Schneiderman, 1992; Niel-‐ sen, 1993; Bevan and Macleod, 1994; Preece et al., 1994; ISO 9241-‐11, 1998; Constantine and Lockwood, 1999) have comprised the term usability with words like effectiveness, efficiency, learnability, memorability, flexibility, low rate of errors, and satisfaction. Summarizing these usability terms, using a framework, it gives that a framework should:
• fulfill its objectives (effective); • be easy to learn (learnable); • be easy to use (efficient);
• be easy to memorize (memorable);
• be designed so the user cannot operate it wrong (low rate of errors); • be able to be used with other methods, tools, and systems (adaptable);
and
• be convenient to use (satisfactory).
However, low rate of errors and adaptability are something that is measured implicitly when measuring satisfaction, efficiency, and effectiveness (Bevan and Macleod, 1994). In order to evaluate usability of frameworks, it is thus believed that one can use learnability, memorability, efficiency, effectiveness, and satisfac-‐ tion, and doing this based on the following definitions:
Learnability: capability of a framework to enable a user to learn it Memorability: capability of a framework to enable a user to mem-‐
orize it
Efficiency: resources expended in relation to achieved results Effectiveness: accuracy and completeness with which users
achieve specified goals
Satisfaction: freedom from discomfort, and positive attitudes to-‐
wards the use of a framework