User and Business Value: A Dual-Stakeholder Perspective on IT Systems

(1)

User and Business Value

A Dual-Stakeholder Perspective

on IT Systems

Tom Airaksinen Erik Byström

Master Thesis in Cognitive Science

Department of Computer and Information Science Linköping University, Sweden

2007-02-08

LIU-KOGVET-D--07/04--SE Supervisor and Examiner: Stefan Holmlid

(2)

(3)

ABSTRACT

Abstract

When discussing the value of an information technology (IT) system, the most common ap-proach is to take on the perspective of either the organization introducing the IT system or the end users. The purpose of this thesis has been to study the values of both stakeholder groups in order to define what system characteristics are the most desirable from a dual-stakeholder point of view.

Through a series of contextual inquiries, interviews and questionnaires, the value perceptions of end users and IT managers at a large European rail operator were investigated. The results of the study point to a high degree of similarity in the value perceptions of end users and IT managers, although the former were generally focused on short-term value while the latter also were con-cerned about long-term, sustaining value.

The findings are applicable to practitioners wishing to take a dual perspective on IT value as well as academics looking to find touch points between usability and business strategy. In the context of end users and IT managers, the most important system characteristics that maximize value for both stakeholder groups were found to be Availability, Recoverability, Efficiency, Reliability and Future

(4)

ACKNOWLEDGEMENTS

Acknowledgements

The authors are deeply grateful to the following persons:

• Stefan Holmlid at IDA, for supervision and support throughout the thesis work • The employees at Appear Networks: Xavier Aubry, Pontus Axelsson and Karl Avedal

for allowing us to spend their time and resources during the course of this thesis. Yvonne

Lundstedt, for teaching us useful Dutch phrases and generally just being nice. Wim Peggeman and Harm Jonker, for valuable information, getting us in touch with the

right people at NS and helping out with the translation. Vincent Dollet, for the good times.

• The people at Nederlandse Spoorwegen: Bert van der Ven, for a lot of support during

our trips to Utrecht. Janet and colleagues at the NS help desk in Utrecht, for answering all our questions and letting us abuse their photocopier. All frontliners, for being so friendly to us! Edwin Heij deserves a special thank you for his never-ending patience and genuine interest in what we did. We owe you a beer or two when you visit Stock-holm, Edwin!

• Henrik Danielsson and Felix Koch at IDA, for answering our questions about the

wonderful world of statistics

• Per Chaikiat, for you-know-what

Tom is also grateful to:

• Karin, för att du är en så varm och öppen människa. Jag älskar dig. • John, lillebror har vuxit upp. Tack för stöd och inspiration – du är grym.

• Mammaåpappa. Tack för allt. (Inte minst lån när ekonomin har knakat på grund av det

utdragna uppsatsskrivandet, hrrrm...)

• Joel och Henrik. Jag har saknat er i Linköping. Tack för alla givande diskussioner de

(5)

(6)

CONTENTS

1 Introduction

In this introductory chapter the purpose and scope of the thesis are defined and the structure of the thesis is presented. The authors also give a brief description of their sponsor.

1.1 Purpose

When discussing the value of an information technology (IT) system, the most common ap-proach is to take on the perspective of either the organization introducing the IT system or the end users. The purpose of this thesis is instead to study the value perceptions of both stakeholder groups concerning the same system. This leads up to the research questions:

1. How do two stakeholders perceive value in an IT system product? 2. What system characteristics bring the most value to both stakeholders?

The questions are addressed through a case study where the values of enterprise end users are defined on one hand and the values of the enterprise IT management on the other. Values are then analyzed and connected into product characteristics that describe a desirable solution from a dual-stakeholder point of view.

1.2 Scope

The scope of the study is to analyze an Information System (IS) within a privately held govern-mental organization. The two stakeholders investigated are end users and IT managers, both em-ployed within the organization. The IS product in focus of the study is a handheld computer used by the end users as an everyday workplace tool. The system has been in use since 1995 and is currently in its third generation. This study will. to some extent, compare the current system with its predecessors, but the focus will be on the user and business values pertaining to the device and its environment as of today.

1.3 Structure

• In part 1, “Background”, the authors present a review of related research and the

theo-retical framework for the thesis, as well as background information on the organization studied.

• Part 2, “Defining User and Business Values”, addresses the first phase of the empirical

case study, where the authors studied what user and business values were involved in this particular case, and how these can be translated into desirable system characteristics for the Information System studied.

• Finally, in part 3, “Measuring User and Business Values”, the authors compare the end

users’ and IT management representatives’ opinions of the importance and fulfillment of the various values. This part of the thesis also contains a concluding discussion on the re-sults of this work.

• The additional part, “Appendices”, contains interview guides, questionnaires and various

empirical data.

1.4 Sponsor

The thesis work has been sponsored by Appear Networks, a provider of wireless communication platforms for enterprise mobility solutions. Their Appear IQ product suite makes it possible to use standard Wi-Fi networks for proactive delivery of context-sensitive, real-time information (data, VoIP and video) to mobile employees.

(11)

INTRODUCTION

As Appear IQ is used for data synchronization in the system being analyzed in this thesis, Appear Networks generously agreed to reserve office space for the authors during February–June 2006, as well as sponsor the trips needed for the on-site empirical studies. However, Appear Networks has had no influence on the results or conclusions presented in this thesis.

(12)

BUSINESS VALUE IN INFORMATION TECHNOLOGY

2 Business Value in Information Technology

This chapter gives the reader an introduction on the definition of information technology (IT), followed by an explanation on what constitutes an investment within the field of IT. The next sections first provide background on what IT value is and if there is any value in IT at all. The chapter then investigates where value can be found and how it is usually measured from a busi-ness point of view.

2.1 Information Technology Defined

Though the term Information Technology, commonly abbreviated IT, is now widely adopted with the advent of personal computers, mobile phones and Internet services, it needs to be de-fined in order to explore the vast spectrum of technologies referred to as “IT”.

As a starting point, the Cambridge University Press’ (2006) Advanced Learner’s Dictionary On-line offers a concise but wide explanation:

“[T]he science and activity of using computers and other electronic equipment to store and send information.”

Wikipedia (2006a) provides a more verbose definition:

“Information technology … is a broad subject concerned with technology and other aspects of managing and processing information, especially in large organizations. In particular, IT deals with the use of electronic computers and computer software to convert, store, protect, process, transmit, and retrieve information.”

Wikipedia’s (2006a) view on IT clearly focuses on more processes than Cambridge University Press’ (2006) Advanced Learner’s Dictionary Online although Wikipedia (2006a) also concerns computer applications.

Lucas (1999) also focuses on the computing part, describing IT as having three components: “computers, databases, and communications networks” (ibid., p. 5). He then elaborates on this brief description and explains that:

“Information technology includes [computers, databases, and communications networks] along with other devices like voice mail systems, fax machines, personal digital assistants … and similar electronic devices that promote computation, storage, and the communication of data.”

It is evident that even though IT mainly concerns computer hardware and software, the term also entails a fuzzy domain of phones and other electronic devices that in one way or another facili-tate communication.

For the purpose of this thesis, the authors will use “IT” as defined by Lucas (1999) above.

2.2 IT Investments Defined

To continue the discussion on what constitutes value from a business point of view, “invest-ment” in relation to IT needs to be defined. In accounting, there are two types of assets that may be acquired by an investment – operating and capital assets. Accounting principles may differ from country to country though in Sweden law regulates the difference between an operating and a capital asset. The difference is expressed in SFS 2000:605, chapter 5, §1:

“Capital asset refers to an asset that is designated for long-term use or ownership. Operating asset refers to other asset.” (Notisum, 2006, our

(13)

A complete overview of the accounting principles on IT investments is out of scope for this the-sis. There is however no uniform definition of such an investment according to Bacon (1992). Not all IT investments are meant for long-term use: the cost of operating an IT system is by definition not an capital investment (but an operating) while expensive infrastructure such as fiber Internet connections or servers are usually meant to be used for longer periods of time. When discussing IT investments in this thesis, the following definition by Bacon (1992) will be referred to:

“Any acquisition of computer hardware, network facilities, or pre-developed software, or any “in-house” systems development project, that is expected to add or enhance an organization’s information systems capabilities and produce benefits beyond the short term” (Bacon, 1992, p. 335-336)

2.3 IT Value Is Nowhere To Be Found?

The question of value in, or by, information technology is controversial. Though IT investments have increased enormously for the past three decades, the growth of productivity attributable to the investments is under much debate. Carr (2003) cites a study by the U.S. Department of Commerce’s Bureau of Economic Analysis stating that in 1965 IT accounted for less than 5 % of capital expenditure in American companies. In the 1980s, after the introduction of the personal computer, IT spending had risen to 15 % whereas by the end of the 1990s the figure was almost 50 %. According to Carr (ibid.), global enterprise IT spending is now more than $2 trillion every year.

Carr (2003) argues that even though organizations and executives have now realized the strategic potential of IT investments, meaning that a competitive advantage can be gained over competi-tors through means of IT, the strategic value is actually not as high as it was in the dawn of in-formation technology. He bases this claim on the notion that something with strategic value is by definition something that the competition does not have access to. IT, on the other hand, is widely available for any organization willing to make the investment. Furthermore, Carr (ibid.) points to the fact that IT has become an unavoidable cost for the business though it needs not provide any distinct returns. Value is seen as strategic: having an IT infrastructure at all is, perhaps mistakenly, thought to have a value in its own respect. Carr’s opinion of uncertain returns on IT investments is backed by other research (e.g. Strassman, 1997) showing that organizations seldom perform post-evaluation studies of their IT investments. Thus, most businesses do not know whether their IT investments have paid off – neither in terms of money nor by other means.

Comparing IT with earlier technologies that made a great impact on the industry and society in the 19th_{century onwards such as the steam engine, the railroad, the telephone and the electric} generator, Carr (2003) presents his argument. At first, each of these technologies provided huge advantages for those who made use of them but eventually became ubiquitous and did not matter strategically. The same thing has happened to IT, Carr (ibid.) concludes.

To back his line of reasoning, Carr (2003) makes a distinction between proprietary technologies and

infrastructural technologies. The former has the characteristics of being exclusive to the company

putting it in use – a patented technology for instance. The strategic advantage of the technology remains as long as it is protected from the use of others. Infrastructural technologies on the other hand become more valuable as their usage is increased. Here, Carr (ibid.) makes a comparison with railways: an individual company might benefit some from having its private railroads to sup-pliers and customers but for the economy as a whole, an open rail network available for all would increase total value.

(14)

characteris-BUSINESS VALUE IN INFORMATION TECHNOLOGY

that IT is a transport mechanism and thus more valuable when it is shared. Business IT has evolved from isolated mainframe computing to personal computing and from local area networks to the Internet. As interconnectivity and interoperability improved, the costs of building proprietary and isolated applications and infrastructure exceeded the benefits of using open networks and stan-dards (Carr, ibid.). The second characteristic is that IT is highly replicable, both in terms of software applications and the business processes around it. As processes become standardized in software rather than created by the company itself, the full business advantage provided by IT can also be replicated by other organizations. As a consequence, no value is to be found within the process advantages that IT provides.

2.4 Applied Right, IT Value Is To Be Found

There are alternatives to Carr’s (2003) point of view though. Brown & Hagel (2003) answer Carr that while it may be true that the strategic value of IT is overestimated it is not a mere commodity (ibid.). They write that even though IT does not by itself provide strategic differentiation, it pro-vides previously unavailable possibilities. Brown & Hagel (ibid.) also points to three broad les-sons to be learned by IT investments so far.

The first lesson to be learned is that “extracting value from IT requires innovations in business practices” (Brown & Hagel, 2003, p. 110). Brown & Hagel (ibid.) points to the fact that businesses cannot implement IT solutions without changing their practices for exploiting the capabilities that IT provides. Not changing practices will destroy IT’s economic value and this happens all too often. Opportunities provided by IT must be actively explored and companies without such exploration processes will not benefit from IT investments as much as those who do.

Brown & Hagel (2003) concludes that companies generally do not see the full possibilities of IT. The companies believe that information technology can reduce transaction costs but then view the transactions costs themselves merely as the transfer of data. This is a flawed assumption Brown & Hagel (ibid.) points out, as there is also a need to create trust for the technology and build knowledge around it among other things – all which are part of the transaction costs. Not understanding this will likely create problems in terms of low acceptance of the technology and lower returns for the business. In total: low value.

The second lesson to be learned is that “IT’s economic impact comes from incremental innovations rather

than ‘big bang’ initiatives” (Brown & Hagel, 2003, p. 110). A “big bang” is a large effort to transform

the company quickly and in a profound way. This, writes Brown and Hagel (ibid.), does seldom create any returns at all; such initiatives destroy value rather than create it. Instead, companies should aim for 6- to 12-month projects to innovate in specific business processes rather than try to transform everything at once. Managing a smaller project will also facilitate measurement of performance. Moreover, Brown and Hagel (ibid.) also argues for the fact that an incremental, focused approach provides near-term rather than long-term returns and thus helps get organiza-tional support from management.

Lesson number three is that “the strategic impact of IT investments comes from the cumulative effect of

sus-tained initiatives to innovate business practices in the near term” (Brown & Hagel, 2003, p. 110). Brown’s

and Hagel’s (ibid.) opinion is that strategic differentiation emerges over time around the evolving capabilities of IT rather than from a specific business practice innovation. They refute Carr’s (2003) comparison of IT infrastructure with railways, electricity, steam engines and telephones by saying that the older technologies were developed in a one-shot fashion thus relatively quickly

(15)

providing diminishing returns. A dominant design1_{also emerged in all of these technologies} giv-ing birth to standards within the different fields.

In contrast, IT is constantly evolving in all areas. A generally dominant design has not appeared; IT has rather gone through several architectural generations and are continuing to do so. The on-going evolution of IT gives new opportunities for innovation. Also, even though the components that make up the technology are becoming widespread commodities, knowledge of how to trans-form the components into value-creating systems is still in short supply. There is still a gap be-tween the opportunities that IT offers and the businesses’ ability to realize that potential. Brown & Hagel (2003) summarize their argument by stating that IT has never mattered isolated. Value is provided through innovation in the new possibilities that the technology brings.

2.5 So If There Is Value, What Kind?

Even though “value” in the context of business might imply monetary benefits from invest-ments, IT management literature does not make the two synonymous. Lucas (1999) writes that:

“[v]alue is not synonymous with [return on investment, ROI]; there are more kinds of value than a measurable financial return on investment.” (Lucas,

1999, p. vii).

He also states two key aspects in his book, first that:

“not all investments in IT should be expected to show a measurable return”

(Lucas, 1999, p. 9) And secondly:

“investments can have value to an organization even without demonstrable financial return” (Lucas, 1999, p. 9)

Citing a personal experience anecdote to make his argument stronger, Lucas (ibid.) mentions that when asking an IT manager what he would do when he felt strongly about a solution but could not show enough benefits, the manager had replied that he just would go on adding intangible benefits until the investment was justified.

A class in IT management that Lucas (1999) had taught once featured a panel of IT professionals where one of the panel members explained why a pure financial analysis was not sufficient. The member, a consultant, had been involved in a very successful telephone company product tar-geted at consumers. When he asked his manager what the IT component in the project was worth, he was told that it was worth nothing. The manager also told the consultant that the ad-vertising on its own was worth nothing at all. Instead, it was the combination of the information technology component and the marketing that had made the investment pay off. Thus, Lucas (ibid.) concludes, value in IT can be very hard do discern as it is today integrated with other ef-forts and processes and does not show much or any value by itself. This view is in line with Brown’s & Hagel’s (2003): IT is merely a platform where new value can be created and used in-novatively that value is a fact.

2.6 Finding and Measuring the Business Value of IT

Seddon et al. (2002) studied 80 senior managers and asked how they evaluated the benefits from IT in their organizations. Within the study, Seddon et al. (ibid.) reviewed IT managers’ approach to measuring value in the overall IT portfolio, individual IT projects and the IT function itself.

(16)

The study focused on a management stakeholder perspective and concluded the most important criteria to value the benefit from an IT system:

1. Support of explicit business objectives 2. Internal Rate of Return

3. Support of implicit business objectives 4. Net Present Value

The first criterion, support of explicit business objectives, means that value stems from how well a certain system fits into specific goals set for the organization or business unit.

The second most used criterion, Internal Rate of Return (IRR), is a measure to show the highest possible interest rate an investment can take before becoming unprofitable from a financial point of view. The IRR calculated is the Net Present Value (NPV, see below) that makes all future cash flows equal zero. Because there is a large body of investment literature that describes the pros and cons of IRR and other measures (see e.g. Ekdal, 2005), the authors of this thesis will not elaborate further on the subject.

Support of implicit business objectives is fulfillment of goals that are not explicitly in writing but perhaps instead in the “company spirit” or minds of senior executives. Measuring something implicit may sound impossible, but if an implicit objective is a quantifiable one – e.g. a certain yield on invested capital – it may very well be easier to measure than an explicit and possibly in-tangible objective.

The fourth criterion was another financial measure: Net Present Value (NPV). NPV is the sum of all future cash flows from an investment calculated with a given interest rate. Formally, NPV is defined as:

!

= " + = N t t t _C r C NPV 1 0 ) 1 ( Equation 2.1

In Equation 2.1 above, N is the lifetime of the investment object in years, r is the cost of capital (interest rate), Ct the cash flow at a given point in time and C0 the initial cash investment. A posi-tive NPV equals a profitable investment from a financial point of view because it will over N years generate a higher cash flow of revenues in than of payments out. (Wikipedia, 2006c)

In summary, management stakeholders valued IT benefits from a mix of “hard” financial meas-ures and “softer” organizational specific objectives. It is worth noting that long-term value – match with explicit business objectives – was ranked higher than financial strength of the project.

2.6.1 Effectiveness as Value

Value is defined as effectiveness by certain authors, e.g. Seddon et al. (2002). Effectiveness means “having a definite or desired effect” according to The Oxford American Dictionary of Current English (2006a). Effectiveness has proven very difficult for organizations to measure as pointed out by Seddon et al. (ibid.) as well Lucas (1999). In addition, Seddon et al. (ibid.) found that about 40 % of the companies in the study did not perform a post-implementation evaluation at all. Bal-lantine et al. (1999, cited in Seddon et al. 2002) studied why IT projects were not evaluated from a feasibility perspective. They found that the single most common problem was to quantify rele-vant benefits, something encountered in 81 % of the evaluations. The second most common problem found in 65 % of the projects was to at all identify the relevant benefits (ibid.). Clearly, effectiveness from a business perspective is not easily determined, much less measured.

(17)

When asking IT managers what slowed or halted evaluations of IT value, Seddon et al. (2002) found that the five largest inhibitors were:

1. Inability to identify intangible benefits 2. Inappropriate measures

3. No satisfactory metrics available 4. Financial focus of organization 5. Poorly defined IT deliverables

The top three have clear connections to measurement and are consistent with the findings of Ballantine et al. (1999): organizations do have difficulties measuring the value of their IT invest-ments.

2.6.2 Assessing the Potential Value of IT

The IT manager views on what criteria make sense for evaluating IT project feasibility that Sed-don et al. (2002) found were ranked from most to least important:

1. Strategic Match with the Business 2. Satisfaction of Customer Needs 3. Productivity Improvements 4. Traditional Cost Benefit 5. Return on Investment (ROI)

The criterion of strategic match with the business means that the outcome of the project should match business goals in one way or another. This may very well be an intangible and subjective measure but nonetheless important as the study showed.

Satisfaction of customer needs is straightforward: there must be clear need from a customer – internal to the company as well as external outside the company. If nobody has a need of the proposed solution, there is no point in implementing it according to this criterion.

Productivity improvement benefits are a quantifiable criterion contrary to the first two, usually expressed as rate of output per unit of input needed (Oxford University Press, 2006b). Improving productivity is beneficial for a company because it over time lowers cost and improves ability to compete.

Cost-benefit analyses try to capture and measure both tangible and intangible benefits and costs. In its most basic version, the net sum of benefits less the costs will show if the investment will yield a positive return or not. Quantifying benefits and costs, intangible in particular, might be very difficult and thus it might be worthwhile pursuing various techniques such as surveys and market indicators to pin down values. (Wikipedia, 2006b)

Return on Investment (ROI) is a percentage describing the benefits less costs in relation to the investment. The formula in its most basic form is (adapted from Robb & Pfefer, 2003):

!

ROI = B " C I

Equation 2.2

Where, in Equation 2.2, B is the total benefits of the investment, C is the total costs and I is the original investment.

(18)

Net Present Value and the Payback Method2_{were ranked as number four and five, making three} out of five project selection criteria financial. Number one and three in the ranking were that the project must support explicit and implicit business objectives correspondingly.

In addition to the measurements mentioned in the study by Seddon et al. (2002) and Bacon (1992), a common IT value measure is Total Cost of Ownership (TCO). TCO was developed in 1987 by the consultant firm Gartner Group and defined as:

“… a holistic assessment of IT costs over time. The term holistic

assessment implies an all-encompassing collection of the costs associated with

IT investments, including capital investments, license fees, leasing costs and services fees, as well as directed (budgeted) and indirect (unbudgeted) labor expenses.” (Gartner, 2006, their emphasis)

If choosing between two or more investments the investment with the lowest TCO, ceteris paribus, is the most favorable from a financial point of view. The foundation of the TCO measure is two types of costs: direct and indirect. Direct costs are usually labor and capital costs, i.e. salaries and the cost of investing in a certain technology. Indirect costs are costs that are hard to pin down and therefore harder to quantify. Their nature may be “soft” rather than “hard” and thus found in employee satisfaction rather than e.g. employee productivity when looking for costs pertaining to lack of either (Gartner, 2006). Indirect costs are far from dismissible though, Gartner (ibid.) estimates that they represent as much as 60 % of total costs for management and ownership of IT. In terms of TCO, Gartner (ibid.) further identifies indirect costs as costs that are an effect of direct costs decisions or misguided decisions that causes indirect costs. A relevant example of the latter would be a management function trying to lower the direct cost of training in a software package by cutting the number of training hours each employee gets. As a consequence, users might become less competent and familiar with the software package and are less productive and satisfied with their work environment. The key, according to Gartner (ibid.), is to make sure that direct costs does not get cut at the expense of increasing indirect costs – the latter might emerge at another part of the organization who suffers from the attempt to save money.

Lucas (1999) makes the distinction between direct and indirect in IT investments as well, but he focuses on benefits rather than costs. Indirect benefits are defined “a second-order effect of the technology, and often these benefits are unanticipated when the original investment is made” (ibid, p. 81). As the benefits are unanticipated, they might by nature be hard or impossible to cap-ture in a cost-benefit analysis. This is not a problem when measuring effectiveness using TCO as the TCO measure concerns costs alone rather than a comparison between benefits and costs.

(19)

VALUE FROM A USABILITY PERSPECTIVE

3 Value from a Usability Perspective

In this chapter, the evolution of the human-computer interaction field with respect to the in-tended value and goals of usability work is described.

3.1 Classical Views

In the late 1970s and early 1980s, computers turned into tools, which were supposed to be used by ordinary office workers, whereas in the early days of computing they were used exclusively by engineers (Preece et al., 2002). A new research area, software psychology, was born. Its goal was to adopt the technology to new user groups, using scientific methods from experimental psy-chology. Practitioners studied people interacting with computers in carefully controlled labora-tory experiments and tried to develop general theories of human cognition and action, that ideally could be used to predict people’s behavior in front of the screen.

The value of this would in turn be the ability to create interfaces that are simple, efficient, and error-free. (Löwgren & Stolterman, 2004.)

3.1.1 Usability Engineering

After a while it became clear that the software psychology approach turned out to have very little, if any, practical impact on the growing software development industry with its demands on cost-efficiency. Thus, software psychology was soon replaced by another approach: usability engineer-ing. This approach was inspired by the methods of software psychology, but its ultimate goal was “good enough” information rather than confirmed statistically significant results from laboratory experiments. (Löwgren & Stolterman, 2004.)

However, usability engineering is still very concerned with similar values: quantifiable and very specific measures, such as how many times per hour of use the user presses the wrong button or how long it takes a novice user to delete an entry from a specific date in a calendar application. (Preece et al., 2002).

Due to this, critics point out the risks of projects focusing on evaluating superficial parameters that can easily be measured in a laboratory environment, but might have little relevance in the real world. This could lead to more important matters being ignored. (Löwgren & Stolterman, 2004.)

3.1.2 Heuristics and Guidelines

Another approach that has been, and still is, popular due to the costs of performing usability work is different types of guidelines. The aim of these is to provide assistance in evaluating or designing a system to make sure it is usable. Two of the most widespread sets of guidelines are Donald Norman’s design principles (Norman, 1988) and Jakob Nielsen’s usability heuristics (Nielsen, 1994).

Norman’s (1988) design principles include e.g. the following:

• Visibility – If functions are clearly visible, users are more likely to find and understand

how to use them.

• Feedback – Related to visibility, feedback tells a user what has happened after performing

an action such as pressing a button. This in turn simplifies both understanding and per-formance.

• Constraints – By restricting what actions can be performed with a system, the user is less

(20)

• Mapping – Mapping refers to the relationship between controls and their effects in the

world. If the controls for burners on a stove are arranged in the same patterns as the burners they are controlling, this is an example of good (or natural) mapping.

Some examples from Nielsen’s (1994) usability heuristics are:

• Visibility of system status – Users should always be able to tell what is going on.

• Consistency and standards – Platform conventions regarding wordings, actions et cetera

should be followed.

• Error prevention – Designers should strive for preventing problems from occurring. • Flexibility and efficiency of use – Expert users should be provided with accelerators and

ways to tailor frequent actions.

• Aesthetic and minimalist design – Dialogues should not contain more information than

necessary as the relative visibility of important information diminishes when extra infor-mation units are added.

Although Nielsen (1994) mentions aesthetics, both his and Norman’s (1988) guidelines in essence boil down to the same kind of good old usability values that we have seen before: basically im-proving efficiency and minimizing errors.

3.2 Towards More Context

One common aspect of all the abovementioned approaches is that they are concerned with a single computer (or other artifact) being operated by one single user. As we will see in the follow-ing section, this has drawn a lot of criticism in the last 10–15 years (Holmlid, 2002).

Another criticism that has been directed towards the traditional views on usability is that they are too concerned about the product itself and general usability criteria. They don’t take the individ-ual usage situation into account. They are also too focused on the user interface, and ignore the effects and utility of the system as a whole. (Löwgren & Stolterman, 2004.)

3.2.1 Quality-In-Use and ISO Definitions

The importance of usability has also been recognized by the International Organization for Stan-dardization, ISO, and this is primarily reflected in the ISO definitions that concern the quality of information systems. Bevan & Azuma (1997) contrasts some of them. In ISO 8402 quality is simply seen as the presence of specified features. ISO/IEC 9126 categorizes the attributes of software quality as functionality, efficiency, usability, reliability, maintainability and portability. However, these standards are very product centric and there has been a growing understanding of the fact that assessment of quality is dependent on the perception of the user. Different users have different needs and consequently the ISO standards should reflect this as well in order to be useful. (Bevan & Azuma, 1997.)

To solve this, the ISO/EIC DIS 14598-1 standard introduces the notion of Quality in use, which is defined as “the effectiveness, efficiency and satisfaction with which specified users can achieve specified goals in specified environments” (Bevan & Azuma, 1997, p. 170). A similar approach is taken in ISO 9241-11. Examples of effectiveness measures are the accuracy and completeness with which the specified goals can be achieved, measured as e.g. the number of spelling mistakes made when transcribing a document. Efficiency is seen as the effectiveness divided by the task completion time. Satisfaction is seen as harder to measure, but Bevan & Azuma (1997) suggest using an attitude rating scale such as SUMI (Software Usability Measurement Inventory), a stan-dardized questionnaire consisting of 50 statements like “This software responds too slowly to

(21)

When discussing possible applications of the standards, Bevan & Azuma (1997) point out that they can be used to evaluate how different components of the working environment might affect the quality of the product in use. They name a few examples that could be considered, among them the amount of user training and possible re-organization of tasks.

Looking at the values, the later ISO definitions in other words take a step away from the product, requirements and functional centric view of quality and take context and organizational issues into account at a much higher degree, but still resemble the usability engineering approach dis-cussed earlier. The important addition is that “user satisfaction” now is seen as a value in itself.

3.3 Looking Beyond End Users

After decades of research and a growing body of evidence that points to the fact that usability work indeed is needed, practitioners still have a hard time convincing companies to allocate time and budget for persona creation, prototyping, user testing and other activities. (See e.g. Bloomer & Croft, 1997; Markensten, 2005; Rosenberg, 2004.) How could this be?

One explanation is that usability work traditionally has been very focused on the end users’ wants and needs; probably even too focused. Markensten (2005) points out that “...in commercial set-tings, it is not only the user who judges what qualities are important, but also the purchasing or-ganization” (p. 12).

Löwgren & Stolterman (2004) recognize that there are situations in which it is hard to identify the user. When designing a car sales system, is the “real” user the sales person or the customer who is looking for a car? Should the designer focus on supporting the sales staff, or is it more important for the company to keep customers happy at the cost of perhaps more work for the staff?

Holmlid (2002, p. 12) points out other limitations of usability work:

Usability presumes that the single system is used by one uninterrupted user knowing her task, utilizing the system to fully or partially solve it in an as efficient and effective manner as possible.

Usability focuses on the features of the system, not on the behavior of the user, nor on the interaction of the artifact with the user’s formal and informal organisation.

[---]

Thus, usability runs into problems where  has become an integral part of the business process. Several consecutive or simultaneous users, several

simultaneous applications, as well as deliberately choosing an ineffective manner of solving the task, are all part of what usability can’t handle.

Due to this, it might be necessary for the HCI practitioners to take a step further and look be-yond the end users and instead focus on the use. Arvola (2004, p. 24), discussing the difference between user-centered design and use-oriented design, explains:

[User-centered design] focuses on users, while [use-oriented design] includes the change of practices and business, and the perspective of the procurer of the interactive system is as important as that of the user (see for example Artman, 2002).

3.3.1 A Broader Perspective: Use Qualities

How can the limitations of usability work be overcome, and how can use rather than end users be put in focus? A theoretical concept that is gaining popularity in the HCI community is use

(22)

until the system interacts with its environment, i.e. until the system is really being used by some-body (Arvola, 2004).

Therefore, the focus of the design should be on the actual use of the system, not on the system itself and its features. Thus, lists of general heuristics and guidelines, intended to be universally applicable (see section 3.1.2), are of less value. One example is Grudin (as cited in Kaptelinin, 1996a), who has studied how designers violate principles of logical consistency to make systems more usable. The fact that a task requires lots of time to solve can also be a quality in itself, for instance in a game. (Löwgren & Stolterman, 2004.)

In this sense, even the ISO definitions and the concept of quality-in-use adopted in them seem too limited to capture the whole essence of systems that are valuable to the user. Value is not simply the sum of effectiveness, efficiency and satisfaction.

Also, with the development of the views on usability, the number of items on lists of heuristics or guidelines has grown very fast – now HCI practitioners have to consider not only efficiency, learnability and other traditional usability goals, but things like entertainment, aesthetical pleasure and motivation as well. It goes without saying that this is a daunting task to handle for the poor designer. Therefore, another point with the use qualities concept is that not all of those goals are equally relevant for every system, and to meet them all in one single system is virtually impossi-ble. (Arvola, 2004)

What, then, are use qualities and what makes them different from the traditional usability goals? Firstly, we would like to stress that many of the traditional usability concepts still hold true and are important to consider in a lot of cases. The problem is they are not enough. Arvola (2004) states that “they help in making different qualities of systems-in-use visible and they allow us to compare products by discussing their properties, but other aspects of systems-in-use are also important and need to be highlighted.” (p. 10).

Löwgren & Stolterman (2004) give some examples of what these other aspects – the use qualities – might be:

• Plasticity could be important when working with a large data set. If the system allows the

user to manipulate the data and get immediate, almost tactile, responses, this accelerates the perceive-think-react loop and encourages discovery.

• Control and independency are two extremes on the same scale. A system that is characterized

by a strong feeling of control is little more than a tool; an extension of the hands or the eyes of the user. An agent, like the animated Office assistant in the Microsoft Office package, on the other hand, is characterized by a large degree of independency; the user does not have to actively bring it up. Rather, the agent sits in the background and tries to make intelligent assumptions about what the user wants to do.

• Flow is a use quality that tries to capture the fact that digital artifacts, especially mobile

de-vices, today often are used in situations where one cannot focus solely on the artifact and the interaction with it. Rather, people tend to shift focus between the artifact and the en-vironment. These focus shifts should ideally proceed as smooth as possible, without dis-turbing the “flow”.

• Personal contact deals with communication, with getting contact and staying in contact in a

way that is meaningful to the user. This is contrasted with the more technical term nectivity, that focuses on the connection as such instead of on the reasons for the con-nection or whom a person is connected to.

• Efficiency is one of the most classical usability goals, and concerns the ability of supporting

(23)

typi-VALUE FROM A USABILITY PERSPECTIVE

cally when a user is driven by external motivation (such as when an office worker is given a task by his or her superiors).

• Elegance could be seen as a quality of artifacts that are simple yet powerful. This is

con-nected to the idea of functional minimalism, which implies that an artifact should offer a relevant set of core functions but nothing more. A counterexample is a web browser that also offers (poor) tools for HTML editing and maybe (even worse) tools for photo edit-ing.

Holmlid (2002) has derived a number of other use qualities from empirical studies in a bank envi-ronment, e.g.:

• Surprise and confusion are traditionally regarded as undesirable, but in learning situations

they can create opportunities for reflection and learning and thus be highly valuable.

• Ante-use is about what precedes the actual use of an artifact. This quality deals with

as-pects such as planning; if the user knows (or exas-pects) that a certain task will take a long time to complete in a certain application, she might choose to e.g. take a note on paper instead.

3.3.2 Value-Centered HCI

An interesting, but still underdeveloped, framework that also takes on this wide perspective on HCI and usability is Cockton’s (2004a; 2004b; 2005) Value-Centered HCI. Cockton argues that the focus of HCI work must be to deliver value in the world. Intended product value is more impor-tant than quality in use or fit to context.

Cockton recognizes the benefits of the context-centered approaches that have developed since the 1990s, but claims that we need to look beyond them. For instance, like their predecessors they lack a “noise filter”. It is fairly easy to expose user difficulties, but “user testing runs the risk of drowning development teams in fat reports of irrelevant and/or trivial user difficulties” (2004b, p. 152). Also, he claims that current HCI methods are restricted to user experience with a system’s implemented functionality and cannot tell us anything about what is missing.

This, however, might not be really fair to the HCI community. See for instance the discussions on prioritizing usability problems in Lindgaard (2003). We also do not see the validity of the ar-gument that HCI cannot investigate what is not there. To us, seeing in what ways a system does not support a user due to missing functionality has been an important goal of HCI work for a long time.

Although both Cockton’s arguments can be questioned, his proposed focus on value seems promising. But what does he mean by “value”?

Value can take many forms. It is not solely a question of capitalist profits and sales. It can be political, personal, organisational, cultural, experiential or spiritual. Our role should be to understand what is valued by a system’s stakeholders and support them in delivering this value. (Cockton, 2004b, p.

155)

As an example, Cockton uses a van hire web site. It has to deliver value to three groups of stakeholders: site/company managers and investors, customers and operational staff at the de-pots. Different things constitute value to the different stakeholders.

For management, the main source of value could be return on investment, brand image and market share. For customers, value comes from the ability to hire an appropriate van for a suitable period for an acceptable cost (money,

(24)

Cockton stresses that a single artifact, in this case the web site, seldom can deliver all kinds of desired value. For instance, some factors beyond the web site are what kinds of vans the com-pany offers, the cost of hiring them and when the pickup depots are open. HCI work should concentrate on making sure that the web site protects existing value, and if possible also adds value through site features.

For the customers, this could mean focusing on obstacles to purchase. The web site should make sure that customers can find a car that meets their needs, that they understand how to get to the depot where they can pick it up and that they can see what the total price will be.

For the staff at the depots, the site should ensure that customers turn up when and where they should and also bring all the required documents, such as ID etc.

For the managers, the site should generate at least the expected amount of sales but preferably also exceed it.

With this in mind, it is easy to see how value-centered HCI requires a very broad view of system design and evaluation. Another example of Cockton’s (2004a) is central heating controllers. These are used to program heating boilers to activate only during specific periods (to save money). The key issue is that nobody has any intrinsic desire to program such controllers. If en-ergy resources were unlimited and available for free no one would use them.

In other words, even if such a controller is easy to learn and use it might miss the point. The real value of a central heating controller, and what one is interested in, is to see how much money and fuel one has saved by setting a particular program, and Cockton points out that he has still not seen a single controller that tells him this.

3.4 Financial Value of Usability

Since the 1990s, there have been several attempts within the HCI community to investigate the relationship between user and business values and to quantify the financial benefits of usability work, spurred by the difficulties HCI practitioners are facing (see section 3.3).

Often, the outcome has been general and rather vague claims of the business benefits of usability. Some frequently noted benefits (e.g. Bloomer & Croft, 1997; Dray, 1995; Lund, 1997) are higher employee productivity, reduced number of errors, lower costs for support and training and re-duced development costs (since developers will “get it right” from the start).

A more ambitious effort has been undertaken by Bias & Mayhew, who in 1994 published the first edition of Cost-Justifying Usability, now considered a “landmark book” within its field (Rosenberg, 2004). Last year, the second edition, Cost-Justifying Usability – An Update for the Internet Age (Bias & Mayhew, 2005), was published.

(25)

Both editions are full of figures, statistics and claims, most of them backed up by references to various articles or case studies. Some typical examples are (Bias & Mayhew, 2005, pp. 20–21):

“Sun Microsystems has shown how spending about $20,000 could yield a savings of $152 million. Each and every dollar invested could return $7,500 in savings” (Rhodes, 2000).

[---]

“You can increase sales on your site as much as 225 % by providing sufficient product information to your customers at the right time” (User Interface Engineering, 2001).

[---]

“Inadequate use of usability engineering methods in software development projects have been estimated to cost the US economy about $30 billion per year in lost productivity” (Landauer, 1995).

The general approach suggested by Bias & Mayhew (2005) to prove the business value of usabil-ity is to lay out a usabilusabil-ity engineering plan for the work to be performed and estimate the costs for executing it (mainly by multiplying the hourly rates of the people involved – usability experts, developers, managers etc. – with the estimated time required).

Then the benefits are estimated. Benefits related to productivity improvements are typically cal-culated by estimating a time saving (e.g. 5 seconds) for a typical task (e.g. performing a database query). Then this time saving is multiplied by the number of times the task is performed each day, the number of working days per year, the number of users and the users’ hourly rate. This way, the monetary savings related to productivity improvement can be estimated. Benefits due to other factors, such as decreased errors and decreased training, are calculated in a similar fashion. Finally, the estimated costs and benefits of the intended usability work are compared for the first year and for the expected application lifetime (typically 3–5 years). Thus the ROI (return on in-vestment; see section 2.6.2) of the usability work can be calculated.

ROI is also clearly the most favored metric for the business benefits of usability (Rosenberg, 2004). A quick look in the index of Cost-Justifying Usability – An Update for the Internet Age (Bias & Mayhew, 2005) reveals 37 occurrences of ROI, several of which span ten pages or more. This could be compared to a meager 2 occurrences for e.g. TCO (total cost of ownership; see section 2.6.2), which is only mentioned very briefly in the introductory chapter of the book.

3.4.1 Is Usability ROI Enough to Convince Management?

So ROI is the most popular economical metric of usability work, but is it necessarily the best metric for convincing other stakeholders of the virtues of usability and catering to the needs and values of end users? Rosenberg (2004) argues that this is not the case. He acknowledges that clas-sic ROI arguments “may carry weight in some areas of the [software] industry” (Rosenberg, 2004, p. 24) but claims that the HCI practitioners seem to be overly focused on measuring ROI.

His main objections against ROI are the following:

• Contrary to what HCI practitioners tend to believe, there are not a lot of empirical case

studies with detailed financial data supporting usability from an ROI perspective. Rather, the same old studies are cited over and over again.

(26)

• Often, ROI claims are very dubious and Rosenberg (2004) points out that many of them

can’t be taken seriously. Consider the following example, which is a parody by Rosenberg (ibid., p. 27), but still not too far from what Rosenberg found when searching for “usabil-ity ROI” on the Internet:

Consider that there are one billion users on the Internet. Assume that only half of them come to your site, but, of these, 80 percent leave without buying. If the average cost of an abandoned shopping cart is $20 you will lose $8 billion a year in sales. With a $5,000 heuristic review, you could reduce abandoned shopping carts with at least 50 percent, thus increasing annual revenues by $4 billion, an ROI of 80,000,000 percent.

• The usability ROI literature does not take into account the opportunity costs of investing

in usability as opposed to other things, which might have even more impact. As an exam-ple, Rosenberg (2004) mentions the DHCP protocol. When it was introduced, it suddenly became possible for people without networking skills to install a computer on an existing network without manually configuring things like IP addresses and domain name server locations.

• In addition to this, the ROI arguments are “at best tactical in nature, while the business

challenges the upper management focuses on are inherently strategic” (Rosenberg, 2004, p. 28). As discussed in section 2.6, this is indeed backed by the findings of Seddon et al. (2002), who found that “Strategic match with the business” was the most important crite-rion when IT managers evaluate IT project feasibility.

Instead of ROI, Rosenberg (2004) proposes that TCO be used as a metric for the business im-pact of usability work. As many vendors in the software business rely on a long-term relationship with their customers, product lifecycles (including field-testing, production deployment and up-grades) can last as long as ten years. This means that the operational costs can easily outweigh the cost of the original purchase. Due to this, Rosenberg (ibid.) claims that customer TCO is one of the most accurate measures of both success and competitiveness. It is also a measure that tech-nology executives understand, and that has been thoroughly studied by Gartner and other analyst firms that executives trust. Additionally, it measures the global end result of an IT system rather than an “isolate intermediate step in the product lifecycle process” (ibid., p. 29), i.e. an investment in usability improvements.

3.4.2 Other Business Benefits from Usability Work

Lindgaard (2003) claims that most of the work done in the HCI community with regard to finan-cial justification of usability has focused on estimating costs and projecting benefits from discov-ery of usability problems. Practitioners seem to miss the importance of business related data that is not directly related to e.g. the user interface of a product, but can be uncovered using investiga-tive HCI methods such as field studies and interviews with users.

In another paper, Lindgaard (2004) gives an example of how serious bottlenecks were found in a grocery distribution centre when a researcher performed a field study instead of just reviewing the user interfaces of two competing hands-free speech devices, which the management intended to introduce to improve staff productivity.

It turned out that the problems at the distribution centre had very little to do with the usability of the devices the warehouse staff was using. Instead, the problems were related to the organization of the work as well as to flawed assumptions in the back-end Warehouse Management System that calculated scheduling and truck routes.

(27)

STUDYING THE USE OF INFORMATION TECHNOLOGY

4 Studying the Use of Information Technology

As we have seen, there is a need for taking context and actual use into account when designing or evaluating information systems. But context is a vast notion, so exactly what should be studied and how can this be done? A framework for studying context and work practices that has gained increasing interest since the 1990s (see e.g. Kaptelinin, 1996b) is activity theory. In this section, we describe this framework and explain how it could be combined with approaches from Con-textual Design to get a good understanding of the context in which work practices take place.

4.1 Activity Theory

Activity theory is a framework for studying various forms of human practices. It has its roots in the classical German philosophy, which emphasized the active and constructive role for humans, but the term originated within Soviet cultural-historical psychology, founded by Vygotsky, Leon-tiev and Luria. (Kuutti, 1996).

Many psychological theories concentrate on studying human action, which makes it easy to set up laboratory experiments. However, the activity theorists argue that actions are always situated into a context and can not be understood in isolation. Therefore actions are not enough for analyzing real-life situations.

In activity theory, the basic unit of analysis of human behavior is instead activity. Kuutti (ibid. p. 27) defines activity as “a form of doing directed to an object”, whereas Bødker (1993, p. 2) de-fines it as “what a group of people engage in to achieve a certain purpose”.

Activities can be broken down into actions, which in turn, are performed as a serious of operations triggered by the situation at hand. Bødker (1993) claims that activity, action and operation can be separated by asking why something takes place (activity), what takes place (action) and how it is carried out (operation).

Another important point is that actions are conscious and have a focus – they are often directed towards an object, and different actions may be undertaken to meet the same goal (Nardi, 1996). Operations, on the other hand, are automated and can be performed without thinking (Bødker, 1993).

A sample activity from Bødker’s (1993) case study of the Danish National Labor Inspection Serv-ice is distributing cases between the inspectors. For this, work planners need to perform the goal-directed action of accessing relevant statistics, which in turn implies that they perform the operations of using a database to look up how many cases the individual inspectors are assigned to.

A central tenet of activity theory is the concept of artifacts. Artifacts are tangible (like tools) or intangible (like norms or division of work) things that are created by humans, and mediate the relations between people and objects or other people (Bødker 1993).

A classical example of mediation is a carpenter driving a nail with a hammer (see e.g. Polanyi, as cited in Ehn, 1988). The hammer is not an object of the carpenter’s attention, but an instrument of it. Although he is aware of the hammer, his focus is on the nail he is driving. The driving of the nail is thus mediated by the hammer. According to Bødker (1996), this is an example of an artifact that works well in our activity. However, if the hammer would break or behave in an un-expected way, the tool as such would become the object of our actions. We would have to reflect over it consciously. This is called a breakdown. Arvola (2004) points out that such breakdowns are vital to be able to question what we otherwise take for granted.

A related concept is that of focus shifts. According to Bødker (ibid., p. 150), a focus shift is “a change of focus or object of the actions or activity that is more deliberate than those caused by

(28)

breakdowns”. An example is a when a person is explaining to somebody how to use a certain artifact. The operations she normally performs now become conscious actions to her.

4.1.1 Activity Theory and Information Technology

Discussing the relationship between activities and information technology, Kuutti (1996) states that information technology can support activities at all levels. Starting at the bottom level, opera-tions, an important driving force behind the computerization has been automation of human data manipulation operations.

At the action level, information technology can be used to support transformative actions di-rected to an object or part of it. One example is editing of a text document such as this thesis; moving paragraphs around etc. But information technology can also be used to support sense making actions, where it can help the user to understand an object of work better. A third kind of actions that can be supported are communicative actions, such as coordinating or negotiation, which are directed towards making the activity run smoothly rather than at a specific object. On the highest level, the activity level, Kuutti claims that information technology can be used in at least two ways. One possibility is that it can be used to make an activity possible or feasible (e.g. by making participants able to communicate with each other). Another possibility is that it could be used to “enable an activity to have an object that would otherwise have been impossible to grasp” (Kuuti, ibid., p. 35). One example of the second possibility that is presented is a case with health inspectors. A computerized “organization memory” was required to solve the broader problems the inspectors were facing after their work requirements had been changed. An important point is made by Kaptelinin (1996a; 1996b), who states that activity theory is inter-ested not only in the relationship between the user and the piece of information technology, but also in what other tools are available to her, computerized as well as non-computerized. People using computers are not doing that because they want to interact with them per se, but to reach goals that are “beyond the situation of the ‘dialogue’ with the computer” (1996a, p. 49).

4.1.2 Methodological Implications

Summarizing the practical application of activity theory in HCI studies, Nardi (1996) mentions four methodological implications:

1. A research time frame long enough to understand users’ objects. Objects and activi-ties change over time, so the researcher has to take this into account.

2. Attention to broad patterns of activity. Episodic fragments often fail to reveal the overall direction of activities.

3. The use of a varied set of data collection techniques. Preferable the researchers should use different techniques in parallel. Interviews, observations, video and other ma-terials all reveal certain aspects of an activity and a single technique is not enough. 4. A commitment to understanding things from users’ points of view. Formal and

ide-alistic models of how activities should be performed do not suffice if we are to under-stand what a situation really is like. Researchers need to get to know the users’ own per-spective.

4.2 Contextual Design

One popular framework for gathering and interpreting fieldwork data to be used in the design of interactive systems is Contextual Design, suggested by Beyer & Holtzblatt (1998). It goes well with the concepts of activity theory. The main idea is that designers must understand the users and their work practices to be able to design a product that meets their real needs.

(29)

The problem is twofold: on the one hand, designers are generally not familiar with the work they are supposed to support, and on the other hand, users are neither aware of what they are really doing nor of what technology is capable of. (Holtzblatt et al., 2005.)

4.2.1 Contextual Inquiries

The proposed solution to the problem of getting access to tacit data about the work practices are so called contextual inquiries. The basic idea is that researchers go out to the users’ workplace to perform a combination of interview, observation and reconstruction of past events. Contextual inquiry was initially developed as a technique to inform design of system, but can also be used for evaluation. (Löwgren & Stolterman, 2004.)

The four key principles of the technique are:

• Context – it is necessary to go to the real workplace to collect data from real users doing

real tasks.

• Partnership – the understanding of the explicit and implicit aspects of the work should

be developed through cooperation between the user and the interviewer, i.e. the inter-viewer should be seen as in less control than in a traditional interviewing or workshop situation.

• Interpretation – the interviewer should actively discuss her hypotheses of what the user

is doing and why with the user during the course of the contextual inquiry to create a shared understanding of the work.

• Focus – clearly, not all aspects of the work and what is going on can possibly be

re-corded and interpreted by the interviewer, so the contextual inquiries should be guided by a focus which determines what aspects of the work that are the most interesting ones. The interviewer should steer the discussions towards the focus, while taking care in chal-lenging her initial assumptions.