Managing the gap between the physical and digital world through a balance between transparent and performative interaction

(1)

Master of Interaction Design

Malmö Högskola (University)

Managing the gap between the physical

and digital world through a balance between

transparent and performative interaction

Advisor: Per Linde PhD Interaction Design

Author: Matthew Goble

(2)

Acknowledgements

I would like to express my gratitude to my academic advisor Per Linde, who has been very fl exible and understanding when it came to this project and also for his enthusiasm and guidance during this entire thesis project

I would also like to express my gratitude to David Cuartielles, with his generosity in time and his patience for my lack of programming skills as he helped me with the physical prototyping part project.

I would like to thank Jörn Messeter for his guidance over the last two years, he has been a

tremendous teacher and great support for all of the masters students, and should be recognized for his commitment to the program.

I would also like to thank Bonifaz Kaufmann from MIT’s Media Lab who graciously allowed me to have access to an advanced copy of his Amarino toolkit for Android and Arduino.

I would like to thank all my classmates from the last two years, Aaron, Soo, Rob, Åste, Sebi, Katrina, Suzanna, Katrin, Anders, Giga, Uli, Adam, Jeanne, Jeune, Maxie, Gordo, & Laurens. It all would not have been the same without you, and I am amazed at how close we have gotten over such a short period of time. For those of you who have or will move on from Malmö, I will miss your friendship dearly.

And last but not least I would like to thank my wife Sandra, who has put up with my complaints and late nights and who has been an amazing support for our family while I was studying. I couldn’t have written this without your help editing and organizing. Every day you remind me of how lucky I am to have you.

(5)

“Traditional approaches would suggest

that we each have an understanding of the

elements of which our world is constructed,

and an abstract mental model of how

these concepts are related...

Th

is information, abstractly encoded in

our heads, guides our actions in the world.

Armed with a model of appropriate concepts

and relations–an ontology–we can look around

us and recognize what we see... In contrast, the

phenomenologists argue that the separation

between mind and matter, or between what

Descartes called the res cogitans and the

res extensa, has no basis in reality.

Th

inking does not occur separately from

being and acting.”

- Paul Dourish; Where the Action Is: Th e Foundations of Embodied Interaction [22]

(6)

Chapter One: Introduction

Abstract

Th is paper focuses on bridging the gap between the physical and digital world by creating a balance between performative (learned) interaction and transparent (direct-realism) interaction. Th e balance between these two types of interactions is also reminiscent of the balance between how people currently interact with digital objects and how people interact with physical objects.

Th is paper does a deep review of the many ways designers, researchers, and even psychologists have studied interaction between the physical and digital world. Th e project started off with a literature review, the literature review consisted of four sections; a review of related papers and projects,

discussion of related theories, my analysis of reviewed literature, and ended with the creation of design guidelines for Near-Field Communication (NFC) devices.

Th e paper proceeds through a hands-on, iterative physical prototyping process in order to develop a prototype that allows our physical interactions to be supported digitally through the use of mobile computing. Wireless communication and metadata are used to create a physical interaction, which is both highly personalized and contextually based in a digital way. An interaction that supports our deeper understanding of the purpose of the interaction, and which also benefi ts from the availability of digital information, which is personal and contextual.

Lastly a proposal for a fi nal design for a digital device that will allow the user to bridge the gap between the physical and digital world is described in this paper. Th is fi nal design incorporates both my analysis of the reviewed projects, theories and literature, and the fi ndings from two rounds of user testing with and iterative physical prototype. Th is is followed by overall conclusions and suggestions for future work.

1.1 Introduction

1.1.1 Problem statement

Th e main problem with many digital objects is that they lack a sense of context and connection with the real world. In the greater sense of things, Interaction design has only been around for a little more than 15 years, yet digital devices have been in development for over 40 years. Interaction design is diff erent than if you were to design a chair, because in addition to function and esthetics, you would also focus your design on interaction and perception. How do people view this chair? Will they see this as a chair? Is this a comfortable chair? Th at is, you will look at how people interact with the chair; do they sit up straight in it? Does the chair force them to slouch, and would that be a bad or a good thing? All this depends on the context of use. If the chair you want to design is in an offi ce setting then it should not force people to slouch. If that same chair is at home, then maybe that is fi ne, even a wanted outcome.

(7)

If we take this same look on function, design and interaction when designing digital devices, we see a gap where the physical world and the digital world do not meet Physical requirements do not, have any merit in the digital world; digital devices are controlled in metaphors or visual mimicry of the physical world. More oft en than not, we understand the means of a digital device but not the how? For example how does you computer send that letter you wrote to the printer? Th is lack of connection to the physical world has severed the context from the interaction. A soft ware designer does not have to worry about how each person is diff erent physically, how sharp their motor skills are or how they best interact. All they have to fi gure out is how to “show” them to interact with your soft ware. Th ere is no consideration of individual’s preferences or when and where they are currently using it. However this is the way we interact with virtually all of the digital devices in the world today.

Until recently we were somewhat limited by stationary digital devices, only in the last 10-15 years have cellular phones replaced the stationary phone, and the desktop has been replaced by the mobile laptop. One thing that helps to bridge this gap between the physical and digital world is a trend towards development of mobile digital devices. If you think about mobile computing you cannot help but see the word mobile. What does being mobile have to do with bridging the gap between the digital and physical world; and how we interact with digital devices? Everything. As a species, humans are mobile. We live by moving throughout our surroundings and manipulating those surroundings with our cognitive and motor skills. In addition, in our current design of digital objects, our cognitive self is fully engaged. However, where product design is lacking is more than just our physical self, digital devices fail to address one of the most basic aspects of human beings, our motor skills.

1.1.2 Problem Relevance

When you design something in the physical world, you look at its shape, you look at how it functions, and you look at how it compliments the human body. When you design something digital, its shape and connection to the human body drop to the background and you focus on the function. Th is leads to a lessened understanding of the object and it ceases to create a direct link to the physical world. In digital objects the surface and the functionality have no direct connection to each other.

One attempt to solve this gap has been with Tangible User Interfaces or TUI’s. TUI’s as Ullmer, Ishiii and Jacob describe them in their paper, Token+Constraint Systems for Tangible Interaction with Digital Information are; “Broadly viewed, tangible interfaces give physical form to digital information. Th e approach has two basic components. First, physical objects are used as representations of digital information and computational operations. Second, physical manipulations of these objects are used to

(8)

Essentially TUI’s are physical representations of digital interactions. Using a physical model of the desktop and manipulating it to interact with the data on a computer screen is a simple form of TUI’s. In this form, TUI’s are less focused on the actual interaction with the physical world and more focused on the recreation of pre-existing digital world interactions, i.e. click, drag and drop mouse-based interactions or keyed input of keyboards. Although TUI’s allowed designers to move away from the screen and interact with our physical world, they lack a successfulness when it comes to closing the gap.

Johan Redström in his paper Tangled Interaction, debates the successfulness of TUI’s as follows; “Th at it is not the fact that GUIs are visual that is of primary interest here, is perhaps better seen if we instead rethink it as a tactile surface: Imagine an interface much like a screen but tactile instead (the keyboard is aft er all already primarily tactile); nothing is really persistent, even though some features re-occur most of the time. Such a design would hardly satisfy the argument, as it does not present a persistent strong relation between surface and functionality. It would be as transient and generic as the GUI being criticized. [11]”

In order to bridge the gap between the physical and digital world, we need to look at how products have been designed so far, and why designers of digital devices have overlooked this issue of

connection to physical world interactions. Johan Redström defi nes this problem in his paper as follows; “How to make the appearance of a product properly express the forms of use intended by its designers is a classical problem (and ideal) in design, even present in notions such as “form follows function.” Its a rather classical problem in human-computer interaction as well, as can be seen in, for instance, the research on graphical user interfaces and on strategies for how to present various functions and options to the user. As we try to merge the worlds of physical products and human-computer interfaces (as tangible user interfaces can be said to do), certain design problems related to the aesthetics of interaction are exposed. [11]”

I agree with Redström that its this loss of connection to the physical world in interactions with digital devices that is to blame for the widening gap between these two worlds. Redström, further explains the loss of connection to the physical world in digital products; “In other words, there is an argument here that interface design needs to build a stronger relationship between the object’s surface and its functionality, between representation and what is being represented, etc. Th is is, however, not the same as saying that we should immediately understand how to use the thing (though this might be benefi cial). [11]”

(9)

1.1.3 Connection to NFC and other wireless communication technologies

In order to understand the link between Near Field Communication (NFC) and bridging the gap between the physical world and digital world, I feel we must start with an explanation of embodied interaction. Paul Dourish, the originator of the concept of embodied interaction explains in his book, Where the Action Is Th e Foundations of Embodied Interaction as; “Embodied interaction is interaction with computer systems that occupy our world, a world of physical and social reality, and that exploit this fact in how they interact with us. [22]”

1.1.4 Enabling of embodied interaction through embedded systems

In order to enable this theory of embodied interaction, an embedded system is required in our physical world, full of smart, aware, and computerized objects. Th ese embedded systems are the keys to unchaining the digital world and allowing it to exist in our physical world. If we allow our digital devices to resemble our physical products more, we are closer to the goal of closing the gap between digital and physical.

Redström explains his thoughts behind the enabling of embedded systems in physical objects, or as Dourish would call it embodied interaction, in his paper Tangled Interactions as; “...the development of computational things and interactive products more closely resembling traditional products and appliances made relations between product design and interaction design more obvious than perhaps initially was the case with the personal computer. [11]” He continues with “More likely, an increasing interest in aesthetics in the last ten years has had something to do with how the development of new kinds of computational things has renewed the discussion on how computers do what they do. Th e diff erence between a command-line terminal, a graphical user interface and a tangible one, is not just a question of what they do, but how they do it. By opening up for a much wider spectrum of approaches to how functionality is expressed through an interface, the question of aesthetics and expressiveness became more pressing. [11]”

Near Field Communication is a short-range wireless technology based on contactless identifi cation and connection technologies such as RFID. As a communication technology, NFC, allows for a simple connection interaction between digital devices and embedded systems. NFC allows for a simple, secure and intuitive connection between devices as a read and write communication technology. Some key aspects of NFC that make it extremely useful for embedded systems are, proximity, explicitness and contextual understanding.

(10)

1.1.5 Physical (proximity) of action

With an NFC or RFID type of wireless interaction, the reader and tag act as a way to physically link the content with the object (tag), which its interacting with. Especially in NFC, this linking is obvious as the operating range of NFC devices is extremely short (less than 10 cm in range). Th e user of NFC devices would have to almost touch the tagged object. Th is fi ts in with Paul Dourish and the psychologist J.J. Gibsons theories that in order to understand the connection between two objects and the meaning behind the interaction, you have to have an object that is not only out in the physical environment, but you should also be focused while interacting with the object to fully create a deep understanding of the meaning behind the interaction.

1.1.6 Explicitness of action

With this close physical proximity of the tagged object to the user, the explicitness of the interaction can only help to create a deeper interaction. Th e user knows that when they touch an NFC reader or device to a tagged physical object they see a reaction immediately from the NFC reader or device, which is explicit in nature. If you read the same object more than once, then you will receive the same response repeatedly. Th is kind of focused explicitness not only strengthens the interaction between physical objects and digital devices, but it allows the user to understand the interaction as they would if they were interacting with any other physical object.

1.1.7 Contextual connection formation

A contextual understanding of the user can be developed with an NFC system. Th e proximity and explicitness of a NFC interaction also allows you to use metadata about the interaction itself to create a context. Th e context can be as simple as time of day and location, or a more complex context could include what kind of object you are interacting with, when was the last time you interacted with that content, and for how long. Hence the context becomes more and more personalized and creates a system that better understands the user and the information the user is interested in. Th e metadata may be analyzed to understand how information is managed. A direct disregard of information may mean that either the information is not interesting or necessary to the user. If over time the user goes back to the information, then that means that the information may be useful later but might not be pressing enough for the user to know immediately. In this way the system can create a personalized context that allows for multiple types of conveyance of information.

(11)

1.1.8 Issues with this problem formation

Although it would be nice to simply state that embedded NFC systems could be used to create embodied interaction in digital devices, which might solve the gap between the physical and digital world. Th at is, unfortunately not the entire case. As I dove deeper into the problem, I found that we, as human beings, have been taught how to interact with digital devices as they currently exist. However creating a deeper understanding behind the interaction will help us interact with digital devices, but it does not entirely bridge the gap between these worlds. As part of this project I conducted a literature review of current projects related to NFC, Embodied Interaction, and bridging the gap between the digital and physical world.

1.2 Description of approach

I chose to work in a mostly theoretical way at fi rst in order to get an understanding of the reasoning behind the interaction of current digital devices, and also the work that has been done in the area of bridging the gap between the physical world and digital world. Th is was done in order to create an understanding of the problem domain. In order to achieve this, I fi rst reviewed literature on projects and theories relating to the subjects of NFC, embodied interaction and bridging the gap between the physical and digital world. Th e review of these theories and literature led me to develop two distinct types of interaction; performative interaction and transparent interaction. Aft er I had grounding in the theories behind the problem domain, I started to develop a physical prototype. Th is physical prototype was created using the Arduino open-source electronics platform. Choosing to use Arduino was key in the iterative process of testing I conducted as it allowed me to change the prototype’s appearance while keeping the same basic physical interaction. It also allowed me to interact with users where I could gain new perspectives which could help develop the prototype, as well as, allowing me to gain an understanding of the diffi culties or working in the physical environment. Th e insights gained from the decision to use physical prototyping made the development of the fi nal project a much more rich and stable design through concerns and issues that were raised during the two rounds of user testing. Th is use testing of prototypes was the basis of the fi nal project design.

My fi nal project design was based on the theoretical knowledge I developed from my literature review. Th is theoretical grounding helped me to understand the fi ndings from my two rounds of user testing and analyze those fi ndings in a meaningful way. Th is led to the development of a fi nal project design, which will help us bridge the gap between the physical and digital world through a combination of multiple theories, technologies and also a balance between two distinct types of

(12)

1.3 Objectives and Knowledge Contribution

My primary objective for this project was to create a device that allowed a user to bridge the gap between the physical and digital world, moreover a way to connect digital information to a physical interaction. A secondary objective was to work with physical prototyping, more specifi cally

the Arduino platform. Aft er spending a great deal of the fi rst year creating projects that were hypothetical, I felt a need to make a physical object.

My knowledge contribution consists of three distinct aspects. Th e fi rst is the detailed review of embodied interaction and NFC literature and projects, and how through careful analyzation of the literature and theories, a set of guidelines for working with NFC projects were created. Th e second aspect is my theories of physical interaction, performative interaction and transparent interaction and how these theories relate to designing digital devices. Th e third, and last part of my knowledge contribution is the idea of balancing multiple forms of interaction to allow us to bridge the gap between the physical and digital world.

(13)

Chapter Two: Literature Review

A large part of my project was to research diff erent types of interactions used between the digital and physical world. A literature review was conducted on areas such as NFC, Embodied interaction, and bridging the gap between the physical and digital world. A brief summary and my perception of each paper are given in the section below. Th eoretical ideologies such as Gibson and Contextual based info is also reviewed in the end of this chapter. I refl ect over the diff erent articles reviewed and theories and propose two design theories, Performative interaction and Transparent interaction

2.1 Related papers and projects

In Ailisto et al.’s paper they demonstrate the need for a diff erent look at tangible user interfaces compare to the more common use of replacing digital inputs with physical objects. Th ey also focus on how TUI’s are used in ambient intelligence (AmbI) systems. Th ese AmbI systems have, according to Ailisto et al., been dependent on embedded computing. Th ey also explain that the key sticking point in most AmbI systems is how the mapping of the digital world is replicated in an understandable way in the real world.

Th is breakdown of mapping between the physical and digital world works both ways, according to Ailisto et al. Th e most common and current methods use humans as the mediator between the two worlds. Th ey looked to create an easier connection between these two diff erent worlds by using a mobile centric system. In their own words, “Rather than environment having the awareness of the user, the personal mobile device should be aware of intelligent aff ordances in the physical environment, and utilize them... [1]”

Th e focus on mobile devices as the backbone for their project is further explained in, “Human interaction with and through a mobile device is becoming more important for at least two reasons: fi rstly, the functionalities residing in the mobile devices, such as smart phones, are becoming more diverse and important for the user. Secondly, the services and devices accessible through local connectivity and cellular communication are off ering a wealth of possibilities as the ambient intelligence vision is coming true. [1]” Adding that the users should not be focusing on the mobile device, but the physical world.

Th ey then focus on physical world selection; “In this work, we present the concept of physical selection, defi ne physical selection paradigms PointMe, TouchMe and ScanMe, analyze the

implementation options as well as tentative user experiments with the physical selection method in order to study its effi ciency and the user experience it results in. [1]”

(14)

Of particular interest is their physical selection paradigm, which focuses on not only the actual physical selection of objects, but also their defi nition as it relates to a mobile device. Th ey illustrate this in three paradigms, PointMe, TouchMe and ScanMe.

In ScanMe, the user scans the environment for tags. All the scanable tags should be visible for the user to choose from allowing the user to select which specifi c tag they would like to interact with. Although they state that this does not provide a strong connection between mapping the physical and digital world because ALL of the tags respond to a scan.

In TouchMe, the tagged object is chosen virtually by the user with a mobile phone reader. Th is requires that the user identifi es the tag, but does not require a line-of sight communication with the object. Th ey defi ne the touching paradigm as the strongest, as it typically eliminates the ambiguity of the action of selection.

In PointMe, they describe pointing as a natural way of selecting visual objects in the immediate surroundings. Th e user points and chooses which tagged object they are selecting with their mobile device. To use the PointMe paradigm the user will need a line-of sight, but do not need to be physically touching the tag.

To illustrate these paradigms they conducted two experiments:

• Physical selection of a website: In this experiment they pointed a mobile phone at a physical icon, and in this case a poster. When pointing at the poster a website with information in regards to the poster would appear.

• Evoking a phone call by pointing: In this experiment the user would hold a button on the phone while pointing it at a picture of who they would like to call and the phone would then dial the corresponding number.

In Chavira et al.’s paper, they focus on Near Field Communication (NFC) to create an AmI environment. Th ey also use the touching paradigm as the users interaction with the physical world. Th e interesting aspect of this paper is that they used the tags to create a context. In order to use the services the user must touch their NFC enabled mobile phone to the tags that are part of the AmbI environment. Each time they interact the system will log this information and allow multiples services available to the user based on the context of their use of the system.

(15)

Chavira et al. conclude their paper with this statement, “Th e use of NFC technology in the AmI environment is a step further in achieving the ideal vision of ubiquitous computing or ambient intelligence environment. Although its not a totally proactive system, touching interaction is a somewhat explicit task that generates savings in terms of eff ort, compared to the traditional way of interacting with devices. Moreover, the fact that the system is embedded in a cell phone will facilitate its widespread use. [2]” Th ey focus on a more technical consideration of the interaction of touch, along with a lot of the technical infrastructure explanation that would be required for the AmbI environment with NFC.

In Gevan et al.’s paper they conducted fi ve diff erent fi eld studies of interacting with NFC in mobile phones. Th ese fi eld studies were, First Experiences Test, Focus Group, Diary Study, Online Survey and Idea Development Workshop. Th ey conducted these fi eld studies in order to fi nd the problem areas within the interaction with NFC devices and services. Th ey found that they could come up with three recommendations for developing NFC services.

• Clarify concrete orientation, place visual markers near the NFC-chip on the mobile device, as well as on the fi xed terminal, so that its clear for the user which surfaces should be placed together.

• Make information about the interaction process available in the world, by placing a pictogram on an active NFC area, and provide a step-by-step description of the actions that users can perform. Improve NFC menus with procedural information on how to use the services. • Important and critical processes (e.g. charging your account) must be transparent. A confi rmation before critical process tasks are completed should be implemented. If an

interaction fails, users expect clear and precise error feedback about what happened and how the error can be dealt with (e.g. via SMS).

Th ey conclude that, “A challenge for designers will be to create services that make it clear where interaction occurs, where feedback is given, and how the fl ow of interaction takes place. [3]”

In Hardy and Rukzio’s paper they describe an NFC system titled Touch and Interact, this project used an NFC enabled phone to control a large display. Th ey use NFC tags to section off a large display wall that is then linked via the phones reading of the NFC tag to a computer, and the computer then displays dynamic feedback on the wall, in relation to where the NFC phone is placed physically on the display tag grid with a projector. Th e idea is to bypass the limited screen size on mobile devices and also to allow you to use both displays for diff erent types of information. Th e mobile phone also allows for storage of data, haptic feedback and support user-generated content onto a larger, shared display.

(16)

In the paper by Ullmer et al., an interaction approach for tangible user interfaces (TUI) is presented. Th is interaction is the token and constraint concept. Tokens are physical objects that represent digital information, while constraints are confi ning regions that allow for use of digital syntaxes (operations). Th is concept of constraints is explained in the paper, “they (syntaxes) give physical form not only to digital information itself, but also to aspects of the syntax for manipulating this information… It (syntax) is the grammar of ways in which objects can be combined together to form expressions that can be meaningfully interpreted both by users and the underlying computational system. [5]” Th ey conclude the explanation of constraints with, “By mechanically structuring and limiting which tokens can be accommodated and what confi gurations these can assume, constraints can express and partially enforce the syntax of their associated digital operations. [5]”

However a key aspect of this paper is when they discuss the positive ramifi cations of the use of physical constraints from a usage and implementational standpoint. Which include, human Perception, human manipulation, machine sensing and machine interpretation. By human

perception they mean, shift ing cognitive load to external representations, which supports perceptual chunking of objects from multiple sources. By human manipulation they mean, the physical

manipulation of multiple objects through moving the entire constraint structure (physically) or moving multiple tokens that are joined. By machine sensing they are pointing to the human sensing the machine easier, through the physical sensing of the machines state through the tangible interface. By machine interpretation they mean that constraints simplify the computational needs for tangible interfaces by limiting them to specifi c states, to reduce errors and also to help with implementation. Tomitsch et al. studied how to make real world tagging (RFID or visual codes) accessible to blind and visually handicapped people. Th ey propose audio/tactile location markers that would make these real world tags accessible. “Th e goal of this work was therefore to develop techniques for making the tags’ presence available to users without changing the tagging technologies themselves. [6]” Th e focus of this paper was on making real world tags accessible to a wider range of users, and where designers can turn to pre-existing tagging situations and base accessible tagging on these familiar methods. It also shows that to make tags work in the physical world, you must think carefully about their placement and how they make themselves known to users.

(17)

In Scott et al.’s paper they address the issue with Bluetooth device discovery, as its slow and can be awkward. Th ey propose using visual tags that bypass this device discovery step and connect the users mobile to the content directly so that Bluetooth can be used as a transfer communication. Of particular interest is the fact that they describe the lack of context awareness when using Bluetooth, as the user might know physically what they want to connect to, but through Bluetooth discovery method they might not necessarily know the name of that specifi c device. It was partially for this reason that, I choose to use NFC as an initial connection communication standard, since the technology works in a short operating range and requires no need for syncing. Th e visual tagging concept also points to a way of advertisement of the tag to the user.

Sandner et al.’s paper focuses on a concept for interaction with NFC enabled devices as an option to interact with digitally tagged news posters located in the physical environment. Th e posters would have tags relating to news stories that the user could scan with their mobile phone and the tag would send the URL information to the phone’s Internet browser for later viewing. An illuminating part of the Sandner et al.’s work was, “...the idea of touching an object with a phone to get information is not common for most people. It is a challenge to shape the aff ordance right and to make interfaces intuitive, for example with graphical symbols. [8]”

In Jacob et al.’s paper they defi ne Reality-Based Interaction (RBI), where human computer interaction has grown from “window, icon, menu, pointing device [9]” (WIMP) or Direct Manipulation

interaction style. As they put it “Interaction with computers has evolved from the fi rst generation of Command Line, to the second generation of Direct Manipulation, to a new generation of emerging post-WIMP interaction styles. Direct Manipulation moved interfaces closer to real world interaction by allowing users to directly manipulate objects rather than instructing the computer to do so by typing commands. New interaction styles push interfaces further in this direction. Th ey increase the realism of interface objects and allow users to interact even more directly with them—using actions that correspond to daily practices within the non-digital world. [9]”

Of particular interest was how they also describe four themes in RBI; Näive Physics, Body Awareness and Skills, Environment Awareness and Skills, and Social Awareness and Skills. Th ey describe Näive physics as, “In the context of emerging interaction styles, user interfaces increasingly simulate or directly use properties of the physical world. For example, a tangible user interface (TUI) may employ physical constraints such as a rack or a slot to guide the way in which physical tokens can be manipulated. Emerging graphical user interfaces, such as the Apple iPhone (see case study below), employ physical metaphors that add the illusion of gravity, mass, rigidity, springiness, and inertia to graphical widgets. [9]”

(18)

Body Awareness and Skills (BAS) are defi ned as, “...the familiarity and understanding that people have of their own bodies, independent of the environment. For example, a person is aware of the relative position of his or her limbs (proprioception), his or her range of motion, and the senses involved in perceiving certain phenomena. Early in life, most people also develop skills to coordinate movements of their limbs... Emerging interfaces support an increasingly rich set of input techniques based on these skills... [9]”

Environment Awareness and Skills (EAS) are explained as, “...people have a physical presence in their spatial environment, surrounded by objects and landscape. Clues that are embedded in the natural and built environment facilitate our sense of orientation and spatial understanding... People develop many skills for navigating within and altering their environment. In the context of emerging interaction styles, many virtual reality (VR), mixed reality (MR), and augmented reality (AR) interfaces along the reality-virtuality continuum. [9]” Th is allows us to use the same of types clues to navigate in the digital world. Th ey also add, “People also develop skills to manipulate objects in their environment, such as picking up, positioning, altering, and arranging objects. Emerging interaction styles oft en draw upon users’ object manipulation skills. [9]” Social Awareness and skills, focuses on humans, “...ability to work with others to collaborate on a task. [9]”

Th e most useful part of this paper has to be when they describe using RBI themes in digital interactions, “Basing interaction on pre-existing real world knowledge and skills may reduce the mental eff ort required to operate a system because users already possess the skills needed. For casual use, this reduction might speed learning... Applying RBI concepts such as naïve physics to an interface design may also encourage improvisation and exploration because users do not need to learn interface-specifi c skills. [9]”

Th ey also talk about how trade-off s are required when creating these types of systems, i.e. reality vs. accessibility or reality vs. practicality. Mostly these trade-off s are due to the implementation of the system and show that design decisions must be made that could lead to a slightly less reality based interaction. Th e design trade-off between reality vs. practicality show that in order to make a physical interaction based system, one must balance between pushing the physical interaction and using pre-existing methods to solve design issues throughout the project development.

(19)

Leong et al. defi ne their paper, “Th is paper proposes an intelligent location context-aware, peer-to-peer, multi-agent system that could be used to address the dynamic requirements of a mobile infrastructure and hence provide reliable autonomic location-based services environment. [10]” Th e interesting part of this paper is how they illustrate the technical and multiple communication technologies that exist in mobile phones, and also the ability of the system to negotiate that. “Th e mobile environment has very diverse communication technologies ranging from 3G, HSDPA, and WiFi to IMS. Mobile users may also be connected to diff erent communication channels depending on their location. Managing such a heterogeneous complex architecture requires the system to be capable of self-manage or autonomic capabilities. Autonomic feature increase the system reliability and manageability. [10]”

In Redström’s paper he explores a basic aesthetic issue in interaction design, namely how a desire to create a connection between appearance of a designed object and its function confl icts in crucial ways with how the miniaturization of technology has changed the connection between the objects surface and its internal complexity. He also discusses how overloading a surface in an eff ort to explore the expressiveness and aesthetic potential of an object leads to what he calls a tangled interaction.

Redström starts the discussion by analyzing the crossing of interaction and product design in digital devices, especially when looking at a designed object in respects to ubiquitous computing, ambient intelligence, and tangible user interfaces, which have exposed interaction issues. He starts by talking about how the change in domain (non-offi ce setting) leads to a change in focus on values, “Th e discussion of the diff erences between designing for effi ciency in use versus the experience of use is just one example of what this shift in use context implied to design and aesthetics. [11]” Continuing with, “Further, the development of computational things and interactive products more closely resembling traditional products and appliances made relations between product design and interaction design more obvious than perhaps initially was the case with the personal computer. [11]” He then

continues to discuss how this change in products relates to computational aesthetics, “Th e diff erence between a command-line terminal, a graphical user interface and a tangible one, is not just a question of what they do, but how they do it. [11]”

He further discusses the issue of time in relation to the physical and digital world, “As we try to merge the worlds of physical products and human-computer interfaces (as tangible user interfaces can be said to do), certain design problems related to the aesthetics of interaction are exposed. [11]”

(20)

He then switches back to discuss the connection of the appearance of an object and what we can do with it, “… there is an argument here that interface design needs to build a stronger relationship between the object’s surface and its functionality, between representation and what is being

represented, etc. Th is is, however, not the same as saying that we should immediately understand how to use the thing (though this might be benefi cial). [11]”

One of his main arguments is discussed next, “With the miniaturization of technology, the relation between internal and surface complexity has been lost. [11]” He continues discussing the loss of relation between surface and internal with, “Th at the relationship between internal complexity and surface indeed has been lost is quite evident when we, for instance, look at a computer or even just a microprocessor… Information technology is not “intangible” just because of how we have designed the interfaces, but because the basic material exists at scales we cannot access or relate to without mediation… Th is means that not only is the strong relation between surface and internal complexity lost, but that it cannot be completely restored. [11]”

He then looks at solutions for the lack of surface understanding, “…exploring acts of use, where strategies for how to express and deal with complexity transfer the focus from spatial surfaces to what happens over time... notion of building skill over time opens up for working with expressive temporal patterns that develop over time. [11]”

Redström then looks at how to deal with the expressiveness of the surface, “In other words, we try to overload the surface by adding diff erent layers of meaning in such a way that, for example, performing a certain action might mean several diff erent things. Obviously, overloading a surface in this way will not be unproblematic, as there also will be interaction between the diff erent layers. Th is, however, we will consider a desired side-eff ect that expands the expressiveness even further, be it that we might have problems controlling exactly what will happen. Th us, to explore the aesthetic potential of such layered interaction, what we will do here is to try to introduce multiple layers of interaction that are in some confl ict with each other. Specifi cally, that we do not only layer interaction, but try to set layers up in ways that makes something potentially interesting happen also in-between, a kind of tangled interaction. It is therefore important to this conceptual design experiment that these layers are continuously present, and not a set we might browse or sequence through like we might switch between using diff erent screen layouts or “pages” for diff erent tasks or applications on a computer. [11]”

(21)

In conclusion of Redström’s paper he states, “…we could say that, from an aesthetic point of view, it is not the fact that they are tangible that is the most crucial part of tangible user interfaces considered to comprise an interface design strategy, but how they aim to deal with the relation between appearance and functionality. [11]” He adds a fi nal statement on how the materials can aff ect the design, “inherently in-between working with the expressiveness of “new” computational materials and the old ones, more or less successfully trying to exploit the potential of both. But it might also be that we are getting close to making a categorical conceptual mistake here; as if we are trying to use the aesthetics developed for one kind of design problem in another, and in some crucial ways, very diff erent kind of design. [11]”

In Lawo et al.’s paper on the WINSPECT computer interaction glove they come up with a concept to use a glove to interact with a computer via various sensors mounted to the glove. Th ey illustrate some current computerized glove interfaces and categorize the interactions aff orded by these gloves. Th ey also describe the WINSPECT wearable computing device and point to its wireless capabilities as, “...it becomes obvious that with such a glove much more information especially on the context of the user can be achieved. [12]”

In the paper by Broll et al., they talk about the need for NFC mobile based interactions to simplify interactions for understanding, “In order to get novice users started with this physical interaction and its applications, we have designed diff erent ways to increase the learnability and guidance of such applications. [13]” Th ey discuss the interaction qualities of NFC type devices and describe the understanding of this interaction as, “Th is physical interaction conveys the usage of the underlying technologies in an easily comprehensible way as it adopts the metaphor of interacting with

something by simply touching it. [13]”

Th ey also discuss the terms of Learnability and Guidance, and how as designers we can manage these interactions with NFC systems. For learnability they look at dedicated start tags, which are tags that are labeled as the “starting point” of the system. Usually visually marked as “Touch-here” or “Start here” to allow novice users to know where to begin the interaction. Visual Cue on the Physical Object is a graphic symbol on the object that identifi es it as a point of interaction to the NFC system. Visual Cue on the Mobile Device is a graphical symbol that would match that graphic on the object but placed on the mobile device so that you know where to “touch” with you phone. Th eir tests showed that with novice users the Dedicated start tag was the shortest interaction time followed by the visual cue on the object then visual cue on the mobile as it was clearly marked and also labeled with text where to interact with the system.

(22)

In Horneker and Buur’s paper they propose a framework for physical and social tangible interaction, “We introduce a framework that focuses on the interweaving of the material/physical and the social, contributes to understanding the (social) user experience of tangible interaction, and provides concepts and perspectives for considering the social aspects of tangible interaction. Th is understanding lays the ground for evolving knowledge on collaboration-sensitive tangible interaction design. [14]”

Th eir framework on tangible interaction contains four themes:

• Tangible Manipulation refers to the material representations with distinct tactile qualities, which are typically physically manipulated in tangible interaction.

• Spatial Interaction refers to the fact that tangible interaction is embedded in real space and interaction therefore occurs by movement in space.

• Embodied Facilitation highlights how the confi guration of material objects and space aff ects and directs emerging group behavior.

• Expressive Representation focuses on the material and digital representations employed by tangible interaction systems, their expressiveness and legibility.

One of the most interesting things is the expressive representation, which is discussed in this paper, “We extend and refi ne this notion as referring to the interrelation of physical and digital representations and to how users perceive them. We have found that users perceive a tangible interface as “not very tangible” and the tangible objects as insignifi cant, if these were only of temporary relevance or not expressive [14]”

In Price et al.’s paper they state that, “...on studies involving two tangible learning systems to analyze the eff ect of design choices on the kinds of (inter)actions engendered and how they create, shape and constrain diff erent learning opportunities. [15]” “To explore these notions of tangible actions, two diff erent tangible learning systems are examined using three key design characteristics: physical space; gestural and manipulative interaction; and input/output coupling. [15]”

Th ey continue to discuss tangible environments as, “...viewed from an object-centric or subject-centric perspective... An object-subject-centric view emphasizes how the design of the object itself determines the interaction style and context and has led to a plethora of tangible objects, which require ‘direct manipulation’. Much previous research in tangible computing has taken an object-centric approach using Gibson’s theory of ‘aff ordance’ as a design guideline... From a subject-centric point of view, interaction focuses on designing the action – the emphasis is on how bodily movement aff ects interaction. In emphasizing the subject, we turn our attention to ‘direct engagement’, an alternative to direct manipulation, and focus on designing user experiences that exploit emotional and cognitive values.

(23)

In this view, the designed actions provide the context and any objects present in interaction are there to suggest context rather than demand it. Th is paper examines this distinction in more detail and the impact on interaction. [15]”

Th ey continue to discuss Action and Representation as it relates to the interaction of tangible systems. “Tangible systems off er opportunities for meaning-making and non-verbal expression through links between action and digital representations. [15]” Th ey also discuss the need to look at location of these tangible representations as, “Location has an impact on the kind of actions that can take place: actions are usually constrained to a surface in co-located systems, while broader kinds of movements are possible within discrete and embedded setups. [15]”

In Wensveen et al.’s paper on designing for emotionally rich interactions they, “propose a 3-step method for designing emotionally rich interactions... By emotionally rich interaction we understand interaction that heavily relies on emotion expressed through action. Th e method addresses three questions: What are the relevant emotional aspects for a context for experience? How can a product recognize and express these aspects? How should the product adapt its behavior to the user on the basis of this information? [16] ” Where their solution is, “Th e essence of our approach is that a product not only elicits emotionally expressive actions, but that the feedback is inextricably linked to these actions. [16]” In the feedback they state that you can access whether or not an expressive design or a non-expressive design will yield expressive or non-expressive feedback. However, they show that it can also go the other way. Receiving non-expressive feedback from an expressive design. In Djajadiningrat et al.’s paper they look at interaction design from a product design perspective, looking at how designed products appeal to our senses and motor skills. Forgoing J.J. Gibson’s theories on perception they conclude that, “…we have come to see both appearance and actions as carriers of meaning, and how we see usability and aesthetics as inextricably linked. [17]” Finally they argue that, “that, in addition to a data-centered view, it is also possible to take a perceptual-motor-centered view on tangible interaction. In this view, it is the rich opportunities for diff erentiation in appearance and action possibilities that make physical objects open up new avenues to meaning and aesthetics in interaction design. [17]”

(24)

Especially interesting is the aspect of designing products to appeal to users senses and motor skills. Another interesting aspect of this paper is how they see appearance and actions as ways of implying meaning, as well as aesthetics being linked to usability. Its this connection between the surface and action combined with designing for senses and motor skills that I will take from this paper when designing my own digital product.

In Dalsgaard and Hansen’s paper, Performing Perception, they look at how a user of a digital product is simultaneously operator, performer and spectator. With these three roles shaping the user’s

understanding and perception of their interaction through their experience of the object’s form and expression. Th ey also demonstrate how perception is also performance and how this aff ects the design of experience-oriented products. Another part they discussed is how context in which you interact, aff ects the relationship between the users performative interaction and their perception of the product. Viewing the user as a performer, spectator and operator and seeing that the context in which you interact with a digital device all aff ect the users perception of the product, I have tried to introduce these two aspects into the design for my project.

In Hummels et al.’s paper, they argue that we perceive our world by physically interacting with it, calling this the core foundation for turning to movement-based interaction. Th is interaction-creates-movement concept does not only apply to the user, but the designer. Th ey also stress the point that the designer should be an expert in movement to design for movement-based interactions, for which they have developed a set of methods and tools for learning to become a movement expert and designing for movement.

Perhaps the most useful aspect of this paper is how they built up the theory behind designing for movement, namely the example of juggling, which I also use to illustrate why motor skills are key to designing for mobile digital devices. I even went so far as to learn to juggle to experience this expertise needed to design for movement in my own project research. Lastly they list the toolkit they developed for designing for movement. Th is includes, meaning through interaction, richness of interaction, design by moving, support for movement, research by doing, educate through and for movement, and design for diversity.

(25)

Mistry’s wearable gestural interfaced augmented reality concept, called the Sixth Sense, is a neck worn device that uses computer vision to scan the environment around us to search the web for related information. Th is information is projected onto a fl at surface in your physical environment, which allows you to manipulate the projected information with gestural actions through colored strips on the tips of your fi ngers. Th e key use of NFC embedded systems which I will use from this project is the connection to a global search through the physical environment, which allows you to access in-context information from the web, through a physical action.

Kimball’s paper discussed service design, and she describes the emergence of service design as a result of two things, “Th e fi rst is the way in which networked media technologies have changed the traditional outputs of design, which now include electronic products but also arrangements of interfaces to distributed devices through which services can be delivered... mobile and fi xed telephony; accessible and aff ordable resources for the creation and distribution of rich media

especially video and high-resolution graphics... Services do not necessarily require these technologies in their design and organization but many do involve them. [21]”

She continues, “Th e second is the increasing attention paid by management theory and

practice to the role of design in organizing production and consumption, and in particular to its role in creating new or innovative products and services... Th e involvement of designers in new product development has increasingly been seen as linked with innovation, whether incremental or radical. [21]”

Her main point is a holistic look at the “touch points” between not just the perceived “service” between the provider and user but also the service as it relates to the services that the provider needs to do to provide a service. Its looking at this larger picture that allows you to design all aspects of the service and provide more meaningful interactions throughout the service.

(26)

2.2 Related Th

eories

2.2.1 Gibsonian psychology/ecological psychology

In order to understand the reasoning behind embodied interaction, psychology and sociology was considered. Of particular interest was the psychologist, J.J. Gibson, of “aff ordances” fame.

J.J. Gibson, the prominent psychologist of ecological psychology, stressed the importance of the environment to fully explain some behavior. Its necessary to study the environment in which the behavior took place. Gibson is known in Human Computing Interaction communities for his concept of aff ordance, which meant an opportunity for action provided by a particular object or environment. Gibson was regularly frustrated by the push for cognitive psychology during his time, and he regarded seeing and acting as being deeply connected. His view of visual perception was not a link between the mind and the environment, but a point of contact between man and his environment that moves and within he acts. Gibson’s ecological psychology studies “knowledge in the world” rather than “knowledge in the head”, this is further described by Paul Dourish in his book Where the Action Is. [23]

Gibson’s ecological psychology was based on his adaptation of the Direct Realism theory, based on Scottish Philosopher Th omas Reid’s work. Direct-Realism is a theory of perception that claims that the senses provide us with “direct awareness” of the external world, and that this perception exemplifi es unmediated contact with the external world. Reid’s description was, “there surely is an external world, and we surely do have knowledge of it” stating that we perceive the external world immediately and directly.

Th ese theories talk more about the actual perception of the external world and objects than “how” we perceive objects. An example of this would be; you see your hand not as a object with skin, nerves, bones, but you can instantly relate your hand as your hand. Th is leads to a direct connection to the external (physical) world.

Obviously the world can be miss-perceived by our senses, which is why I believe that there is not just a direct perception, although there is the senses that allow us for direct perception. In this aspect ecological psychology starts to fi ll the voids in perception. As I understand it, ecological psychology is not totally ignoring the cognitive perception, its more implying that perception is not just based on cognitive alone, this allows an aff ordance in our interaction within the environment. Th is aff ordance is that we as humans can also perceive things cognitively. Th is allows us to build a context based on previous experiences, which can help our direct contact with the environment through our senses understand and perceive our environment.

(27)

Which ties to what Ailisto et al. state in their paper, Bridging the physical and virtual worlds by local connectivity-based physical selection, “Rather than environment having the awareness of the user, the personal mobile device should be aware of intelligent aff ordances in the physical environment, and utilize them... [1]”

Its the aff ordances in the environment that allow the embodied interaction to work, we do not have to simply rely on our physical senses, we can create a new sense, through a digital device that allows us to directly “sense” these aff ordances.

2.2.2 Contextual based information

Th ese environmental aff ordances and embodied interaction systems, along with ecological psychology leads to consideration of the context of the situation and how this contextual based information could serve as a basis to manage the gap between the physical and digital world. In Chavira et al.’s paper, context plays an important part in creating and managing the AmbI

environment [2]. Th e interesting aspect of this paper is that they used the tags to create a context. In order to use the services, the user must touch their NFC enabled mobile phone to the tags that are part of the AmI environment. Each time they interact with the system, it will log this information and allow multiples services available to the user based on the context of their use of the system. Th e connection between contextual based interactions in NFC enabled environments is personalization, (i.e. what you the consumer feel is important; sustainability, healthy products, allergies) and what you, the consumer, can do with the information (i.e. which foods to purchase, recipes, drink pairings). In addition there are social aspects that can be gained through personalized contexts such as, who else uses this, sharing information, recipes, ratings, and purchase records. All of these services can be connected to create a rich, contextual but overall personalized interaction, the information already exists in metadata and throughout the Internet

In Mistry’s, project Sixth Sense, they explain that “...data, information and knowledge that mankind has accumulated about everything and which is increasingly all available online. [20]”

(28)

Seeing this contextually personalized information on a mobile platform allows us to have the information we want, when and where we want or need it. Contextual tags (such as NFC tagged packaging) may be used to interact and inform us about our physical lives. Th is can be done if we focus on a contextual based NFC interaction, which uses information over time to build up this context, and is also connected to your digital schedule linked to your mobile phone and desktop. It can further connect to your email accounts and through metadata, and can add a contextual link. All of this is the aff ordance that can be gained from using a smart phone and connecting it to our physical world through embedded NFC tags in our environment.

Again, simply creating an intelligent context-gathering device is not enough; it might, like direct-realism, lead to a misunderstanding, which in turn would jeopardize the appeal of the system as a whole. Aft er all, if you have a system that is supposed to augment your physical life with digital data and that data is of no signifi cance to you personally, where is the value in the system?

Th is is why I have developed two terms for interaction with digital systems embedded in our environment; Performative interaction and transparent interaction. Which, with a balance of these two types of interactions leads to a way to bridge the gap between the physical world and digital world.

(29)

2.2.3 Service/System Design

Looking at service design is a key need in the development of any system. Lucy Kimball describes the emergence of service design in her article, Th e Turn To Service Design as,

“Th e emergence of service design accompanies two developments. Th e fi rst is the way in which networked media technologies have changed the traditional outputs of design, which now include electronic products but also arrangements of interfaces to distributed devices through which services can be delivered... mobile and fi xed telephony; accessible and aff ordable resources for the creation and distribution of rich media especially video and high-resolution graphics... Services do not necessarily require these technologies in their design and organization but many do involve them. [21]”

She continues with, “Th e second is the increasing attention paid by management theory and practice to the role of design in organizing production and consumption, and in particular to its role in creating new or innovative products and services... Th e involvement of designers in new product development has increasingly been seen as linked with innovation, whether incremental or radical. In these accounts the creative input of designers is observed in three main ways: through their insights into consumers and end users evidenced in their human-centered approach and methods; through their iterative processes of idea generation, modeling and prototyping, testing and selection, oft en involving multi-disciplinary teams; and through their competences in working with aesthetics and with visual forms. [21]”

Kimball continues to explain that in order to look at the service design, one must look past just the off ered services and combine all the ”touch-points” between the product or organization and the consumer or user. Th ese must be seen holistically as an intentionally designed experience of the service of a product or organization.

With this holistic viewpoint in mind, I started to look at the various touch points of service related to a Mobile centric NFC system embedded into our environment. Obviously there is the touch points related to the mobile device itself, which brings into play the aff ordances of the device, its communication and computational ability inherent to the device itself. Th en there are the touch points of the embedded smart objects in the physical environment. Th ese are, of course, only the physical touch points, when we talk about service design for systems we have to look past these, as Kimball states in reference to the holistic experience. Th is would include the digital information that is shared by the system, the upkeep of the system and embedded devices, the involvement of mobile handset manufacturers to supply suitable mobile devices for this type of interaction. Also the involvement of content providers, if this system were to be placed in public areas, then local and city governments would

(30)

Foruquare.com’s instructional video on how to use Foursquare on a mobile device.

Last but not least, the human factor, as this service would be driven by the mobility and usage of humans with their developed fi ne motor skills and ability to learn by doing in a performative way. Th ese are all touch points when designing a service for embedded NFC systems, which have to taken into account in the overall system.

2.2.4 Social aspects

Included in the system is the connection to social aspects of the service. Th is aff ordance to the system allows the service as a whole to off er sharing and comparing options of pre-existing social networking sites such as Twitter, Facebook, and Foursquare and allows the user to share digitally their physical interactions.

Foursquare is perhaps the best example of this sort of physical/ digital social information sharing. Foursquare is a web and mobile application that allows registered users to connect to friends and share their physical location via a

connection to their Twitter and Facebook accounts. Th e users check-in at locations with either an sms or device specifi c application and are awarded points and or badges in relation to how many times they check-in at that location. Th ese check-in’s can be posted via Twitter or Facebook to share their location with Forusquare and non-Foursquare users. If users check-in to a specifi c location more frequently than any other user they will become the mayor of that location, which could lead to incentives from the location. Also incentives are available to Foursquare users at some locations regardless of mayor status. Users can also create tips for other users about locations. Foursquare also allows users via connections of Facebook or Twitter to fi nd other friends and their locations and also what tips they have about those locations.

Foursquare as a service is an interesting physical/digital world mobile social service as it allows the user to control how many times they frequent locations and how they choose to disperse and to what depth they choose to participate (tips) in the service.

(31)

Th is sort of social service shows how digital world information can be connected via multiple social outlets (Twitter, Facebook) and also shows an incentive service for both the user and the locations (businesses) that the users frequent.

An interesting aspect of Foursquare and similar social services is that they, like I have previous stated in my research on NFC related projects, use a multiple of digital services and technologies to achieve their goal. Th is is a common thread when dealing with digital services, which are located in the physical world, and one that helps the social aspect of service design to be more personalized to the user. Th is level of aff ordance allows the designer to off er more pre-existing content and mimics the balance between readily existing digital services and interactions and creates a personalized added value to any physical/digital service. It also allows the user to bring in a sense of familiarity to a new service, while showing the capabilities of digital services where interaction is out in the physical world.

2.2.5 How computers see us

To learn how to bridge this gap between the physical world and digital world, we must fi rst understand what we are up against in current HCI practices. Th e famous example by Tom Igoe of ‘how computers see us’ shows that to a computer, we are not humans as we interact in the physical world, we are a bodiless creature with one fi nger, a single eye, two ears and no mouth. Computer interaction does not take into account our motor skills, or all of our senses it focuses mostly on what it “needs” from us to carry on its computing. If we see this as an obstacle we cannot get past, then we are stuck with what we have been doing for the last forty years or so. Likewise we cannot simply disregard this understanding, as it has been built into nearly every computational device on the planet If we slowly add ways to interact with computers with other more physical means, such as embodied interaction practices used today, we open up the whole body to interaction with computational devices and create a stronger link between computers and humans, but also the physical world and the digital world.

Managing the gap between the physical and digital world through a balance between transparent and performative interaction

Master of Interaction Design

Malmö Högskola (University)