Licentiate Thesis Anna Ståhl December 2005
Abstract
In our daily lives we communicate emotions not only in face-to-face situations, but also in the digital world. When communicating emotions to other people we are not always aware of exactly what we are expressing. Emotions are communicated not only by the actual words we say, but through physical expressions like gestures, body posture and tone of voice.
Designing for emotional expressivity requires a design that can capture the characteristics of emotions as well as the subjective experiences. This design should also mirror the com-municative reality that we live in and open up for personality, context and situation to be expressed.
In order to explore emotional communication in the digital world we have designed, imple-mented and evaluated eMoto, a mobile service for sending text messages that can be en-hanced with emotional content. In this thesis we will present a detailed description of the design process, including user studies, leading to the design of the emotional expressivity in the eMoto prototype.
Through the use of a body movement analysis and a dimensional model of emotion experi-ences, we arrived at the final design. The service makes use of the sub-symbolic expressions; colours, shapes and animations, for expressing emotions in an open-ended way.
The results from the user studies show that the use of these sub-symbolic expressions can work as a foundation to use as a creative tool, but still allowing for the communication to be situated. The inspiration taken from body movements proved to be very useful as a design input.
From the design process and the user studied we have extracted four desirable qualities when designing for emotional expressivity: to consider the media specific qualities, to provide cues of emotional expressivity building on familiarity, to be aware of contradictions between the modalities, and to open for personal expressivity. Incorporating these open up for more expressivity when designing within this area. The actual design process is itself another ex-ample that can be used as inspiration in future designs aiming at emotional expressivity.
Acknowledgement
I would especially like to thank my supervisors; Mike Stott, for comments and discussions around design, Kristina Höök for being supportive, enthusiastic and for everything she has done all along the way and Jonas Löwgren for valuable input and discussions on design as research.
Another special thanks to my colleague Petra Sundström, through cooperation this work has come much further and been so much more fun.
Others I like to thank are Jussi Karlgren for help in naming the prototype, for reading and commenting. Jarmo Laaksolathi, Åsa Rudström, Jakob Tholander, Ylva Fernaeus, Martin Svensson and people in the Mobile Life-project for helpful comments. Charlotte Wiberg for important input and for sorting out the contributions.
CHAPTER 1, Introduction 1 1.1 Introduction 1 1.2 Problem 2 1.3 eMoto 2 1.4 Methodology 3 1.5 Contributions 4 1.6 Outline 5 CHAPTER 2, Methodology 7
2.1 The Design Process as a Result 7
2.2 Methodology Used in the Design Process 9
2.3 Focus of the Methods 10
CHAPTER 3, Background Theory 11
3.1 Computers and Emotions 11
3.2 Cues of Familiarity through Sub-symbolic Expression Forms 16 CHAPTER 4, Categorizing Emotional Communication Artefacts 23
4.1 Designed Communicating Artefacts 23
4.2 Multimedia communication 25
4.3 Communication of Affect through Adding Sub-symbolic Expressions 27
CHAPTER 5, Design Case of eMoto 31
5.1 Design Requirements 31
5.2 eMoto 32
5.3 Targeted User Group 33
5.4 Sandra – the Persona 33
5.5 Questionnaire 34
5.6 Brainstorming 35
5.7 An Analysis of Emotional Body Language 35 5.8 Russell’s Circumplex Model of Affect 36
5.9 Affective Gestures 37
5.10 Design Process of the Emotional Graphical Foundation 37
5.11 Final User Study 40
CHAPTER 6, Designing for Emotional Expressivity 45 6.1 Consider the Media Specific Qualities 45 6.2 Provide Cues of Emotional Expressivity Building on Familiarity 47 6.3 Being Aware of Contradictions in the Modalities 48
6.4 Open for Personal Expressivity 48
CHAPTER 7, Discussion 51
7.1 Strengths and Weaknesses 51
7.2 Future Work 52
PAPER A:
Petra Fagerberg, Anna Ståhl, and Kristina Höök (2003) Designing Gestures for Affective Input: an Analysis of Shape, Effort and Valence, In Proceedings of Mobile Ubiquitous and Multimedia, MUM 2003, Norrköping, Sweden.
PAPER B:
Petra Fagerberg, Anna Ståhl, and Kristina Höök (2004) eMoto - Emotionally Engaging action, In Journal of Personal and Ubiquitous Computing, Special Issue on Tangible Inter-faces in Perspective, Springer.
PAPER C:
Anna Ståhl, Petra Sundström, and Kristina Höök (2005) A Foundation for Emotional Expres-sivity, In Proceedings of DUX 2005, San Francisco, CA, USA.
only in face-to-face situations, but also in the digital world using SMS (Short Messaging Service), MMS (Multimedia Messaging Service) or different chat systems. These techniques for communication have been extended and enhanced for emotional com-munication through smilies and emoticons. These user innovations show that there is a need for emo-tional communication in other forms than written words, in these media. But smilies do not capture all the characteristics of emotional communication. They are quite crude, simple representations that do not capture the complexities of personality, context or situation. Nor don they capture the subtleties of emotion as such. When communicating emotions to other people we are not always aware of exactly what we are expressing. Emotions are communicat-ed not only by the actual words we say, but through physical expressions like gestures, body posture and tone of voice. These physical expressions can happen unconsciously and might then even be con-tradictory to what we say, but they can also be used to reinforce the expressions. Consciously they can also be used to show other emotions than what we actually feel or used for expressing irony.
Emotions are important and interesting to us in our daily life, without emotions we are not capable of rational thinking (Damasio 1994). We express emo-tions directly to other people, but we often also dis-cuss our emotions with others and reflect ourselves over our own emotions. Both negative and positive emotions are important to us, it is the whole array of emotions that makes it so interesting. Emotional experiences and expressions are very personal and are partly dependent on the shifting context and situations that take place in our lives.
done through letting users’ reflect their own lives in the medium, where their own specific personal-ity, context and situation can come through in the design. In this design there should be a balance in the execution, to allow for this reflection to take place. The expressive medium should balance between ready made expressions with pre-deter-mined meaning, like smilies, and expression forms that are open, like MMS. In MMS users start from scratch in creating their own messages and expres-sions. This balance can be achieved through pro-viding the right amount of cues of familiarity in the design of the expressive media, the user can have this as a starting point when creating their own messages in their own communicative reali-ties. We are not always aware of what emotions we have or want to convey. These cues can make a starting point for users to begin reflecting and working with their emotional expressions.
Through this users may get involved in an affec-tive loop (Sundström 2005), where emotions are treated as processes. The emotions can be elabo-rated on and defined through a process view of emotions, instead of being viewed as distinct emo-tional states, represented by labels or ready made expressions. To design for an affective loop experi-ence the communication has to have depth, which opens up for exploration and reflection of the tings that are fascinating about emotions (Sundström 2005)
Every form of expressive media have their own communication structure or rhythm, for example in a phone conversation a direct answer is expect-ed, in SMS a delay in answer is accepted and in e-mails maybe an even longer delay. When
extend-INTRODUCTION
ing digital media with emotional content or when designing new expressive media, these media spe-cific qualities are important to have in mind in the design process
This work is done within the tradition of indus-trial design and not from a HCI (Human Computer Interaction) perspective. Here the aim is to sepa-rate what is of importance within the industrial design approach and how this can re-inform HCI. Traditionally when describing a design process in HCI or interaction design, some steps are well de-scribed and argued for, while other creative devel-opment steps, like the actual process of graphical or physical product design often are black-boxed and not argued for (Fällman 2003). In this thesis we want to articulate and argue for how these de-cisions were made in our specific design process. This is to exemplify and show that a design seldom happens by chance, although in the moment of sketching or modelling it might be hard to articu-late what is happening. An articulation can often be done afterwards and thereby be used as input and inspiration in future similar projects.
1.2 Problem
The problem for this thesis lies within designing a foundation for emotional expressivity, from which users can start out from in composing their own messages in their own communicative realities. This foundation should capture the characteris-tics of emotional communication, where the sub-jective experience and the familiar aspects of the emotions should be reflected. The execution of the expressive media needs to be balanced, building on cues of what is familiar to us in natural emo-tional communications. The balance is somewhere in between leaving the user with a blank paper to
start building a completely new language from or on the other side providing the user with labelled ready made expression. This balance is very specif-ic to the aims of expressive media at hand. The user should be able to express their personality, the con-text and situation they are in through the expressive media. In designing an emotional expressive media the qualities that are specific to this media needs to be considered and respected in the design.
1.3 eMoto
To explore the design of emotional expressivity, we have designed, implemented, and evaluated a prototype named eMoto that aims to illustrate a de-sign process of emotional expressive media. eMoto is an emotional messaging service built on top of SonyEricsson’s P800 and P900 mobile terminals. In eMoto the interaction is done through affective gestures as input (Figure 1.1) and the emotional expressivity is conveyed through combinations of colours, shapes and animations, as backgrounds to users’ text messages (Figure 1.2).
Figure 1.2: The emotional expressivity (the animations can be downloaded from: http://emoto.sics.se/anima-tions).
In eMoto the user first writes the textual content of the message and then adjusts the background of the message to fit the emotional expression she wants to achieve. The adjustments are done through the affective gestures with an extended stylus, pick-ing up on movement and pressure, this will render an animated background acting as an emotional expression to the user’s text message. (Figure 1.3 shows examples of this, see side bar.)
To focus on the subjective experience of emotions we have used a dimensional model of emotions (Russell 1980) to let users combine their gestures with various emotional expressions. Dimensional models are presented in more detail in chapter three and five, but in short, emotions are treated as pro-cesses that blend into each other and not as discrete
states and they are mapped along two dimensions, for example, arousal and valence. The gestures as well as the graphical background are partly constructed from an analysis of body movements based on the theories of Laban, (Davies 2001, La-ban and Lawrence 1974), who was a famous dance choreographer and movement analyser. The char-acteristics of emotional expressions found in the analysis are represented both in the gestures, but also through colours, shapes and animations in the design of the animated background. This ex-pressive media with its tangible interface together with the interactive feedback will affect users both physically and cognitively and attempt too make them involved in an affective loop, where they can reflect upon and work with their emotions.
eMoto is used as a tool and is the first example in exploring the design of emotional expressivity in a digital media and not aimed at being a product for commercial use even if, as we shall see, we designed and implemented it to such a level that it could be deployed in the real world for testing. 1.4 Methodology
Through a detailed description of the design cess of eMoto, the aim is to make the whole pro-cess more transparent. The aim is to articulate and argue for every step in the design process. Only when we can collect details of this and similar design stories will we have the basis for turning the black-boxed parts of a design process into an object of research study. This articulation can be used as input and inspiration for other designers in future design processes.
The limitations given in the beginning of the eMo-to design process were eMo-to design for mobility, and emotional and bodily experiences, with this start-I think start-I’ll stay home
tonight.
I’m OK I feel better now.
I hate him! I’m so happy!
ing point the design process was developed. The design process in the eMoto case is not a linear structure, but very dynamic, always attempting to take the design forward. The description of differ-ent steps in the process below is a simplification: • A persona (Cooper 1999) was set up to illustrate
the targeted user group followed by a question-naire distributed among potential users, which encircled a potential area of emotional commu-nication.
• Brainstorming sessions were conducted to come up with potential ideas and also for evaluation of the ideas.
• An analysis of emotional body language using movement analysis, LMA (Laban Movement Analysis) (Davies 2001, Laban and Lawrence 1974) to understand the essence and characteris-tics of bodily emotions was done.
• Design of the graphical and the gestures for emo-tional expressivity started simultaneously with the design of the stylus used for input.
• Theory on colour, form and animation.
• An image board extending on what is already fa-miliar to us.
• An iterative sketching process using pen and pa-per, based on the background.
• Sketching in the computer, sketching in Adobe Illustrator, for refining the sketches and trying out properties that can be hard to achieve in hand-sketching.
• The animations and movements, and time dura-tions were tested in Macromedia Flash.
• Refinement of the expressions due to memory re-strictions in the implementation on the P900. • A user study on the graphical expressions • A redesign of the graphical expressions
• A final user study was conducted in real use of the eMoto-service.
In the chapter Design Case of eMoto this design process will be further described and reflected upon in more detail. The branch describing the develop-ment of the affective gestures will not be in focus in this thesis, although the whole interaction was tightly coupled in the design process, as it is de-scribed elsewhere (Sundström 2005).
1.5 Contributions
The main contribution of this thesis lies in eMoto as an example of an emotional expressive media, four desirable qualities when designing for emotional expressivity and the description of the design pro-cess. In the design process we have described and argued for every step and articulated the choices made that are often black-boxed. This can be used as inspiration and input in future projects, aiming at designing for emotional expressivity. The descrip-tion of the eMoto process is, of course, only the first one and more has to be added to get a more complete view.
From the analysis of the eMoto design process and user studies we have extracted four desirable quali-ties when designing for emotional expressivity: • Consider the media specific qualities
• Provide cues of emotional expressivity building on familiarity
• Be aware of contradictions between the modali-ties
• Open for personal expressivity
These qualities build only from the eMoto project and more projects aiming at emotional expressivity has to be conducted in order to refine and complete
these desirable qualities. The qualities are not gen-eral applicable, but are very dependent on the aims of the application. They can be used as input and inspiration for projects aiming at a design for emo-tional expressivity.
This thesis is composed of three papers. The re-search in the eMoto project is done by all three authors of the papers in co-operation. This thesis main focus is on the design of emotional expressiv-ity through graphical expressions and interaction design, where I have done the whole design of the graphical expressions.
All design activities were discussed by all three au-thors, but Petra Sundström was mainly focusing on the design of the affective gestures and eMoto as an example of the affective loop experience (Sund-ström 2005). She also implemented the service. The final user study was also planned, conducted and analysed by Petra Sundström.
1.6 Outline
This thesis is composed of three papers:
• Paper A presents an analysis of emotional body language, analyzed in terms of shape, effort and valence using Laban’s notation (Laban and Law-rence 1974) and Russell’s (Russell 1980) work on mental maps of emotions. Published at MUM’03 in Norrköping.
• Paper B describes how this analysis is imple-mented in the design of the graphical emotional expressions and the corresponding affective ges-tures in eMoto. Published in Journal of Personal and Ubiquitous Computing, Special Issue on Tan-gible Interfaces in Perspective, 2004.
• Paper C gives a detailed description of the design process of the graphical expressions, containing a user study of the emotional expressivity.
Pub-lished at ACM DUX’05 in San Francisco. In chapter two the methodological stance taken is briefly described. Chapter three gives a theoreti-cal background to the area of affective interaction, emotion theory and theories used in the design pro-cess of the graphical expressions. In chapter four a categorization of related work is provided. Chapter five describes the design process of the graphical expressions, a more thorough description is given in Paper C. In chapter six, designing for emotional expressivity is extracted into four desirable quali-ties. Finally, chapter seven summarizes the contri-bution and future work.
The main addition here compared to the papers is a deeper analysis of the media specific qualities, four desirable design qualities and a discussion around the articulation of the design process.
METHODOLOGY
sense of industrial design, not including all other types of design in general within computer science or HCI.
Coming into a field like HCI which is a knowl-edge-generating environment from a background as a design practitioner can be both problematic and confusing. The background to this probably lies within the shifting views on how design work is conducted and what is considered to be real and valuable knowledge. HCI has its roots and back-ground mainly in cognitive science that is search-ing for standard work practices, resultsearch-ing in mea-surable data meeting the pre-defined goals and generating generally applicable usability design guidelines and principles (McCarthy and Wright 2004). The HCI focus has traditionally been aimed at improving the efficiency of a work task between human and computer. History has had an impact on the methodology used and results presented even when the design moves into different areas than work tasks, such as designing for emotions, experi-ences and fun.
In a contribution on “boxesandarrows” 1 Löwgren
explores Just how far beyond HCI is Interaction Design? he discusses the adoption and adaptation of the design label, currently very fashionable in HCI, and makes the argument that it often is used a bit too loosely. This text also touches upon the area of how HCI and interaction design are related to one another. There is a need for an understanding of what design is and in particular interaction design. Löwgren claims that just re-labelling HCI to inter-action design is not enough and will not change it from what it already is. The difference is more fun-damental, the problem is not only that it is hard for
the area itself, the concept of interaction design is still new and there is no good or well established definition of what it really is, that can capture the essence of what it can contribute with.
A key question concerns what a design research contribution is and how we can guarantee its va-lidity and soundness as such. How can we assure quality aspects in interaction design? In HCI, us-ability goals can be measured trough user evalu-ation, giving results that can be seen as proof for whether it is good or bad design. The quality of interaction design work is harder to measure, be-cause the user experience goals are more complex and compound, consisting of both aesthetic experi-ences and user experiexperi-ences, all of them very tightly coupled and hard to separate from one another and to measure separately. In the end, the final result can still be tested, but it is hard to tell what parts of the results that can be tied back to the specific parts of the user experience. There are no ready-made answers on how to measure this, but if the end-results of a design process are good and work-ing there must be worthwhile focuswork-ing and shed light on the actual design process that lead to the specific design. Löwgren and Stolterman (2004) see results from a design process as the knowledge created in the process and not only the designed artefact.
Fällman (2003) addresses what design is and how it is related to HCI. He sees HCI as a design-ori-ented field of research and from this follows that it should be possible to get a feel for what design is and how it is done from studying literature in the HCI-area. He touches upon the issues mentioned above, but never concludes which parts of design 1http://www.boxesandarrows. com/
view/just_how_far_beyond_hci_is_in-teraction_design_ 2002, 8-Oct-05
that HCI could benefit from in detail. To answer the question what design is, Fällman turns to litera-ture of design theory, where design seemed to be described in a way that could be divided into three competing accounts; the conservative, the roman-tic, and the pragmatic. In short, in the conservative account the designer is an information processor, where the product is a result of a transparent pro-cess, where step by step rules are applied and the interface grows out of that process. In the pragmat-ic account the problem is decided by the designer, the process is a reflective conversation, where the product is the result of it. The romantic account de-scribes the designer as an artist where the process is a mystery, influenced by art.
When looking at the descriptions of these divi-sions, another problem occurs and that is the use of the word design and what is included in that. On one hand, design is something that HCI could make use of in the development of a prototype, but on the other hand design includes almost every-thing. There can be problems with viewing design as belonging to only one account, because design is rarely done using only one way of tackling prob-lems, it is a mixture between very structured work and creative work. For example, the conservative account is described as being possible to extract guidelines from and therefore a transparent de-sign process, whereas in the romantic account, the design happens in some sort of creative mystery, not making it possible to explain how the design happened, the design is black-boxed. In his thesis, Fällman (2003) comes back to this issue and states that none of the accounts should be too much emphasized nor abandoned in the way designed is conducted and discussed. In design work ev-ery step in the design process can be described in
some way, although not following exactly the same structure. Fällman writes about the role that design has in HCI research, where research prototypes just seem to happen. And he questions how theories and fieldwork actually turn into an application? He also gives an explanation to why this is happening: “The design process tends to remain implicit as researchers are embarrassed by not being able to show evidence of the same kind of control, struc-ture, and rigourness in doing design as they are able to show in other parts of their research.” (Fällman 2003:230)
What Fällman has addressed here is an interesting area, where design can inform and improve HCI. If this is one of the reasons that the design prototypes within HCI just seem to happen and that it also has become practice, does not open up for reflection around this whatsoever in the design process. It basically allows for coming up with any solution, without having any arguments in the design process explaining the outcome and without reflections around alternative solutions, which might have given a different or better result.
In this thesis I will reflect upon the design process of eMoto, describe and argue for the decisions made in the process. The decisions in a design process seldom happen by chance, but are well balanced against other alternatives or have their background in studies or theory. The design process is too com-plex and shifting to describe on a general level, but the designer still needs methodology and theory. The design process leading to an artefact is always a unique, normative approaches do not suffice, to handle such a complex task and process. (Löwgren and Stolterman 2004)
As Ilstedt Hjelm (2004) discusses, the design
pro-Figure 2.1: An example of how method, theory and research proto-type can carry the design research process forward. Where both theory and method can be used as input, but also become an outcome of the process.
cess is partly construction and partly problem-solv-ing. For the problem at hand, designing for emo-tional expressivity, our aim is to provide insight into the specifics in this particular design process, lead-ing to the system eMoto, but where some steps will be specific to eMoto and not applicable to any other design process in a one-to-one manner. Through providing a thorough account of the design process, we will provide one example of use of methods and theory for designing for emotional expressivity that other designers can use as inspiration in their own design processes. In the future, through providing more thorough descriptions of design processes in similar manners compare them and structure them in different themes, this could give more insights for other designers on how to start out a design process. In a similar manner Löwgren (2005) struc-tures design experiences in inspirational patterns, where several design examples in the same area are studied, from this abstractions of core ideas and im-portant design elements are done. As Löwgren and Stolterman (2004) discuss design theory as consist-ing of knowledge that will not lead the designer into having preconceived notions about how design processes can and will be conducted. The knowl-edge would also aim at creating new conditions for design and producing good examples.
Given this motivation as to why design stories are important knowledge construction in themselves, let us provide a brief summary of the methods and theory, used in the design process of eMoto – the full story is then provided in chapter five, where we shall end with a discussion on how this particular design process can be tied back to the overall ques-tions of how to do research in the design area.
2.2 Methodology Used in the Design Pro-cess
The limitations given in the beginning of the eMo-to design process were eMo-to design a mobile service, where emotional and bodily experiences were key ingredients in the communication with other users, with this starting point the design process was developed. A persona (Cooper 1999) was set up to illustrate the targeted user group followed by a questionnaire distributed among potential us-ers, which encircled a potential area of emotional communication. Brainstorming sessions to come up with potential ideas and to evaluate these were conducted. An analysis of emotional body language using movement analysis (Davies 2001, Laban and Lawrence 1974) to understand the essence and characteristics of bodily emotions was used. With this background input, the design of the graphical expressions started simultaneously with the design of the gestures and stylus used for input. The aim was to mirror the gestures in the visual design. The branch for describing the latter will not be in focus of this thesis, although the whole interaction was tightly coupled in the design process.
The graphical expressions have their basis in the analysis of emotional body language, theory on colour, form and animation and the inspiration of an image board extending on what is already fa-miliar to us. Using this as a base the expressions were constructed in an iterative process sketched out using pen and paper, to be transferred into the computer. The sketching then continued in Adobe Illustrator, for refining the sketches and trying out properties that can be hard to achieve in hand-sketching. The animations and movements, and time durations were tested in Flash Macromedia. The implementation of the expressions to work on
the SonyEricsson P800 and P900 phones lead to further changes, due to memory restrictions. The expressions were then tested in a user study that in turn led to a redesign of the expressions, sub-sequently tested in a final user study conducted in real use of the eMoto-service. In chapter five, De-sign Case of eMoto, this deDe-sign process will be fur-ther described and reflected upon in more detail. 2.3 Focus of the Methods
The methods used in the design process of eMoto, described above, have different focus. Some of the methods support design for research, while others are more closely connected to and aimed at prod-uct development and the market. The aim with the methods is slightly different. Methods for prod-uct development incorporate compromises and trade-offs, to get a product out on the market in a reasonable time and to a reasonable price. De-sign research on the other hand, is done with the aim of answering research questions. Therefore the methods can be chosen and altered to focus on the research questions. For example, the methods of image boards, personas and target groups are more market focused, whilst analysing emotional body movement are closer aimed at answering the research questions. The theories used on colour, shape and animation might land somewhere in be-tween these two. It can become problematic when using methods working towards two different aims. The result can become a product that does not exemplify and answer the research questions in the way intended. However, the methods aimed at market are often well established and tried out methods, which can lead the design process for-ward in an effective way. These methods are also valuable for the design process, but it is important
to have this in mind, so the trade offs these methods incorporate does not take the focus off the research questions. If this happens, the prototype becomes more of a product aimed at the market. But these market-oriented methods often bring with them a view of the user and use situation, which help mak-ing the prototype understandable for a user today, even though the prototype often has a futuristic perspective.
In the next chapter the theories used as background and inspiration in the design process will be pre-sented.
eMoto as an expressive media and the incorporation of cues of familiarity in the design of the graphical expressions, which will be explained in more detail in this chapter. In the following sections a back-ground to the area of emotions in computers and emotion theory is also provided, theories which has been used as background, inspiration and input into the design process of the graphical expressions will be presented; Laban’s Movement Analysis (Davies 2001, Laban and Lawrence 1974), theory on colour, form and animation.
3.1 Computers and Emotions
Emotions used to be regarded as unimportant and even disturbing when it comes to rationality, deci-sion-making and solving of problems. Emotions have been viewed as something that is uncontrol-lable and does not contribute anything to cognitive aspects of rational thinking. Recent research has shed light on and altered this belief. Emotions are now seen as closely connected and interdependent with rational thinking and rational decision mak-ing. It turns out that emotions are crucial for prob-lem solving (Damasio 1994). Emotional reactions cannot be separated from conscious reasoning. In general, recent research shows that emotions are closely linked both to our rational reasoning pro-cesses, and also to our physical, bodily reactions. The old separation of mind and body, going back to the works by Descartes (Damasio 1994), turns out to be unfortunate to our understanding of how hu-man decision making, reasoning and behaviour are initiated, controlled and enacted. Body and mind are intimately connected. Emotional experiences can originate not only from external stimuli, but
ing around, we can alter our bodily experience from sadness to happiness which in turn influence hormonal processes and affect our brain processes. This is why therapeutic methods such as dance or laughter therapy may have some relevance to actu-ally treating depression.
Emotion Theory
Emotion theory has had an explosive growth during the end of the last century (Scherer 2002). Emotion theory includes various areas such as neurosci-ence, psychophysiology, cognitive psychology and genetics. It is a very big area, describing all parts of emotion theory is not of interest here. Here the focus is on using emotion theory as background in the design process, to provide users with cues to start out from in building their emotional expres-sions in digital communication. The theory here is used as inspiration from an emotional model. We were looking for a model that could capture the view of emotions as processes and personal expe-riences.
The use of emotion theory in computer systems consists mainly of inspiration from different emo-tion models to be able to model and interpret data. Psychologist Scherer has structured and summa-rized the most common emotional models accord-ing to their components and phases in the emotion process. Answers to questions such as how many emotions there are and what they are, have vary-ing answers dependvary-ing on the theoretical stance taken. The models described below briefly provide some account of different perspectives on emo-tions in humans. We include it here to provide a background and understanding of the complexity
BACKGROUND
of human emotions, the reasons for having emo-tions and how that is shown. Some examples of models according to Scherer are:
Dimensional Models
In dimensional models emotions are described along two or several dimensions that describe different qualities of emotions. For example, the dimensions can be their valence (positive versus negative) and their activation level. In a sense, dimensional models attempt to describe the sub-jective experience of emotions. This allows for a model that aims to capture what is in common to emotional experiences, while still acknowledg-ing that there are individual and cultural differ-ences in how we experience emotions. Psycholo-gist Russell’s Circumplex Model of Affect (1980) is one example of a dimensional model, where emotions are placed according to their valence and arousal. Russell’s model builds on a study of people from our culture, distributing emotions in a circular space. This means that the model mirrors the subjects’ view upon emotions and could there-fore work as an inspirational basis in the design. It also captures our inner emotional experiences. The model will be described in more detail in chapter five.
Discrete Emotion Models
According to the theory of discrete emotion mod-els there are a set of basic emotions, for example Ekman’s six basic emotions; happiness, fear, sur-prise, anger, disgust and sadness (Ekman et al. 1972). These emotions are characterized by adap-tive responses to events, like fear/flight in response to an attack.
Meaning-oriented Models
The structure of emotions builds on the semantic meaning, categorizing emotions with almost the same meaning into groups. For example, the cate-gory of anger would cover words such as irritation, rage and being cross. Close to this view are social constructivist models arguing that certain emotions only exist in specific cultures (Lutz 1989).
Adaptational Models
The adaptational models view emotions from an evolutionary perspective. It explains the way hu-mans respond automatically to a certain stimulus that is potentially dangerous to us, like snakes and spiders. In this view, it is claimed that in evolution the human has developed this response called the biological preparedness. (Öhman and Wiens 2003) Appraisal Model
An appraisal models is a cognitive model, which explains the actions taken as result of an emotional experience. The emotions appraise the situations we are in, to help us choose the alternative that seems most appropriate for us. Emotions are explained from the way they affect the actions taken.
The dimensional models, described above, concur nicely with the notion of people experiencing emo-tions differently. It builds from the view people in our culture have on emotions and thereby grasp much of the characteristics of emotions in our soci-ety. The dimensional models were chosen to work as a background and inspiration from emotion the-ory.
In the next section a background on emotions in computer systems, from affective computing to af-fective interaction, will be provided.
Affective Computing
In the area of affective computing (Picard 1997) the importance of emotions to our everyday experi-ences and interactions is put forth as an argument to design computer artefacts in such a way that they take our emotional states into account. In affective computing the aim is, in short, to capture users’ emotional states through the use of bio-sensors, readings of facial data or body movements, and from that build computational models interpreting the input data and represent it back to the user in order to create better interaction (Figure 3.1). What the field of affective computing tries to achieve is very hard and might, similar to most Artificial In-telligence (AI) solutions, only work in closed envi-ronments and for a very specific,
restricted purpose. It is extremely hard to capture the social situations and the context around us through sensors and even harder to interpret what the data captured might represent. When it comes to design-ing for emotional expressivity and experiences of technology, there are also other, more fundamental, issues in how to address emotional processes.
Figure 3.1: Affective Computing.
Affective Interaction
As Boehner et al. (2005) discusses HCI has it roots in cognitive science and this philosophy creates a foundation that has its ground in that human mind can be understood and modelled. Traditionally in HCI there has been a focus on efficiency in work tasks, often with the aim of generating general de-sign principles (Preece et al. 1994). HCI has start-ed to approach emotions in computer systems, but has not always taken into account that the goal of the system may not be efficiency in a work task, the goal can be something completely different like having fun, expressing yourself or other val-ues which can hardly be fulfilled with a traditional HCI approach.
According to Boehner et al. affect in computer systems can be seen as information or as interac-tion. Seeing emotional experience as information means seeing it as consisting of well-defined, dis-crete units that can be transferred from human to the computer repeatedly, without changing the content of the emotions. An example of this view would be affective computing. Affect seen as inter-action, on the other hand, is seen as co-construct-ing and active interpretation of one’s emotions. Affective communication becomes in this per-spective, an ambiguous, complex and ill-defined process. The aim here is to make the design open up for interpretation, experience and for the pro-duction of emotions. The emotions build and ex-tend on socially already existing communicative relationships between people. The view of affect as interaction allows for a user-centred approach to affective interaction, where the user can be in control of what emotions are expressed and also open up for emotional expressions that users can, to some extent, create their own interpretations of.
This view opens up for an interaction as a process, for an affective loop (Höök et al. 2004).
The view of communication of affect as interaction would follow a completely different process than that of Affective Computing. This is described as a user-centred affective loop, where both aims and methods are user-centred, which in the end give the users stronger emotional experiences (Sund-ström et al. 2005), (Figure 3.2).
Figure 3.2: Affective interaction.
Boehner et al. (2005) lists five design principles from their experience of designing for affect as interaction. The principles discussed are (quoted from Boehner et al. 2005: 65-66)
The interactional approach recognizes affect as a social and cultural product
This builds on emotions being something that re-lates to the context and social situation we are in, that we need to have grounds in the real world for meaning-making with the system.
The interactional approach relies on and supports interpretive flexibility
To leave the definition of emotions and its interpre-tation for the user to decide upon, emotional mean-ing emerge between users in situated way.
The interactional approach avoids trying to for-malize the unformalizable
Here the in the interactional approach the emotions should not be formalized by the system, instead the users should supply the emotional meaning in the system.
The interactional approach supports an expanded range of communication acts
Emotion can in this way be communicated in a richer way, than clearly defined signs allow. Instead of labelling the emotional expressions the user can mirror their emotions in the affective expression and find something that suits them. This should ac-cording to Boehner et al. be achieved through the designer not putting in meaning within the system, for example, that a fast-moving dot should be inter-preted as high energy.
For the user to take the design in use, the designer should design for meaning, building on what is familiar to us, so that we can recognize cues and patterns in the design, but the designer should not explicitly label the expression, it is still up to the user to decide what these expressions means to her. As in the discussion above, it is quite much to ask from the user to build meaning from scratch into a design. The distinction between too much cues of meaning and too little is very subtle.
The interactional approach focuses on people using systems to experience and understand emotions In the design open up for the complex, ambiguous
nature of emotions by designing for the experience of emotions, through use make people more aware of their emotions. Hence the systems do not have to read or label users emotions.
This view of affect as interaction does not as, for example, in affective computing try to simplify emotions, instead it extends on the qualities of emo-tions, and makes use of these to augment the com-plexity of interaction. The view of what amount of meaning that should be built into the application by the designer differs, as can be seen in Figure 3.3. In examples such as emoticons, the expressions are explicitly labelled not leaving much room for the users to interpret and create their own mean-ing around the expression. The view on smilies has changed over time, in the beginning when first used, smilies were thought of as being quite open for in-terpretation in their expression form. With more ad-vanced techniques for expression emotions, smilies have become more static in their appearance. On the other hand the approach Boehner et al suggests does not build in much meaning at all, leaving the interpretation and meaning-making totally in the hands of the user. Our approach and design would be placed somewhere in between. It would provide the user with a foundation, a set of tools, building on patterns and cues that are familiar to them in their own realities, but without explicitly labelling them. Something that can help users get started in interac-tion and meaning-making but that can be modified over time and interpreted differently depending on context and the communication partners.
Figure 3.3: Amount of built in meaning in a system, which varies from explicitly labelled expressions to not building in any meaning.
Some of the problems that Boehner et al. (2005) have touched upon in this view lie in the problem with the HCI-approach to the area of affect, where one can not use the same methods and grounds for evaluation as in traditional HCI-work. Unfortu-nately they sum up their own work by naming the result design principles as in traditional HCI. This is probably a result of designers lacking terms and language to write about these things in way that reflect the content. The content in the principles are findings that resonate with our view on how to design for affective interaction. Not having a lan-guage to talk about design easily pulls us back to the problem of what design research is all about. In particular in relation to attempts to design for ‘experiences’, namely describing results in an area which should be ambiguous, flexible in interpreta-tion, subtle and open up for experiences. Achiev-ing these qualities in a design would require a bal-ance, which probably can not be generalised to fit all systems designing for affect as interaction. This does not mean that the “principles” content is of bad quality themselves – only that they should be reformulated in a shape that does not claim to be
design principles.
For emotions to be communicated through digi-tal techniques, like SMS, it is important that they build from socially, already existing communica-tive relationships and are situated and context-de-pendent. When implementing this into a design, as discussed above, the cues that are visible to us when expressing emotions to each other in real life have to be included. Therefore the founda-tion for expressions in the eMoto-case, builds on an emotional model, emotional body language and existing theories on colour, shape and animation, which are used in the design of the expressions of meaning making cues. Completely avoiding pro-viding cues in the expression of the design would probably mean ending up with a design that would build on completely random expressions. This would require much effort from the users in creat-ing expressions from scratch. If users should build their own communication, there has to be some cues for users to start out from, for them to be able to start interacting. There is a balance required in how much cues to provide depending on the kind of application one deals with.
3.2 Cues of Familiarity through Sub-sym-bolic Expression Forms
In the view of emotions as inner, bodily processes and partly depending on socially, already existent communicative relationships that are situated and context-dependent, cues for the user to start out from is needed. The cues build on what already is familiar to us, like body language, how colours, shapes and movements are perceived. This is re-alised in sub-symbolic expression forms of colour, shape and animation. Sub-symbolic in the sense that these visualisation forms address people in
a more direct manner, without adding symbols or words that have to be consciously interpreted. In the following sections the background to this is pre-sented.
The area of bodily emotions, that emotions and body movement are intimately connected has not been widely explored. Dance theory and choreog-raphy touches upon this subject. Laban (Davies 2001 and Laban and Lawrence 1974) a famous choreographer analyses movement in terms of the inner experience, which corresponds nicely with the view of emotions as inner, bodily processes and has therefore been used here.
Laban’s Theory
Laban’s theory oftentimes referred to as LMA (Da-vies 2001) allows for the possibility to articulate the characteristics and essence of emotional body movement. This theory is used in analysing the study of emotional body language conducted in the design process of eMoto.
Laban worked as a choreographer and movement analyzer, in his work he defined five underlying di-mensions in movement: body, space, effort, shape and relationship. The focus in our analysis of the study is on effort and shape, as these best describe the bodily emotion expressions. Laban’s formalism for describing movements and theories about shape and effort will be introduced here, at least at a shal-low level, in order to explain the study in the design process.
Shape and Effort According to Laban
Shape describes the changing forms that the body makes in space, while effort involves the “dynam-ic” qualities of the movement and the inner attitude towards use of energy (Zhao, 2001).
Shape can be described in terms of movement in three different planes: the table plane (horizontal), the door plane (vertical) and the wheel plane, which describes sagittal movements. Horizontal move-ments can be somewhere in-between spreading and enclosing, vertical movements are presented on a scale from rising to descending, and sagittal move-ments go between advancing and retiring (Figure 3.4, see side bar).
Effort comprises four motions factors: space, weight, time and flow. Each motion factor is a con-tinuum between two extremes (Table 3.1).
Motion factor Dimensions Examples Space attention to the surround-ings Indirect (flex-ible): spiralling, deviating, flex-ible, wandering, multiple focus Waving away bugs, survey-ing a crowd of people, scan-ning a room for misplaced keys Direct: straight, undeviating, channelled, single focus Threading a needle, pointing to a particular spot, describing the exact outline of an object Weight attitude to the movement impact Light: buoyant, weightless, eas-ily overcoming gravity, marked by decreasing pressure Dabbing paint on a canvas, pulling out a splinter, describing the movement of a feather Strong:
power-ful, forcepower-ful, vig-orous, having an impact, increas-ing pressure into the movement
Punching, push-ing a heavy ob-ject, wringing a towel, express-ing a firmly held opinion
Motion factor Dimensions Examples Time lack or sense of ur-gency Sustained: lei-surely, lingering, indulging in time Stretching to yawn, stroking a pet Sudden (quick): hurried, urgent, quick, fleeting Swatting a fly, lunging to catch a ball, grabbing a child form the path of danger, making a snap move Flow amount of control and bodily tension Free (fluent): uncontrolled, abandoned, unable to stop in the course of the movement
Waving wildly, shaking off water, flinging a rock into a pond Bound:
controlled, re-strained, rigid
Moving in slow motion, tai chi, fighting back tears, carrying a cup of hot tea Table 3.1: The dimensions of effort according to Laban as described by Zhao (Zhao 2001).
In Figure 3.5a we depict the graphs Laban uses to express effort. As an example, Figure 3.5b presents an effort graph of the movement of inserting a light bulb where the movement is direct in space, light in weight, sustained in time and bound in control.
Figure 3.5: (a) Laban’s effort graph, (b) an example effort graph of inserting a light bulb (Laban and Lawrence, 1974). Figure 3.4: The three different
planes of shape, adapted from Da-vies (DaDa-vies 2001).
Laban’s notations on how to describe the character-istics in emotional body language gives us a way to articulate and describe similarities and differences between different expressions. This in turn allows for conveying these characteristics in the design of the emotional expressivity and through the user study see if these characteristics came through. Colour, Shape and Animation
In the design of the expressions in eMoto to work as a foundation, the patterns and cues provided to build on familiarity have their basis in theories on colour, shape and animation. The expression could as well have built from expressions in, for exam-ple, in music or haptics, but in this case we have chosen to work with the above three. In the design it is the combination of all media that gives the ground for emotional expressivity. Here we have separated them as an attempt to understand the contribution from each and one of them.
In the area of colour theory, there is an ongoing debate on the cultural dependence of colours and to what extent it is valid. Many things in the world are cultural dependent, shapes, movements, ges-tures are also cultural dependant and bear differ-ent kind of meaning in differdiffer-ent cultures, but in each culture we have some kind of common basic understanding of these meanings. The theories of colours, shape and animation presented below are the result of work done within a specific culture and might not be valid all over the world, although some parts might be more common. And these common basic understanding within a specific cul-ture, can be used as an inspiration and something to build from in a design.
Theory on Colour
Colours can and have been studied with many dif-ferent approaches, for example, in physics, through the cultural differences and how we are affected by colours emotionally. There have been many attempts to explain how we are influenced by co-lours, but not any real scientifically approaches. But there is something that affects us with colours, which many has tried sort out. Some of these colour theories and systems will be briefly described here. Already Pythagoras, Aristotle and Plato examined colour mixtures and created colour systems. In the eighteenth century, Newton (Figure 3.6, see side bar) was the first to arrange colours in a circular system, Newton’s approach to colours was in the area of physics only (http://www.colorsystem.com, 14-Nov-05).
About 100 years after Newton, the German poet Goethe (1810) studied the nature of colours with a different approach. His interests were more in the psychological effects of colour, to bring order to the (in his opinion) chaotic and aesthetic aspects of colour. In Goethe’s colour-circle (Figure 3.6, see side bar), the basic pair of yellow and blue were not placed opposite to one another, but placed together with red in a colour triangle. Goethe writes about colours as having negative and positive qualities. He refers to the part of his circle from yellow to red as the plus-side while the opposite side is the mi-nus-side. Goethe discusses colours in terms of yel-low being associated with effect, light, brightness, force, warmth and closeness and blue as depriva-tion, shadow, darkness, weakness, cold, distance. The colours on the positive side in the system, in-duce an exciting, lively, aspiring mood, whilst the colours on the negative side create unsettled, weak and yearning feelings.
Figure 3.6: Newton’s, da Vinci’s and Goethe’s colour systems from http:// www.colorsystem.com.
Both Itten (1971, 1973) and Albers (1975) ex-amine the colour as a means for interaction. Itten (1973) discusses the cold - warm qualities that co-lours contain which can be verbalized in numbers of other contrary terms, he mentions: cold-warm, shadow-sun, airy-earthy, light-heavy, wet-dry etc. These impressions illustrate the expressive pow-ers of the cold-warm contrast, which can be used in expressing creating depth and perspective effects within a picture.
Colour Psychology
In the area of colour psychology the aspects of co-lours’ effect on humans are even more investigated than in the above theories. According to this field different colours are said to influence us in different directions and should also initially be interpreted in a similar way.
For example, a room with the exact same tem-perature was experiences differently depending on the colour of the walls. A room painted in orange was experienced as warmer than a room painted in blue-green colours (Ryberg 1991). Colour should also create automatically responses or abstract as-sociations. Ryberg describes the amount of energy in different colours, where red represents the most powerful and strong emotions, moving along a co-lour scale ending with blue, means moving towards less energy. He also describes the qualities of each specific colour (Figure 3.7, see side bar).
Theory on Form
When discussing the execution of form in products, its meaning and how this is perceived by users, the field of industrial design often relates to product semiotics. Monö (1997) discusses the study of sign and sign systems and their structure and role in
socio-cultural behaviour and relates it to product design. In industrial design different qualities that a product should possess is often expressed in the design. This is realised in the physical form of the product, the choice of material, the sounds it produce (for example when closing a car door) etc. The meaning that we arrive at, when our senses perceive these different qualities should make it easier for us to understand the message of the product.
In a research project in general and especially when dealing with emotions there is a slightly difference in approach, work process, and goal when working with form and design, than when developing a product for the market as an industrial designer. Using semiotics and semantics as a way of discussing form factors is one way of putting words on these subtle things as Monö does. The approach that Monö takes can be problematic, since there is no conventionalised way of doing or seeing form; it is by nature quite vague and arbitrary. Applying Monö’s view of meaning in form, you would almost have to have a library of curves, form elements, structures, sounds etc. to pick from. Putting these together in a certain way would create certain qualities, like for example heavy and fast. There lies a problem within theorising these things, or maybe the problem lies within the theoretical stance taken? The issues Monö discusses are of great importance to the area of form and design, although they can also be discussed without a specific theoretical stance taken. Norman (1990, 2004) discusses the same issues arriving at similar conclusions, but with a different theoretical perspective, maybe somewhat influenced by his background as a psychologist. Red: Instinct,
heat, sexuality, strength, courage, fight preparedness, impulsiveness Orange: lively, joy, friendship, generosity, abundance Yellow: Intellect, hastiness, clarity, precision, logic, communication Green: Dream, hope, tenderness, sensitivity, youth, artistic Blue: intuition, idealism, truth, authority, calm, concentration Violet: will, power, religion, mystery, genial, artistic
Figure 3.7: Examples of the quali-ties of each colour (translated from Swedish) (Ryberg 1991).
Discussion around Form and Meaning
The different parts of a product, like in our case colour, form and animation, are not isolated factors but work as a whole together, influencing each other. Hesselgren (Monö 1997) presents factors that help us discern this. Examples of these are: proximity, similarity, area, common movement and experience (Figure 3.8, see side bar).
The ways formal elements are arranged and related create meaning through the way they dynamically either harmonize or contrast to each other. Depending on what the aim is with the product’s expression, the different formal elements can be put together to form a coherent whole or be contradictory and in that case it can cause confusion or suspicion by the user. Visual cues must correspond with tactile or audio signs to not cause confusion, for example, a car giving the impression of being a very robust, cross-country car, but when the engine starts it sounds as though it has a very tiny motor. You would probably not believe in the car’s cross-country qualities. The physical form a product has creates expectations from the user of other qualities that it holds. A static form can be perceived as heavy or slow and a sleek, graceful form could be perceived as light (Figure 3.9). A form can also express two or more different qualities, which also may seem contradictory in their appearance. The picture below shows interplay between two different expressions that could be regarded as counteractive (Figure 3.10). Another example is products with the same functions, for example a mobile phone, can in its form express status, youth, “techiness” or other properties that it holds.
Figure 3.9: Static and graceful.
Figure 3.10: Counteractive expressions in a ski boot, ex-pressing both speed qualities and robustness
Except for describing the properties, the form in itself can also describe how it should be used, for example, how to manage different light switches. The form of the light switch indicates that it should be flipped, while the second light switch does not hold the same properties, but instead describes itself as to be pushed (Figure 3.11). This is applicable on digital products as well where the form can describe the function, buttons to be pressed, slide bars to be
Proximity, grouping of objects
Similarity, objects that look alike
Area, the smaller area pops out
Common movement
Experience, round shapes are per-ceived as more friendly
Figure 3.8: Showing examples of the different factors from the design in eMoto.
moved in a certain direction, make information look active or passive etc (Figure 3.12).
Figure 3.11: To be flipped and to be pushed.
Figure 3.12: To be pressed.
Monö also discusses our current conception of how a product’s principal function is expressed in its form. This current conception’s appearance are such strong bearer of meaning to us, so even when presented in a very abstract way, we interpret them as the product (Figure 3.13, see side bar). If the principal function is not expressed, it is difficult to interpret. This explains why a shape of random proportions gets ambiguous and is open to many different interpretations. People often try to read in symbolic meaning into abstract patterns of shapes and try to construct a story around it (Heider and Simmel 1944).
Often when we design we have visions of what we want to convey in our design, expressed in words and spoken language. To convert these into form elements is something we unconsciously do daily, by speaking of strong and weak, warm and
cold colours. In the same manners we perceive round shapes as more friendly and positive while an angular, spiky shape is perceived as more negative.
Animation
There has been very much work done in practical animation, but not so much in the area of how to convey emotions in animation, how to capture the subtle characteristics that make an animated expression create affect or what makes an animated character appear to have human emotions in a believable way. Thomas & Johnston, Disney animation (1981) talks about harmony and disharmony in animation as a way of conveying emotions. Movements that harmonize can be perceived as positive, imagine for example the billowing waves of a sea (Figure 3.14, see side bar). On the other hand movements of object that are disharmonizing, looking as though they are close to colliding can creative negative emotions. Imaging the same sea but in full storm where the waves are moving in all directions in the wind, breaking against the cliffs (Figure 3.15, see side bar). Another important source of inspiration according to Thomas & Johnston is using patterns of movement that we already are familiar with and by experience connect to a certain emotion. Examples of this can be the pattern of movement in nature, like the motion of lightening or billowing waves on a calm sea. Body movements and posture is another example of familiar patterns that can be used in animation to convey emotions. Overlays, creating depth in the picture by the way the objects are laid out can create a feeling of emotional expression, using overlays and a dramatic layout is only useful when creating negative emotions Figure 3.13: An abstract
representa-tion where the principal funcrepresenta-tion is expressed is less open for interpre-tation.
Figure 3.14: Movements that har-monize.
Figure 3.15: Movements that dis-harmonize.
(Figure 3.16, see side bar). Lasseter (1987) discusses the importance of timing in animation, the speed of an object gives emotional meaning to the movement. It can be used to portray the weight and size of an object.
Summary
The theories described in this chapter have been used as background and inspiration in the design process. The view of emotions, as laid out in the dimensional models described above, builds from the view people in our culture have on emotions. It can thereby inspire a background to the design of the emotional expressivity. Seeing emotions as interaction and not as information, as discussed by Boehner et al., is the ground for the emotional expressivity to build from socially, already existing communicative relationships that are situated and context-dependent. Analysing emotional body language using Laban’s notation provides even more input into the practical design of the emotional expressivity. The theories of what colour, shape and animation portray, can be used as modalities in designing cues which are used in the design of the expressions for meaning making. These insights will be implemented into the design process of eMoto, which will be described chapter five, but first a categorization of emotional communication artefacts will be provided in the next chapter. This categorization describes the state of the art in the area.
Figure 3.16: Creation of depth in the picture, example from the eMoto design.
expressing emotions through other media, we will describe a number of systems implemented by other researchers. The applications described all aim to aid users to express affect in a way that is to the goals of affective interaction rather than affective computing. All the systems we have chosen to describe here focus on communication between users. In different ways they combine the modalities described in the previous chapter. People need to communicate their emotions to each other – sometimes through technology. This has led to a whole array of attempts to build technology and services for expressing emotions through interactive media. Emotions are expressed in many and different ways by different people and it is a combination of various factors that make us understand what someone is attempting to express. We get some cues from body posture, the intonation of the voice, body movements, facial expressions and the wordings chosen. Important for the understanding of emotional communication is also that it is context- and situation dependent. The specific people communicating, their relationship to one-another and the situation that they are in, often provide the basis for understanding the subtle signals conveyed by, e.g., body postures or movements.
Designs that allow users to communicate their emotions through interactive media have been created with a range of different aims. Besides smileys and variants of smileys, such as emoticons, we can distinguish between three classes of solutions: communicating through specifically designed artefacts, communicating through composing multimedia messages (MMS, Instant
sub-symbolic expressions, such as colours or animations, to textual media of different kinds. Let us describe some systems from each of these categories.
4.1 Designed Communicating Artefacts The designed communicating artefacts is a category that has been widely explored. It has resulted in numerous systems allowing for expressing emotional concern or presence among, in most cases, two people having a close relationship, who are separated by physical distance. The solutions often involve equipping two people with one device each that communicate wirelessly. Quite often, the systems are built for pairs of users allowing them to communicate emotions or show concern to one-another, but some devices also allow for multiple users. The aim is often to focus on “minimal” emotional communication like showing presence or concern.
Feather, Scent and Shaker
are three systems design by Bill Gaver and Rob Strong (1996). Their design goal is to support simple, expressive interaction. Feather is a system supposed to be used in situations where one person is travelling and the other stays at home. The couple have one device each; the person who is away from home has an interactive picture frame and the person at home a cone containing a feather. When one user is handling the picture frame, a fan starts blowing the feather at home, illustrating affection in as a poetic experience of connection (Figure 4.1, see side bar).
Scent, the second system, builds on the same idea Figure 4.1: Feather (Gaver and
Strong 1996).
Figure 4.2: Scent (Gaver and Strong 1996).
CATEGORIZING
EMOTIONAL
COMMUNICATION
ARTEFACTS
Figure 4.3: Shaker (Gaver and Strong 1996).
