Designing Gestures for Affective Input: An Analysis of
Shape, Effort and Valence
Petra Fagerberg*, Anna Ståhl*, Kristina Höök
Stockholm University/KTH
DSV
Forum 100
164 40 Kista
* Authors in alphabetical order
Abstract
We discuss a user-centered approach to incorporating affective expressions in interactive applications, and argue for a design that addresses both body and mind. In particular, we have studied the problem of finding a set of affective gestures. Based on previous work in movement analysis and emotion theory [Davies, Laban and Lawrence, Russell], and a study of an actor expressing emotional states in body movements, we have identified three underlying dimensions of movements and emotions: shape, effort and valence. From these dimensions we have created a new affective interaction model, which we name the affective gestural plane model. We applied this model to the design of gestural affective input to a mobile service for affective messages.
Keywords: Affective interaction, gestures, user-centered design, mobile service
1 Introduction
By addressing human emotions explicitly in the design of interactive applications, the hope is to achieve both better and more pleasurable and expressive systems. The work presented in here is inspired by the field of affective computing [Paiva, Picard], even if our aim is to take a slightly different stance towards how to design for affect than normally taken in that field – a more user-centered approach.
Affective computing, as discussed in the literature, is computing that relates to, arises from, or deliberately influences emotions [Picard]. The most discussed and spread approach in the design of affective computing applications is to construct an individual cognitive model of affect from first principles and implement it in a system that attempts to recognize users’ emotional states through measuring biosignals. Based on the recognized emotional state of the user, the aim is to achieve an as life-like or human-like interaction as possible, seamlessly adapting to the user’s emotional state and influencing it through the use of various affective expressions [e g Ark et al., Fernandez et al.]. This model has its limitations [Höök], both in its basic need for simplification of human emotion in order to model it, and its difficult approach
into how to infer the end-users emotional states through various readings of biosignals.
To get the users involved in a more active manner we would, instead, like to propose the user-centered approach to affective computing. Our aim is to have users consciously expressing their emotions rather than having their emotions interpreted or influenced by the system, while still maintaining the mystery and open interpretation of emotional interaction and expression. Inspired by the results of our previous work [Paiva et al.] we arrived at a set of four design principles, outlined in detail below: embodiment as a means to address physical and cognitive concepts in the interaction with the application [Dourish], natural but designed expressions as a means to communicate affect instead of aiming for complete naturalness, an affective loop to reach emotional involvement with both body and mind, and ambiguity of the designed expressions [Gaver et al.] to allow for open-ended interpretation by the end-users instead of simplistic, one-emotion one-expression pairs.
Our specific focus in this paper is to describe the process of finding affective gestures for interacting with a mobile service. Our idea is that gestures will address the body-part of emotions in people. When placed in an interaction that also speaks to our mind, the result may be an increased sense of actually communicating affect. Based on previous work in movement analysis [Davies, Laban and Lawrence], emotion theory building upon people’s everyday understanding of emotion states [Russell], and a study of an actor expressing emotional states in body movements, we identified three underlying dimensions of movements and emotion: shape, effort and valence.
To exemplify our design principles and our ideas of affective gestures, we approached the design of an application for a mobile setting, an affective messaging service. An important part of telephone communication is its usage to maintain intimate and close relationships between people [Castelfranchi]. In mobile phones this is done both through phone conversations but also through text messaging (e g SMS1 and MMS2) [e g Grinter and
Eldridge]. In the messaging interaction, the affective bandwidth is narrow, and most of the richness of the emotional content is lost. This also has a negative impact on the communicative bandwidth. The designed affective message application makes use of a combination of gestures and a pulse sensor as affective input, and uses emotional expressions in graphics (color, shape, animation) as output. An important goal is to mirror form and content of the gesture input in the emotional expressions added to the message. Below we first describe our design principles in more detail, before we turn to the specific problem of designing the affective gestures. We describe our affective interaction model, which we name the affective gestural plane model. The mobile service for
1 SMS: Short Message Service, used to send text messages
between mobile phones.
2 MMS: Multi-Media Messaging Service, used to send
multi-media between mobile phones. Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. MUM 2003 Norrköping, Sweden.
affective messaging, which we describe last, exemplifies how our design framework and the affective gestural plane model might be applied.
2
Designing for affect
While early theories on emotions regarded emotions as discrete states [Ortony et al., Roseman et al.], later work has seen emotions more as processes and appraisal functions that regulate behavior [Paiva], not on or off singular states. As discussed by Castelfranchi, [Castelfranchi], emotions are subjectively experienced states, and we all react differently depending on our background, our previous experience, our mental and physical state and other individual factors. Depending on the social setting we may also express our emotions differently. Expressing happiness during a football game will be quite different from expressing happiness at a business meeting. Thus, recognizing emotional states from biosignals or other physical or external signals is an extremely difficult task – especially in a mobile scenario with its ever-changing psychical and social contexts. Therefore, emotions as part of human communication is better seen as a human, rich, enigmatic, complex, and ill-defined experience. This experience does not solely sit in the brain as part of a rational, cognitive reasoning process. Instead, body and mind are intimately connected [Davies, Dourish, Ekman, Laban and Lawrence, Picard], and emotions cannot be seen solely as a mental state but also a physical, bodily, state [Ekman, Picard]. Emotions can be generated through someone’s imagination without physical interaction, but they can also be generated from body movements [Ekman]. Try moving as if you are extremely happy and you will probably also experience a warm feeling that slowly grows inside you. It is quite hard to feel sad while jumping up and down and smiling.
In order to design for subjective affective experiences with a user-centered perspective that addresses both body and mind, we extracted four, interrelated design principles that we adjusted to the particular motives and needs of our design situation.
2.1 Embodiment
Dourish [Dourish], defines embodiment as “the creation, manipulation, and sharing of meaning through engaged interaction with artifacts”. By artifacts he does not only mean physical objects, but also social practice. Rather than embedding fixed notions of meaning within technologies, embodied interaction is based on the understanding that users create and communicate meaning through their interaction with the system and with each other through the system. The concept of embodiment allows Dourish to combine two trends from the human-computer interaction area; tangible interaction where interaction is distributed over the abstract digital world and objects in the physical world [Ishii and Ullmer], and social computing where social practice and the construction of meaning through social interaction is core in design [e g Bannon].
Designing for embodied affective interaction thus entails both looking for the physical artifact embodiment of abstract emotion concepts, as well as allowing for social practice and interpretation of meaning of the emotional expressions. The physical embodiment concurs nicely with the strong connection between body and emotion, as discussed above.
2.2 Natural but designed expressions
To get users physically involved, one approach is to build the interaction upon our previous physical and cognitive experiences of emotional processes. This approach can be applied to the
design of the whole interaction, including both input and as well as output channels and the connection of the two in the application.
Human-computer interaction and human-computer-human interaction are not and should perhaps not be the same as human-human interaction. An application is a designed artifact and can therefore not build solely upon (whatever is meant by) “natural” emotional expressions. On the other hand, using mainly designed expressions bearing no relation whatsoever to the emotional experiences people have physically and cognitively in their everyday lives, would make it hard for the user to recognize and get affected by the expressions. Therefore we argue that emotional expressions should be aiming to be natural but designed expressions.
The specific focus of this paper is how to design for affective gestures. When studying the research done on gestures in computer interaction in general there are two main strands that exemplify the conflict: designed gestures [e g Long et al., Nishino et al.] and natural gestures [Cassell, Hummels and Stappers]. Designed gestures can, for example, be resembled to sign language. The gestures make up a language and depending upon the complexity of the language, it may take quite some effort to learn. Natural gestures, on the other hand, aim to be easier to learn as they build upon how people tend to express themselves in various situations. Body language, posture and more conscious gestures, however, vary between individuals, cultures and situation. Thus, designers of gesture interaction often aim for designed gestures based on natural behavior, looking for the underlying dimensions giving rise to the specific movements.
2.3 Affective loop
The aim of the affective loop idea, is to couple the affective channels of users closely to those of interactive applications, so that the user’s emotions are influenced by those emotions expressed by or through the application, and vice versa. Through designing for physical expressions of the end-user (e g body posture, gestures, tangible input through toys, speech) that makes sense with regards to the design of the overall interaction or narrative or the system they interact with, we try to make users involved both physically and cognitively. By having users express their emotions in interacting with the system, they can be engaged in an affective loop, where their emotions are either affected or increased in intensity, either by the modality by which the emotions are submitted or as a response to output.
An example of a system that inspired and explored the affective loop idea is SenToy [Paiva et al.]. SenToy is a doll, which is used as an input device to a game. The end user interacts by acting out various emotions through movements with the doll. For example, to express anger, the user needs to shake the doll back and forth. The idea was that these body movements, together with the resulting activities appearing in the game progression would also influence users emotionally, both their body and mind.
The other part of the affective loop, the emotional output, concerns how the system in turn expresses its response to the user input. Some modalities, such as color and shape [Itten], movement, and music stand a better chance to address our physical experience. For example, according to Ryberg [Ryberg] humans have the same first instinctive reaction to colors. In movies music is used to put us in different emotional states [Bordwell and Thompson]. Bresin and colleagues [Bresin and Friberg] have produced a system, which given a piece of music can replay it to express different emotions.
2.4 Ambiguity
Most designers would probably see ambiguity as a dilemma for design. Gaver, however, looks upon it as “a resource for design that can be used to encourage close personal engagement” [Gaver et al.]. He argues that in an ambiguous situation people are forced to get involved and decide upon their own interpretation of what is happening. As affective interaction oftentimes is an invented, on-going process inside ourselves or between partners and close friends, taking on different shades and expressions in each relationship we have with others, ambiguity of the designed expressions will allow for interpretation that is personal to our needs. For example, if a system was to have buttons where each was labeled with a concrete emotion, users might feel extremely limited since they would not be able to convey the subtleties of their emotional communication to others.
Ambiguity may also follow from the ideas of embodiment, that sees meaning as arising from social practice and use of systems – not from what designers intended originally. An open-ended ambiguous design might allow for interpretation and for taking expressions into use based on individual and collective interpretations – both by sender and receiver of affective messages. Ambiguity in a system will perhaps also create a certain amount of mystery that will keep users interested. However, there needs to be a balance, since too much ambiguity might make it hard to understand the interaction and might make users frustrated [Höök et al.].
3
A model of affective gestures
While any service that attempts to instantiate the design ideas outlined above should be concerned with the whole interaction and not only one part of it, this paper will be focused mainly on the affective input side. As discussed above, we wanted to involve users physically with the application and our idea from the SenToy-work was that natural but designed gestures for affective expressions could be an interesting design alternative.
In order to find affective gestures that can express emotion, we turned to the work by Laban and his colleagues [Davies]. Laban was a famous dance choreographer, movement analyzer and inventor of a language for describing the shape and effort3 of
different movements. His work will not lend itself to turning emotional expressions into a table with one-to-one mappings of movements to emotions – but his theories of movement can be used to understand the underlying dimensions of affective body behaviors.
To map emotional body behavior to Laban’s dimensions of movements, we invited Erik Mattsson4, an actor, who works with
counseling and education in human communication. We asked the actor to express nine different emotional processes in body language, while we videotaped him. In a questionnaire distributed to 80 SMS-users in Sweden we found the emotions they mostly wanted to communicate in mobile messages: excitement, anger, surprise-afraid, sulkiness, surprise-interested, pride, satisfaction, sadness and being in love.
Before we turn to the analysis of the movements, we need to introduce Laban’s formalism for describing movements and
3 Laban’s theory oftentimes referred to as LMA (Laban’s
Movement Analysis) is composed of five major components: body, space, effort, shape and relationship. The focus in our analysis is on effort and shape as these best describe the emotion expression contained in gestures.
4 http://www.ordrum.com/erik.html
theories about shape and effort, at least at a shallow level, in order to understand the analysis of the actor’s expressions.
3.1 Shape and Effort according to Laban
Shape describes the changing forms that the body makes in space, while effort involve the “dynamic” qualities of the movement and the inner attitude towards use of energy [Zhao].
Motion factor Dimensions Examples Space attention to the surroundings Indirect (flexible): spiraling, deviating, flexible, wandering, multiple focus
Waving away bugs, surveying a crowd of people, scanning a room for misplaced keys Direct: straight, undeviating, channeled, single focus Threading a needle, pointing to a particular spot, describing the exact outline of an object Weight attitude to the movement impact Light: buoyant, weightless, easily overcoming gravity, marked by decreasing pressure Dabbing paint on a canvas, pulling out a splinter, describing the movement of a feather Strong: powerful, forceful, vigorous, having an impact, increasing pressure into the movement
Punching, pushing a heavy object, wringing a towel, expressing a firmly held opinion Time lack or sense of urgency Sustained: leisurely, lingering, indulging in time Stretching to yawn, striking a pet Sudden (quick):
hurried, urgent, quick, fleeting
Swatting a fly, lunging to catch a ball, grabbing a child form the path of danger, making a snap move Flow amount of control and bodily tension Free (fluent): uncontrolled, abandoned, unable to stop in the course of the movement
Waving wildly, shaking off water, flinging a rock into a pond Bound: controlled,
restrained, rigid Moving in slow motion, tai chi, fighting back tears, carrying a cup of hot tea
Table 1: The dimensions of effort according to Laban as described by Zhao [Zhao].
Shape can be described in terms of movement in three different planes: the table plane (horizontal), the door plane (vertical) and the wheel plane, which describes sagittal movements. Horizontal moments can be somewhere in-between spreading and enclosing, vertical movements are presented on a scale from rising to descending, and sagittal movements go between advancing and retiring (Figure 1).
Effort comprises four motions factors: space, weight, time and flow. Each motion factor is a continuum between two extremes (Table 1).
Figure 1: The three different planes of shape, adapted from Davies [Davies].
Figure 2: (a) Laban’s effort graph, (b) an example effort graph of inserting a light bulb. [Laban and Lawrence] In figure 2a we depict the graphs Laban uses to express effort. As
an example, figure 2b presents an effort graph of the movement of inserting a light bulb where the movement is direct in space, light in weight, sustained in time and bound in control.
3.2 Analysis of emotional expressions in body
movements
All of the emotions that the actor was asked to perform may of course give rise to a whole range of different body movements depending on the setting, the background and previous experience of the person, personality, culture and various other factors. This act is only one way that these emotions can be expressed.
Even though, the actor was asked to perform nine distinct emotions, his act was more like a process working on the concept of each given emotion, going from starting the expression to “feeling” it more and more, expressing it stronger, and then varying it using various alternative interpretations of when this emotion would arise. In figure 3, an example of the actor’s expression of each emotion is depicted. The analysis, however, was performed on the whole sequence of expressions for each given emotion. Two independent persons (two of the authors) did the same analysis of the videotape, after which notes were compared and discussed.
3.2.1 Shape and effort
Using Laban’s theories of shape the actor’s interpretation can be described as follow:
• Excitement – extremely spreading, rising and advancing movements.
• Anger – somewhat spreading, rising and advancing movements.
• Surprise-afraid – enclosing, somewhat descending and retiring movements.
• Sulkiness – enclosing, somewhat rising and retiring movements.
• Surprise-interested – somewhat spreading, neutral in the vertical plane and advancing movements.
• Pride – somewhat spreading, rising and somewhat advancing movements.
• Satisfaction – neutral in all planes of movements. • Sadness – enclosing, descending and retiring movements. • Being in love – somewhat spreading, somewhat rising and
somewhat advancing movements.
Figure 4 presents the corresponding effort graphs using Laban’s notation.
From looking at our analysis of emotional body language the nine emotions, presented in figure 4, can be divided into three groups with different effort levels, starting with the one with highest effort:
1) Excitement, anger, surprised-afraid
2) Sulkiness, surprised-interested, pride, satisfaction 3) Sadness, being in love
This far we had worked with two variables, shape and effort, but the different emotions are still clustered, for example excitement and anger have nearly the same shape descriptions and exactly the same effort graphs (Figure 4). Therefore, we looked for a third variable, which we found in Russell’s “circumplex model of affect” [Russell].
Figure 3: Emotional body language expressed by the actor.
Figure 4: Emotional body language expressed in effort graphs.
3.2.2 Valence
In the “circumplex model of affect” psychologist Russell looks at emotions in terms of pleasure and displeasure (here named valence) and arousal. Since a high degree of effort brings a high degree of arousal and vice versa Russell’s analysis of emotions concurs nicely with Laban’s theories of movements. Thus, valence is our third variable. In a series of studies Russell established that people have the same mental map of how emotions are distributed in a system of coordinates where the y-axis is the degree of arousal and the x-y-axis is the valence (Figure 5). The subjects, for example, placed angry and delighted on the same arousal level but with different valence.
3.3 Designing emotional expressions with a
basis in shape, effort and valence
To conclude the above analysis it is necessary to set up a combination of shape, effort and valence to create an affective interaction were it is possible to express all kinds of emotional states without resorting to a one-to-one mapping. It is not
necessary, however, to incorporate all dimensions: shape, effort and valence, into a new modality. It can likewise be a combination of the modality and emotional expressions in the interface. We will show an example of the latter in the next section.
Figure 5: Russell’s “circumplex model of affect” [Russell].
4
A mobile service for affective messaging
The goal of the affective message service is to provide users with a means to enhance their messages with emotional expressions. With today’s technology, such as MMS, users can add photos, colors, sound or animations to messages, but it is quite time-consuming, difficult to create on the fly and to get the right expression of such a messages. Instead, our idea is to build an interactive service on top of the MMS-technology that expands on the expressive power while still allowing for ambiguity and open interpretation of the affective content.In the questionnaire (mentioned briefly above) the answers indicated that most users feel limited or alien to expressions such as smilies as a means to express emotions in text messages. Not only is the emotional content restricted but also the emotional interaction with the other party. In a phone conversation, the voice itself can be a bearer of emotional content that complements what is being said. Thus, both parties in the conversation receive too little emotional feedback and are provided with too little emotional expressive power when composing or receiving text-messages. The users in our questionnaire expressed a need for a richer medium.
Below follows a description of the mobile service and thereafter we will explain how shape, effort, valence and the four design principles are incorporated.
Our design example is an emotional text messaging service built on top of a SonyEricsson P800 mobile terminal, where the user can write a text message and then adjust it to fit the emotional expression they want to achieve. The adjustments are mainly done through affective gestures, but with a little mystery added through obscuring the input through mixing it with measurements of the users’ pulse. The affective gestures performed with the stylus used with the P800 terminal, together with the pulse will render an animated background with an emotional expression to the user’s text message. Figure 6 shows a usage scenario.
Figure 6. A usage scenario
4.1 Shape, effort and valence in the interaction
We use Russell’s “circumplex model of affect” (Figure 5), as the basis for the interaction. The user will be moving around in the circular space of emotions expressing effort and valence of their emotional state through combinations of two basic movements that when combined can render an infinite amount of gestures. We call these combinations of the two movements the circumplex affective gestures (Figure 7):
• Moving along the valence scale towards displeasure is done through increasing the pressure on the stylus, decreasing the pressure on the pen results in higher pleasure on the valence scale.
• Shaking and making faster movements, with the hand holding the pen, requires more effort and therefore result in higher arousal, while more swinging, not so direct movements result in lower arousal.
The circumplex affective gestures are inspired by the shape, effort and valence analysis. Emotions with negative valence are associated with strain and tension, while positive emotions often involve less pressure and strain. Emotions with high effort are stronger in weight, more flexible in space and quicker in time, while emotions with less effort are less controlled, lighter and smaller in space. While the user is performing the circumplex affective gestures, the system is responding through showing the emotional expressions in color, shape and animations as indicated in figure 8. The emotional expression works like an animation in The user starts to write a text
message. At the starting point the pressure on the pen and the pulse of the user decides the emotional expression of the animated background.
While the user writes her message the background is still animated but keeps the initial expression.
When the message is written, new input from the pulse of the user influences how strong the expression of the message will be.
If the expression does not fit the message written, the user can adjust it through the affective gestures…
…until the user finds an expression that suits the message.
Figure 7: The circumplex affective gestures.
Figure 8: The affective gestural plane, a. showing how the output is expressed when interacting and b. showing how the pulse decides the width of this plane, presented in light to strong colors.
the background of the message, giving the writer immediate feedback on the appearance of the message (Figure 8a). The user activates this input by holding in a button on the pen. Once the user finds the expression she wants, the button on the pen is released and the expression is thereby chosen.
The animations allows the different emotions to float into each other similar to how Russell argues that emotions blend into one another and do not have any defined borders. Still, the characteristics of each emotion found in the analysis of body movements are clearly represented through the choice of colors, shapes and movements. Most of the emotions, or their position in Russell’s circular model, can be expressed through colors. Red represents, according to Ryberg, the most powerful and strong emotions. Moving along a color scale ending with blue would be moving towards calm and peaceful emotions. The strength of an emotional state could then be expressed in terms of deepening the color. In this example we are not working with the actual text in the message, neither with sound, but it is something that can be added in future work. Much can be done with different typefaces,
sizes and animations of text, [Forlizzi et al.], sound and music can also convey emotional content [Bresin och Friberg ].
As an example, the characteristics of the emotion excited entail much energy, it is high in effort, and the movements are extremely spreading, rising and advancing. This can be used to create an animation and coloring as in figure 9 (where the animation cannot be shown in this paper).
The circumplex affective gestures would probably render a predictable and thereby less interesting interaction. We therefore decided to add the pulse sensor, which is integrated in the pen, measuring the user’s pulse while writing.
The model combining pulse with the pressure on the pen, as shown in the usage scenario, decides where in the circular space of emotions the user initially starts:
• If your pulse is high and you are holding the pen firmly, you will start where there is high effort and negative valence • With high pulse and a lighter grip around the pen you will
• Low pulse and a firm grip will put you where there is low effort and negative valence
• Low pulse and a lighter grip will put you where there is low effort and positive valence
Figure 9: How “excited” is expressed in the message The user always starts the message with a light emotional expression. When the user has finished writing her text, the pulse decides the width of the circular space of emotions, which is presented as the strength of the emotional expression – varying from light to strong (Figure 8b). This combination of circumplex affective gestures and the pulse sensor we named the affective gestural plane model. The intention is to achieve a kaleidoscopic effect, so that e g “sad” always has the characteristics of sadness but never takes on exactly the same expression. This will hopefully maintain the user’s interest.
If the pulse signal were the only way for the user to provide input to the system, the user would not be in control of the interaction at all, which in turn would be both frustrating and probably render erratic interpretation of users’ affective states most of the time. But since the circumplex affective gestures allows the user to move around the circle of the affective gestural plane, the user is still allowed most of the control.
4.1.1 The interaction device
Designing for emotional input requires a coherency between the actual product’s physical design and the task performed. In this case, the stylus has to be designed in such a way that it appeals to our emotional sensing. You are probably more likely to hug and pat, for example, a teddy bear than a laptop.
Figure 10: A design example of the interaction device.
On the other hand, it is also important that the interaction device does not take on any personality or emotional state in itself. It must not look like some character or carry a specific expression [Andersson et al.], but instead be bland enough to carry users’ intentions. Making a pen that is quite characterless, but still emotionally appealing will provide a suitable artifact for affective computing but still keep the user focused on the interaction. Figure 10 shows a design example.
4.2 Incorporation of design principles
The design principles introduced above, all played an important role in the design of the affective message service. Embodiment is realized both in terms of the actual physical interaction with the extended stylus, as well as through how the user will experience the circumplex affective gestures as such. The two taken together, embody and aid users to externalize the internal emotional states they want to convey.
The principle of Natural but designed expressions is incorporated through the circumplex affective gestures and the interactive feedback that are designed to resemble the shape, effort and valence of natural emotional movements.
Since the design is trying to address both body and mind the emotional state of the user is reinforced not only through the gestures, but also through the response that the system generates, and therefore the interaction will involve the user in an affective loop. While not discussed in this paper, the interaction with the receiver of the affective message will also constitute another affective loop interaction.
Ambiguity is achieved in the affective gestural plane model as well as in the interactive feedback. The pulse sensor creates a small proportion of mystery in the interaction, thus keeps the user interested in exploring their emotional expressions further. By using circumplex affective gestures to navigate the affective gestural plane, we avoided that one gesture corresponds to one emotion, and instead created an interaction where users can create their own language and make their own interpretations of the interactive feedback.
5 Summary
We have shown how to go from a user-centered perspective, involving both body and mind, via theory of movements and emotional expressions, a study of an actor and his emotional expressions, to a specific design of a set of circumplex affective gestures for expressing emotion to a mobile messaging service. We are aware of that this work is somewhat cultural dependent, however, we find this piece of work valid and interesting as input even if not entirely possible to generalize irrespective of culture and personality.
In particular, we have identified three underlying dimensions of bodily emotional expressions: shape, effort and valence that we have incorporated in the design of our mobile service both in the affective gestural plane model as well as in the interactive feedback. This frees us from design solutions that assume that users will be in discrete, well-defined emotional states, where one gesture (or input signal) corresponds to one emotion. Instead our specific design approach allows for an interpretative, interactive cycle with the emotional output that will place users, and their interpretation of emotional expressions and needs for how to express themselves, at core. This diverts from the existing trends in affective computing, where the focus is not on the emotional experience as such but on recognizing and adjusting to what the system believes that the user is feeling.
Acknowledgements
Foremost we like to thank Erik Mattsson for his cooperation in expressing the emotions. We would also like to thank Lars Wallin, Jakob Tholander, Mattias Esbjörnsson, Peter Lönnqvist, Jussi Karlgren, Annika Waern, Phillip Jeffrey and Martin Jonsson for discussions and comments on earlier drafts of this paper.
This work is partly funded by SSF through the “Mobile Services”-project.
References
Andersson, G., Höök, K., Mourao, D., Paiva, A. and Costa, M. (2002) Using a Wizard of Oz study to inform the design of SenToy, Proceedings
of the conference on Designing interactive systems, ACM Press.
Ark, W., Dryer, D. and Lu, D. (1999) The emotion mouse, Proceedings of
HCI International 1999. Munich, Germany.
Bannon, L. (1991) From humans factors to humans actors: The role of psychology and human-computer interaction studies in system design,
Design at Work: Cooperative Design of Computer Systems, J. Greenbaum
and M. Kyngs, eds, Hillsdale, N.J.:Erlbaum.
Bordwell, D. and Thompson, K. (2001) Film Art, An Introduction, Sixth Edition, McGraw-Hill, New York, USA.
Bresin, R. and Friberg, A. (2000), Emotional Colouring of Computer-controlled Music Performances, Computer Music Journal, (24)4, pages 44-63.
Cassell, J. (1998) A Framework for Gesture Generation and Interpretation,
Computer Vision in Human Machine Interaction, R. Cipolla and A.
Pentlan, eds. Cambridge University Press, New York, USA.
Castelfranchi, C. (2000) Affective Appraisal versus Cognitive Evaluation in Social Emotions and Interactions, Affective Interactions, Towards a
New Generation of Computer Interfaces, Ana Paiva, ed, Springer-Verlag,
Berlin.
Cummings, J. N., Kraut, R. and Kiesler, S. (2001) Do we visit, call, or email?: media matter in close relationships, Proceedings of the Conference
on Human Factors and Computing Systems (CHI '01), extended abstracts,
Seattle, Washington, USA.
Davies, E. (2001) Beyond Dance, Laban’s Legacy of Movements Analysis, Brechin Books ltd., London, UK.
Dourish, P. (2001) Where the action is. The Foundations of embodied
Interaction, MIT Press, Cambridge, MA, USA.
Ekman, P. (1972) Emotion in the Human Face, Pergamon Press Inc., New York, USA.
Fernandez, R., Scheirer, J. and Picard, R. (1999) Expression glasses: a wearable device for facial expression recognition. MIT Media Lab Tech.
Rep. 484.
Forlizzi, J., Lee, J. and Hudson, S. E. (2003) The Kinedit System: Affective Messages Using Dynamic Texts, Proceedings of the conference
on Human Factors in Computing Systems (CHI’03), Ft. Lauderdale,
Florida, USA.
Gaver, W.W., Beaver J. and Benford, S. (2003) Ambiguity as a Resource for Design, Proceedings of the conference on Human factors in computing
systems (CHI’03), Ft. Lauderdale, Florida, USA.
Grinter, R, and Eldridge, M. (2003) Design for the socially mobile: Wan2tlk?: everyday text messaging, Proceedings of the conference on
Human factors in computing systems (CHI’03), Ft. Lauderdale, Florida,
USA.
Hummels, C. and Stappers, P.J. (1998) Meaningful Gestures for Human Computer Interaction: Beyond Hand Postures, Proceedings for the third
IEEE international conference on Automatic Face & Gesture recognition (FG’98), Nara, Japan.
Höök, K., Sengers, P. and Andersson, G. (2003) Sense and Sensibility: Evaluation and Interactive Art, Proceedings of the conference on Human
factors in computing system (CHI’03), Ft. Lauderdale, Florida, USA.
Höök, K (forthcoming) User-Centred Design and Evaluation of Affective Interfaces, Evaluating ECAs, Pelachaud, C. and Ruttkay, Z. Kluwer. Ishii, H., and Ullmer, B. (1997) Tangible bits: Towards seamless interfaces between people, bits and atoms. Proceedings of Conference on
Human Factors in Computing Systems (CHI’97). ACM Press.
Itten, J (1971) Kunst der Farbe, Otto Maier Verlag, Ravensburg. Laban, R. and Lawrence F.C. (1974) Effort, Economy of Human Effort, Second Edition, Macdonald & Evans Ltd., London, UK.
Long, A. C. Jr, Landay, J.A., Rowe L. R. and Michiels, J. (2000) Visual Similarity of Pen Gesture, Proceedings of the conference on Human
factors in computing system (CHI’2000), The Hague, Amsterdam.
Nishino’, H., Utsumiya’, K., Kuraoka, D. and Yoshioka, K. (1997) Interactive two-handed gesture interface in 3D virtual environments,
Proceedings of the ACM symposium on Virtual reality software and technology, ACM Press, Lausanne, Switzerland.
Ortony A., Clore G.L. and Collins A. (1988) The Cognitive Structure of
Emotions, Cambridge University Press, Cambridge, UK.
Paiva, A. (Ed.) (2000) Affective Interactions, Towards a New Generation
of Computer Interfaces, Springer-Verlag, Berlin.
Paiva, A., Costa, M., Chaves, R., Piedade, M., Mourão, D., Sobral, D., Höök, K., Andersson, G., and Bullock, A. (2003) SenToy: an Affective Sympathetic Interface, International Journal of Human Computer Studies, Volume 59, Issues 1-2, July 2003, Pages 227-235, Elsevier.
Picard, R. (1997) Affective Computing, MIT Press, Cambridge, MA, USA. Roseman I.J., Antoniou A.A. and Jose P.E. (1996) Appraisal Determinants of Emotions: Constructing a More Accurate and Comprehensive Theory,
Cognition & Emotion, 10(3), Pages 241-277.
Russell, J.A. (1980) A Circumplex Model of Affect, Journal of
Personality and Social Psychology, Vol. 39, No. 6, pages 1161-1178,
American Psychological Association.
Ryberg, K. (1991) Levande färger, ICA Bokförlag, Västerås, Sweden. Zhao, L. (2001) Synthesis and Acquisition of Laban Movement Analysis
Qualitative Parameters for Communicative Gestures, PhD thesis, CIS,
