Mälardalen University Press Licentiate Theses No. 181
TOWARDS IMMERSIVE MOTION CAPTURE ACTING
DESIGN, EXPLORATION AND DEVELOPMENT
OF AN AUGMENTED SYSTEM SOLUTION
Daniel Kade
2014
School of Innovation, Design and Engineering
Mälardalen University Press Licentiate Theses
No. 181
TOWARDS IMMERSIVE MOTION CAPTURE ACTING
DESIGN, EXPLORATION AND DEVELOPMENT
OF AN AUGMENTED SYSTEM SOLUTION
Daniel Kade
2014
Copyright © Daniel Kade, 2014 ISBN 978-91-7485-161-8 ISSN 1651-9256
Printed by Arkitektkopia, Västerås, Sweden
Abstract
Current and future animations seek for realistic motions to create a perception of authentic and human-like animations. A technology widely used for such purposes is motion capture. Therefore, to create such human-like animations, motion capture actors enrich the movements of digital avatars with realistic and believable motions and emotions.
Acting for motion capture, as it is performed today, does not provide a natural acting environment. This is mostly because motion capture actors do not see and feel the virtual environment they act for, while acting. In many cases this can result in unnatural motions such as stiff looking and emotionless movements.
To investigate ways to solve this, we first identify the challenges actors are facing as well as concepts to support a motion capture actor. Furthermore, we discussed, how the task of supporting motion capture actors was approached and which factors were discovered to provide support when designing and im-plementing a solution. Initial prototypes have been created to address the men-tioned issues and to find suitable solutions to support and immerse motion cap-ture actors during their performance. For this thesis, one goal was to conduct research by focusing on the question: What are the experiential qualities of immersion in an interactive system to create an immersive acting environment that supports motion capture actors.
The developed application provides a flexibility to set up and modify digital assets and scenes quickly and with an easy to use interface. Furthermore, the prototype helps to provide an understanding on which hardware and software prototypes can be designed and used to build an immersive motion capture environment. The built prototype allows to investigate user experiences, user tests and the satisfaction of users and their effects on motion capture acting.
Abstract
Current and future animations seek for realistic motions to create a perception of authentic and human-like animations. A technology widely used for such purposes is motion capture. Therefore, to create such human-like animations, motion capture actors enrich the movements of digital avatars with realistic and believable motions and emotions.
Acting for motion capture, as it is performed today, does not provide a natural acting environment. This is mostly because motion capture actors do not see and feel the virtual environment they act for, while acting. In many cases this can result in unnatural motions such as stiff looking and emotionless movements.
To investigate ways to solve this, we first identify the challenges actors are facing as well as concepts to support a motion capture actor. Furthermore, we discussed, how the task of supporting motion capture actors was approached and which factors were discovered to provide support when designing and im-plementing a solution. Initial prototypes have been created to address the men-tioned issues and to find suitable solutions to support and immerse motion cap-ture actors during their performance. For this thesis, one goal was to conduct research by focusing on the question: What are the experiential qualities of immersion in an interactive system to create an immersive acting environment that supports motion capture actors.
The developed application provides a flexibility to set up and modify digital assets and scenes quickly and with an easy to use interface. Furthermore, the prototype helps to provide an understanding on which hardware and software prototypes can be designed and used to build an immersive motion capture environment. The built prototype allows to investigate user experiences, user tests and the satisfaction of users and their effects on motion capture acting.
Swedish Summary /
Sammanfattning
Inom dagens och framtidens animation efterstr¨avas realistiska r¨orelser f¨or att ge betraktaren en uppfattning om autentiska och m¨anniskoliknande animeringar. Motion capture anv¨ands i stor utstr¨ackning fr detta ¨andam˚al. Motion capture-sk˚adespelare ger digitala avatarer realistiska och trov¨ardiga r¨orelser och k¨anslor.
Dessv¨arre erbjuder inte nuvarande motion capture-teknik en naturlig sk˚ade-spelarmilj¨o. Det ¨ar fr¨amst p˚a grund av att sk˚adespelare inte ser eller p˚a annat s¨att uppfattar den virtuella milj¨on de agerar i. Det h¨ar leder i m˚anga fall till onaturliga r¨orelser.
F¨or att unders¨oka l¨osningar till detta problem, identifierar vi f¨orst de ut-maningar som sk˚adespelarna st¨alls inf¨or samt vilka st¨od som finns f¨or motion capture-sk˚adespelare. Vidare diskuteras hur det ¨ar m¨ojligt att st¨odja motion capture-sk˚adespelare och vilka faktorer som m˚aste beaktas n¨ar en ¨andam˚alsenlig l¨osning designas och implementeras. Efter detta skapar vi de f¨orsta prototype-rna f¨or att ta itu med utmaningaprototype-rna runt motion capture och f¨or att finna lmpliga l¨osningar f¨or att st¨odja och skapa immersion f¨or motion capture-sk˚adespelare. Ett av avhandlingens m˚al har varit att fokusera fr˚agan: Vilka upplevelsek-valiteter runt immersion f¨or ett interaktivt system skapar en immersiv milj¨o som st¨odjer motion capture sk˚adespelare?
Den interaktiva forskningsprototypen ger ett flexibilitet och l¨attanv¨ant anv¨andargr¨anssnitt f¨or att ¨andra i scener och digitala objekt. Forskningspro-totypen bidrar ocks˚a till frst˚aelse f¨or hur h˚ardvara och mjukvara kan utfor-mas och anv¨andas f¨or att bygga en immersiv motion capture-milj¨o. Prototypen m¨ojligg¨or experiment, anv¨andartester och unders¨okningar av anv¨andarupplevelse, anv¨andartillfredst¨allelse och dess effekter p˚a motion capture-sk˚adespeleri.
Swedish Summary /
Sammanfattning
Inom dagens och framtidens animation efterstr¨avas realistiska r¨orelser f¨or att ge betraktaren en uppfattning om autentiska och m¨anniskoliknande animeringar. Motion capture anv¨ands i stor utstr¨ackning fr detta ¨andam˚al. Motion capture-sk˚adespelare ger digitala avatarer realistiska och trov¨ardiga r¨orelser och k¨anslor.
Dessv¨arre erbjuder inte nuvarande motion capture-teknik en naturlig sk˚ade-spelarmilj¨o. Det ¨ar fr¨amst p˚a grund av att sk˚adespelare inte ser eller p˚a annat s¨att uppfattar den virtuella milj¨on de agerar i. Det h¨ar leder i m˚anga fall till onaturliga r¨orelser.
F¨or att unders¨oka l¨osningar till detta problem, identifierar vi f¨orst de ut-maningar som sk˚adespelarna st¨alls inf¨or samt vilka st¨od som finns f¨or motion capture-sk˚adespelare. Vidare diskuteras hur det ¨ar m¨ojligt att st¨odja motion capture-sk˚adespelare och vilka faktorer som m˚aste beaktas n¨ar en ¨andam˚alsenlig l¨osning designas och implementeras. Efter detta skapar vi de f¨orsta prototype-rna f¨or att ta itu med utmaningaprototype-rna runt motion capture och f¨or att finna lmpliga l¨osningar f¨or att st¨odja och skapa immersion f¨or motion capture-sk˚adespelare. Ett av avhandlingens m˚al har varit att fokusera fr˚agan: Vilka upplevelsek-valiteter runt immersion f¨or ett interaktivt system skapar en immersiv milj¨o som st¨odjer motion capture sk˚adespelare?
Den interaktiva forskningsprototypen ger ett flexibilitet och l¨attanv¨ant anv¨andargr¨anssnitt f¨or att ¨andra i scener och digitala objekt. Forskningspro-totypen bidrar ocks˚a till frst˚aelse f¨or hur h˚ardvara och mjukvara kan utfor-mas och anv¨andas f¨or att bygga en immersiv motion capture-milj¨o. Prototypen m¨ojligg¨or experiment, anv¨andartester och unders¨okningar av anv¨andarupplevelse, anv¨andartillfredst¨allelse och dess effekter p˚a motion capture-sk˚adespeleri.
German Summary /
Zusammenfassung
Aktuelle und zuk¨unftige Animationen streben nach realistischen Bewegungen um eine Wahrnehmung von authentischen und menschen¨ahnliche Animatio-nen zu erstellen. Eine Technologie, die h¨aufig f¨ur solche Zwecke verwendet wird, ist Motion-Capture. Deswegen bereichern Motion Capture Schauspieler die Bewegungen der digitalen Avatare mit realistischen und glaubw¨urdigen Bewegungen und Emotionen um solche menschen¨ahnlichen Animationen zu erstellen.
Schauspiel f¨ur Motion Capture, wie es heute durchgef¨uhrt wird, bietet keine nat¨urlich wirkende Schauspielumgebung. Dies beruht haupts¨achlich da-rauf, dass Motion Capture Schauspieler die virtuelle Umgebung in der sie spie-len nicht w¨ahrend des Schauspiels sehen und f¨uhspie-len k¨onnen. In viespie-len F¨alspie-len kann dies dann zu unnat¨urlichen Bewegungen f¨uhren.
Um L¨osungswege zu finden, haben wir zun¨achst die Herausforderungen denen Schauspieler konfrontiert sind, sowie die Bed¨urfnisse und Anforderun-gen an einen Motion Capture Schauspieler identifiziert. Dann argumentieren wir, wie die Unterst¨utzung von Motion Capture Schauspielern m¨oglich ist und welche Faktoren bei der Konzeption und Implementierung einer L¨osung ber¨ucksichtigt werden m¨ussen. Danach wurden erste Prototypen erstellt, um die genannten Probleme zu addressieren und geeignete L¨osungen zu finden um Motion Capture Schauspieler auch w¨ahrend des Schauspiels zu unterst¨utzen und diese in die Umgebungen des Aktes zu vertiefen. Ein Ziel dieser Licentiate-Arbeit war es, Forschung bez¨uglich folgender Frage durchzuf¨uhren: Was sind die Erlebnisqualit¨aten der Immersion an ein interaktives System, um eine im-mersive Umgebung zu erstellen, welches Motion Capture Schauspieler un-terst¨utzt.
German Summary /
Zusammenfassung
Aktuelle und zuk¨unftige Animationen streben nach realistischen Bewegungen um eine Wahrnehmung von authentischen und menschen¨ahnliche Animatio-nen zu erstellen. Eine Technologie, die h¨aufig f¨ur solche Zwecke verwendet wird, ist Motion-Capture. Deswegen bereichern Motion Capture Schauspieler die Bewegungen der digitalen Avatare mit realistischen und glaubw¨urdigen Bewegungen und Emotionen um solche menschen¨ahnlichen Animationen zu erstellen.
Schauspiel f¨ur Motion Capture, wie es heute durchgef¨uhrt wird, bietet keine nat¨urlich wirkende Schauspielumgebung. Dies beruht haupts¨achlich da-rauf, dass Motion Capture Schauspieler die virtuelle Umgebung in der sie spie-len nicht w¨ahrend des Schauspiels sehen und f¨uhspie-len k¨onnen. In viespie-len F¨alspie-len kann dies dann zu unnat¨urlichen Bewegungen f¨uhren.
Um L¨osungswege zu finden, haben wir zun¨achst die Herausforderungen denen Schauspieler konfrontiert sind, sowie die Bed¨urfnisse und Anforderun-gen an einen Motion Capture Schauspieler identifiziert. Dann argumentieren wir, wie die Unterst¨utzung von Motion Capture Schauspielern m¨oglich ist und welche Faktoren bei der Konzeption und Implementierung einer L¨osung ber¨ucksichtigt werden m¨ussen. Danach wurden erste Prototypen erstellt, um die genannten Probleme zu addressieren und geeignete L¨osungen zu finden um Motion Capture Schauspieler auch w¨ahrend des Schauspiels zu unterst¨utzen und diese in die Umgebungen des Aktes zu vertiefen. Ein Ziel dieser Licentiate-Arbeit war es, Forschung bez¨uglich folgender Frage durchzuf¨uhren: Was sind die Erlebnisqualit¨aten der Immersion an ein interaktives System, um eine im-mersive Umgebung zu erstellen, welches Motion Capture Schauspieler un-terst¨utzt.
vi
Die entwickelte Applikation bietet Flexibilit¨at bei der schnellen Einrich-tung und Modifizierung von digitalen Objekten und Szenen durch eine einfach zu bedienende Oberfl¨ache. Desweiteren hilft die Applikation ein Verst¨andnis zu entwickeln, welche Hardware- und Software-Prototypen entwickelt und be-nutzten werden k¨onnen mit denen eine immersive Motion Capture Schaus-pielumgebung aufgebaut werden kann. Der entwickelte Prototyp erm¨oglicht es Nutzererfahrungen, Nutzertests und die Zufriedenheit der Nutzer und dessen
Effekt im Motion-Capture Schauspiel zu erforschen
Acknowledgements
At this point I would like to express my sincere gratitude to my supervisors, Professor Dr. O˘guzhan ¨Ozcan, Dr. Rikard Lindell and Professor Dr. Dr. Gor-dana Dodig-Crnkovic who have guided and encouraged me throughout my studies. I have learned a lot from their advices, feedback and guidance. More-over, I would like to thank O˘guzhan for his efforts in providing the opportunity to study abroad and to participate in a larger research project.
A special thanks goes also to the people from Imagination Studios who provided support and a motion capture studio. Another thank you goes to Kaan Aks¸it for the good research cooperation, the good fellowship during research exchanges and the commonly achieved results.
Furthermore, I would like to thank all students that contributed with their implementations to my research work. Claudio Redavid implemented an initial design concept idea, providing pseudo 360 degree vision on 4 screens to an ac-tor, during his master thesis. Robert Gustavsson and David Reypka developed software to map motion capture data in real-time to the UDK game engine. Patrick Sj¨o¨o coded the walk algorithms and the camera and sensor usage of a mobile phone that we used for our proof-of-concept implementation.
I would also like to thank all teachers, administrative staff, colleagues and friends at the department, the company and during my research stays, who have helped me to evolve as a researcher.
This work has been supported by the Swedish Knowledge Foundation (KKS), M¨alardalen University and Imagination Studios within the PhD school ITS-EASY. This support has made it possible for me to be an industrial PhD student, so I would like to also express my gratitude for that. It has been an interesting and fun time for me in which I learned a lot.
Daniel Kade V¨aster˚as, October, 2014
vi
Die entwickelte Applikation bietet Flexibilit¨at bei der schnellen Einrich-tung und Modifizierung von digitalen Objekten und Szenen durch eine einfach zu bedienende Oberfl¨ache. Desweiteren hilft die Applikation ein Verst¨andnis zu entwickeln, welche Hardware- und Software-Prototypen entwickelt und be-nutzten werden k¨onnen mit denen eine immersive Motion Capture Schaus-pielumgebung aufgebaut werden kann. Der entwickelte Prototyp erm¨oglicht es Nutzererfahrungen, Nutzertests und die Zufriedenheit der Nutzer und dessen
Effekt im Motion-Capture Schauspiel zu erforschen
Acknowledgements
At this point I would like to express my sincere gratitude to my supervisors, Professor Dr. O˘guzhan ¨Ozcan, Dr. Rikard Lindell and Professor Dr. Dr. Gor-dana Dodig-Crnkovic who have guided and encouraged me throughout my studies. I have learned a lot from their advices, feedback and guidance. More-over, I would like to thank O˘guzhan for his efforts in providing the opportunity to study abroad and to participate in a larger research project.
A special thanks goes also to the people from Imagination Studios who provided support and a motion capture studio. Another thank you goes to Kaan Aks¸it for the good research cooperation, the good fellowship during research exchanges and the commonly achieved results.
Furthermore, I would like to thank all students that contributed with their implementations to my research work. Claudio Redavid implemented an initial design concept idea, providing pseudo 360 degree vision on 4 screens to an ac-tor, during his master thesis. Robert Gustavsson and David Reypka developed software to map motion capture data in real-time to the UDK game engine. Patrick Sj¨o¨o coded the walk algorithms and the camera and sensor usage of a mobile phone that we used for our proof-of-concept implementation.
I would also like to thank all teachers, administrative staff, colleagues and friends at the department, the company and during my research stays, who have helped me to evolve as a researcher.
This work has been supported by the Swedish Knowledge Foundation (KKS), M¨alardalen University and Imagination Studios within the PhD school ITS-EASY. This support has made it possible for me to be an industrial PhD student, so I would like to also express my gratitude for that. It has been an interesting and fun time for me in which I learned a lot.
Daniel Kade V¨aster˚as, October, 2014
List of Publications
Papers Included in the Licentiate Thesis
1Paper A An Immersive Motion Capture Environment. Daniel Kade, O˘guzhan ¨Ozcan and Rikard Lindell, Proceedings of the ICCGMAT 2013 International Conference on Computer Games, Multimedia and Allied Technology vol. 73, WASET, Zurich, pp.500-506, Available from: World Academy of Science, Engineering and Technology, January 2013.
Paper B Towards Stanislavski-based Principles for Motion Capture Acting in
Animation and Computer Games. Daniel Kade, O˘guzhan ¨Ozcan and
Rikard Lindell, CONFIA 2013 International Conference in Illustration & Animation, Porto, November 2013.
Paper C Ethics of Virtual Reality Applications in Computer Games
Produc-tion. Daniel Kade, IACAP 2014 International Association of Computers
and Philosophy, Thessaloniki, Greece, July 2014. (Paper accepted) Paper D Head-worn Mixed Reality Projection Display Application. Kaan Aks¸it,
Daniel Kade, O˘guzhan ¨Ozcan and Hakan ¨Urey, ACE 2014, Advances in Computer Entertainment Technology, November 11-14th 2014, Funchal, Portugal. (Paper accepted)
1The included articles have been reformatted to comply with the licentiate thesis layout
List of Publications
Papers Included in the Licentiate Thesis
1Paper A An Immersive Motion Capture Environment. Daniel Kade, O˘guzhan ¨Ozcan and Rikard Lindell, Proceedings of the ICCGMAT 2013 International Conference on Computer Games, Multimedia and Allied Technology vol. 73, WASET, Zurich, pp.500-506, Available from: World Academy of Science, Engineering and Technology, January 2013.
Paper B Towards Stanislavski-based Principles for Motion Capture Acting in
Animation and Computer Games. Daniel Kade, O˘guzhan ¨Ozcan and
Rikard Lindell, CONFIA 2013 International Conference in Illustration & Animation, Porto, November 2013.
Paper C Ethics of Virtual Reality Applications in Computer Games
Produc-tion. Daniel Kade, IACAP 2014 International Association of Computers
and Philosophy, Thessaloniki, Greece, July 2014. (Paper accepted) Paper D Head-worn Mixed Reality Projection Display Application. Kaan Aks¸it,
Daniel Kade, O˘guzhan ¨Ozcan and Hakan ¨Urey, ACE 2014, Advances in Computer Entertainment Technology, November 11-14th 2014, Funchal, Portugal. (Paper accepted)
1The included articles have been reformatted to comply with the licentiate thesis layout
Term Descriptions
Interaction Design
Actor We consider an ’actor’ as someone that acting for a motion capture shoot. This ’actor’ can have different skills or training but must not necessarily have acting training of some sort. Application Here we mean software that is designed to help performing
specific tasks.
Stakeholder Stakeholders are persons involved or to be considered within the design of a solution. In our research these are mainly actors, directors, motion capture staff and researchers.
System A system combines software and hardware with their interactions and components to act as a whole.
Tester A tester is someone that is testing our prototypes, designs and systems. This user can but does not need to have specific skills. User A user is someone that is using or testing our prototypes,
designs and systems. This user can but does not need to have specific skills.
Term Descriptions
Interaction Design
Actor We consider an ’actor’ as someone that acting for a motion capture shoot. This ’actor’ can have different skills or training but must not necessarily have acting training of some sort. Application Here we mean software that is designed to help performing
specific tasks.
Stakeholder Stakeholders are persons involved or to be considered within the design of a solution. In our research these are mainly actors, directors, motion capture staff and researchers.
System A system combines software and hardware with their interactions and components to act as a whole.
Tester A tester is someone that is testing our prototypes, designs and systems. This user can but does not need to have specific skills. User A user is someone that is using or testing our prototypes,
designs and systems. This user can but does not need to have specific skills.
xii
Computer Science
AR Augmented Reality: An augmented reality superimposes digitally created content onto the real world. This allows a person using AR to see both, the real and the digital world. Usually Ar uses some means of connecting real world objects with the augmented reality.
AV Augmented Virtuality: Augmented virtuality aims to
include and merge of real world objects into the virtual world. Here physical objects or people can interact with or be seen in as themselves or digital representations in the virtual environment in real-time. Avatar An avatar is usually the representation of a person or player in
a digital or virtual environment. In our definition an avatar is equal to a digital character.
DOF Degree-of-freedom: We use the term from computer graphics and animation to describe possible movements and rotations around different axes.
HD High Definition
HDMI High-Definition Multimedia Interface LED Light Emitting Diode
MHL Mobile High-Definition Link
MR Mixed Reality merges the real world with the virtual world. This can be achieved with different means and technologies for different purposes. The Milgram Taxonomy [1] uses MR as an umbrella term that includes VR, AR, Augmented Virtuality (AV) and the real world. VR Virtual Reality: A virtual reality creates a reality that is not real. The
vision of a person is in many cases occluded by digitally created content or the person is placed in a virtual reality environment. Synonyms are: virtual world, virtual environment, and cyberspace.
Contents
I
Thesis
1
1 Introduction 3 1.1 Introduction . . . 3 1.2 Motivation . . . 5 1.3 State-of-the-Art . . . 6 2 Research Description 13 2.1 Research Questions . . . 132.2 Proposed Solution to the Research Problem . . . 14
2.3 Identified Scientific Challenges . . . 15
2.4 Research Methods . . . 16
2.5 Expected Results . . . 21
3 Thesis Contributions 23 4 Approach, Prototype and Findings 27 4.1 From Vision to Prototype (Part I) . . . 27
4.2 Technology Review . . . 28
4.2.1 Why not VR? . . . 28
4.2.2 Projection Mapping . . . 29
4.2.3 Augmented Reality . . . 30
4.2.4 Mixed Augmented Reality (AR) . . . 31
4.3 Personas . . . 32
4.4 Improved Motion Capture Process . . . 35
4.5 From Vision to Prototype (Part II) . . . 36
4.6 Initial Design Concepts . . . 36
4.6.1 Component Based Setups . . . 39
xii
Computer Science
AR Augmented Reality: An augmented reality superimposes digitally created content onto the real world. This allows a person using AR to see both, the real and the digital world. Usually Ar uses some means of connecting real world objects with the augmented reality.
AV Augmented Virtuality: Augmented virtuality aims to
include and merge of real world objects into the virtual world. Here physical objects or people can interact with or be seen in as themselves or digital representations in the virtual environment in real-time. Avatar An avatar is usually the representation of a person or player in
a digital or virtual environment. In our definition an avatar is equal to a digital character.
DOF Degree-of-freedom: We use the term from computer graphics and animation to describe possible movements and rotations around different axes.
HD High Definition
HDMI High-Definition Multimedia Interface LED Light Emitting Diode
MHL Mobile High-Definition Link
MR Mixed Reality merges the real world with the virtual world. This can be achieved with different means and technologies for different purposes. The Milgram Taxonomy [1] uses MR as an umbrella term that includes VR, AR, Augmented Virtuality (AV) and the real world. VR Virtual Reality: A virtual reality creates a reality that is not real. The
vision of a person is in many cases occluded by digitally created content or the person is placed in a virtual reality environment. Synonyms are: virtual world, virtual environment, and cyberspace.
Contents
I
Thesis
1
1 Introduction 3 1.1 Introduction . . . 3 1.2 Motivation . . . 5 1.3 State-of-the-Art . . . 6 2 Research Description 13 2.1 Research Questions . . . 132.2 Proposed Solution to the Research Problem . . . 14
2.3 Identified Scientific Challenges . . . 15
2.4 Research Methods . . . 16
2.5 Expected Results . . . 21
3 Thesis Contributions 23 4 Approach, Prototype and Findings 27 4.1 From Vision to Prototype (Part I) . . . 27
4.2 Technology Review . . . 28
4.2.1 Why not VR? . . . 28
4.2.2 Projection Mapping . . . 29
4.2.3 Augmented Reality . . . 30
4.2.4 Mixed Augmented Reality (AR) . . . 31
4.3 Personas . . . 32
4.4 Improved Motion Capture Process . . . 35
4.5 From Vision to Prototype (Part II) . . . 36
4.6 Initial Design Concepts . . . 36
4.6.1 Component Based Setups . . . 39
xiv Contents
4.7 From Vision to Prototype (Part III) . . . 39
4.8 Initial Prototypes . . . 39
4.8.1 Screens around an actor . . . 39
4.8.2 Using a motion capture system . . . 40
4.8.3 Portable screen . . . 41
4.8.4 Using a Pico projector . . . 42
4.8.5 Design Decisions . . . 43
4.9 Proof-of-Concept Prototype . . . 43
4.9.1 Technological Categorization of our Prototype . . . . 44
4.9.2 Hardware . . . 45
4.9.3 Software . . . 46
4.9.4 Use of Reflective Materials in Optical Motion Capture? 47 4.9.5 User Experiences . . . 49 5 Conclusions 51 5.1 Conclusion . . . 51 5.2 Future Work . . . 52 Bibliography 55
II
Included Papers
61
6 Paper A: An Immersive Motion Capture Environment 63 6.1 Introduction . . . 656.2 Current State-of-the-Art . . . 66
6.2.1 Visualization . . . 66
6.2.2 Tracking . . . 68
6.2.3 Interaction . . . 68
6.2.4 Combined Research Areas . . . 70
6.3 Conducted Research . . . 71 6.4 Findings . . . 72 6.5 Open Issues . . . 74 6.6 Future Solution . . . 75 6.7 Conclusion . . . 77 Bibliography . . . 79 Contents xv 7 Paper B: Towards Stanislavski-based Principles for Motion Capture Acting in Animation and Computer Games 83 7.1 Introduction . . . 85
7.2 What is acting? . . . 85
7.3 Which principles should we support in motion capture? . . . . 87
7.4 What is the nature of Motion Capture Actors? . . . 90
7.5 Do we need to adapt major acting techniques in motion capture actor training? . . . 91
7.6 What motion capture actors think and need? . . . 93
7.7 How to improve current motion capture structures? . . . 94
7.8 Conclusion . . . 96
Bibliography . . . 97
8 Paper C: Ethics of Virtual Reality Applications in Computer Games Produc-tion 99 8.1 Introduction . . . 101
8.2 State-of-the-Art . . . 102
8.2.1 Ethics in Computer Games . . . 102
8.2.2 Ethics in Virtual Realities . . . 103
8.2.3 Ethics in Acting . . . 105
8.3 Possible Ethical Issues Within Motion Capture . . . 105
8.4 Ethical Analysis . . . 109
8.5 Discussion Towards an Ethical Guideline . . . 112
8.6 Conclusion . . . 114
Bibliography . . . 115
9 Paper D: Head-worn Mixed Reality Projection Display Application 119 9.1 Introduction . . . 121
9.2 State-of-the-Art . . . 123
9.3 Head-worn Projection Display . . . 126
9.3.1 Hardware Description . . . 126
9.3.2 Software Description . . . 130
9.4 Application of the Prototype in Motion Capture . . . 133
9.5 Functionality Test . . . 134
9.6 Conclusion . . . 135
xiv Contents
4.7 From Vision to Prototype (Part III) . . . 39
4.8 Initial Prototypes . . . 39
4.8.1 Screens around an actor . . . 39
4.8.2 Using a motion capture system . . . 40
4.8.3 Portable screen . . . 41
4.8.4 Using a Pico projector . . . 42
4.8.5 Design Decisions . . . 43
4.9 Proof-of-Concept Prototype . . . 43
4.9.1 Technological Categorization of our Prototype . . . . 44
4.9.2 Hardware . . . 45
4.9.3 Software . . . 46
4.9.4 Use of Reflective Materials in Optical Motion Capture? 47 4.9.5 User Experiences . . . 49 5 Conclusions 51 5.1 Conclusion . . . 51 5.2 Future Work . . . 52 Bibliography 55
II
Included Papers
61
6 Paper A: An Immersive Motion Capture Environment 63 6.1 Introduction . . . 656.2 Current State-of-the-Art . . . 66
6.2.1 Visualization . . . 66
6.2.2 Tracking . . . 68
6.2.3 Interaction . . . 68
6.2.4 Combined Research Areas . . . 70
6.3 Conducted Research . . . 71 6.4 Findings . . . 72 6.5 Open Issues . . . 74 6.6 Future Solution . . . 75 6.7 Conclusion . . . 77 Bibliography . . . 79 Contents xv 7 Paper B: Towards Stanislavski-based Principles for Motion Capture Acting in Animation and Computer Games 83 7.1 Introduction . . . 85
7.2 What is acting? . . . 85
7.3 Which principles should we support in motion capture? . . . . 87
7.4 What is the nature of Motion Capture Actors? . . . 90
7.5 Do we need to adapt major acting techniques in motion capture actor training? . . . 91
7.6 What motion capture actors think and need? . . . 93
7.7 How to improve current motion capture structures? . . . 94
7.8 Conclusion . . . 96
Bibliography . . . 97
8 Paper C: Ethics of Virtual Reality Applications in Computer Games Produc-tion 99 8.1 Introduction . . . 101
8.2 State-of-the-Art . . . 102
8.2.1 Ethics in Computer Games . . . 102
8.2.2 Ethics in Virtual Realities . . . 103
8.2.3 Ethics in Acting . . . 105
8.3 Possible Ethical Issues Within Motion Capture . . . 105
8.4 Ethical Analysis . . . 109
8.5 Discussion Towards an Ethical Guideline . . . 112
8.6 Conclusion . . . 114
Bibliography . . . 115
9 Paper D: Head-worn Mixed Reality Projection Display Application 119 9.1 Introduction . . . 121
9.2 State-of-the-Art . . . 123
9.3 Head-worn Projection Display . . . 126
9.3.1 Hardware Description . . . 126
9.3.2 Software Description . . . 130
9.4 Application of the Prototype in Motion Capture . . . 133
9.5 Functionality Test . . . 134
9.6 Conclusion . . . 135
xvi Contents
Bibliography . . . 137
I
Thesis
xvi Contents
Bibliography . . . 137
I
Thesis
Chapter 1
Introduction
1.1 Introduction
Todays video games are becoming more and more realistic [2], not only be-cause of hardware and software innovations but also bebe-cause of the use of highly realistic animations of humans, animals, objects and environments in these games. In many cases, cinematic elements, shown during game-play al-most feel like watching a movie.
This is to a large part thanks to motion capture technology that we per-ceive motions in a gaming environment as more realistic than in older video games which did not use this technology. To create this sense of realism, hu-man motions are recorded from skilled performers and then mapped to virtual avatars. Motion capture actors play an important role when creating these re-alistic motions and performances. Therefore, actors, stuntmen and athletes perform recordings of motions to bring virtual characters closer to realism.
Motion capture technology is used for various applications from medical to training and entertainment purposes. In this work we focus especially on mo-tion capture for animamo-tions, mainly used in computer games. Captured move-ments from human actors are used to map these movemove-ments to virtual avatars. The avatars then perform these movements during gameplay or in short video clips during the transition of parts of the game, so called in-game cut scenes. This research started with the goal to explore solutions to support a motion capture studio with their daily work. Quick responses to customer and actor needs, scenery changes, as well as the need to improve their motion capture procedures started our research investigations.
Chapter 1
Introduction
1.1 Introduction
Todays video games are becoming more and more realistic [2], not only be-cause of hardware and software innovations but also bebe-cause of the use of highly realistic animations of humans, animals, objects and environments in these games. In many cases, cinematic elements, shown during game-play al-most feel like watching a movie.
This is to a large part thanks to motion capture technology that we per-ceive motions in a gaming environment as more realistic than in older video games which did not use this technology. To create this sense of realism, hu-man motions are recorded from skilled performers and then mapped to virtual avatars. Motion capture actors play an important role when creating these re-alistic motions and performances. Therefore, actors, stuntmen and athletes perform recordings of motions to bring virtual characters closer to realism.
Motion capture technology is used for various applications from medical to training and entertainment purposes. In this work we focus especially on mo-tion capture for animamo-tions, mainly used in computer games. Captured move-ments from human actors are used to map these movemove-ments to virtual avatars. The avatars then perform these movements during gameplay or in short video clips during the transition of parts of the game, so called in-game cut scenes. This research started with the goal to explore solutions to support a motion capture studio with their daily work. Quick responses to customer and actor needs, scenery changes, as well as the need to improve their motion capture procedures started our research investigations.
4 Chapter 1. Introduction
Through observations and interviews with 18 actors, 10 directors and 4 motion capture operators, we came to the conclusion that the current acting environment in a motion capture studio does not provide a natural acting en-vironment for an actor, especially when we compare it to stage or film acting. This is because the virtual environment, the actors are acting for, is neither vis-ible nor perceptvis-ible while acting. Objects, obstacles, other virtual characters or events have to be memorized and are not yet visualized in real-time without having the actor to turn towards a screen while acting. In a film acting envi-ronment, an actor is placed in a real-world environment and is surrounded by proper physical objects as well as real persons in appropriate costumes. Even the surrounding conditions like environmental sounds, smoke, rain and simi-lar can be created which allows actors to concentrate more on the acting itself. For current motion capture shoots, an actor has to memorize the scenario, build the character, imagine the environment and do the acting; sometimes with only a brief preparation times. Through our observations and questioning we ex-perienced that motion capture performances are very much dependent on the actors’ capability to imagine the scenery and to put themselves in the desired role and mood quickly [3].
With this research we aim at creating a better motion capture environment to support actors with their task of delivering realistic and believable perfor-mances. Yet to achieve this, we first needed to create an understanding of the motion capture actor, as well as the needs and wants of an actor. Also, outlin-ing the challenges a motion capture actor is facoutlin-ing in current motion capture shoots was of importance in order to find a basis to design innovative solutions to solve the identified issues and to support actors with their work.
Understanding the users of an immersive motion capture system has been a major task before implementing any prototypical solutions. This complies with the fact that for this thesis, approaches from the fields of computing sciences as well as interaction design have been merged to create and explore solutions towards an immersive motion capture acting environment. Therefore, we also discussed the nature of a motion capture actor and pointed out skills, demands and developed principles to support motion capture actors with their work. Prototypes have been developed to test the applicability of the solutions. Fur-ther developed prototypes are planned to be introduced into the research and development cycles.
A combination of technologies that address multiple human senses could be of use to create such an environment, namely: vision, sound, smell, the feeling of touch and haptic feedback. Nevertheless, we limited the focus for this thesis to visual prototypes.
1.2 Motivation 5
As literature studies have shown, there seems to be no out-of-the-box so-lution that serves all demands, especially not for an application in a motion capture environment. For this thesis we intend to explore how an immersive motion capture environment can be created and what needs to be considered when developing such a system. Working prototypes are meant to show the potential of overcoming identified solutions. Designing prototypical solutions as well as explorations of these designs through constructing prototypes are meant show and evaluate these potentials.
1.2 Motivation
The industrial motivation for the research presented in this thesis is how to create an acting environment that enables actors to perform more efficiently and naturally. There is also a need for faster motion capture outcomes with improved quality.
For the academic motivation of this research the areas of interaction design and computation are of main interest. Here, the interests lay in understanding the modalities of interaction between the users and interactive artefacts de-signed for a motion capture environment, as well as having a guideline on how interactive systems can be developed and identifying what the user experiences in the specific area of motion capture are.
My motivation for this research is built on the above-mentioned interests and is also driven by the interest to create immersive virtual reality environ-ments. Furthermore, a personal interest in computer games and the desire to find smart solutions to improve their quality drive this research. Therefore, my motivation can also be described through the interest to gain knowledge on how immersive environments for gaming and motion capture can be realized. The overall motivation of this thesis results in a vision. This vision can be described by saying that:
Actors will perceive the virtual environment they are acting in, visually and emotionally, through the design of an immersive environment.
By constructing an immersive environment, our vision is that actors should be capable of perceiving a virtual environment, visually and emotionally while acting in a way that does not hinder them in their task to act for motion capture shoots.
4 Chapter 1. Introduction
Through observations and interviews with 18 actors, 10 directors and 4 motion capture operators, we came to the conclusion that the current acting environment in a motion capture studio does not provide a natural acting en-vironment for an actor, especially when we compare it to stage or film acting. This is because the virtual environment, the actors are acting for, is neither vis-ible nor perceptvis-ible while acting. Objects, obstacles, other virtual characters or events have to be memorized and are not yet visualized in real-time without having the actor to turn towards a screen while acting. In a film acting envi-ronment, an actor is placed in a real-world environment and is surrounded by proper physical objects as well as real persons in appropriate costumes. Even the surrounding conditions like environmental sounds, smoke, rain and simi-lar can be created which allows actors to concentrate more on the acting itself. For current motion capture shoots, an actor has to memorize the scenario, build the character, imagine the environment and do the acting; sometimes with only a brief preparation times. Through our observations and questioning we ex-perienced that motion capture performances are very much dependent on the actors’ capability to imagine the scenery and to put themselves in the desired role and mood quickly [3].
With this research we aim at creating a better motion capture environment to support actors with their task of delivering realistic and believable perfor-mances. Yet to achieve this, we first needed to create an understanding of the motion capture actor, as well as the needs and wants of an actor. Also, outlin-ing the challenges a motion capture actor is facoutlin-ing in current motion capture shoots was of importance in order to find a basis to design innovative solutions to solve the identified issues and to support actors with their work.
Understanding the users of an immersive motion capture system has been a major task before implementing any prototypical solutions. This complies with the fact that for this thesis, approaches from the fields of computing sciences as well as interaction design have been merged to create and explore solutions towards an immersive motion capture acting environment. Therefore, we also discussed the nature of a motion capture actor and pointed out skills, demands and developed principles to support motion capture actors with their work. Prototypes have been developed to test the applicability of the solutions. Fur-ther developed prototypes are planned to be introduced into the research and development cycles.
A combination of technologies that address multiple human senses could be of use to create such an environment, namely: vision, sound, smell, the feeling of touch and haptic feedback. Nevertheless, we limited the focus for this thesis to visual prototypes.
1.2 Motivation 5
As literature studies have shown, there seems to be no out-of-the-box so-lution that serves all demands, especially not for an application in a motion capture environment. For this thesis we intend to explore how an immersive motion capture environment can be created and what needs to be considered when developing such a system. Working prototypes are meant to show the potential of overcoming identified solutions. Designing prototypical solutions as well as explorations of these designs through constructing prototypes are meant show and evaluate these potentials.
1.2 Motivation
The industrial motivation for the research presented in this thesis is how to create an acting environment that enables actors to perform more efficiently and naturally. There is also a need for faster motion capture outcomes with improved quality.
For the academic motivation of this research the areas of interaction design and computation are of main interest. Here, the interests lay in understanding the modalities of interaction between the users and interactive artefacts de-signed for a motion capture environment, as well as having a guideline on how interactive systems can be developed and identifying what the user experiences in the specific area of motion capture are.
My motivation for this research is built on the above-mentioned interests and is also driven by the interest to create immersive virtual reality environ-ments. Furthermore, a personal interest in computer games and the desire to find smart solutions to improve their quality drive this research. Therefore, my motivation can also be described through the interest to gain knowledge on how immersive environments for gaming and motion capture can be realized. The overall motivation of this thesis results in a vision. This vision can be described by saying that:
Actors will perceive the virtual environment they are acting in, visually and emotionally, through the design of an immersive environment.
By constructing an immersive environment, our vision is that actors should be capable of perceiving a virtual environment, visually and emotionally while acting in a way that does not hinder them in their task to act for motion capture shoots.
6 Chapter 1. Introduction
1.3 State-of-the-Art
Designing and implementing an immersive motion capture environment im-plies different research areas like motion capture acquisition, tracking and re-search considering the visualization of virtual content while acting as well as creating the experience with hardware and software. From literature there are technologies and solutions that can be of use to work towards the goal of cre-ating a more immersive acting environment [4, 5, 6, 7]. Some related research in similar research areas or that we see as potentially useful are listed below. Acting in a virtual environment
There have been research projects exploring the use of virtual reality to support actors with their work. In one research project, acting training and increasing the final performance was approached through sensory-motor rhythm neuro-feedback training. Actors have been exposed and trained to different lighting conditions, reactions of the audience, and the look of the theatre from stage [8]. Moreover, acting support through a virtual environment was researched as a distance rehearsal system. This system was used to study to which extent virtual reality can be used by actors and director to rehearse their performances without being physically present [9].
Furthermore, research and explorations within the area of ’cyberdrama’ [10], ’digital theatre’ [11] and ’narrative in cyberspace’ [12] has been con-ducted. In these forms of acting, participants create the story through active engagement and interactions between technology and participants. According to other research this interactivity can be grouped into ’Navigation’, ’Partici-pation’, ’Conversation’ and ’Collaboration’ which allows participants to steer the play and get involved before the final performance [13].
Another research explained the setup of a completely virtual theatre where actors steer a virtual character in real-time from their computers with data gloves and keyboard inputs and the audience can listen and interact to the per-formance and the theatre by choosing the seat in the theatre and by applauding and booing [14].
In other research, even the use of motion capture techniques during a live theatrical performance have been used [7]. On-stage, actors were interacting with digital avatars that were controlled by actors wearing a motion capture suit throughout the theatrical act. A screen, which is installed in front of the on-stage actors, displayed the avatars controlled by the motion capture actors who performed their acting in real-time on a close-by motion capture area. Virtual scenery in context of each act was displayed on a background screen, visible
1.3 State-of-the-Art 7
to the audience. In the mentioned research, the displayed scenery and avatar content around the on-stage actors was used to interact with their acting.
The above mentioned research projects show that theatre and acting adapts to the digital age and integrates not only technology but also audience and other participants during the play, preparations and rehearsals. Nonetheless, less research focuses on acting for motion capture or acting within a virtual environment in the sense of physically being in the virtual environment and where the environment is seen as digital acting support while acting.
Designing immersive environments
Designing an immersive environment includes 2 basic concepts, ’Immersion’ and ’Presence’ which have been explored and defined already [15]. ’Immer-sion’ provides the functionality and usability of an immersive environment and provides the opportunities to be immersed into the virtual environment. ence’ creates the perception of being in the virtual environment, where ’Pres-ence’ can be seen as an ”increasing function of immersion” [15].
Through the perception of ’Presence’ it is believed that behaviours in the virtual world are consistent with behaviours in the real world under similar cir-cumstances [15]. It was furthermore mentioned that ’Presence’ could influence the performance of a person using a virtual environment. This statement was to mention that a well designed interface and hardware can increase ’Presence’ and a ”greater vividness in terms of richness of the portrayed environment” which improves task performance [15]. Moreover it is mentioned that ’Pres-ence’ is of importance to train e.g. fire-fighters or surgeons within a virtual en-vironment corresponding to behaviours in the real world. Therefore, a virtual environment must be well designed and allow for immersion and the feeling of being in the environment.
Another interesting statement is that the impact of the feeling of immersion is depended on ”the application or task context and the perceptual requirements of the individual” [15]. Basically this means that different aspects need to be considered when creating an immersive environment. First, it is important to create the application according to the task to be performed. So the virtual environment should focus on the most important aspects that the real world in a similar situation would resemble. When a motion capture actor is supposed to act as a musician, an aural feedback might be more important to create an immersive feeling. This might differ when visual aspects are more of impor-tance, e.g. when acting on a futuristic space ship using futuristic technologies. In other occasions a mixture of a visual and aural environment might allow to increase immersion. We could think of acting for a war zone where audiovisual
6 Chapter 1. Introduction
1.3 State-of-the-Art
Designing and implementing an immersive motion capture environment im-plies different research areas like motion capture acquisition, tracking and re-search considering the visualization of virtual content while acting as well as creating the experience with hardware and software. From literature there are technologies and solutions that can be of use to work towards the goal of cre-ating a more immersive acting environment [4, 5, 6, 7]. Some related research in similar research areas or that we see as potentially useful are listed below. Acting in a virtual environment
There have been research projects exploring the use of virtual reality to support actors with their work. In one research project, acting training and increasing the final performance was approached through sensory-motor rhythm neuro-feedback training. Actors have been exposed and trained to different lighting conditions, reactions of the audience, and the look of the theatre from stage [8]. Moreover, acting support through a virtual environment was researched as a distance rehearsal system. This system was used to study to which extent virtual reality can be used by actors and director to rehearse their performances without being physically present [9].
Furthermore, research and explorations within the area of ’cyberdrama’ [10], ’digital theatre’ [11] and ’narrative in cyberspace’ [12] has been con-ducted. In these forms of acting, participants create the story through active engagement and interactions between technology and participants. According to other research this interactivity can be grouped into ’Navigation’, ’Partici-pation’, ’Conversation’ and ’Collaboration’ which allows participants to steer the play and get involved before the final performance [13].
Another research explained the setup of a completely virtual theatre where actors steer a virtual character in real-time from their computers with data gloves and keyboard inputs and the audience can listen and interact to the per-formance and the theatre by choosing the seat in the theatre and by applauding and booing [14].
In other research, even the use of motion capture techniques during a live theatrical performance have been used [7]. On-stage, actors were interacting with digital avatars that were controlled by actors wearing a motion capture suit throughout the theatrical act. A screen, which is installed in front of the on-stage actors, displayed the avatars controlled by the motion capture actors who performed their acting in real-time on a close-by motion capture area. Virtual scenery in context of each act was displayed on a background screen, visible
1.3 State-of-the-Art 7
to the audience. In the mentioned research, the displayed scenery and avatar content around the on-stage actors was used to interact with their acting.
The above mentioned research projects show that theatre and acting adapts to the digital age and integrates not only technology but also audience and other participants during the play, preparations and rehearsals. Nonetheless, less research focuses on acting for motion capture or acting within a virtual environment in the sense of physically being in the virtual environment and where the environment is seen as digital acting support while acting.
Designing immersive environments
Designing an immersive environment includes 2 basic concepts, ’Immersion’ and ’Presence’ which have been explored and defined already [15]. ’Immer-sion’ provides the functionality and usability of an immersive environment and provides the opportunities to be immersed into the virtual environment. ence’ creates the perception of being in the virtual environment, where ’Pres-ence’ can be seen as an ”increasing function of immersion” [15].
Through the perception of ’Presence’ it is believed that behaviours in the virtual world are consistent with behaviours in the real world under similar cir-cumstances [15]. It was furthermore mentioned that ’Presence’ could influence the performance of a person using a virtual environment. This statement was to mention that a well designed interface and hardware can increase ’Presence’ and a ”greater vividness in terms of richness of the portrayed environment” which improves task performance [15]. Moreover it is mentioned that ’Pres-ence’ is of importance to train e.g. fire-fighters or surgeons within a virtual en-vironment corresponding to behaviours in the real world. Therefore, a virtual environment must be well designed and allow for immersion and the feeling of being in the environment.
Another interesting statement is that the impact of the feeling of immersion is depended on ”the application or task context and the perceptual requirements of the individual” [15]. Basically this means that different aspects need to be considered when creating an immersive environment. First, it is important to create the application according to the task to be performed. So the virtual environment should focus on the most important aspects that the real world in a similar situation would resemble. When a motion capture actor is supposed to act as a musician, an aural feedback might be more important to create an immersive feeling. This might differ when visual aspects are more of impor-tance, e.g. when acting on a futuristic space ship using futuristic technologies. In other occasions a mixture of a visual and aural environment might allow to increase immersion. We could think of acting for a war zone where audiovisual
8 Chapter 1. Introduction
scenery and effects could provide the feeling of presence. Secondly, one needs to consider that individuals are different and react to audiovisual stimulation differently. Therefore, we need to design a virtual environment allowing to create different audiovisual scenarios and conduct research to find out which virtual environments we can build that allow to support motion capture actors. Another research in the area of presence in virtual environments states that presence can only be supported when the technologies or devices used to create the virtual environment feel non-existent to the users [16]. This means that when creating an immersive virtual environment for motion capture acting, one needs to consider designing solutions that do not limit the actors’ freedom of movement or might be uncomfortable to wear.
Considering the elements of immersion and presence in the designs for our immersive environments as well as to test and evaluate them must be of impor-tance. It has to be researched how this can be applied and created for immersive environments as acting support in motion capture.
Interaction
We consider designing useful interaction scenarios for an immersive motion capture system as important and believe that research on interaction and per-ception in virtual environments could deliver important insights to interact within an immersive motion capture system. In this respect, some research, which works on body-centered interaction in immersive virtual environments, provides such insights and shows how interaction and navigation in such virtual environments might be possible [17]. This body of research was done for other purposes but might be adaptable to develop more immersive environments for game-based motion capture shoots. It still needs to be answered, how more natural interactions between actors and virtual environment can be achieved during motion capture shoots bwhen using these techniques. Furthermore, it would need to be tested how this might work for multiple performers.
In a virtual environment, the action and reaction of virtual personas and objects is in most cases fairly primitive. In some research, characters in virtual environments have been trained to react to predicted and unpredicted events in order to maintain realism. The approach to solve the problem of motion syn-thesis for interactive, humanlike characters was by combining dynamic simu-lations and human motion capture data [18]. It needs to be investigated, how and if the virtual content in a motion capture environment can be used to steer the actors while acting. Other than in the above-mentioned research the actors do not directly steer or interact with virtual content, they far more use virtual and mediated objects like real objects.
1.3 State-of-the-Art 9
Developing a virtual environment
At this point, we assume that by making the virtual content visible, yields the greatest positive result in helping an actor perform. This is why we focus on exploring ways to provide vision first, before tackling other senses. Several applications that offer such an ability are already available:
A mixed reality environment, which could possibly be used in a motion capture application, is used for many industries such as military-based training environments. In this respect, some research projects are capable of provid-ing an immersive environment without usprovid-ing virtual or augmented reality (VR /AR) glasses [4]. To achieve this aim, an immersive environment is created by using a mixture of real and projected objects, transparent digital flat screens and the capability to add smell and temperature changes to the environment. There is also a wide-area mixed reality application, which was realized to create an immersive virtual reality environment in which users can walk and run freely among simulated rooms, buildings and streets. In such applications, large rear-projection screens that employ digital graphics are used to depict a room’s interior, a view to an outside world or a building’s exterior. The applications also provide life-size projection displays with physical props and real-time 3D graphics [4]. A scenario like the one mentioned above has its limits when con-sidering it for a motion capture shoot. Large projection walls or other props cannot be placed in front of the optical motion capture cameras because the recordings would be blocked and occlusions would occur. Setting up such a training environment requires time and planning which is not economical in a motion capture environment because the scenery for a motion capture shoot changes often, sometimes multiple times during a shoot day, and needs to be dynamic. Therefore, this solution cannot be used, as is, to create an immersive environment for motion capture actors with the goal to support and allow more natural acting.
A novel optical see-through head-worn display that is capable of mutual occlusions could also be considered for motion capture shoots. Here, mutual occlusion is an attribute of an augmented reality display where real objects can occlude virtual objects and virtual objects can occlude real objects [19]. Mutual occlusions are one of the problems in visualizing 3D content in a real world environment. Research is also conducted to test the perception of im-age motion during head movement [20]. The perception of imim-age and head motion is tested when wearing a head-mounted display (HMD). In another sig-nificant work, in addition to virtual and augmented reality, two control condi-tions were studied: viewing real-world objects, and viewing real-world objects through a head-mounted display. The presence and absence of motion parallax
8 Chapter 1. Introduction
scenery and effects could provide the feeling of presence. Secondly, one needs to consider that individuals are different and react to audiovisual stimulation differently. Therefore, we need to design a virtual environment allowing to create different audiovisual scenarios and conduct research to find out which virtual environments we can build that allow to support motion capture actors. Another research in the area of presence in virtual environments states that presence can only be supported when the technologies or devices used to create the virtual environment feel non-existent to the users [16]. This means that when creating an immersive virtual environment for motion capture acting, one needs to consider designing solutions that do not limit the actors’ freedom of movement or might be uncomfortable to wear.
Considering the elements of immersion and presence in the designs for our immersive environments as well as to test and evaluate them must be of impor-tance. It has to be researched how this can be applied and created for immersive environments as acting support in motion capture.
Interaction
We consider designing useful interaction scenarios for an immersive motion capture system as important and believe that research on interaction and per-ception in virtual environments could deliver important insights to interact within an immersive motion capture system. In this respect, some research, which works on body-centered interaction in immersive virtual environments, provides such insights and shows how interaction and navigation in such virtual environments might be possible [17]. This body of research was done for other purposes but might be adaptable to develop more immersive environments for game-based motion capture shoots. It still needs to be answered, how more natural interactions between actors and virtual environment can be achieved during motion capture shoots bwhen using these techniques. Furthermore, it would need to be tested how this might work for multiple performers.
In a virtual environment, the action and reaction of virtual personas and objects is in most cases fairly primitive. In some research, characters in virtual environments have been trained to react to predicted and unpredicted events in order to maintain realism. The approach to solve the problem of motion syn-thesis for interactive, humanlike characters was by combining dynamic simu-lations and human motion capture data [18]. It needs to be investigated, how and if the virtual content in a motion capture environment can be used to steer the actors while acting. Other than in the above-mentioned research the actors do not directly steer or interact with virtual content, they far more use virtual and mediated objects like real objects.
1.3 State-of-the-Art 9
Developing a virtual environment
At this point, we assume that by making the virtual content visible, yields the greatest positive result in helping an actor perform. This is why we focus on exploring ways to provide vision first, before tackling other senses. Several applications that offer such an ability are already available:
A mixed reality environment, which could possibly be used in a motion capture application, is used for many industries such as military-based training environments. In this respect, some research projects are capable of provid-ing an immersive environment without usprovid-ing virtual or augmented reality (VR /AR) glasses [4]. To achieve this aim, an immersive environment is created by using a mixture of real and projected objects, transparent digital flat screens and the capability to add smell and temperature changes to the environment. There is also a wide-area mixed reality application, which was realized to create an immersive virtual reality environment in which users can walk and run freely among simulated rooms, buildings and streets. In such applications, large rear-projection screens that employ digital graphics are used to depict a room’s interior, a view to an outside world or a building’s exterior. The applications also provide life-size projection displays with physical props and real-time 3D graphics [4]. A scenario like the one mentioned above has its limits when con-sidering it for a motion capture shoot. Large projection walls or other props cannot be placed in front of the optical motion capture cameras because the recordings would be blocked and occlusions would occur. Setting up such a training environment requires time and planning which is not economical in a motion capture environment because the scenery for a motion capture shoot changes often, sometimes multiple times during a shoot day, and needs to be dynamic. Therefore, this solution cannot be used, as is, to create an immersive environment for motion capture actors with the goal to support and allow more natural acting.
A novel optical see-through head-worn display that is capable of mutual occlusions could also be considered for motion capture shoots. Here, mutual occlusion is an attribute of an augmented reality display where real objects can occlude virtual objects and virtual objects can occlude real objects [19]. Mutual occlusions are one of the problems in visualizing 3D content in a real world environment. Research is also conducted to test the perception of im-age motion during head movement [20]. The perception of imim-age and head motion is tested when wearing a head-mounted display (HMD). In another sig-nificant work, in addition to virtual and augmented reality, two control condi-tions were studied: viewing real-world objects, and viewing real-world objects through a head-mounted display. The presence and absence of motion parallax
10 Chapter 1. Introduction
was crossed with all conditions. Like many previous studies, another research found that depth perception is ”underestimated” in virtual reality, although the magnitude of the effect was surprisingly low. Underestimation was described as a misperception of distances in the virtual reality in comparison to the real world. The most interesting finding was that no underestimation was observed in AR [21]. For motion capture shoots, the interactions between actors and the possibility to see props are important. This is another reason why a VR environment is not the best solution when an actor needs to be able to see real world objects and persons, as well as virtual content at the same time. A gen-eral problem with HMD’s remains, regardless of future AR glasses solutions: a significant part of the actor’s face is covered and can limit motion capture shoots. The glasses especially limit the freedom of movement of an actor and the shoots are limited when facial motion captures are of importance.
To make virtual environments as visually immersive as possible another so-lution has been widely researched by using flat panel 3D displays [5]. Equip-ping an entire motion capture room with these 3D displays, or even by just using a single flat screen needs to be investigated if this might be a solution that could create an immersive acting environment and still is economic and usable in daily motion capture business.
Another method which could be of possible use to display virtual content to the actor, while acting, is by using the emerging laser based pico projector technology [6]. These projectors are of small size [22, 5] and come in differ-ent technologies based on micro-LCDs, the Texas Instrumdiffer-ents’s DLP technol-ogy, which uses an array of microelectromechanical systems (MEMS) micro-mirrors and LEDs, or projectors based on laser scanning [23]. The projected data can be shown on small screens or can be reflected to polarized video con-tact lenses which a user is wearing [5]. A screen, which is placed right in front of the actor might limit the actor in some cases but might be applicable to some motion capture shoots. Nevertheless, it would be good to answer the question on how virtual content can be shown to the motion capture actors and it needs to be researched how new technologies can be applied to a motion capture sce-nario.
Assessment and Evaluation
Finding ways of assessing and evaluating our research and prototypes accord-ing to its usability, functionality and its ability to provide immersion is what we are aiming for. Some research has already been performed that can help within this matter and is described below.
Measuring immersion and presence in virtual environments and computer
1.3 State-of-the-Art 11
games has been researched in multiple research already [24, 25, 26]. This could provide a basis to assess the feeling of immersion in our research as well.
Even though there are not many guidelines on how to design and evaluate virtual environments, a methodology for ensuring the usability of virtual envi-ronments through user-centered design and evaluation of virtual environment user interaction has been provided [27].
Measuring user satisfaction [28] and emotions [29] as explained in the cited research might be another way to support our task to assess and evaluate our efforts towards a more immersive motion capture system.
According to in literature mentioned research we will need to identify which of the mentioned methods to assess and evaluate our research suits best and how it might need to be modified for the area of motion capture acting. This state-of-the-art section shows that there are research projects that provide technologies that could be used to create and explore solutions towards an im-mersive motion capture acting environment. Nevertheless, research needs to be done on how such an environment can be created and how it can be used to support actors while acting. To approach this, we describe in the following how our research and approach towards a solution is set up.
