IN
DEGREE PROJECT MEDIA TECHNOLOGY, SECOND CYCLE, 30 CREDITS
STOCKHOLM SWEDEN 2019,
The Sound of Suspense
Designing an audio-physical, interactive storytelling system
TANIA CHRISTENSEN
KTH ROYAL INSTITUTE OF TECHNOLOGY
SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE
Abstract
In this paper, the design process and evaluation of an interactive audio-physical storytelling
system that uses motion capture to deliver a narrative is described. Following a research through
design approach, the aim is to establish which factors are important when designing audio-
physical systems for storytelling. To investigate how the visuals and audio collaborated, two
diferent experiments were conducted. Six participants were flmed when interacting with the
system while using the Sensual Evaluation Instrument (SEI), followed by an interview focusing on
their experience of the narrative and an analysis of their SEI usage. Results showed that the audio
structures the dramatic experience, while the visual elements put the narrative in a context and
support curiosity.
Sammanfattning
I den här rapporten beskrivs designprocessen och utvärderingen av et interaktivt audio-fysiskt system som använder motion capture-teknik för at leverera et narrativ. Genom användning av Research through design-metodologi var målet med studien at fastställa vilka faktorer som bör tas hänsyn till vid design av audio-fysiska system för historieberätande. För at undersöka hur visuella och auditiva element samarbetade utfördes två olika experiment. Sex personer flmades när de interagerade med systemet samtidigt som de använde Sensual Evaluation Instrument (SEI).
Eferåt följde en intervju med fokus på upplevelsen av narrativet och en analys av interaktionen
med SEI. Resultaten visade at de auditiva elementen strukturerar den dramaturgiska upplevelsen
medan de visuella elementen ger narrativet et sammanhang och uppmuntrar till nyfkenhet.
The Sound of Suspense
Designing an audio-physical, interactive storytelling system
Tania Christensen MID, Media Technology and
Interaction Design Royal Institute of Technology
Stockholm, Sweden taniac@kth.se
ABSTRACT
In this paper, the design process and evaluation of an interactive audio-physical storytelling system that uses motion capture to deliver a narrative is described.
Following a research through design approach, the aim is to establish which factors are important when designing audio-physical systems for storytelling. To investigate how the visuals and audio collaborated, two diferent experiments were conducted. Six participants were flmed when interacting with the system while using the Sensual Evaluation Instrument (SEI), followed by an interview focusing on their experience of the narrative and an analysis of their SEI usage. Results showed that the audio structures the dramatic experience, while the visual elements put the narrative in a context and support curiosity.
Author Keywords
Audio-physical systems; Interactive storytelling;
Location-based audio;
INTRODUCTION
Storytelling is an ancient craf and one of the primary ways of communication between humans[6, 16]. One way of making sense of the world around us is by storytelling and the ability to order events into narratives. New media technologies have quickly been adopted for storytelling. Technological advances in the past two decades have seen spatial audio technology permiting advanced audio-augmented environments revolutionise the feld of interactive soundscape design.
An example of how this technology can be used is in immersive audio-augmented environments. Tese have previously been explored mainly in exhibition spaces and museums, with the object of immersing the visitor in a visually dominated space with the help of location based auditory scenes [5, 7–9, 15, 21].
In this paper, the study of a new interactive, audio- physical storytelling system will be presented, with focus on the relationship between the design of the
physical space and auditory scenes. It builds upon the same idea of how to use spatial audio technology as the research mentioned above but on a smaller scale as it does not involve full-body interaction. Te system consists of a tabletop miniature of an apartment in which a drama takes place. It can be experienced by using motion capture technique; by dipping a motion capture marker in diferent locations in the miniature, diferent parts of the narrative and story can be heard, creating the illusion of being a fy on the wall.
Te system can be viewed as a combination of immersive theatre and radio drama, but with a tangible interface. Traditional radio drama has ofen been criticised for being an incomplete medium due to its lack of visual elements[4, 11]]. Tus, one could argue that with vision being our dominant sense, the introduction of visual elements to a radio drama will create a medium for delivering stories that in theory should be perceived diferently from traditional radio dramas[22]. With this theory in mind and considering the interactive elements of the system presented in this paper, I aim to propose what aspects are important to consider in a design process in order for a user to fully understand the narrative. Te research question is as follows:
RQ: What factors infuence a story delivered through an interactive audio-physical medium?
BACKGROUND Location-based audio
Location-based audio experiences have been covered by research in various felds. Te idea was frst explored more than half a century ago, when visitors to museums wore headphones with pre recorded information about the artwork situated along a certain path[7]. Although the interaction and personalisation of the experience were limited to pausing and
rewinding, a seed had been planted that would later grow into immersive audio-augmented environments.
Availability of advanced spatial audio technology opens the doors to various implementations, ofen extending the idea of creating individual experiences in exhibition and creative spaces[7–9, 19, 23, 25]. Te work presented in this paper is similar to the LISTEN project, where the key concept was to place the notion of visual, auditory and imaginary and the relationship between them at the center of the design [7, 25].
Audio-based narratives in historical locations, investigating the perceived emotion from a combination of content design, sound design, tangible and embodied interaction, have also been explored[5, 15, 18]. In[5], listeners are instructed to follow a certain path, while[15, 18] encourage the listener to explore the allocated space as they wish; something that was enabled through the implication of non linear narratives. Reid et. al in[18] do however mention that the non linearity created some confusion due to not presenting an overarching structure and that future designs of similar systems should take this into consideration.
In [13], Drewes et al. designed an audio-only experience based on the classic murder mystery game Cluedo. Te player takes on the role as detective and has to conclude what character is guilty based on conversations and acoustic cues that can be heard across the virtual environment where the game is set.
Drewes gives valuable insight into how acoustic cues should be designed compared to other acoustic elements of the system, underlining the importance of testing and fne-tuning each individual sound. Te study also identifes a range of design issues regarding the interactive narrative. To motivate the user to keep exploring the space, there must be a well-writen narrative at the core of the game. As the story is non linear, care has to be taken when designing the diferent parts of it. Each collection of acoustic cues in a particular room must tell its own story, while still contributing to the overall narrative.
Narratological structure
When constructing a captivating drama, whether it is for a traditional medium or interactive technology, the theory still used in Western tradition dates back to 300 B.C, when Aristotle presented Aristotle’s Arc for analysing Greek tragedies[1]. In 1863, Freytag introduced his own structure based on Aristotle's arc, but tweaked to ft modern tales. See fgure 1. It has since become the standard for various kinds of storylines. Madej, through Ong, suggests that the most
obvious use of such a model is the mystery story, as it has a simple and clear approach to reaching a climax [14].
Figure 1: Figure showing the Freytag pyramid
Designing and evaluating interactive digital narratives
Murray expands Aristotle’s model into the domain of interactive narratives, by introducing three concepts.
She categorizes these as immersion, agency and transformation in her famous framework for digital media, with the two former being interesting in this research[17]. Drawing inspiration from Murray, combined with Roth’s own previous work, Roth and Koentiz presented a toolbox for user experience research that can be used to identify design principles for interactive digital narratives (IDN) [20, 21].
Murray describes immersion as the feeling of being transported to a diferent reality[17]. According to Roth, it exists on both a perceptual and a narrative level [20]. Experiencing f o w a n d presence belongs to the perceptual level, while role-identifcation, believability, curiosity and suspense are linked to the narrative. Te perception of fow is accomplished by designing an experience that is not too easy nor too demanding, but keeps the interactor engaged at all times with full dedication. Presence is a product of fow, leaving the interactor feeling absorbed in the story-world. Te atributes linked to the narrative are obtained by creating a realistic environment (believability), assigning the user an elaborated role (role- identifcation), designing a space and narrative that is somewhat uncertain and allows exploration (curiosity) and that plays on the interactor's emotions (suspense).
Agency is described as the satisfying power to take meaningful action. One form of agency characteristic of digital environments is spatial navigation [17].
Lessons can also be learned from theory about interactional trajectories, which describes a user’s journey through an experience[2]. Benford et al. defne
key transitions when the feeling of continuity is at risk.
Te ones interesting for this paper include beginnings, endings a n d traversals between physical and virtual world.
Te system investigated in this paper relies heavily on the idea that the users are free to explore the miniature as they wish. At the same time, it has to be designed to make sure that they do not miss out on critical parts of the narrative. Benford et al. describe this as tension between the canonical trajectory, the ideal route intended by the author, and the participant trajectory, the users actual route, which can be resolved through careful orchestration [2, 3].
METHOD
Research through design (RtD)
RtD is an approach that uses methods from design practice[24]. Te aim is to do research that could lead to theory for design by engaging in design activities and/or designing an artifact that itself is an implicit, theoretical contribution [23].
Writing the manuscript
Te story taking place is a about a murder and robbery, inspired by the von Sydow murders which took place in Stockholm in the 1930s. Here, it is the early 70s and we are in an apartment on Mariaberget in the south of Stockholm. Te theme was chosen due to it being especially appropriate to apply Freytag’s Pyramid on [14]. Described below are the seven diferent scenes with critical story segments taking place, each of them writen to more or less ft diferent parts of Freytag’s pyramid. As proposed by[13], the majority of scenes are independent from each other, while still contributing to the overarching narrative.
S1. Rofe and Ufe, the two bachelors of the apartment, talking about Anna who has just moved in with Tomas, and that she held the dish brush like a knife the other day.
S2. Lasse and Lena, an old hippie couple, playing the guitar and talking about how what has just happened in town resembles the von Sydow murders that took place in the 30s. Afer their guitar playing and conversation a Velvet Underground song is played to indicate the year.
S3. Eva and Janne, another couple, talking about what painting to hang above their bed and
“speaking about painting… can one see the fat where it happened from the bedroom window?” Eva then says to meet in the kitchen for cofee.
S4. News segment about a robbery and murder in a fat on Norr Mälarstrand, where a painting by
Anders Zorn was stolen and colonel Gyllencrantz and his maid were murdered. Te segment interrupts a famous Swedish TV show (Hylands Hörna). (climax)
S5. Eva and Janne, in the kitchen talking about that Tomas and Anna keep their door locked which is against the housing policy. Tey also mention that it is odd that the new couple wears coats in the middle of the summer.
S6. A child whom Eva is babysiting approaches Eva and says that maybe she and Janne can hang the painting which was just lying under the bed in Tomas and Annas room. Eva puts two and two together and calls on Lasse and Lena.
S7. Tomas and Anna realise that they are caught and that there must have been a spare key to their room. Tey jump out the window as the sound of sirens gets louder and the police is knocking on the door. Credits are rolled to signal the end of the story.
Additionally, for each scene, the voice actors improvised a few minutes of dialogue which did not have anything to do with the narrative but rather hinted about the time and seting of the story.
Building the prototype
Afer the seting for the manuscript was decided, a miniature of an apartment with the dimensions 60x45 cm was built. Te size was determined to be easy to navigate in; the user only has to move his or her arm to reach every part of it. Te furniture and other objects in the miniature were built in wood or 3D printed and then painted to ft 1970s design. See fgure 2. Te objects selected to get a visual representation were of some importance to the story. For example, a door was created for Tomas's and Anna’s room. Besides being aesthetically pleasing, they could act as visual cues to guide the user during the experience.
Figure 2: Living room coming together
Script for structuring the narrative
Te script for handling the audio fles was writen in Python by collaborator Ariel Blomkvist Rova. As input, the script takes coordinates of the motion capture
marker collected by the motion-capture system. In each room, a diferent conversation and/or ambient sound can be heard. As can be seen in fgure 3, the diferent audio fles are clustered in diferent groups, which in turn make up diferent stages of the narrative that are triggered in a strict order. Tis is due to logistical reasons, two conversations can’t take place at the same time in one room as it would be very hard to listen to, and to make sure that the user hears the diferent scenes in an order that makes sense dramaturgically.
Te algorithm made it possible to have characters appear in more than one scene and gave the opportunity to sidestep from Drewes' notion about designing individual collections of acoustics for each room [13].
Most of the time, the system is instructed to play the improvised part of the scene. However, every twentieth second, the script for handling the audio activates story segments to keep the story moving. Which segment is activated depends on what stage the system is currently in. However, the segment will not be played until the user is in the corresponding location, making sure important parts of the narrative are not missed. If a user is already in an activated space, they will not be exposed to the story segment until leaving and revisiting the space. Tis is to avoid unnatural breaks in the dialogue. Each story segment can only be played once afer which the system will change state. See fgure 3.
Figure 3. Table showing the progression of the narrative
Design of auditory display
Room 2, 3 and the hallway were purposely designed without a conversation taking place in them so it would be possible to investigate if it infuenced the experience.
In room 2 and the part of the hallway which had windows, outdoor ambiance was placed. In room 3 it was completely quiet. Te last parts of the drama, S6 and S7 were not tied to a certain location but were instead dependent on time passed since the last story event, thus they could be experienced anywhere in the miniature. Tis was due to, from an author’s
perspective, a desire for high pace as the story came to an end.
Two diferent cases, case A and B, where the audio fles were reorganised between the rooms were designed. In case A, the diferent scenes were placed in rooms that correlated to them. S1 took place in the kitchen as they talked about doing the dishes and grocery shopping. In case B, both ambiance and scenes were reorganised according to the description in fgure 4. Te purpose of the diferent cases was to investigate if and how the navigation of the miniature and interpretation of the narrative difered depending on the layout.
Figure 4. Placement of audio fles in the diferent cases
Te conversation taking place in S1 had some issues due to incorrect setings when it was recorded.
Sentences were cut inappropriately but the audio fle had to be used anyway as the actors were not available to record again. Afer editing it was deemed acceptable to include although the faults could still be heard.
Evaluation
Sensual Evaluation Instruments
Sensual Evaluation Instruments (SEI) are used to support non-verbal self-report of emotion as an experience unfolds[12]. Te idea is to use a set of objects with diferent shapes that is accessible to interact with as desired by the participant. Te shapes are meant to evoke diferent emotions, thus the interactions serve as markers for discussions in the evaluation of the experience which is recorded on video. In this study, a set of eight objects was used.
Figure 5: SEI and their names [12]
Pilot study
A pilot study was conducted with one person to evaluate the experiment setings before the actual study took place. Te result of the pilot study lead to the elaboration of background stories of the participating characters, as the pilot subject had a difcult time recognising voices and remembering names throughout the progression of the story. Te intention of the SEI was also more thoroughly explained in the actual experiments.
Procedure
Te setup of the user tests was as follows:
A consent form regarding usage of collected data was signed. Te participants were informed about what kind of experience the system allows for. Tey were told that there is a story taking place in the miniature that they could listen to by moving the motion capture marker through it and were given a brief instruction of how the motion capture system works. Tey were encouraged to stay and listen to diferent conversations but to move around if they didn’t think what they heard made the story progress.
Te SEI were presented and the participants were told that they were free to handle the objects as they wished. Tey were informed that the objects were created with an emotion in mind, but not which one, and that they should pick up/handle a new object whenever the system triggered a new emotion.
Te participants got to read the introduction of the characters in the story, afer which the interaction with the miniature could begin. Te procedure was flmed with one camera, placed in front of the subjects, capturing the interactions with the SEI, facial expressions and the movement of the motion capture marker through the system.
Figure 6:Participant interacting with the SEI.
Aferwards, an interview consisting of three parts was carried out. First, the participants were asked to retell
the story, indicating whether they had understood the narrative and how it ft the Freytag pyramid. In the second part, the subject, while watching the video of themselves, was asked about the interaction with the SEI and gestures made during the experiment. Finally, general questions about the experience, the design of the miniature and how it correlated with the story they had just heard were asked. Te last part was especially interesting for case B. A thematic analysis was then carried out to capture recurring answers and observations.
Six people, three males and three females, with an average age of 28 participated in the study. Four participants had a background in engineering and programming while two of them were completely new to the feld.. Tree of the participants were presented with case A and three with case B. Below they are referred to as the leter of their case and a number from one to six. Eg. the second participant presented with case B is referred to as B5.
RESULTS AND EVALUATION Navigating the story
When asked to retell their experience, A1, A3 and B4 focused on summarising the story and immediately recounted the climax of the plot without describing the miniature or any conversations or actions that were not directly connected to it. In contrast, A2, B5 and B6 described how they entered the apartment and how they explored, listened to diferent conversations and tried to make sense of the characters and the physical space before gradually understanding the details of the narrative.
All participants understood that the story revolved around the robbery of a famous painting and that the persons guilty of the crime were residents of the apartment. Only one person, B4, gathered that there had been a murder as well as a robbery. B6 said she had hoped for a murder the whole time and was disappointed in the end that it had “only been a robbery”.
Almost all participants needed to be interrupted when the credits were rolling as it was not clear that the story had ended. A1 and A2 moved the marker to outside of the miniature to see if the story would continue there, afer Tomas and Anna had jumped out the window.
Story elements
All subjects claimed that the most essential part for understanding the story was the news segment and the
conversations between Eva and Janne; Eva’s and Janne’s frst conversation to understand what had happened and their second to understand that it was someone in the apartment who was guilty.
S1 was overlooked by most and only mentioned as a clue by A1 when he was asked whether he had noticed anything suspicious in the narrative before the climax was revealed. B6 briefy mentioned the same but said she had been confused by it and thought that it was one of the characters participating in that dialogue that had accidentally killed someone on a night out. S2 was not mentioned at all in this part of the evaluation.
Te experiment did not indicate if there was a diference in how participants in the diferent groups understood the overarching narrative.
Identifying the antagonists
A3 was the only person who was able to correctly state the names of the persons guilty of the crime. With the exception of B5, everyone else realised it was the people who resided in the room with the locked door who were guilty, but expressed confusion about their names or if they had been visible and/or participating earlier in the story. B5 said that it was Lasse and Lena who were guilty of stealing the famous painting, “as they probably wanted money to eat more oysters”, something that was mentioned in the background as an old hobby of Lena’s.
Navigating the miniature
All participants started exploring in the nearest half of the miniature. Tree of them went through the hallway, two started in room 4 and one in the kitchen. No one started in room 3, situated on the lower half furthest away from hallway. B6 explicitly said: “I started through the hallway because it is from there a house is entered.”. B5 said he put the marker in the room with the guitar and Beatles poster because it was the nearest to him. A1, A3 and B6 expressed that they systematically wanted to visit every room in the miniature to understand what was going on and not miss out on anything. Test subjects rarely moved diagonally. If wanting to move across the fat from one corner to another they usually followed the hallway. A3 decided to move anti-clockwise through the space because that was the most comfortable thing to do. A2, B4, B6 all decided to leave a room because the static movement of holding the marker was uncomfortable.
All participants visited all rooms at least once. On average, the least visited room was room 3, the quiet room, followed by the living room and room 2.
Temporal diferences
Te participants spent approximately ffeen minutes exploring the miniature before they reached the end of the narrative. Tere was no diference in time spent between group A and B. However, participants in group B heard the frst story events afer more time had passed than the participants in group A.
Handling the SEI
In the following section selected interactions with the SEI will be presented.
Act 1: Exposition and complication (S1 and S2)
A2 and B4 used Bubbly when hearing about Anna holding the brush as a knife. B4, whom had earlier been exposed to Lasse’s and Lena’s conversation said “Here I had completely forgoten about Gyllencrantz! I just thought maybe they have killed someone! I got really confused because there was so much information.”
Ten, during the improvised part of the conversation A2 moved Barbapappa. “Te frst [Bubbly] one means that I was confused. What are they saying? Te second one [Barbapappa] means that it is exciting, because I don’t know what is going to happen.” Although not handling the SEI, A3 too expressed confusion in this phase. “I thought about picking up the stone in the kitchen [as well], but thought that I had maybe misinterpreted the story.” During the other part of the frst act, S2, B5 said he registered that Lasse and Lena were talking about a murder and that the story might have something to do with it. B6 was the only other person to pick up a SEI here but said she couldn’t remember why she did it.
Act 2: Climax (S3, S4 and S5)
A2 picks up Barbapappa afer hearing Eva and Janne talking about the apartment where something has happened and that they are going to have cofee in the kitchen. “(…) I thought that something was about to happen in the kitchen, but it didn’t. Afer a while I thought, maybe she got stuck somewhere on the way so I tried looking for her in the other rooms. Maybe I should have tried following her way to the kitchen through the hallway.”
A3, B4 and B6 expressed similar thoughts. For A3, it was the one time she used the SEI. “I got curious, so I went into the living room to see if I could fnd out more on the TV”. B4 said he got excited, and that he understood that it was related to things he had previously heard, but due to the amount of information he had been given it was hard to understand exactly how. B5 picked up Bubbly and said that “It felt like [he]
entered a new chapter.”.
All participants in group B picked up a SEI in the kitchen where the TV show was playing as they all had difculties understanding who was talking. B4 said “I had no idea what was going going on. I thought maybe it was someone’s dad who was holding a speech. I wondered if there was anybody called Lennart. Ten they said “show” and I realised what it was.” B6 thought that Hyland was “in the way” and said “I was really annoyed, because I knew I was close to fguring the whole thing out. I was very engaged at this point.”
Act 3: Reversal and denouement (S6 and S7)
When hearing S5, B4, B5 and A2 picked up Bubbly.
B4 thought it was mysterious while B5 said “Tis was one of the highlights, when things started coming together.” A2 had another take on it: “It was funny that they said that they shouldn’t judge people but immediately started doing just that.”.
B6 expressed “It’s happening now! I am really excited and interested in what’s going on. I understand now that someone in the apartment has stolen the painting.
I know it is all gonna unfold soon.”.
B5 picked up Barbapappa when Tomas and Anna jumped out the window. “Now the mystery is solved, I understand that it’s done.” B4 said that now it all made sense and picked up Spiky.
SEI and Ambiance
A1 used the SEI three times, only in connection to diferent ambient sounds. Starting of in the kitchen, he handled Spiky. “I did not like the ambience, it felt dark and made me uncomfortable.” In the room where Lasse and Lena were playing the guitar he picked up Bubbly.
“It was nice to stop and just listen to the music for a while.” Lastly, he handled Bubbly again when listening to the bird song. B5 indicated with Stone that he enjoyed the music in the room, both the guitar playing and the song. B6 also expressed that she thought it was cozy in the room with the music, crediting the aesthetics of it as well.
Te SEI was used 5,3 times on average. A3 only used them once, while B6 used them thirteen times. Te average in group A and B were 4 and 9,3 respectively.
Te highest usage of the SEI was during the climax of the story. Participants in both groups mainly used them to indicate when they were afected by the narrative, but also to describe emotions evoked by the ambiance, poor sound quality and on one occasion the appealing aesthetics of the miniature. In group B the instruments were also used due to confusion arisen from the unmatched audio and visuals. A recurring statement
across both groups was that it was stressful at times to deliberately decide on which instrument to pick up.
Bubbly was mostly used to describe curiosity and confusion and Spiky when experiencing discomfort which is along with Isbister’s fndings[12]. However the analysis showed that in general, the usage did not really refect how the SEI was used in Isbister's study.
Audio, visuals and narrative collaborating
All participants in both groups said that they navigated and explored the miniature according to what they heard, but difered in how they experienced the purpose of the aesthetics and details of it. A1 said “I was defnitely more driven by the audio, but when I could make a connection to the visuals, like in the music room the experience got extra intense.“ For participants in group A, the visuals seemed to amplify what was communicated in the audio rather than being the driving force itself. All participants in group B on the other hand said that the visual elements were almost redundant. B4 said “I could have had my eyes closed during the whole time. (…) You could have removed all furniture and details and just have had empty spaces for me to move around.”. B5 and B6 expressed similar thoughts, but also reported that the incorrect or lack of visuals urged them to use their imagination instead. B5 said “Where the visuals didn’t match the audio I just flled in the gaps. When I heard the TV I thought it was a radio instead, and when Eva and Janne were in the living room instead of kitchen I thought they had been in the kitchen and fetched the cofee before going to the living room to sit down.“ and B6 said “I didn’t think much about the visuals. I thought it was weird that there were no windows when they talked about windows, but then again they also talked about wardrobes and there were no wardrobes either.”
Te ambiance of the kitchen in the living room turned out to be harder to make sense of. B4 thought that he could maybe hear through the walls while B5 did not understand what he was hearing at all.
Te only time any participant expressed their actions being driven by the miniature was when wanting to fnd out more about the mysterious incident Eva and Janne were talking about. Both A2 and A3 then went to the living room because they thought that they could fnd out something from the TV. Unlike in the scene where Eva says she is going to the kitchen, it had not been indicated by the narrative that they should visit the living room to make the story progress.
Regarding using the miniature as means for understanding the narrative, A1, A2, A3 and B6 all tried
to fgure out which rooms the diferent characters resided in by looking at the number of beds and the size of them. A2 said that “(…) logically there would be a couple in the bedroom with the double bed, but there was only one double bed so [she] wasn’t sure.” B3 had similar thoughts, but reasoned that it probably was common in the 70s for two people to sleep in a single bed. B6 said several times that she thought that there should be visuals corresponding to the crucial parts of the story. “It’s funny that they are gonna jump out a window but there isn’t a window in that room! I was thinking that they are stuck.”
B6 did not realise Lasse and Lena had appeared at all, while A2 thought that Lasse and Lena resided in the double room as 1) Tey were a couple and there was a double bed and 2) Eva and Janne, the ones staying in the double room in case A, were talking about yoga, and in the introduction it says that Lena likes yoga.
A3 was certain that it was Janne who resided in the room with the guitar, Beatles poster and record player as it was mentioned in the introduction that he was a music teacher and Beatles fan.
Positioning the work
A1 and B4 both mentioned audio books when asked about similar systems to this, but that this was more interactive and ofered the possibility to decide the pace of the whole experience according to one’s own preference. A1 said that “It was like a combination of an audio book and a video game. I was scared of missing out though, and I wish I would have felt comfortable exploring a bit more.” A2, B4 and B6 compared the experience with diferent games: Cluedo, Ace Atorney and Escape Rooms respectively. B4:
”You’re presented with loads of facts and have to work out what is important for the narrative and what is not.”
A3 said she thought of radio drama or something that belongs in a museum, thus being the only one who confrmed the author’s idea of how to position the system.
DISCUSSION
Te miniature, the story taking place in it and the evaluation were designed to explore how a physical model and audio narrative collaborate with each other to create a novel experience within the feld of interactive storytelling. In this section I will present my view of what are the most important aspects to consider in a process of designing such systems.
Navigating the story and its characters.
Te pilot study resulted in background stories being writen for each of the characters participating in the story, something that proved both helpful and confusing. It provided an appropriate beginning to the experience as it built suspense and was mentioned several times as an important part of making sense of the narrative. It did however lead to incorrect assumptions occasionally being made, as the topic of the conversations seemed to be the most important aspect when trying to understand who was present, more so than names and dynamic between the characters. Tis suggests that extra care should be taken when writing the manuscript, to make sure that the conversations are in line with the personalities and interests of the characters to avoid confusion. Also, considering how difcult it was for the test subjects to name who were guilty of the crime and to understand who was talking, it would have probably been suitable with fewer characters.
As for understanding when the story came to an end, the credits did not provide a clear transition. Even though participants stated that they realised that the story was almost fnished, they still had to be interrupted when the credits were rolling. It needs to be communicated clearer through the narrative when the story is fnished, perhaps through auditory icons.
.in relation to Freytag’s pyramid
Te climax of the story, the top of Freytag’s pyramid combined with Eva’s and Janne’s conversations were pointed out by all participants as the most important piece to understand the narrative, hence proving that the dramatic curve came across at least in that sense.
Te SEI showed that as well, as Eva and Jannes conversations generated a lot of interactions.
Participants also expressed that they were curious, interested and excited which indicates that they felt some level of immersion[20]. However, the frst act of the story was almost completely overseen when the subjects were asked to retell the story. Hardly anyone stated that they understood what kind of incident had taken place before being exposed to the climax. On the other hand, the usage of the SEI and the subsequent interviews did show that some people reacted to the earlier segments, so they were not completely obsolete even though they were not remembered aferwards. A reason for this outcome was that the subjects were not familiar with the von Sydow murders, making it a difcult clue to understand. S1 might have been too subtle or it was not mentioned aferwards because it happened very early on in the story afer which there was a large stream of information to process. Failing to notice these two clues also lead to a majority of
participants misunderstanding the crime that had taken place as the later part of the narrative focuses on the stolen painting. Because it was hard to tell what information was signifcant, the idea of having improvised conversation that was not essential to the narrative should be reconsidered. It should at least contribute to a deeper understanding of the characters and their relationships, which is more likely if scripted.
Navigating the miniature and temporal diferences
As all participants started the exploration of the system in the part that was physically the closest to them and also tended to return there more frequently than to rooms further away, important conversations and parts of the narratives should be placed within that area if what is desired is a from the beginning high paced story. Another possible solution is to create a physical model that is possible to interact with from diferent angles. In case B, all participants took longer to hear the frst two story segments than in group A, which can be interpreted was because they were placed further away. When the story progressed there was no diference in when story segments were activated and participants in both groups spent the same amount of time on exploring before the story came to an end.
Tat participants did not move diagonally across the miniature but tended to move along the hallway implies that these should not be overlooked in a design process. In this study its purpose was underestimated.
Clear paths to move along can add to guiding the user, minimizing the tension between the canonical and participant trajectories.
Visual design
From the results it seems like, even though the miniature was appreciated by the participants, it was something that added to the experience rather than made it. I believe the outcome would have been diferent had all the details that were mentioned in the conversations been present and not just some of them.
Now, the lack of physical representations made the ones that were actually there insignifcant. On the other hand, similarly to when listening to radio drama [4], many subjects reported making sense of the narrative by imagining details in the miniature that were not there but that suited what could currently be heard, for example imagining a radio when the segment with the TV show was played in the kitchen. Leaving room for the imagination will provide a more personal experience as it will then be co-created by the user. A question that arises is how to keep a good balance between providing a believable system that is still
encouraging creativity and imagination. Höök addresses this confict in her paper about designing for familiar open surfaces, where she suggests designing ambiguous representations[10]. One way could be to a d d on ly details that were mentioned or somehow correlated to the characters background stories. Te miniature would still be believable, but would be open for interpretation while not adding unnecessary objects to an already information-heavy system.
Sound and auditory display design
In the last scene, three characters appear for the frst time; the child who fnds the painting (all subjects failed to understand who discovered the painting) and Tomas and Anna who understand that they have been caught. In hindsight, having that scene played anywhere in the miniature to keep up some kind of pace and excitement in combination with presenting three characters for the frst time was a mistake. For example, it taking place in a room without windows made one of the participants question the believability and discredit thedenouement. Instead there should have been a cue, similar to when Eva said she was going to the kitchen, that guided the user in the right direction.
Te faulty audio in S1 was a big issue as indicated by participants both when using the SEI and in the interview. Te importance of having good quality audio became very clear to not disturb the fow and continuity. Leaving parts of the miniature completely quiet should also be avoided as it interrupted the experience in a similar manner. Even though the ambiance was misinterpreted and described as eerie in one case suggesting that it was not well-produced enough, it was in general highly appreciated and contributed to the atmosphere, even evoking more emotions than the narrative for some participants.
Spending more time on designing the ambiance is presumably a convenient way to immerse the user even deeper into the experience.
Te interaction with the miniature and what parts of the design of both story and miniature that were appreciated also seemed to depend on personal interests. A1, A2, A3 and B4 all have a background in engineering and were interested in the afordances and constraints of the system. Tey tried to work out what was possible to do, like checking if the sound was the same in an entire room by for example moving closer to the TV in order to hear it louder. B5, who enjoys listening to and making music expressed that he had rather just stayed in the music room the whole time,
while B6 who has an interest in drama and writing was very engaged in the narrative and story taking place.
Future work
Tere is a lot of potential for developing the system.
Tere are numerous ways of improving the prototype by spending more time on sound production, refning the algorithm, building the miniature and writing the script.
As most participants stated that it resembles some kind of game there should be insights in game theory that could be of interest. Te game resemblance also suggests that it would have been interesting to have had children participating in the experiment.
Considering imagination skills of children, perhaps their opinions on what is important to tell a enjoyable story in this manner would have been completely diferent providing insights about how to design for diferent target groups.
Regarding the design of the auditory display, since it could be concluded that participants tended to stick to the nearest part of the miniature, it should be investigated whether there is a diference in how a physical model is navigated if parts of the narrative take place in the farther parts of it. It would also be interesting to see if separating the sounds in one room would create a more realistic experience. For example, designing the sound of the TV show to be heard louder if moving the marker closer to the physical representation of the TV.
Using motion capture-technique was appropriate as it has very high precision, something that was needed when working on a small scale. However, most subjects reported that holding the marker was physically tiring.
Other technology, eg. QR-codes, RFIDs or even eye tracking, should be investigated in the future, especially if wanting to create a more elaborate experience or aiming to make it portable.
Method discussion
Recording the interactions and interviewing the test subjects was a suitable method for analysis. Te video both served as something to discuss with the participant during the interview and a tool when analysing the SEI usage and comparing the interaction with the miniature to how the experience was perceived.
Although using the SEI was benefcial to this study, they did really serve their purpose as memory markers, it also interrupted the continuity of the experience meaning it reduced the sense of immersion.
Participants said that they sometimes forgot to listen to the story because they were busy trying to decide what fgure to use or, on the contrary, that they just picked any instrument without reasoning before which was counterproductive in the evaluation. If they had been given the opportunity to interact with the objects beforehand it might have given clearer results.
CONCLUSION
In this study I have looked at what factors should be taken into consideration when designing a novel audio- physical system for interactive storytelling. Tis has been done through a research through design process and user testing of the subsequent prototype.
Results indicate that the story delivered via audio structures the experience, while the miniature with its visual elements supports curiosity and amplifes and puts the narrative in a context if designed appropriately. Te miniature should contain physical representations of all objects mentioned in the narrative, otherwise risking decreased believability.
However, other details should be added with caution, even if they ft aesthetically, as it is an information- heavy system where it can be difcult to navigate what is of essence and what is noise. Another reason to not add too many objects is to encourage imagination and creativity which can result in a more personalised experience.
Writing a narrative that revolved around a murder and robbery proved successful. As was gathered from the literature study, it has a climax that stands out which was convenient in the large stream of information that the system ofered. Only a limited number of characters should be introduced, especially as no physical representations of them exist. Te characters should have distinctive personalities and their conversations should be carefully scripted to be in line with their diferent traits.
ACKNOWLEDGMENTS
I would like to thank my supervisor Jarmo Laaksolathi for his support, guidance and endless positivity throughout this process. Also thanks to everyone who participated as voice actors or in any other way when creating the system. Last but not least, I wish to thank Ariel Blomkvist Rova for being an invaluable collaborator on this project.
REFERENCES
[1] Aristotle 2017.Te Poetics of Aristotle. CreateSpace Independent Publishing Platform.
[2] Benford, S. et al. From Interaction to Trajectories:
Designing Coherent Journeys Trough User Experiences. 11.
[3] Benford, S. and Giannachi, G. 2008. Temporal trajectories in shared interactive narratives. (2008), 73–82.
[4] Cazeaux, C. 2005. Phenomenology and Radio Drama. Te British Journal of Aesthetics. 45, 2 (Jan. 2005), 157–174.
DOI:htps://doi.org/10.1093/aesthj/ayi018.
[5] Dow, S. et al. 2005. Exploring spatial narratives and mixed reality experiences in Oakland Cemetery. (2005), 51–60.
[6] E. Worth, S. 2008. Storytelling and Narrative Knowing: An Examination of the Epistemic Benefts of Well-Told Stories.
Te Journal of Aesthetic Education. 42, (Sep. 2008), 42–56.
DOI:htps://doi.org/10.1353/jae.0.0014.
[7] Eckel, G. 2001. Immersive audio-augmented environments: the LISTEN project. Proceedings Fifh International Conference on Information Visualisation (Jul.
2001), 571–573.
[8] Fosh, L. et al. 2013. see me, feel me, touch me, hear me:
trajectories and interpretation in a sculpture garden. (2013), 149–158.
[9] Hansen, F.A. et al. 2008. Mobile Urban Drama – Seting the Stage with Location Based Technologies. Springer Berlin Heidelberg.
[10] Höök, K. 2006. Designing familiar open surfaces.
Proceedings of the 4th Nordic conference on Human-computer interaction changing roles - NordiCHI ’06 (Oslo, Norway, 2006), 242–251.
[11] Huwiler, E. 2016. A Narratology of Audio Art: Telling Stories by Sound. AudionarratologyInterfaces of Sound and Narrative. De Gruyter.
[12] Isbister, K. et al. 2006. Te sensual evaluation instrument:
developing an afective evaluation tool. (2006), 1163–1172.
[13] M. Drewes, T. and Mynat, E. 2000. Sleuth: An Audio Experience. (Jul. 2000).
[14] Madej, K. 2008. “Traditional Narrative Structure” – not traditional so why the norm? (Aug. 2008).
[15] Marshall, M.T. et al. 2016. Audio-based narratives for the trenches of World War I: Intertwining stories, places and interaction for an evocative experience. International Journal of Human - Computer Studies. 85, C (2016), 27–39.
DOI:htps://doi.org/10.1016/j.ijhcs.2015.08.001.
[16] Mateas, M. and Sengers, P. Narrative Intelligence. 10.
[17] Murray, J.H. 1997. Hamlet on the holodeck: the future of narrative in cyberspace. MIT Press.
[18] Reid, J. et al. 2004. Riot! 1831: Te design of a location based audio drama. (Jan. 2004).
[19] Rossito, C. et al. 2016. Interweaving place and story in a location-based audio drama. Personal and Ubiquitous Computing. 2 0 , 2 ( 2 0 1 6 ) , 2 4 5 – 2 6 0 . DOI:htps://doi.org/10.1007/s00779-016-0908-x.
[20] Roth, C. and Koenitz, H. 2016. Evaluating the User Experience of Interactive Digital Narrative. Proceedings of the 1st International Workshop on Multimedia Alternate Realities - AltMM ’16 (Amsterdam, Te Netherlands, 2016), 31–36.
[21] Roth, P.C.H. 2015. Experiencing interactive storytelling.
[22] Stokes, D. and Biggs, S. 2014. Te Dominance of the Visual. Perception and its Modalities. D. Stokes et al., eds.
Oxford University Press.
[23] Vazquez-Alvarez, Y. et al. 2016. Designing Interactions with Multilevel Auditory Displays in Mobile Audio- Augmented Reality. ACM Transactions on Computer-Human Interaction (TOCHI). 2 3 , 1 ( 2 0 1 6 ) , 1 – 3 0 . DOI:htps://doi.org/10.1145/2829944 .
[24] Zimmerman, J. et al. 2010. An analysis and critique of Research through Design: towards a formalization of a research approach. (2010), 310–319.
[25] Zimmerman, J. and Forlizzi, J. 2014. Research Trough Design in HCI. Ways of Knowing in HCI. J.S. Olson and W.A.
Kellogg, eds. Springer New York. 167–189.
[26] Zimmermann, A. and Lorenz, A. 2008. LISTEN: a user- adaptive audio-augmented museum guide. User Modeling and User-Adapted Interaction. 1 8 , 5 ( 2 0 0 8 ) , 3 8 9 – 4 1 6 . DOI:htps://doi.org/10.1007/s11257-008-9049-x.
www.kth.se
TRITA-EECS-EX-2019:856
