INOM
EXAMENSARBETE TEKNIK,
GRUNDNIVÅ, 15 HP ,
STOCKHOLM SVERIGE 2020
Investigating the Perceived
Quality of Image-Based
Impostors
OSCAR ALMQVIST
ERIC VICKSTRÖM
KTH
Bachelor in Computer Science Date: June 8, 2020
Supervisor: Christopher Peters Examiner: Pawel Herman
School of Electrical Engineering and Computer Science Swedish title: En undersökning av den upplevda kvalitén av bildbaserade impostors
iii
Abstract
The rendering process of a large number of 3D objects is computationally expensive and to achieve a fluent experience for the observer, optimization techniques need to be applied. One such technique for rendering 3D objects is with image-based impostors, which involves replacing a 3D object with a 2D image. This thesis investigates the number of degrees the observer can move in relation to the impostor before updating its representation to still appear as a 3D object. Specifically, the aim of this thesis is to explore what update-rate an impostor of a virtual human should have to be perceived as smooth, and if the distance between the user and impostor has any effect on this update-rate. A user study was conducted with an experiment outline based on previous work by Hamill et al. [1], with the objective to answer if a certain update-rate was considered smooth or jerky. Results display that an update every ~5.9° was considered smooth at a close distance, while an update-rate of ~6.5° was considered smooth when the impostor was far away from the observer. A one-way ANOVA test on the two different distances indicated that the distance did not significantly influence the degree that was considered smooth. This implies that impostors of virtual humans should be used with an update-rate of approximately 6°, regardless of the distance between the observer and the impostor.
iv
Sammanfattning
Renderingsprocessen av en stor mängd 3D-objekt är beräkningsmässigt dyrt och för att uppnå en behaglig upplevelse för iakttagaren, måste optimerings-tekniker appliceras. En av dessa optimerings-tekniker för att effektivt rendera 3D-objekt är med hjälp av bildbaserade impostors, som består av att byta ut ett 3D-objekt till en 2D-bild. Detta arbete undersöker antalet grader som iakttagaren kan röra sig i relation till impostorn innan dess representation behövs uppdateras för att framstå som ett 3D-objekt. Mer specifikt är målet att utforska vilken uppdateringsfrekvens på en impostor av en virtuell människa ska ha för att uppfattas som slät, och om distansen mellan iakttagaren och impostorn har någon betydelse på denna uppdateringsfrekvens. En användarstudie utfördes med en undersökningsmetod baserad på ett tidigare arbete av Hamill m. fl. [1], med målet att besvara om en viss uppdateringsfrekvens ansågs slät eller ryckig. Resultaten visar att en uppdateringsfrekvens på ~5.9° ansågs slät när impostorn var placerad nära iakttagaren, medan en uppdateringsfrekvens på ~6.5° ansågs slät långt ifrån iakttagaren. Ett envägs-ANOVA test mellan de två distanserna indikerade att det inte var en signifikant skillnad mellan vilken uppdateringsfrekvens som ansågs slät. Detta implicerar att virtuella männi-skor bör använda en uppdateringsfrekvens på cirka 6°, oberoende av distansen mellan iakttagaren och impostorn.
Contents
1 Introduction 1
1.1 Aim and Research Questions . . . 2
1.2 Approach . . . 3 1.3 Scope . . . 4 1.4 Outline . . . 4 2 Background 5 2.1 Level-of-Detail Techniques . . . 5 2.2 Impostors . . . 6 2.3 Psychophysical Experimentation . . . 7 2.3.1 Staircase Method . . . 7 2.3.2 Psychometric Functions . . . 8 2.4 Impostor Perception . . . 8 2.5 Web-Based Experiments . . . 9 3 Methods 11 3.1 User Study . . . 11 3.2 Experiment . . . 11 3.2.1 Outline . . . 11 3.2.2 Implementation . . . 14 3.3 Pilot Study . . . 16 4 Evaluation 17 4.1 Experiment Results . . . 17
4.1.1 First Research Question . . . 17
4.1.2 Second Research Question . . . 20
4.2 Experiment Feedback . . . 21
4.3 Discussion . . . 21
4.3.1 Limitations . . . 23
vi CONTENTS
4.3.2 Technical Imperfections . . . 24 4.3.3 User Study Feedback . . . 24 4.3.4 Future Work . . . 25
5 Conclusion 26
Chapter 1
Introduction
One of the main obstacles when displaying a large amount of complex (some-times animated) 3D objects in real-time, is the computational power it takes to render them. For example, to achieve a realistic model of a human, a polygonal mesh of high complexity is needed to represent it. If the mesh is simplified there might occur apparent anomalies that are visible for the observer and dis-turb the realism of the human [2]. An example of a simple polygonal mesh can be seen in figure 1.1, which represents a dolphin. Therefore, the trade-off between a more realistic 3D model and performance becomes an issue.
Figure 1.1: An example of a model displayed with a polyg-onal mesh.
A method to increase the performance when rendering 3D objects in a scene is to make use of impostors. Impostors are a simplified ver-sion of the original polygon-based object, and it can either be dynamically generated or pre-generated. The simplified version could be a reduction in the number of polygons rendered, or by replacing it by a simple image as seen in figure 1.2. Being generated dynamically means that they are generated during run-time in the application, while pre-generated impos-tors are created before the application runs and then loaded into memory. [3]
2 CHAPTER 1. INTRODUCTION
Figure 1.2: An example of a tree displayed both as a 3D model (left) and as a image-based impostor (right).
Examples of using impostors in the industry can be seen in Halperin, Santiago, and Bezrati [4], where they described how they created their own version of an impostor system in the game Marvel’s Spider-Man. In this case, impostors were used to render an urban landscape; specifically for buildings further away from the player. Another game where impostors are applied is Fortnite1, which in turn saved memory for their mobile version of the game.
1.1
Aim and Research Questions
In order to gain a deeper understanding of users’ perception of impostors, our aim is to further investigate the usage of pre-generated impostors, specifically image-based impostors of virtual humans. Image-based impostors are always facing the viewer and change the current image depending on the viewing an-gle in order to appear as an actual polygon-based object. Deciding at what angle the impostor should update, also called the update-rate of the impostor, is a point of interest as it determines how often the impostors are in need of updates. As characters and objects have different placements in the scene in which they are displayed, we also believe it would be interesting to explore if the perceived quality of impostors at different distances from the observer would be statistically significant. Perceived quality, in this context, is specified
CHAPTER 1. INTRODUCTION 3
by at what update-rate the user considers the impostor to be smooth. We aim to answer this problem statement:
What is the perceived quality of image-based impostors and does distance be-tween the impostor and the observer affect this perceived quality?
As this problem statement is rather general, we express it in the two following research questions:
• What update-rate of pre-generated image-based impostors of virtual
hu-man characters do users consider smooth?
As the perceived quality is specified as the update-rate the user perceives as smooth, we aim to investigate the threshold for the update-rate. Therefore, the perceived quality decides how many images an image-based impostor is in need of and if it could be lowered at no loss of perceived quality for the observer. This research question is answered in section 4.1.1.
• Does the distance between the observer and the impostor have any
sig-nificant effect on what update-rate is considered smooth?
This research question relates back to if the distance does matter for how the user perceives the quality of the image-based impostor. If the distance has a statistically significant effect, it would suggest that the update-rate should be changed based on where the impostor is placed. Our hypothesis is that there will be a significant difference between what the users consider smooth between the two distances because the impostors will occupy a smaller portion of the screen and thus make it harder to spot details of the character. The answer to this research question can be found in section 4.1.2.
1.2
Approach
To answer the research question we will conduct a user study surveying what different update-rates for impostors that users feel are smooth at two distances: the impostor placed close to the observer and the impostor placed far away from the observer. The distances will be evaluated by using a one-way ANOVA test with a significance level of 0.05 between the update-rates of the two dis-tances. As this is a one-way ANOVA test, the null hypothesis is predefined, saying that the means of the two groups are equal to each other. The user study will consist of an interactive web page created in Unity3D2; a real-time
2
4 CHAPTER 1. INTRODUCTION
3D development platform, with which we will conduct the user study. This will be evaluated using psychophysics; a methodology of conducting exper-iments to find thresholds between sensory responses and the physical world [5]. More specifically, a dual staircase procedure will be used to figure out at what update-rate the impostors are considered smooth by the participants of the study. The experiment in the study is based on an experiment conducted by Hamill et al. [1], which investigated the update-rate of impostors of buildings by using a psychophysical experiment.
1.3
Scope
There are many factors to consider when conducting experiments in com-puter graphics such as what type of lightning, textures, animations, and more. Therefore, the main focus of this study will be examining the update-rate of im-postors and how the distance between the impostor and the camera influences the user. The model will be displayed statically without any animations.
1.4
Outline
The background chapter will put forward different techniques of rendering ob-jects at different level-of-detail and what forms of impostors there are. It will then go on to discuss psychophysics, as well as give insight on related work on impostor perception, and finally touch upon conducting web-based user studies. Experiment outline and implementation will be stated in the methods chapter, with results displayed and commented on in the evaluation chapter. Conclusively, an analysis is made in a discussion section, followed by a con-clusion.
Chapter 2
Background
This chapter will introduce the context of this study and different aspects of rendering 3D objects at different level-of-detail. Thereafter, psychophysics experimentation is outlined, followed by related work on impostor perception. Lastly, the process of conducting web-based studies will be investigated.
2.1
Level-of-Detail Techniques
In order to efficiently be able to render up to tenths of thousands of objects with different behavior and appearance, one must apply different techniques and strategies. Even though computing power in recent years has improved, the hardware can only do so much. One concept for accelerating object ren-dering is to use different level-of-detail (LoD) renren-dering techniques, which changes the LoD depending on, for instance, the distance between the object and the observer [3] [6]. Beacco, Pelechano, and Andújar [3] outlines three primary LoD techniques: Polygon-based techniques, point-based techniques, and image-based techniques. The polygon-based technique introduces modi-fications to the polygonal mesh of the agent by reducing the geometric com-plexity of the mesh by collapsing edges. The point-based is the use of points to render geometry by using a vast amount of points it is possible to render a geometric surface. The third and last technique, the image-based technique, is different compared to the previous methods. Instead of simplifications in the geometry, the agent is replaced by a different representation and tries to give the illusion that it is the same object from that certain viewing angle.
6 CHAPTER 2. BACKGROUND
2.2
Impostors
As seen in the previous section impostors can be represented in several dif-ferent ways. Beacco, Pelechano, and Andújar [3] brings up the idea of pre-generated impostors. The method creates several images of a rendered agent from several different viewing angles and for every animation frame in an ani-mation cycle. This technique can achieve high performance by replacing each rendered agent with a single polygon with proper texture according to the cur-rent animation frame and viewing angle. Figure 2.1 displays an side-by-side comparison of a 3D model and an impostor. The negative aspect of this method is the amount of memory required to store an image for each angle and frame. There are alternative methods that solve the memory issue pre-generated im-postors have. One method is to generate the impostor during run-time. A snapshot of the agent is taken and rendered over several frames on a textured quad that continuously faces the observer.
Another method for reducing the number of textures needed for impostors, especially for animations, is described by Kavan et al. [7] by using polypostors; a combination of impostors and geometric models. The polypostors in the paper were pre-generated and displayed by transforming the texture depending on the animation instead of having a single image for that specific animation key-frame.
Figure 2.1: An example of a side-by-side comparison of an impostor and a 3D model. The left image shows the models viewed from the front, while the right image displays the models from a side angle, exposing the fact the left character is not a 3D model.
CHAPTER 2. BACKGROUND 7
2.3
Psychophysical Experimentation
The type of experiment this thesis will explore is a psychophysical experiment. Psychophysics is a means for investigating the relationship between sensory behavior and physical events [5]. It provides a way to create and analyze ex-periments in a controlled manner and is built up by multiple components. As described by Kingdom [5], a psychophysical experiment is constructed by a stimulus, a task, a method, an analysis, and a measure. The stimulus is what is presented to the participants of the experiment while the task is what the participant is supposed to do. The method is how the stimuli are shown and in what order. Thereafter, an analysis is made based on the task and method, with the measure showing the result of that analysis. Experiments based on psychophysics have previously been used in a computer graphics setting, by for example O’Connor [8] to assess perceived realism with regards to a crowd simulation within an urban environment.
2.3.1
Staircase Method
One type of method frequently used in psychophysical experiments is the stair-case method. The stairstair-case method consists of systematically alternating stim-ulus over the course of the trials in an experiment to find a threshold for which no change can be detected [9]. It is implemented by increasing the intensity of the stimulus when the user accepts (or answers ’yes’, depending on the ex-periment itself) the stimulus, and then decrease the intensity when the user rejects it (or answers ’no’). It fundamentally has four parameters that must be decided before usage, which is outlined by Cornsweet [10]. Firstly, the start-ing point must be set preferably close to the threshold in order to minimize the number of trials needed. Secondly, an appropriate step-size has to be chosen to obtain a threshold without taking unreasonably many steps at the same time as it should be able to identify a somewhat exact value during convergence. Thirdly, a stopping point must be specified either by a set number of trials or after a set number of reversals. One reversal is characterized by a change of answer; switching from ’yes’ to ’no’, or vice versa. Fourth and lastly, it must be decided if the step-size should be altered during the course of the staircase. Depending on the experiment, the step-size can be set to a constant size, and in other cases, it can vary.
A drawback of the staircase method that Prins et al. [9] highlights is the case of participant training within a single staircase; the participant can discover how their answers affect the next trial and thus introducing trial-to-trial
dependen-8 CHAPTER 2. BACKGROUND
cies. To prevent this from happening one can implement multiple staircases, such as a dual staircase, that are randomly interleaved. These can also have different starting points to see if the end threshold is equal to each different staircase.
2.3.2
Psychometric Functions
To evaluate the results from a staircase procedure, the experiment results of a user can be fitted to a psychometric function in order to extrapolate addi-tional information from the results. As the psychometric function is a statisti-cal curve, it is possible to fit it against a number of different distributions [1]. As described by Prins et al. [9], a common threshold derived from this estima-tion is the Point of Subjective Equality (PSE) which is the point for which a user 50% of the time would answer ’yes’ or ’no’. The fitting of the psychomet-ric function in regards to the data is commonly done with maximum likelihood maximization [9].
2.4
Impostor Perception
Representing objects such as virtual humans or buildings as impostors and how it affects how users, can be seen in the previous work of Hamill et al. [1]. This involved using both pre-generated impostors as well as dynamic impos-tors in a number of different experiments. In one particular experiment, they aimed to examine how sensitive the users were to different update-rates in impostors of virtual buildings (libraries, museums, etc.). The buildings were displayed in a grey-scale, which meant that other colors were not used. Results showed that around 8° of rotation per update was what the users considered smooth. The psychophysical experiment was conducted using two staircases with one ascending and one descending being interleaved at random. Both of the staircases lasted for 12 reversals each, with an initial update-rate of 10°. Out of the 12 reversals, the four first reversals consisted of the step-size being halved, thereon running the remaining eight reversals. The average stimulus value from the last four reversals was considered the true value of the staircase. Important to note is that Hamill et al. [1] only considered the results of partic-ipants with both staircases converging. However, they did not define the range of the stimulus level between the two staircases were considered converging, but the stimuli were limited between an update rate of 0°and 20°. They also concluded, via a one-way ANOVA test, that the different building models had no significant effect on the results. As there were few studies examining
specif-CHAPTER 2. BACKGROUND 9
ically the update-rate of image-based impostors, the study by Hamill et al. [1] will be the basis for which the study will be created on.
Flyvholm Iversen [11] investigated how the effects of fire and smoke could be made less computationally heavy by introducing impostors and limiting their viewing angles. An experiment was conducted with two tests of image-based impostors, with regards to two different fire and smoke compositions; smoke rising upwards in a cone-like manner, and one with a wind blowing the smoke sideways. Each of these tests had three versions: the first updated the impostor at all degrees, the second at every two degrees, and the third at every five degrees. The results displayed that the characterization of the smoke had a great influence on if a difference between the updates was noticed by the participants.
2.5
Web-Based Experiments
As this thesis will be based on a user study conducted online, it is relevant to explore what similarities and differences there are when conducting online user studies compared to studies in person. There exist multiple positive at-tributes of online studies, as it (in certain cases) makes the study anonymous and prevents what is called the social desirability response bias; a bias that can occur due to the participants trying to tell the author of the study what they want to hear. An online user study, compared to traditional lab studies, can also make it easier to achieve a larger sample size. However, there exist limitations in conducting a user study online. In particular, participants’ at-tention will often fade faster than in person, and the available time for each participant is therefore lower. Another phenomenon to be aware of is "mental cheaters", participants who are fulfilling the study but with the goal of com-pleting the study as fast as possible and not reflect on their choices. One way counteract this is by using "speed traps", a certain question or action that con-firms that they are not speeding through the study. A technique for helping the participant and giving them a sense of how far into the study they are, is to use progress indicators. One example of this is a progress bar, which also gives the participant an indicator for when the study is complete. [12]
Conducting online user studies to collect perceptual and behavioral data is discussed in Semmelmann and Weigelt [13], where they replicated multiple psychophysical experiments focusing on reaction times, but in the following three settings: "lab", "web-in-lab", "web". This was in order to examine poten-tial differences and similarities between changes in the hardware used as well
10 CHAPTER 2. BACKGROUND
as the environment. Their results point to online psychophysics being a poten-tial substitute for acquiring data instead of being conducted in a lab. Reimers and Stewart [14] investigated the accuracy of response timings when a stimu-lus was presented, on a multitude of different hardware, tried in a web-based setting (both Flash and HTML5). Their findings suggest that the variability of different hardware along with within-system noise did not have a significant effect on the overall variance. Similar to what Albert [12] mentioned about the decrease of attention span, Reimers and Stewart [14] argues that sources such as misunderstanding instructions and distractions could pose larger variance in a web-based setting compared to a lab setting.
Chapter 3
Methods
This chapter will consist of a description of the user study as well as an outline of the psychophysical experiment and its parameters. Thereafter, a summary of the implementation is described as well as the modifications prompted by our pilot study.
3.1
User Study
To answer our research question a user study was conducted. Due to restric-tions (see section 4.3.1) and that the research aims to be quantitative, the pro-cess of conducting the psychophysical experiment was automated. Each par-ticipant received a link to a website to conduct the experiment. The website consisted of an implementation of the experiment outline described further in the upcoming sections. After completion, data consisting of the staircase pro-cedure was remotely sent to us, which made it possible for us to fit the data to a psychometric function and evaluate it. The participants were also asked to fill out a questionnaire regarding their thoughts about the experiment.
3.2
Experiment
3.2.1
Outline
To find an appropriate update-rate threshold that is considered smooth, the evaluation method conducted by Hamill et al. [1] was performed. Each par-ticipant was presented with a human impostor at the center of their screen.
12 CHAPTER 3. METHODS
The imposter was facing the observer, and alternating between images from different angles; simulating a complete rotation around the original 3D object. The impostor completed a full rotation every 4 seconds with an update-rate be-tween 0.5° and 20° degrees. The update-rate was decided by a dual staircase: one ascending staircase with an initial update-rate of 0.5° and one descend-ing staircase startdescend-ing at 20°. Figure 3.1 shows a visualization of how different update-rates affect the rotation. To advance one of the staircases, the partici-pant had the option to decide if they perceived the rotation as either "Smooth" or "Jerky". If "Smooth" was chosen, the update-rate would increase with the current step-size and therefore increase the number of degrees of rotation be-fore updating the image of the impostor. The opposite would happen if "Jerky" was chosen. After each decision, the staircases were randomly interleaved. Each staircase initial step-size was 10° and was decreased by a factor of 2 for the first 4 reversals. Subsequently, the final step-size was 0.625° and after 12 reversals had been performed the staircase was considered completed. The end result for a staircase was the true value; the average value of the four last reversals. If the true value between the ascending and descending staircase was similar, they were considered converged. Again, this was configured ex-actly as Hamill et al. [1] configured their staircase procedures. Resembling Hamill et al. [1], cases where the ascending and descending staircase did not converge were excluded from the results. Regarding what is meant by conver-gence, as Hamill et al. [1] did not define the range of what they considered two staircases to converged, we decided that the two staircases would be con-sidered converged if they were between two times the smallest step-size from each other (1.25°).
CHAPTER 3. METHODS 13
Figure 3.1: A visualization of two different update rates. In the upper row, the impostor has access to more images at each angle between the impostor and the observer. If we reduce the available images as shown in the lower row, the transition from the left-most to the right-most image would be perceived as rather jerky.
The psychometric function used for evaluating the results from the staircase procedures was equation 3.1, which is identical to the Psychometric Func-tion used by Hamill et al. [1]. The parameters that are to be estimated in this psychometric function is α and β, which corresponds to the stimulus at the halfway point and the curve steepness, respectively [1]. γ is equivalent to the probability of choosing the correct stimuli, and as this task only displays one stimulus at the time, this leads to γ = 1.0 [1] [9]. As explained in section 2.3.2, this was then used for estimating the PSE for each converging partici-pant. In this case, it is equal to the update-rate each participant perceives as smooth.
P (x) = 1 − (γ × ( 1
1 + (xα)−β)) (3.1)
In order to answer our second research question regarding the distance to the observer and if it played a role in the accepted update-rate, we performed two
14 CHAPTER 3. METHODS
iterations of the dual staircase procedure mentioned previously. Each itera-tion had a different distance between the observer and the impostor; the first iteration had the impostor placed close to the observer, while the second one was further away. A one-way ANOVA test was then conducted with regards to the PSE from each participant, with the two groups corresponding to the two different distances, on a significance level of α = 0.05. This resulted in the following hypotheses:
Hypothesis H0: Both of the two different distances have the same mean PSE. Hypothesis Ha: The two distances have different mean PSE.
3.2.2
Implementation
For us to be able to perform the experiment according to the experiment out-line, an interactive 3D environment was required. We decided to go with Unity3D (version 2019.3.3f1), a game development environment for real-time 3D rendering. However, the main reason was that the finished product could be exported to WebGL, a way to run Unity content in web browsers 1. The human 3D object model, and later impostor, was downloaded from the Unity Asset Store2. To create the impostor, 720 images were pre-generated, which consisted of different evenly spaced viewing angles horizontally around the 3D object. Each image had a resolution of 720 x 720 pixels, however, the im-postor was centered in the middle covering a smaller portion of the image. In Unity coordinates, the main camera of the scene was positioned (0, 1, −10) . For the closer distance the impostor was placed at (0, 1, −4) and the distance further away (0, 1, 6). For the perspective and the final size of the impostors, see reference figures 3.2 and 3.3, which display the accurate proportions of what the observer saw during the experiment.
1
Unity3D documentation for WebGL: https://docs.unity3d.com/Manual/webgl-gettingstarted.html
2
The model used in this thesis can be found on the following URL:
CHAPTER 3. METHODS 15
Figure 3.2: The impostor being placed close to the observer. Depending on if they perceived the update-rate as "Smooth" or "Jerky", they would press the corresponding button displayed at the bottom of the screen.
Figure 3.3: Identical to figure 3.2, with the impostor placed far away from the observer.
The web page consisting of the experiment was split into different parts. The initial part presented each participant with a set of instructions telling them to label the movements either "Smooth" or "Jerky", depending on how they per-ceived it. This part also consisted of an example of a jerky movement with an update-rate of 20° and a smooth movement of 0.5° in order for the users to see what the smoothest and jerkiest movements looked like. The second part was a survey for the user to fill in information about themselves and their agreement that the data would be used in this thesis. If the participant agreed with the
16 CHAPTER 3. METHODS
terms, they were presented with the third one, which was the experiment itself following the outline presented in the previous section. A screenshot from this view can be seen in figure 3.2 and figure 3.3. The fourth and final part directed the user to a questionnaire regarding their thoughts about the experiment and if there were any graphical bugs occurring during the course of the experiment. This is important as we could catch potential errors from the experiment, as it is done without direct supervision from us.
3.3
Pilot Study
In the early stages of the project, an initial version of the study was conducted with the authors and two more participants in order to detect graphical imper-fections and other aspects that might affect the observer in a way that it was not meant to. In the beginning, the experiment consisted of a total of three dis-tances instead of the final two. The main reason for removing a distance was due to the duration of the experiment; the participant would have to go through a dual staircase procedure at three different distances, which potentially could make the participant end the experiment before completion.
Another aspect was adding a time buffer for each new trial in order to prevent users from accidentally clicking a button twice, which could unintentionally skew the results. This also prompted the participants to hesitate over their next choice and think through their decision. We used this as our form of speed trapping to also prevent mental cheaters.
Chapter 4
Evaluation
This chapter will display the findings from the user study, and evaluate them with regards to each research question. Thereafter, the feedback from our par-ticipants regarding the experiment will be presented, followed by a discussion that sets our results in a broader context.
4.1
Experiment Results
This section is split up into two parts, relating to the two different research questions.
4.1.1
First Research Question
What update-rate of pre-generated image-based impostors of virtual human characters do users consider smooth?
18 CHAPTER 4. EVALUATION
Figure 4.1: The mean PSE for the two distances, with the error bars repre-senting the standard deviation of each distance. The update-rate for the close distance is 5.857°±2.513° and the far distance 6.501°±2.828°.
A total of 16 participants took part in the experiment, with 14 participants showing results with good convergence; the staircases had at most 1.25° dif-ference between the true values. In figure 4.1 the mean PSE1can be seen for both of the two distances, and the update-rate which users considered smooth were ~5.9° when the impostor was close, and ~6.5° when the impostor was far away. As can be seen in figure 4.1 the resulting PSE values are consider-ably spread, which means that there was a large difference between what the participants considered smooth.
1
CHAPTER 4. EVALUATION 19
Figure 4.2: A completed dual staircase procedure, with one ascending and one descending staircase with two different starting points. For this participant the converged update-rate was approximately 6°.
Each participant completed two staircase procedures. Figure 4.2 displays a completed staircase procedure with both the ascending and descending stair-case. As shown, one of the two interleaved staircases could run for more trials than the other due to the stopping point being after 12 reversals. This is an ex-ample of convergence; the true value, the average stimulus value for the four last reversals, is within the set convergence range.
20 CHAPTER 4. EVALUATION
Figure 4.3: An example of a fitted psychometric curve, based on Equation 3.1, shown in green. The black dots represent the acceptance rate for a participant when shown a specific stimulus. The PSE for this participant is around 8°.
A psychometric curve was then fitted for each converging participant. One sample from the user study can be seen in figure 4.3. In this case, the partic-ipant never accepted an rate of 20° while always accepting an update-rate of roughly 6°. With the fitted curve, it is then possible to extract the PSE, the point where the stimulus is accepted 50% of the time. In other words, the PSE is in this case the number of degrees between updates that are required for the participants to consider it smooth enough for use. The PSE for specifically figure 4.3 is around 8°. By fitting the psychometric curve for every participant we can gather the PSE for all the distances to further see what the mean PSE is for when the impostor is both close and far away from the observer, which can be seen in figure 4.1.
4.1.2
Second Research Question
Does the distance between the observer and the impostor have any significant effect on what update-rate is considered smooth?
By investigating if the PSEs found when the impostor was close respectively far away from the observer, the statistical significance between the two dis-tances could be evaluated. A one-way ANOVA test conducted on the PSEs
CHAPTER 4. EVALUATION 21
for the two groups (close and far) resulted in an F-ratio of F (1, 24) = 0.38. The p-value resulted in p = 0.545, a value larger than the significance level of α = 0.05 specified previously. This leads to the null hypothesis H02 fail-ing to be rejected. This indicates that the distance did not statistically signif-icantly affect the update-rate at which participants considered the rotation to be smooth.
4.2
Experiment Feedback
From the questionnaire that was to be answered after participating in the ex-periment, it was found that two participants perceived that the impostor was flickering and that it may have affected their choice when deciding between the two options. Two other participants answered that they noticed a few im-perfections on the impostor such as small white pixels, however, both of them continued with that it did not affect their choice. One participant also remarked that the foot of the impostor disappeared for a short period of time.
When the participants were asked to give their additional thoughts about the experiment, one of the participants suggested that a progress bar would help to motivate the user. Six out of the total sixteen participants believed that the length of the experiment was too long. Another participant commented that it was difficult to focus on the same spot for the whole duration of the experiment.
4.3
Discussion
The results from the user study imply that an update-rate of approximately 6° for image-based impostors of virtual humans is what users perceive as smooth. Furthermore, the distance seems to not have mattered, which contradicts our initial hypothesis. Compared to the results of Hamill et al. [1], which got an update-rate from the mean PSE to be roughly ~8°, our results proved to be slightly lower with ~5.9° and ~6.5° depending on the distance to the observer. There could be many reasons for this happening, while we tried to stay true to the experiment outline performed by Hamill et al. [1] there were a few aspects in the experiment implementation that we were not able to replicate. There exist a disparity in the 3D environment the user study was conducted in, along with differences in how our user study was conducted over the Internet while
2Hypothesis H
22 CHAPTER 4. EVALUATION
theirs was in person. Another main point of difference was the model used; they used grey-scaled buildings while we used virtual humans with colors. As Flyvholm Iversen [11] pointed out in their experiment, different characteriza-tions of impostors (in their case smoke) can have a great effect on the perceived quality of impostors.
A point of uncertainty when parsing our results was to define what is meant by two staircases converging. In this paper, we defined it as when the true value3 of the two different staircases are within a stimulus value equal or within two times the size of the smallest step-size (1.25°). Hamill et al. [1] lacked an explanation in this regard, as they did not clearly define what they considered converging, more than just the true value of both the ascending and descending staircase being close to each other. As a consequence of not properly defining convergence, the results will vary by disregarding the participants that did not have converging staircases and thus potentially skewing the results. Looking at previous work from other applications of psychometrical experimentation and staircase procedures, within and outside of computer graphics and perception, it was hard to find any guidelines when defining convergence as it was mostly domain-specific. The motivation for our definition of convergence came from internal testing and evaluation, from which we believe that a change of 1.25°to the update-rate was visibly different. This could be further explored in future research.
Due to the spread of the PSEs being substantial, it is not appropriate to say that the mean result should be closely followed as a specific update-rate for a particular distance. Rather, the results should be seen as guidelines as for what update-rates that could be viable for specifically virtual human charac-ters. Regarding the result of the two different distances and the null hypothesis not being rejected, one possible explanation could be that when the character moved further back the participants became more focused and/or resized the browser window the experiment was conducted in. Another reason could sim-ply be that the distance does not matter when dealing with a single impostor that the user has its focus upon. As the impostor was the only object being displayed in the scene, the user had no other thing to focus on than the impos-tor. In reality, this is not always the case, as there are most likely other objects such as buildings, trees, or objects being displayed simultaneously.
3
CHAPTER 4. EVALUATION 23
4.3.1
Limitations
Due to COVID-19 and the increased necessity of social distancing, the user study had to be conducted over the Internet. The user received a website link to participate in the study via their browser. This led to the study becoming less controlled and introduced new possibilities for errors. Seeing that the user study aimed to find a threshold of update-rate sensitivity, it became more sensi-tive to for example screen resolution, drops in frame rate, network connection, and other aspects that may diffuse the result. We observed this with the help of our questionnaire, where two participants mentioned flickering.
Under normal circumstances, all the experiments in the study would be con-ducted on the same piece of hardware and in an environment without distrac-tions. This would ensure that all the experiments under the same conditions which would increase the reliability of the results. However, in some cases, this could be unrepresentative of the situations where impostors could be used. For example in games, where there could be a large difference in what hard-ware is used. Because of these scenarios, one could argue that it is beneficial to perform the experiment on different hardware to establish an update-rate that performs well in different settings. In our case, we conducted the exper-iment on 16 different participants, and while this is a few more than Hamill et al. [1], the number of participants does not cover all the possible varieties in hardware. If there would have been more resources, the study could have been conducted on a larger scale with even more participants.
Even though Reimers and Stewart [14] reported that hardware would not have a large influence on the overall variance we still argue that in our case it has a larger impact than in their setting. In the case for Reimers and Stewart [14] they investigated the difference in the response accuracy reaction times. This is vastly different from our study, as we investigated the visual perception of impostors of different qualities. However, we could have taken precautions to see if this was actually the case. One possible solution would have been to take a device fingerprint, and thus get data for different operating systems, browsers, monitor resolutions, etc. With this data, we could have grouped par-ticipants together by certain hardware characteristics. Still, as the number of participants was quite small, it might have been hard to draw further conclu-sions from hardware differences.
24 CHAPTER 4. EVALUATION
4.3.2
Technical Imperfections
As mentioned in the results, a number of participants experienced technical imperfections during the course of the experiment, with at least two partic-ipants saying that specifically "flickering" made them perceive it as jerkier. These were imperfections that we did not encounter on devices in our internal testing or during the pilot study, which makes it difficult to debug. A possible (but expensive) solution would be to conduct a pilot study on multiple operat-ing systems as well as browsers in order to discover these imperfections and fix them. Another option would be to conduct the experiment over a video call (and share the screen) on a device that did not encounter any bugs. However, this opens up to other possible error sources such as the quality of the video feed.
4.3.3
User Study Feedback
As multiple participants believed that the length of the experiment was too long, and one participant suggested using a progress bar, we believe a num-ber of improvements could be done in the practicality of the user study. We did consider using a progress bar for the user study, which Albert [12] recom-mended as a technique for guiding participants, but we opted not to as we did not want participants to think a certain answer was correct. This is due to the length of the experiment being somewhat dynamic; it runs for 12 reversals4. In hindsight, a dynamic progress bar could have been constructed with ran-dom elements to hinder participant training. Participant training is a common occurrence when using single staircase methods as described by Prins et al. [9], where the participants could learn how their answers influence the next stimulus intensity shown.
We implemented extra precautions for issues that we found during the pilot study, such as the time buffer as a way to combat accidental double-clicks, which seemingly worked well. As Albert [12] mentioned, the attention span could be shorter in participants during online studies as compared to studies done in a lab. Therefore, we believe that we made the appropriate decision of reducing the number of distances that were examined from three to two, as the experiment length in our pilot study grew quite large when measuring three different distances.
4
A reversal is defined as a change of answer; switching from "smooth" to "jerky" or vice versa.
CHAPTER 4. EVALUATION 25
4.3.4
Future Work
As this study only contains impostors with a subset of all possible available features, there are many paths for future work to explore. For instance, inves-tigating what type of objects could work as impostors with a lower or higher update-rate. Another factor to further look into is the model and the context of the scene; if there is a difference in how users perceive different update-rates if the model is shown in isolation or with other objects in the scene as virtual characters are seldom shown in isolation. The character could also be placed having a different angle in relation to the observer. Furthermore, ex-perimenting with other distances to see if a larger difference might affect users differently could also be interesting.
In terms of performance, there exist the need to benchmark the impostors with the established PSEs. While it is very likely for performance increases with an impostor with a high update-rate, an impostor with this level of detail might not achieve the sought out increase. Further research analyzing the perceived quality of a crowd of impostors compared to a crowd of real 3D-objects using the presented PSEs, while at the same time performing a benchmark compari-son to see if there exist a significant gain in performance. If an impostor would be used in real situations such as games or simulations, in most cases you want to cover more perspectives rather than just horizontally around the object, and have an animation frame, this would increase the required memory substan-tially. Further studies investigating if it is reasonable to try to replace a 3D object with an impostor using the presented PSEs in the context of memory usage.
Chapter 5
Conclusion
The main goal with this paper has been to establish at what update-rate of ro-tating image-based impostors of virtual human characters that observers con-sider smooth, and if the distance between the observer and the impostor has any influence. This was done by conducting a user study; specifically, a psy-chophysical experiment. Results from the user study show that the participants considered an update-rate every ~5.9° and ~6.5° smooth enough, for impos-tors being close and far away from the observer, respectively. However, from a one-way ANOVA test, the two different distances did not seem to have a statistically significant effect on the update-rate. From this research, it seems that the impostor should have the same update-rate regardless of the distance between the observer and the impostor. Compared to previous research this update-rate is similar to other results despite the fact that different objects were examined, with ours having a slightly lower update-rate.
Bibliography
[1] J. Hamill, R. McDonnell, S. Dobbyn, and C. O’Sullivan. “Perceptual evaluation of impostor representations for virtual humans and build-ings”. In: Computer Graphics Forum. Vol. 24. 3. Wiley Online Library. 2005, pp. 623–633.
[2] F. Tecchia, C. Loscos, and Y. Chrysanthou. “Visualizing crowds in real-time”. In: Computer Graphics Forum. Vol. 21. 4. Wiley Online Library. 2002, pp. 753–765.
[3] A. Beacco, N. Pelechano, and C. Andújar. “A survey of real-time crowd rendering”. In: Computer Graphics Forum. Vol. 35. 8. Wiley Online Library. 2016, pp. 32–50.
[4] X. Halperin, D. Santiago, and A. Bezrati. “Spider-Man IG-Impostors: Cityscapes and Beyond”. In: SIGGRAPH Asia 2018 Technical Briefs. SA ’18. Tokyo, Japan: Association for Computing Machinery, 2018. isbn: 9781450360623. doi: 10 . 1145 / 3283254 . 3283259. url: https://doi.org/10.1145/3283254.3283259.
[5] F. Kingdom. “Psychophysics”. eng. In: Encyclopedia of Human
Behav-ior. Second Edition. Elsevier Inc, 2012, pp. 234–239. isbn:
9780123-750006.
[6] D. Luebke, M. Reddy, J. D. Cohen, A. Varshney, B. Watson, and R. Huebner. Level of detail for 3D graphics. Morgan Kaufmann, 2003. [7] L. Kavan, S. Dobbyn, S. Collins, J. Žára, and C. O’Sullivan.
“Polypos-tors: 2D polygonal impostors for 3D crowds”. In: Proceedings of the
2008 symposium on Interactive 3D graphics and games. 2008, pp. 149–
155.
[8] S. O’Connor. “Assessing the perceived realism of agent crowd behaviour within virtual urban environments using psychophysics”. PhD thesis. Coventry University, 2016.
[9] N. Prins et al. Psychophysics: a practical introduction. Academic Press, 2016.
28 BIBLIOGRAPHY
[10] T. N. Cornsweet. “The staircase-method in psychophysics”. In: The
Amer-ican journal of psychology 75.3 (1962), pp. 485–491.
[11] M. Flyvholm Iversen. “Perceived quality of smoke effects in virtual en-vironments, using fluid systems and imposters.” MA thesis. Aalborg University, 2011.
[12] B. W. Albert. Beyond the usability lab conducting large-scale user
ex-perience studies. eng. San Francisco, Calif.: Morgan Kaufmann, 2010.
isbn: 1-282-54119-6.
[13] K. Semmelmann and S. Weigelt. “Online psychophysics: Reaction time effects in cognitive experiments”. In: Behavior Research Methods 49.4 (2017), pp. 1241–1260.
[14] S. Reimers and N. Stewart. “Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments”. In:
www.kth.se
