IN
DEGREE PROJECT TECHNOLOGY AND LEARNING,
SECOND CYCLE, 30 CREDITS STOCKHOLM SWEDEN 2017,
Overviewing and VR
information visualizations
How interacting with, and perceiving an
information visualization in VR affects our
overview of the information visualization
ANTON SIVERTSSON
Overviewing and VR information visualizations
How interacting with, and perceiving an information visualization in VR affects our overview of the information visualization
Överblick och informationsvisualiseringar i VR
Hur interaktioner med och upplevelse av en informationsvisualisering i VR påverkar vår överblick av informationsvisualiseringen
Anton Sivertsson antonsiv@kth.se
Master’s Thesis in Media Technology
Master of Science in Engineering in Media Technology School of Computer Science and Communication at
The Royal Institute of Technology
Supervisor: Björn Thuresson Examiner: Tino Weinkauf
Date: 2017-06-20
Abstract
Society generates more and more data every day, and with competent ways to visualize it, we can learn new things about the world we live in. While traditional visualizations try to stay clear of 3D graphs because they are hard for a user to process without proper depth cues, VR technology allows us to better perceive 3D structures, but what happens to our overview of the data when we perceive and interact with it in 3D?
6 participants were subject to a 3D VR visualization of customer data with filtering possibilities, where they were to perform a series of short tasks as well as a more open-ended free form task.
Qualitative data was gathered through extensive semi-structured interviews. Quantitative supportive data was gathered as well in the form of interaction logs, time to complete tasks and the Presence Questionnaire. After this first round of tests, feedback and data was compiled into a new version of the visualization that was then tested on seven new participants.
Results showed that seeing the graph from different perspectives was important to gaining an overview of the 3D graph, but also to actively interact with the data helped participants gain an overview by digging deeper into the data. In order to dig into the data in the 3D graph, participants expressed that they would’ve liked to be able to compare subsets of the data.
Sammanfattning
Vi genererar ofantliga mängder data varje dag, men utan att visualisera denna data är det inte säkert att vi lär oss någonting av den. Med detta följer även ett behov av att visualisera denna data i flera dimensioner, något som varit svårt i informationsvisualiseringar tidigare då det är svårt att representera djup på en platt skärm så att människor förstår. Med VR-teknik kan vi däremot använda detta djupseende i informationsvisualiseringar, men vad händer med vår överblick av informationen när vi ser och interagerar med den i 3D?
6 testpersoner fick interagera med en 3D-visualisering av kunddata med filtreringsmöjligheter i VR och fick i denna utföra en serie mindre uppgifter samt en större, fri uppgift. Under testet samlades kvalitativ data in i form av utförliga semi-strukturerade intervjuer och kvantitativ supportdata i form av loggar av interaktioner, tid att slutföra uppgifter samt svar från Presence Questionnaire för alla testpersoner. Åsikterna från denna testrunda användes sedan för att skapa en ny version av 3D- visualiseringen som sedan testades på 7 nya personer enligt samma studieformat.
Resultaten visade att det var viktigt att kunna se grafen från olika perspektiv för att få en överblick,
Glossary 1
1 Introduction 2
1.1 Relevant areas of research 2
1.1.1 Information Visualization 2
1.1.2 Virtual Reality 4
1.1.3 Visualization in Virtual Reality 5
1.2 Case scenario 5
1.3 Problem definition 6
1.4 Objective 6
2 Important Concepts 7
2.1 Information Visualization concepts 7
2.1.1 Overview 7
2.1.2 Overview as situation awareness 8
2.1.3 Consistency 8
2.1.4 Adaptive visualizations 8
2.1.5 Data filtering 9
2.2 Virtual Reality concepts 9
2.2.1 Depth perception in VR 9
2.2.2 Presence & Immersion 9
2.2.3 VR ergonomics 10
2.2.4 Cyber sickness 11
3 Application design 12
3.1 Environment of the virtual scene 12
3.2 Visualization form and data 12
3.3 Slider filters 13
3.4 Highlighted area 14
3.5 Product buttons 14
3.6 Numerical overview display 15
3.7 Size of graph 15
4 Evaluation method 17
4.1 Resources 17
4.2 Scientific methods 17
4.2.1 Structural analysis 17
4.2.2 Semi-structured interviews 17
4.2.3 Presence questionnaire 18
4.3.1 Test plan research questions 18
4.3.2 Participants 19
4.3.3 Study outline 20
4.3.4 Background questionnaire 21
4.3.5 Think aloud 22
4.3.6 Set of tasks 22
4.3.7 Free form task 23
4.3.8 Presence questionnaire 23
4.3.9 Semi-structured interview 23
4.4 How the study was performed 25
4.4.1 Pilot study 25
4.4.2 Test environment 25
4.4.3 Demonstration scene 25
4.4.4 Think aloud 25
4.4.5 Order of set of tasks 26
4.5 Method critique 26
4.5.1 Recorded data that was purposefully omitted 27
5 Results 28
5.1 Participant distribution 28
5.2 Task completion results 29
5.2.1 Data logs for set of tasks 29
5.3 Free form task results 31
5.3.1 Free form task times 31
5.3.2 Free form task data logs 32
5.4 Presence questionnaire results 33
5.5 Semi-structured interviews 34
5.5.1 How do users gain an overview of the data in the visualization? 34
5.5.2 How do users understand the visualization graph? 35
5.6.2 Faded product colour instead of faded grey 41
5.6.3 Reset button 42
5.6.4 Updating the numerical overview display 42
5.6.5 Updating the frame of the graph 43
5.6.6 Changing orientation of the time as customer slider 43
6 Discussion 44
6.1 Discussion of results 44
6.1.1 How do users gain an overview of the data in the visualization? 44
6.1.2 Graph design 45
6.1.3 Suitability of visualization with respects to case 45
6.1.4 To what extent are users aware of the state of the visualization at a certain point in time? 46
6.2 Future work 47
6.3 Sustainability and ethics 47
7 Conclusion 48
8 Bibliography 49
9 Appendices 51
Appendix 1 - Additional tables 51
Appendix 2 - Disclaimer on collaborative work 54
Glossary
VR - Virtual Reality VS - Virtual Scene
HMD - Head-Mounted-Display
Glyph - Visual representation of an object with some properties (colour, size, position, etcetera) Visualization - The representation of an object, situation, or set of information as a chart or other image
Infographic - data represented as an image or graph, subcategory of Visualization
Information Visualization - a visualization based on data processed by a computer into some sort of a visualization
1 Introduction
Our world is generating larger and larger amounts of data. It was estimated by the Economist that in 2010 alone, we produced 1,200 exabytes of information. That is 1,200,000,000,000 gigabytes of information (The Economist, 2010); data that with the help of competent visualization methods may help us predict behaviours and find new patterns in the world and societies we live in, but with such an exorbitant amount of data, the need for new and more capable visualization methods is ever growing.
A lot of the data generated today is high-dimensional, but most information visualizations we use are made for the two-dimensional, flat screen medium, which has some limitations in representation. The main benefit of using Virtual Reality (VR) for information visualizations is utilizing the third dimension that VR allows us to better perceive structures because of stereoscopic vision. It also gives us more real estate to present visualizations on. These benefits can help us realize new visualization methods that is more ready to cope with the ever-growing amount of data.
In interactive information visualizations, it is often relevant for the user to be able to explore the data in order to uncover new knowledge. This is often done by interacting with the visualization in order to filter the data being represented, for example by restricting which parameters are being examined or setting an interval for a certain parameter. When we filter the data in an interactive information visualization we also see it change, and if we’re dealing with a complex, high-dimensional visualization it may be easy to lose our overview of what the visualization is showing. So, how can we construct high-dimensional information visualizations in VR in a way that allows us to preserve an overview of the data even as it is being filtered?
Let it be stated that this is first and foremostly a thesis in the area of information visualization, but with a VR approach.
1.1 Relevant areas of research
1.1.1 Information Visualization
“Visualization provides an interface between two powerful information processing systems—the human mind and the modern computer. Visualization is the process of transforming data, information,
and knowledge into visual form making use of humans’ natural visual capabilities. With effective visual interfaces we can interact with large volumes of data rapidly and effectively to discover hidden
characteristics, patterns, and trends. In our increasingly information-rich society, research and development in visualization has fundamentally changed the way we present and understand large complex data sets. The widespread and fundamental impact of visualization has led to new insights
and more efficient decision making.”
- Gershon, Eick and Card, 1998
The quote above is a suitable introduction to visualization and why it matters. Visualization as a term has two different meanings according to Oxford Dictionaries, the first being “The formation of a mental image of something”, but has in recent times also taken on the second definition “The representation of an object, situation, or set of information as a chart or other image.” (Oxford Dictionaries, no date). Both definitions are intertwined, even though they are very different, because visualizations as a tool are based on the fact that our human visual resources are so great that they allow us to quickly process massive amounts of data, if presented in the right ways. We acquire more information from our visual sense than from all other senses combined (Ware, 2013). When a visualization is referred to in this degree project, it is the second definition that we refer to, visualization as a visual representation of something rather than a cognitive process.
The modern field of Information Visualization as research has been around since 1987 when Scientific Computing published a special issue of Computer Graphics on Visualization, but the usage of infographics (a term for data represented as an image or graph and therefore a subcategory to the second definition of a visualization) can be traced back to 1769 when John Priestly presented his revolutionary infographic on the human history (Bailey and Pregill, 2014). Other notable pioneers in this area include William Playfair, Florence Nightingale and John Snow. For this degree project, we will define information visualization as a visualization based on data processed by a computer into some sort of a visualization.
According to Colin Ware, information visualizations improves our cognitive systems because we both have the human visual system, which has a “flexible pattern finder coupled with an adaptive decision- making mechanism,” together with the computing power and resources of a computer system with access to the world wide web (Ware, 2013). The process of visualization can be illustrated as in Figure 1. The main takeaway is that unprocessed data is gathered, transformed and represented, and that we as humans interact with all these steps to increase the knowledge gathered from the data. The cloud representing the social environment affects how we gather and interpret the raw data that is coming from our physically perceived environment.
The question in this process that information visualization is concerned with is how to best present the data in order for people to understand it.
Figure 1 – The visualization process adapted from Colin Ware (Ware, 2013)
Presentation in information visualization is one of the most important components. Encoding the data into visualization saves us a tremendous amount of time in gathering knowledge from the data, but since the representation can be done in many different ways, one of the most important questions anyone making a visualization should ask themselves is “What do I want the user to do with this data?” (Cairo, 2013). Data is presented in a way that facilitates one or more ways of interpreting the data. Different forms of presentation lead the user to interpret the data differently, which leads some conclusions from the data to be hidden and some to be more prominent. So, the representation of the data is highly dependent on the social environment.
1.1.2 Virtual Reality
Virtual Reality, or VR for short, uses technology to replace as many physical sensations as possible with technically induced ones. VR today predominantly focuses on changing our visual perception by using a head-mounted-display (or HMD) with one screen for each eye. The HMD is tracked in space, which allows the technology to change the images of a virtual scene in correspondence with the movements of the HMD.
Current high-end VR systems available for the consumer market are the HTC VIVE and the Oculus Rift. This study will focus on the HTC VIVE system, but as of this writing, the two are almost interchangeable with both having similar specifications in terms of resolution and tracking, as well as both now having motion tracked hand controllers.
Important technological factors of VR equipment are that they have accurate tracking of the HMD, good enough refresh rate (usually 90 frames per second as of this writing), as well as high enough resolution to increase immersion (the HTC VIVE and Oculus Rift both use a 1080x1200 display per eye).
1.1.3 Visualization in Virtual Reality
Virtual Reality today is mostly focused on the entertainment businesses, but has been used for scientific visualizations in such varied domains as paleontology, physics, medicine and chemistry (Zhang et al., 2001; Koutek, 2003; Demiralp et al., 2006; Chen et al., 2012; Ragan et al., 2013; Laha, Bowman and Socha, 2014).
Information visualizations have for the most part been developed for a flat medium, be it paper or screen, but our brains and our eyes are used to and allow us to perceive 3D structures. We are used to perceiving things in our world as spatial objects.
Information Visualization usually attempts to put all relevant data on one screen or one piece of paper.
This usually leads to compromises since we are used to dealing with screens or papers of a certain size. VR, in comparison, serves us an entire surrounding at our leisure, which allows us to put more data into the environment and removes the limitations of trying to condense information visualizations to a certain screen size. Our information space is no longer limited to flat screens, but to our entire surrounding in the form of a virtual scene.
The main benefits we get from VR information visualization today are
• Increased information space
• Better comprehension of 3D structures (because of stereoscopic vision)
• Spatial interaction through motion tracked controllers
1.2 Case scenario
I am doing this degree project in tandem with the web bureau Adaptive Media AB for one of their clients, Ownit Broadband. Ownit Broadband is an internet service provider (ISP) and provide their customers with high-speed internet connections as well as television packages, routers and telephony to both personal customers, housing cooperatives and businesses. They will be referred to as the principal henceforth.
The principal is interested in finding new sales hypotheses by examining different customer segments in order to send them customized sales campaigns. This is mainly in the interest of the principal’s sales
The data, which is owned by the company, will be made accessible during the study for use in the prototype. This data contains anonymized customer data: where they live, what products they own, when they became customers, how happy they are with their service, and more.
How happy a customer is quantified as an NPS score (Net Promoter Score), in which the customer gives a score from 0 to 10 on the question “How likely is it that you would recommend our company/product/service to a friend or colleague?”
1.3 Problem definition
How to design a 3D VR visualization of customer data in order to preserve overview of data while dynamically filtering it?
In this degree project, I will examine how a 3D information visualization in VR can be constructed to preserve a user’s overview of the properties of the data as it is being dynamically filtered.
The user in this case is a sales manager, meaning someone who would want to use information visualization in order to discover new sales potential by analysing customer data. The information visualization in question will be a multi-graph visualization showing the features of a customer segment in comparison to the features of all customers.
The filters that dynamically change the data will in this case be:
• Customers by age (interval in years from 20 to 93)
• Customers by a set of broadband products they own (100 mbps, 250 mbps or 1000 mbps)
• Customers by NPS score (interval between 0 and 10)
• Customers by how long they’ve been a customer (interval from 4 months ago to 10 years ago)
These filters decide which customer elements are highlighted and which are faded in the graph.
1.4 Objective
The objective of this degree project is to investigate how to create 3D information visualizations in VR that help users preserve an overview of the data as it is being dynamically filtered.
2 Important Concepts
2.1 Information Visualization concepts
2.1.1 Overview
There exists two kinds of definitions for “overview” within information visualization, one refers to the cognitive task of gaining an overview, the other refers to a user-interface component in an information visualization that gathers a collection of objects of interest for the visualization; which is therefore separated from the visualization in itself (Hornbæk and Hertzum, 2011). In this work, we are interested in the cognitive task of gaining an overview, which is important when faced with high- density information visualizations, so overview from here on will refer to this cognitive task. An overview as a user-interface component can however be used as an aid to a user in gaining and preserving an overview.
Edward Tufte’s ideology for creating information visualizations is that they should be of high-density with a high level of complexity, avoid what he calls Chartjunk (unnecessary filling that is not data for the visualization) and have three levels of depth available (Tufte, 2008):
1. What is seen from a distance - in effect what is gathered by overviewing the visualization 2. What is seen up close - the finer details of the data are shown when examining the
visualization closer
3. What is implicitly seen - the cognitive process that allows us to perceive properties not explicitly show in the visualization
Tufte argues that the viewer should be in control of what the data means, by supplying them with as much data as possible and allow the viewer to selectively form their own conclusions from it. Given this, an information visualization should be able to allow the user to perceive properties from it without having to view the finer details of the data.
Hornbæk et al. writes on overview in information visualization that “[...] overview is an awareness of an aspect of an information space acquired by pre-attentive cues, information reception, or active creation” (Hornbæk and Hertzum, 2011). This definition is based not only on Tufte’s reasoning on overview, but from analysing a large body of work on information visualizations and overview. In order to understand this definition, it is important to define pre-attentive cues, information reception and active creation and how they relate to the acquisition of an overview.
Pre-attentive cues in a visualization are usually groupings, colour coding, density and similar visual
The process of moving around and actively surveying the data in 3D as is possible with VR is best defined as a form of active creation, since it requires an effort. However, in the case that a user would move about without the express intent of actively surveying the visualization, it would fit in more with a form of pre-attentive cues.
2.1.2 Overview as situation awareness
Mica Endsley defines situation awareness as “the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning and the projection of their status in the near future” (Endsley, 1988), which ties well in with the previous definition of overview as a form of awareness in 2.1.1 Overview.
What this means is that overview, as it is a form of awareness, is not dictated simply by what we perceive immediately, but also by our understanding of the meaning of the data and what it may point to in the future.
Situation awareness can be evaluated, for example using the SAGAT (Situation Awareness Global Assessment Technique) which is a high validity test that quantifies how much situation awareness a user experiences in a given situation, with regards to the three levels of awareness in Endsley’s definition: immediate perception of elements, comprehension of their meaning and a projection of their status in the near future.
2.1.3 Consistency
Traditional graphic design often put emphasis on consistency, and many apply the same consideration into information visualizations. Alberto Cairo states that interface consistency is important because it allows users to easily navigate information visualizations without having to re-learn them (Cairo, 2013).
Many works on information visualization put emphasis on consistency, and consistency can relate to many parts of an information visualization (Wills, 2012). It may mean using the same colours for the same variables, using the same layout for interfaces, using the same typefaces across interfaces, and more.
2.1.4 Adaptive visualizations
One recent subfield of information visualization research is that of adaptive visualizations. An adaptive visualization is a quite broad concept, because there are many different ways in which a visualization can be an adaptive visualization.
Kawa Nazemi defined adaptive visualizations as:
“Adaptive visualizations are interactive systems that adapt autonomously the visual variables, visual structure, visualization method, or the composition of them by involving some form of learning, inference, or decision making based on one or many influencing factors like users’ behavior or data characteristics to amplify cognition and enable a more efficient information acquisition.” - Nazemi,
2016
This definition is very broad since it can include every visualization where user input changes visual variables, visual structure, visualization method or composition of the previous. This degree project is only concerned with one aspect of adaptive visualizations, which is Visual content adaptation, defined as “the choice, reduction or expansion of the data that are visualized” as Nazemi notes (Nazemi, 2016).
A visualization that allows dynamic filtering of data, which is what I’m researching, is by definition an adaptive visualization with visual content adaptation.
2.1.5 Data filtering
The term filtering in this report refers to “data filtering”, which means to reduce or expand the amount of data according to a set of parameters.
2.2 Virtual Reality concepts
2.2.1 Depth perception in VR
There are both binocular (two eyes) and monocular (one eye) depth cues used to perceive depth. When rendering 3D on a flat screen, we use monocular depth cues like perspective, shading, shadows, texture gradients and motion parallax to give the user a sense of depth. VR allows for the use of convergence and binocular parallax as well, which improves depth perception. Humans use all of these depth cues to determine how far away objects are situated.
As an object’s distance from a person approaches infinity, the visual lines of our eyes form parallel lines instead of converging. This causes us to lose the depth cue convergence and causes objects to look flat. This is dependent on the resolution of the screens in the HMD used; at a certain distance objects will be drawn the same on both screens of the HMD.
The distance after which objects appear flat for the HTC VIVE, as well as for the Oculus Rift Consumer version (two popular contemporary VR HMDs on the market) is ≈ 20m.
2.2.2 Presence & Immersion
Two important concepts of Virtual Reality are presence and immersion. According to Daniel R Mestre, presence is the psychological and cognitive notion of “being there”, and a consequence of the technological term immersion. Immersion focuses on replacing as many real world sensations as possible with ones corresponding to the virtual world (Mestre and Vercher, 2011).
Thus, immersion is quantifiable according to a set of variables as presented by Slater et al. (Slater et al., 1996)
• Extensiveness, the amount of sensory systems that are accommodated (visual, auditory, spatial interactions)
• Surrounding, the extent of directions in which sensory information can arrive
• Inclusiveness, to what extent external sensory data are shut out
• Vividness, the variety and richness of the virtual information that is presented (graphical quality for example)
• Matching, how well the technological mapping of body movement functions (head movement, hand movement etcetera)
Presence, according to Slater et al. is seen as a product of two independent variables: how well the virtual representation is matched with the user’s world model (how well attuned it is to what they expect of the world, such as objects being affected by gravity) and how well there is a match between proprioception and sensory data (the extent and quality of feedback back and forth between the virtual scene and the user) (Slater et al., 1996).
As can be gathered from these definitions, experienced presence in a virtual scene is affected by certain variables of immersion. However, immersion may be very high in a virtual scene whereas presence can be very low, as may be possible if the laws of that world are significantly different from our own.
Slater et al. performed a study in which the results suggest that increased immersion does improve task performance in VR (Slater et al., 1996), where both increased realism and increased immersion produced better task performance results for participants.
2.2.3 VR ergonomics
The Oculus Rift guidelines recommend placing objects in an area of 0.75m to 3.5m. Having objects any closer can make the lenses of the eyes to misfocus and strain to retain focus (Oculus VR, 2017).
Alex Chu, who worked with the Samsung Gear VR headset technology, discovered that users on average define their zone of comfortable head movements as 20° up and 12° down in the vertical plane (Chu, 2014). Figure 2 below illustrates this range.
Figure 2 - Comfortable head movement in the horizontal plane
2.2.4 Cyber sickness
When designing for VR, it is important to be aware of cyber sickness. It is a subcategory of motion sickness experienced by people in Virtual Reality when they experience movement in the virtual scene although they themselves are not moving. Symptoms include nausea, eye strain, dizziness, stomach discomfort, sweating, postural instability, disorientation and tiredness. The actual cause for cyber sickness is not certain, and several different theories exist that try to explain the physiological effects.
Some people react more to cyber sickness than others, and the effects seem to wane off somewhat as a user gets more experience with a VR system.
To prevent this, often negative, experience, it is important to never move the user in the virtual scene when they are not physically moving themselves. It is unadvisable, for example, for the user to move themselves in the virtual scene using an analog stick, which is common method in video game applications for moving the player.
3 Application design
3.1 Environment of the virtual scene
The environment of the virtual scene should be very minimalistic and dark in order to use bright colours and accentuating objects for indicating points of interest, which in this case is the visualization in the scene. It was decided that the visualization should use shadows and realistic lighting in the scene since shadows help us better understand 3D structure and make the visualization more realistic which makes users more immersed by definition (Slater et al., 1996).
A grey floor of a lighter shade than the rest of the environment was used to show users the physical region of interest and movement, so that they would be less confused about where to be.
The user’s hands were represented by models of the HTC VIVE hand controllers in order to preserve consistency and familiarity between the interaction feel in the real world and the virtual scene.
3.2 Visualization form and data
The simplest 3D form to represent data points in would be a cubic shape, so it was decided that the three dimensions customer age, time as customer and NPS score should be represented along the three dimensions of the cube. To keep consistency of the 3D environment, data points in the graph should be represented as 3D objects rather than 2D billboards. Their most basic shape would be cubic as well, so the customer glyphs in this visualization would be these cubes.
This study limits itself to representing three types of products in the visualization, and the colour of a customer glyph was chosen to be indicative of which product that customer owns. These colours were set to cyan, magenta and yellow since these colours are very different from one another, which should facilitate a clear distinction between customer glyphs in the visualization for a user. In order for a user to better perceive the density of glyphs in an area, the glyphs were made semi-transparent, which also allows a user to perceive customer glyphs that would normally be occluded by another customer glyph in front of them because of alpha compositing (a computer graphics technique used to create the appearance of full or partial transparency). This is expected to facilitate a general perception of structure of the data, since the alpha compositing also helps a user to further determine depth of customer glyphs
The size of the customer glyphs was set to a size that would allow them to be clearly visible even at the far end of the visualization with regards to the resolution of the HMD.
The graph had a grey wireframe for the graph to make it easier for a user to perceive the shape of the graph no matter from which perspective they are looking. This can be seen in Figure 3.
Figure 3 - The visualization with data points, slider filters and button filters
3.3 Slider filters
In terms of filter interaction in the visualization, I wanted users to be able to highlight an area of the visualization by setting an interval for each dimension represented on the graph.
To create a strong sense of correlation between the slider values as well as the spatial representation of the customer glyphs, it was reasoned that filters and their values for the three dimensions should be placed on the graph. A user should then be able to control these filters to specify a cubic area in the graph which would represent a continuous interval of values for each dimension. All customer glyphs that do not fit into this area should be faded, but still visible in order to clearly show the concerned customer glyphs for these intervals, but also allow for a simultaneous perception of the whole structure of the data. The reason for this is to clearly show which customer glyphs are affected by the current filters, and which are not. The faded material for these customer glyphs are shown in Figure 4.
they were close to each other and in part to encourage the user to move around, which may cause them to better perceive the data in the visualization. The text on the slider handles shows the current value it is set on, and as such both slider handles show the interval for that dimension.
The data dimensions were oriented so that the lowest values for each dimensions converge in one corner of the graph. The reason for this was to keep it similar to a 3D graph of coordinates where the values reach zero in the origin, which is a very common representation of 3D graphs in, for example, mathematics.
The text labels shown in the visualization for both descriptions and values on the sliders was visible from the behind. The reason was that this may make it easier for a user to find the interaction area for the filters, since not all sides of the visualization have sliders.
Figure 5 - The slider filter and the button filters
3.4 Highlighted area
In order to better give users a spatial sense of the area highlighted in relation to the entirety of the graph, a green, semi-transparent highlighted area was used. The dimensions of this green cubic shape were directly mapped the handles of the sliders, and in length with the values specified by these slider filters. With this, users are better equipped to perceive the spatial domain that their filters are set to since it is the size of the cubic shape.
3.5 Product buttons
The user should be able to filter which product or products are to be visible in the visualization. This was implemented as a button for each product that would toggle the faded material used to show which customers are in the current filters. The text on the buttons and colour show which customer glyphs it concerns. These buttons were placed slightly below and in front of the graph to allow an averagely tall user between 175 and 185 cm to comfortably use them with their arms in a comfortable position of 90° bend, as seen in Figure 5.
It was decided to have the buttons available from all sides of the graph so that a user didn’t feel they needed to keep on one side to use these buttons. As such, the buttons would follow the user around and be placed at whichever face of the graph the user was facing.
3.6 Numerical overview display
In order for users to be able to get a numerical value of how many customers were filtered out, I implemented a virtual display which is a form of an overview as a user-interface component mentioned in chapter 2.1.1 Overview. It was placed on the user's non-dominant hand. The reason was that this would allow users to see numerically what the results of their filters show. This presents the selected amount of customers in the graph in relation to the total amount of customers as seen in Figure 6. By placing it in the non-dominant hand, the user would be able to look at this display whenever and wherever it is suitable for them. With this overview as a user-interface component, users should get an awareness of the data in a numerical sense as well, which should increase the overall sense of overview in the visualization.
Figure 6 - The numerical overview display
3.7 Size of graph
The graph was dimensioned for an amount of around 700 customer glyphs, mainly due to hardware limitations. The computer used for this study (specifications can be reviewed in 4.1 Resources) was not able to render a larger amount and keep the frame rate for the VR system above 90 frames per second, which would have a negative impact on the tracking of the system while increasing the risk of
This would allow them to comfortably see the entirety of the visualization without excessive head movements in accordance with the guidelines presented in 2.2.3 VR ergonomics.
4 Evaluation method
This part of the report will explain the study, which scientific methods were used, how they were used for this study and how the study was formed to comply with these methods.
4.1 Resources
The principal will provide access to a HTC VIVE VR head-mounted-display (HMD) with motion tracked hand controllers, as well as a computer workstation capable of rendering software for the device. The Unity game engine version 5.5 will be the main development tool, since it is free-of- charge and has rigorous support for the HTC VIVE.
The HTC VIVE VR system tracks the user within a specified tracking area, in which the user's movement is translated into movement in the virtual scene by tracking of the HMD. This makes the HTC VIVE system less prone to induce cyber sickness.
The computer system used had the following specs
Processor Intel Core i5-7600K (4.20 GHz)
Graphics card ASUS STRIX GeForce GTX 1070 8GB GAMING
RAM Corsair Vengeance LPX 16GB / 2666MHz / DDR4 / CL16
Operating
System Windows 10 Home 64-bit VR system HTC VIVE
4.2 Scientific methods
4.2.1 Structural analysis
Structural analysis is a method of research highly appropriate to study computer interfaces because of their rapid pace of development (Ware, 2013). This is especially true for the contemporary state of information visualizations in VR where the need to discover new valuable hypotheses and observations is higher than the need to accurately validate any of the findings.
There are several scientific methods within structural analysis to use, but this study will mainly focus
Colin Ware elaborates that the main advantages of semi-structured interviews for evaluation, as opposed to measuring reaction time or error rates, is that it is possible to gain information on a wide range of issues and that one might gain knowledge about something that was not asked in the interview (Ware, 2013). The disadvantage is that we do not get any objective measurements from these interviews or any detailed knowledge of the different aspects of the subject of research, however the semi-structured interview often opens up for observations and hypotheses that will allow us to evaluate the different aspects better than before.
4.2.3 Presence questionnaire
The presence questionnaire was created by Witmer & Singer in 1998 (Witmer and Singer, 1998). It is used for measuring presence in a VR environment. The current version of the questionnaire contains a total of 24 items that queries the participant about different factors of the VR application and system that correlates with presence. Each question is answered on a Likert-scale from 1 to 7. The questionnaire can be evaluated item-by-item or using cluster analysis of the item’s subscales.
4.3 Implementation of methods in study
The most relevant way to evaluate information visualizations according to both Ware and Ellis et al.
(Ellis and Dix, 2006; Ware, 2013) is using an explorative research method since all information visualizations are generative artefacts, meaning that they have no value in themselves, but generate value based on who is using it and how that person is using it. Ellis et al. make the case that observational and ethnographic methods reveal the most since their case study of evaluations of information visualizations showed that the most significant data was received from comments from test subjects.
Ellis et al. further discovered that quantitative measures as sole evaluation method rarely discovers anything, mainly because information visualizations are generative artefacts, and should only be introduced alongside qualitative methods in the manner that they are justified by the research question, in which case they help build validity (Ellis and Dix, 2006).
Based on this, a semi-structured interview is a highly valid approach for answering my research question and will be the main method of information collection for this study. It will be used to have a controlled discussion with the test users regarding the visualization and aspects of it connected to VR and how this affected their overview of the data in the visualization. Apart from this, I will gather a range of quantitative data from participants’ interactions with the visualization that may be relevant to form a discussion around.
The presence questionnaire will allow us to compare the data from the semi-structured interviews in order to see if there were any differences in experience between the two iterations of the application.
This will also give a quantitative indication on how immersed users felt in the visualization.
4.3.1 Test plan research questions
The purpose of the test is to answer the main research question of this thesis, which is
“How to design a 3D VR visualization of customer data in order to preserve overview of data while
This question will be evaluated by having users learn and interact with the visualization, and to gather as much data as possible that arises from that interaction.
The most important part in usability testing according to Rubin and Chisnell is the research questions that you’re trying to answer, which should be accurate, clear and measurable or observable (Rubin, J.,
& Chisnell, 2008). To answer my rather broad research question, I have decided to split it into several sub-questions. These are
• How do users gain an overview of the data in the visualization?
• How do users understand the visualization graph?
• Do users have a comprehension of the data as well as patterns and correlations in the data?
• Does VR affect overview of the data in the visualization?
• To what extent were users able to perform case-related tasks in the visualization?
• How did filtering interactions affect overview of the data?
• To what extent are users aware of the state of the visualization at a certain point in time?
These questions deal mainly with the different aspects of overview and situation awareness from the chapter 2.1.1 Overview and 2.1.2 Overview as situation awareness, as well as other aspects that may affect overview in the visualization.
4.3.2 Participants
Participants for this study should to a varying degree be familiar with information visualization and data filtering, but to what extent should vary. It is also desirable for participants to have varying experience with virtual reality in order to get perspectives both from expert as well as novice users. A diverse group in these aspects should be positive since they should introduce a broader spectrum of interaction with an information visualization in VR. This may lead to a larger amount of interesting observations from the user study.
Thus, participants will be categorized based on their experience with:
• Information visualizations
• Data filtering
• Virtual Reality systems
Jakob Nielsen presents that user studies do not need a great number of participants, but should instead use around five participants, because using a greater number is less time effective. Instead, time should be used to gather as much data as possible from the tests, and use an iterative design process (Nielsen
experiences, it is more likely that concerns or problems that the different groups of experience may have are addressed during the study. The resulting distribution can be reviewed in 5.1 Participant distribution.
4.3.3 Study outline
Introduction (10 minutes)
• Participant signs consent form
• Participant fills out background questionnaire
• Moderator outlines the study
• Participant learns VR interactions in demonstration scene (if they are not familiar with VR or the HTC VIVE system)
• Participants are presented with a description of the visualization
Tasks (15 minutes)
• Participant performs a think aloud while getting acquainted with the visualization
• Participant performs a set of tasks on data set #1
• Participant performs a free form task on data set #2
After tasks (35 minutes)
• Participant fills out the Presence Questionnaire
• Participant and moderator has a semi-structured interview about the experience
To best evaluate and create a good discussion around the visualization during the semi-structured interview, it’s desired that the participants may learn the visualization, are able to perform tasks in it and finally are allowed to freely use it as it is meant to be used.
In order to get an as indicative experience as possible for test users, they will not only perform a set of decided tasks, but will also perform a free form task of using the visualization to discover as many patterns, properties, interesting features and correlations in the data as possible. The reason is that it is wiser to have the user freely interacting with the visualization and actively interpreting the results.
This way of interacting with the visualization is most representative of the typical use case, which is using the visualization to gather new knowledge from the data.
Using the answers and observations from this first round of testing, a second iteration of the application will be constructed which I will perform the same study on as described for the first iteration.
This data includes
• How many times the participant changed view perspective on the visualization
• How many times the participant used the product buttons
• How many times the participant used the slider filters
• How long time the participant used the slider filters
• How many times the participant changed sides on the interactions
• How much time the participant spent on the free form task
• How many times the participant used customer detail-on-demand*
* - Functionality implemented and logged for second iteration
4.3.4 Background questionnaire
The questionnaire aimed to categorize the participants and asked them to rate their experience for the following questions
1. Describe your previous experience with VR a. None: I have never used VR b. Minimal: 1-3 times used VR c. Intermediate: 4-9 times used VR d. Expert: 10 or more times used VR 2. Which VR hardware/-s have you used?
a. Google Cardboard b. Samsung GearVR c. HTC Vive
d. Oculus Rift with Oculus Touch hand controllers e. Oculus Rift (without hand controllers)
f. PlayStation VR with hand controllers g. PlayStation VR (without hand controllers) h. Other/-s
3. Describe your previous experience using VR with motion tracked hand controllers a. None: I have never used VR with motion tracked hand controllers
b. Minimal: 1-3 times used VR with motion tracked hand controllers c. Intermediate: 4-9 times used VR with motion tracked hand controllers d. Expert: 10 or more times used VR with motion tracked hand controllers 4. Describe your information visualizations experience
a. None: I have never used any information visualization to gain knowledge or support decisions
b. Minimal: I have used 1-5 information visualizations to gain knowledge or support decisions
c. Intermediate: I have used information visualizations several times to gain knowledge or support decisions
d. Expert: I regularly use information visualizations to gain knowledge or support decisions
5. Describe your data filtering experience
4.3.5 Think aloud
Users should first be allowed a period of getting used to the visualization in order to learn the control methods for filtering the data in the visualization before being instructed to complete tasks in the application. During this time, they will be asked to verbalize their experience so as to both have them actively and critically examining the components of the visualization, but also to capture thoughts that may be queried about in the semi-structured interview.
4.3.6 Set of tasks
After allowing the users to briefly learn the visualization, they will be asked to perform a set of tasks.
The meaning of these tasks is to observe how users proceed about solving them using the visualization and to measure to what extent they managed in solving these.
These tasks are more focused on seeing how well users can perform certain common operations in the visualization, but it doesn’t really say anything about how well the visualization works for discovering data or gaining or preserving an overview of the data.
These tasks were performed on a data set of 675 data points i.e. customers and the data set was the same for all participants in order to not have any irregularities in answers.
The set of tasks comprises of the following:
1. What is the lowest and highest value on the age of customers slider?
2. Filter out customers who are 50 years or older with a NPS score of 6 to 10 3. How many customers are there in total with 100 mbps?
4. How many customers are there in total for each product?
5. How old is the oldest customer with 1000 mbps?
6. For 250 mbps, are there more customers with a NPS score of 8 and over compared to 7 and under?
7. Which product has the most customers in the age interval 20 to 40 years?
8. Are there more customers who’ve been customers for 10 months or shorter than have been customers for 11 months or longer?
All tasks are evaluated as
1. Correct - Task was completed satisfactorily
2. Read error - Filters were set correctly, but reading of data was erroneous
3. Interaction error - There was an error in setting the filters or other interactions leading to a read error
In order to mitigate the effects of learning transfer, meaning that one task may teach a user how to solve another task, a counterbalancing technique should be implemented according to Rubin &
Chisnell (Rubin, J., & Chisnell, 2008). This means that the order of the tasks should be distributed in a variable fashion among participants. For this study, a simplified amount of only three task orders was specified in order to mitigate this effect to a small degree. Ideally all permutations of the order of tasks would have been distributed among participants for each iteration of the application, but that would require a test population of 80 640 participants, which was not feasible for this study. The three task orders presented below should be equally distributed among the test participants for each iteration of
Task order #1 1, 2, 3, 4, 5, 6, 7, 8 Task order #2 8, 7, 6, 5, 4, 3, 2, 1 Task order #3 5, 7, 4, 8, 2, 6, 1, 3
4.3.7 Free form task
The free form task is meant to allow the participants to interact with the visualization in an as close way as possible to how a case scenario user would generally interact with the visualization.
Participants were instructed to use the visualization in order to find as many correlations or interesting things in the data as possible. It’s not interesting for the study what they find, but rather that they use the visualization in this manner in order to better find difficulties in using the visualization for this general purpose.
A new dataset is used for this free form task. It contains the same amount of data points i.e. customers (675) since the visualization is dimensioned for roughly that amount of data points. This new data set is however spaced over a larger time in the “time as customer” dimension.
4.3.8 Presence questionnaire
Directly after exiting the virtual reality application, users will be asked to fill out the presence questionnaire. This is done directly after the testing in order for the person in question to have the experience in recent memory. All questions concerning sound will be removed from the questionnaire since there is no sound being used in the application.
A cluster analysis will be performed on the presence questionnaire as is suggested by Witmer &
Singer when dealing with a smaller number of participants (Witmer and Singer, 1998). Each question from the presence questionnaire will be added to one of the following subscales
INV/C Involvement/Control 11 items
NATRL Natural 3 items
HAPTC Haptic 2 items
RESOL Resolution 2 items
IFQUAL Interface quality 2 items
The mean and standard deviation will then be calculated for each subscale in order to evaluate the different subscale factors of the application in terms of presence.
o What could have been other useful interaction methods or filters?
• Do you feel like you have learned how the visualization works?
• Did you understand what the different axes in the visualization showed and meant?
o How long did it take to learn them did you feel?
• Do you feel that you learned about the data and the customers from the visualization?
o What?
o How?
• How was it to get an overview of the data through the visualization?
• Did you feel that you had to move about?
o Why/why not?
• Did you feel like you could get an overarching understanding for patterns or correlations in the data?
o In what ways?
• How easy or hard was it to see where in space the different customers were located?
• Could you understand which values a customer had just by looking at where in the visualization it was located [the customer glyph]?
o To what precision?
• In what way did you use the visualization?
o Did you begin by setting filters and then inspect visually or the other way around, or neither of them?
o Was it different for different tasks?
• Did anything make the visualization hard to use for you?
• In what way do you think VR affected the visualization?
o Did VR make it easier or harder to understand the data?
• Could you distinguish individual customers in the visualization
o What made it easy or hard to distinguish customers from each other?
• Was there something about the visualization or application that you did not understand?
• Were you able to keep track of the minimum and maximum values for the sliders?
o When you had moved the sliders?
After the first iteration, three questions were added to query users on new features added to the visualization. These were
• Did you use the reset button?
o Did it help you?
o How?
• Did you use the possibility to get information on an individual customer [customer detail-on- demand]?
o Did it help you?
o How?
• Did you use the panel on your non-dominant hand [numeric overview display]?
o Did it help you?
o How?
There is also the possibility to add questions based on observations made during the testing, as well as the possibility for the participant to air any thoughts that were not addressed by the other questions.
4.4 How the study was performed
The study was performed on six participants for the first iteration and on seven participants for the second iteration. Each test took roughly 1 hour and 30 minutes. Participants in the study are given aliases in order to separate their answers and their opinions for this study. Participants P1 - P6 performed the study for the first iteration of the application while participants P7 - P13 performed the study for the second iteration.
4.4.1 Pilot study
A pilot study was performed on two different participants in order to find potential problems with the test plan. It was discovered from this that some of the questions needed rephrasing for more clarity on the set of tasks. Also quantitative measures of interactions and movement were added to the study after the pilot study. No data from the pilot study is used in this degree project.
4.4.2 Test environment
The user tests were performed at an enclosed area of the Visualization Studio at KTH (the Royal Institute of Technology) in Stockholm. The HTC VIVE virtual area was 3m x 2m and a synchronization cable was connected between the two lighthouses of the HTC VIVE in order to improve tracking stability during the tests, and to decrease errors arising from poor tracking synchronization between the lighthouses.
4.4.3 Demonstration scene
Any user who was not sufficiently familiar with VR or had not used VR with tracked hand controls were subject to a demonstration scene. In this scene, they could practice interactions that were necessary for interaction with the visualization. Users with little or no VR experience would also be able to familiarize themselves with the medium so that any initial feelings of awe were mitigated.
This scene contains several objects, such as sliders, buttons and balls, which the user can interact with.
The interaction methods for the sliders and the buttons are the same as the ones used later in the actual application, which allows users to learn the interaction methods beforehand.
After the participants had had time to acclimatize themselves to the medium and the methods for interaction, the demonstration scene was ended and the participant was brought back into the real world to process their experience.
4.4.4 Think aloud
They were asked to continue with the think aloud throughout their time with the visualization both on the set of tasks as well as the free form task.
4.4.5 Order of set of tasks
Forgetfulness during the first tests caused a skewed distribution where a large portion of the participants were subject to task order #1. It was also discovered that task order #2 was malformed since question #4 answers question #3. Below is the distribution of task orders for all participants in Table 1.
Task order #1 1, 2, 3, 4, 5, 6, 7, 8 Task order #2 8, 7, 6, 5, 4, 3, 2, 1 Task order #3 5, 7, 4, 8, 2, 6, 1, 3
Task order Participant count in iteration 1
Participant count in iteration 2
Total
#1 5 2 7
#2 1 1 2
#3 0 4 4
Table 1
4.5 Method critique
Likert scales, which are used in the presence questionnaire, should not be attributed to an absolute value, because most users will use the whole scale, they are more useful for measuring relative differences. The presence questionnaire should therefore only provide us with a relative measurement of presence and immersion. However, the scale in the presence questionnaire provides a clear distinction between what the highest, lowest and middle values entail, along with a clear instruction for users to consider the entirety of the scale when answering.
To get an indication of users’ situation awareness the question “Would you be able to tell which filter values were active if the system would stop at a random point in time?” was posed. This question however, mostly leads to speculation on part of the participants, and as such I have decided not to use the answers for this question in my study. I will instead address how situation awareness could have been measured in chapter 6.2 Future work.
One of the test users experienced problems with tracking during their test. However, it was only brief, and the user in question was categorized as an expert in our background questionnaire, and has most certainly been subject to these tracking issues before. As such, I do not believe that these tracking issues carried any significant error into the study.
The order of the tasks was compromised as stated in 4.4.5 Order of set of tasks, however there is little reason to believe that this compromised the tasks in any way, since no data that was recorded would be affected by the order of tasks. That being said, having a random distribution of the tasks may have
There were two tests in particular in which the order of the questions may have affected the answers.
The question number 4 contains the answer of question number 3, and as such it was confusing for participants when they were asked in reverse order. Because of that, the reverse order was exchanged for a pseudo-random order that put question 3 before question 4. There is however no reason to believe that this affected the study other than slightly confusing two of the participants, it was only done in order to not make participants confused by this order.
Think alouds were not recorded, and as such any observations made during them have been recorded just as writing. This may affect their reliability, but in general when interesting observations were made from the think aloud, these were noted and brought up during the semi-structured interview. As such, most of the observations made were elaborated and recorded, which serves to increase validity.
Participant P11 was left-handed, which may have affected how they interacted with the visualization as compared to other participants. P11 however did not feel that this affected their experience with the visualization, when asked.
Because of limitations in time and resources, it was not possible to find a group of sales managers to test the visualization on. This group is the intended end user for the case scenario, and would be the best candidate for testing it. Instead, a group of students were used because of previously mentioned constraints. However, the only significant difference from a sales manager's perspective is a familiarity with the data and its implications, which may affect forward projection of the data, that is what the features in the data actually means.
4.5.1 Recorded data that was purposefully omitted
The time it took for a participant to complete each task during the set of tasks were recorded, however, since participants were performing a think aloud simultaneously which had an impact on how quickly they solved the different tasks, I have decided to omit this data from the results. The time to complete the free form task is still used however.
5 Results
5.1 Participant distribution
The participant distribution can be viewed in Table 2. In terms of experience with information visualizations, the distribution was quite close to the desired one, but in terms of experience with VR systems however, it was skewed towards a great deal of experienced participants as well as no one who had never used VR before.
Question Category Number of participants Desired number of
participants Describe your previous
experience with VR
None 0 3
Minimal 4 3
Intermediate 2 3
Expert 7 4
Describe your previous experience using VR with motion tracked hand controllers
None 1 3
Minimal 3 3
Intermediate 4 3
Expert 5 4
Describe your
information visualizations experience
None 1 3
Minimal 3 3
Intermediate 4 3
Expert 5 4
Describe your data filtering experience
None 0 3
Minimal 1 3
Intermediate 7 3
Expert 4 4
Table 2 - Participant distribution from Background Questionnaire
5.2 Task completion results
Figure 7 - Task completion results for first iteration
A brief comparison of the averages for button clicks, slider usages, slider time and side changes between the iterations can be seen in Figure 9. Because of a measurement error, one of the participants’ values were not logged for the set of tasks. As a result, the data logged was taken from 6 persons per iteration, instead of 6 persons for the first iteration and 7 for the second. For the free form task, logging was working for the concerned participant. A detailed table of the logs can be reviewed in Data logs for set of tasks.
Figure 9 - A comparison of interactions between iterations
5.3 Free form task results
5.3.1 Free form task times
Figure 10 - Average time spent on free form task per person Detailed times can be viewed in Free form task times.
5.3.2 Free form task data logs
A comparison between the different iterations can be seen in Figure 11. There was a general increase in interactions during this task for the second iteration, which may be attributed to participants spending more time on average than participants for the first iteration. Button clicks were however more common in the first iteration as a contrast. A detailed table of the logs can be reviewed in Free form task data logs.
Figure 11 - Log averages for free form task
5.4 Presence questionnaire results
The items from the presence questionnaire (PQ) were divided into 5 different subscales.
INV/C Involvement/Control 11 items
NATRL Natural 3 items
HAPTC Haptic 2 items
RESOL Resolution 2 items
IFQUAL Interface quality 2 items
Figure 12 - Results from the Presence questionnaire
Figure 13 - Standard Deviation for the different subclasses
The results of the PQ between iterations are too similar to detect any difference in experienced presence between participants from the first and second iterations. Changes made between the first and second iterations seems not to have increased or decreased immersion according to these results.
5.5 Semi-structured interviews
The results for the semi-structured interview was divided into answers concerning the different research questions detailed in 4.3.1 Test plan research questions.
5.5.1 How do users gain an overview of the data in the visualization?
All participants except for P9 described the same way of using the visualization, which is that during the free form task they began by examining the data visually to find correlations or hypotheses and then proceeded to use the filters to investigate if the hypothesis was true. During the set of tasks, all participants thought about how to use the filters to achieve the given task, and did generally not care to look at the data until after performing the filtering task.
P9, in contrast, described that for the free form task:
“I set the filters first and then looked at the data. Like I said before, I don’t like to have too much information at one time” - P9
P4 and P8 described that for task 5, in order to get the oldest customer, they looked at the data first and then used the filters to find out the value, in contrast to all other tasks.
