Research proposal : Information Visualization for Information Fusion

(1)

Research proposal

Information Visualization for Information Fusion

by Maria Riveiro maria.riveiro@his.se

Supervisor: Tom Ziemke

Technical Report HS- IKI -TR-07-004

School of Humanities and Informatics, University of Sk¨ovde

SE-541 28 Sk¨ovde, Sweden

(2)

Abstract

Information fusion is a field of research that strives to establish theories, techniques and tools that exploit synergies in data retrieved from multiple sources. In many real-world applications huge amounts of data need to be gathered, evaluated and analyzed in order to make the right decisions. An important key element of information fusion is the adequate presentation of the data that guides decision-making processes efficiently. This is where theories and tools developed in information visualization, visual data mining and human computer interaction (HCI) research can be of great support.

This report presents an overview of information fusion and information visualization, highlighting the importance of the latter in information fusion research. Information vi-sualization techniques that can be used in information fusion are presented and analyzed providing insights into its strengths and weakness. Problems and challenges regarding the presentation of information that the decision maker faces in the ground situation awareness scenario (GSA) lead to open questions that are assumed to be the focus of further research.

Keywords: information visualization, information fusion, decision support, situation aware-ness, uncertainty, human computer interaction, visual data mining, visual data exploration

(3)

Acknowledgment

This work was supported by the Information Fusion Research Program (University of Sk¨ovde, Sweden) in partnership with the Swedish Knowledge Foundation under grant 2003/0104 (http://www.infofusion.se) and carried out in collaboration with Saab Microwave Systems (Gothenburg, Sweden). I would like to thank my supervisor Tom Ziemke for his useful comments and the other members of the ground situation awareness scenario for fruitful discussions and valuable feedback.

(4)

Publications

Riveiro, M. 2007. Evaluation of Uncertainty Visualization Techniques for Information Fu-sion. Proceedings of the 10th International Conference on Information Fusion (ICIF ’07), Qu´ebec, Canada, July 912, 2007, pp. 1-8. IEEE. Catalog Number: 07EX1591. ISBN: 978-0-662-45804-3.

Niklasson, L., Riveiro, M., Johansson, F., Dahlbom, A., Falkman, G., Ziemke, T., Brax, C., Kronhamn, T., Smedberg, M., Warston, H. and Gustavsson, P. 2007. A Unified Situation Analysis Model for Human and Machine Situation Awareness. Proceedings of the 3rd Ger-man Workshop on Sensor Data Fusion: Trends, Solutions, Applications (SDF 2007), Bremen, Germany, September 27, 2007.

Riveiro, M. 2007. Cognitive Evaluation of Uncertainty Visualization Methods for Decision Making. Symposium on Applied Perception in Graphics and Visualization (APGV 2007), T¨ubingen, Germany, July 25-27, pp. 133. ACM SIGGRAPH. ISBN: 978-1-59593-670-7.

(5)

List of Figures

2.1 Information fusion from databases, sensors and simulations . . . 10

2.2 Revised JDL model (1998) . . . 11

2.3 OODA loop . . . 13

2.4 Waterfall model . . . 14

2.5 The importance of information presentation through the OODA loop . . . 18

3.1 Diagram of the visualization process . . . 23

3.2 Reference model or framework for visualization . . . 24

3.3 SeeSoft: visualizing software . . . 24

3.4 Tree Map example: SmartMoney . . . 25

3.5 Cone Tree . . . 25

3.6 WebForager (site map application) . . . 26

3.7 WebBook (web pages organizer) . . . 26

3.8 Integrating information across multiple applications: historical information . 27 3.9 Integrating information across multiple applications: routes over maps . . . . 27

5.1 Visualization pipeline: sources of uncertainty . . . 36

5.2 Color change (saturation) indicating uncertain information . . . 38

5.3 Colour indicates uncertain surface . . . 38

5.4 Uncertain surface indicates uncertainty . . . 38

5.5 Box-and-whisker plot . . . 38

5.6 Glyphs representing angular and magnitude uncertainty . . . 39

5.7 Castle reconstruction. Uncertainty is encoded into transparency . . . 39

5.8 Extension of the 2D box-plot . . . 40

5.9 Uncertainty glyphs . . . 40

5.10 Hypothetical probabilistic Center Monitor . . . 42

5.11 Center monitor display . . . 43

5.12 Pairs of icons representing hostile or friendly objects . . . 43

5.13 Blurred icons: probability that an object is hostile or friendly . . . 44

5.14 Graphical display used in the dynamic decision making experiment . . . 44

(8)

List of Tables

2.1 Comparison of data/information fusion models . . . 15 2.2 Summary of HCI Enabling Technologies . . . 19

(9)

Chapter 1

Introduction

We are drowning in information but starved for knowledge. This level of in-formation is clearly impossible to be handled by present means. Uncontrolled and unorganized information is no longer a resource in an information society, instead it becomes the enemy

John Naisbitt, Megatrends

In today’s information age, the lack of information is seldom a problem. Rather the problem is the opposite, the overload of information. The difficulty of processing and handling vast amounts of information from multiple sources is a common feature of multiple real-life domains. Military operations, crisis management or homeland security applications involve a large number of actors with different characteristics, needs and behaviors. The solution lies in the ability to process and filter the information in a manner that results in knowledge, providing responders and decision makers with improved situation awareness.

Achieving situation awareness is crucial for making effective decisions. Such awareness in these complex situations may be difficult to achieve due to not only the overload of infor-mation but also other factors like time pressure, high stress or the imperfect and uncertain nature of the information. Hence, there is a need for decision support systems that help the decision maker to comprehend the situation and anticipate future consequences. Fortunately, techniques, methods and tools used in information fusion can attenuate this problem.

Information fusion has been identified as key enablers for providing decision support. It includes theory, techniques and tools for exploiting the synergy in the information acquired from multiple sources, for example sensors observing the environment, databases storing knowledge and simulations predicting future behavior.

The result of applying information fusion techniques can generate vast amounts of com-plex data that need to be analyzed by a decision maker. The presentation of the information, the graphical interface and the availability of interaction methods play a central role in the acquisition of the awareness necessary to make effective decisions. Advances in information visualization, interactive computer graphics (software and hardware) and human computer interaction open new possibilities for the access, analysis, navigation and retrieval of infor-mation.

(10)

The focus of this report is the role that information visualization plays in decision mak-ing in the ground situation awareness (GSA) scenario that is part of the Information Fusion research program at the University of Sk¨ovde. The GSA scenario includes military appli-cations in a network centric warfare, surveillance, national security and civilian operations, like catastrophe management. All this applications share common characteristics, like the availability of huge amounts of data, large number of objects/targets that are hard to identify and classify, hidden or camouflage entities, uncertain data and hidden patterns and relations. This information is not only handled and processed by a computer system but it is typically presented to the decision maker who is normally under time pressure and overwhelmed by the overload of information. In order to support the decision maker and overcome these difficulties, first steps in this research identify tools and techniques developed in information visualization, human computer interaction and visual analytics that can help users to synthe-size information; derive insights from massive and often conflicting data; detect the expected and discover the unexpected.

1.1 Organization of the report

The report is organized as follows: chapter 2 introduces information fusion (definitions, con-cepts and models) and highlights the importance of HCI and visualization within information fusion. Chapter 3 gives a brief overview on visualization both information and scientific vi-sualization. Visualization tools, techniques and breakthroughs in information visualization are presented in section 3.2. Chapter 4 describes the ground situation awareness scenario and its characteristics, HCI research needs and research questions that provide guidance on future research. Chapter 5 describes methods to represent uncertainty, introduces cognitive theories for their evaluation and presents three evaluation examples of information fusion applications.

(11)

Chapter 2

Information fusion

This chapter provides an introduction to data and information fusion, basic terminology and concepts. Widely used models in information fusion, such as the JDL model and OODA loop are briefly described and alternative models are presented as well. The last two sections discuss the importance of HCI and information presentation (visualization) in information fusion.

2.1 Information and data fusion

The term data fusion first appeared in the literature around 1960 as mathematical models for data manipulation (Esteban, Starr, Willetts, Hannah, and Bryanston-Cross, 2005). Variously called multisensor data fusion, sensor data fusion or sensor fusion, early definitions note that data fusion seeks to combine data from multiple sensors to perform inferences that may not be possible from a single sensor or source alone (Hall and McMullen, 2004, chap. 1). For example, a definition of data fusion is given in (Steinberg, Bowman, and White, 1998-1999):

Data fusion is the process of combining data or information to estimate or predict entity states.

Another definition of data fusion is given in (Hall and Llinas, 2000, chap. 1):

Data fusion techniques combine data from multiple sensors and related infor-mation to achieve more specific inferences than could be achieved by using a single, independent sensor.

Information fusion, as used herein, can be considered an important sub-area of data fusion (data fusion is a more general concept given that data is potential information (Meadow and Yuan, 1997)). It should be noted that no longer the source of processed data are only sensors (present), but also databases (past information) and predictions (future information) (see figure 2.1).

Information fusion has emerged as an independent research field in the last two decades (Kokar, Tomasik, and Weyman, 2004). The origins of information fusion can be traced back to developments in many areas, specially from the defense arena (Dasarathy, 2000). Examples

(12)

Figure 2.1: Information fusion from databases, sensors and simulations. Adapted from (An-dler, Niklasson, Olsson, Persson, de Vin, Wangler, Ziemke, and Planstedt, 2005)

of fields that already exploit benefits from information fusion are: robotics, maintenance engineering, medical diagnosis, information management systems, traffic control, biometrics and military applications such as battle space intelligence, surveillance or crisis management. Dasarathy defines information fusion as follows:

Information fusion encompasses theory, techniques and tools conceived and employed for exploiting the synergy in the information acquired from multiple sources (sensor, databases, information gathered by human, etc.) such that the resulting decision or action is in some sense better than would be possible, if these sources were used individually without such a synergy exploitation.(Dasarathy, 2001, p. 45)

2.1.1 The JDL model

The most widely used model to categorize data fusion related functions is the Data Fusion Model (or JDL model) (Steinberg, Bowman, and White, 1998-1999), developed in 1985 by the U.S. Joint Directors of Laboratories (JDL) Data Fusion Group. It is a functional model, where functions are divided into levels that relate to the refinement of objects, situations, threats and processes (Hall and Llinas, 2000, chap. 2) (see fig. 2.2). It should be noted that the JDL model is not a process model, because it does not specify the interaction among these functions within the information system.

Since its development, the JDL model has been revised several times. The first review of the model (in (Steinberg, Bowman, and White, 1998-1999)) broadened the definitions of levels 1-3 to accommodate fusion problems beyond military and intelligence ones which had been the focus of earlier versions of the JDL model (Steinberg and Bowman, 2004). To address problems of detecting and characterizing signals a level 0 (Sub-Object Data Assessment ) was proposed. Level 4 (Process Refinement ) was also emphasized in that revision as a resource management function (involving planning and control and not estimation). Nevertheless, it has been argued whether or not the level 4 (process refinement) should be consider an independent level (Llinas, Bowman, Rogova, Steinberg, Waltz, and White, 2004). Later revisions of the model (Hall, Hall, and Tate, 2000) and (Blasch and Plano, 2003) added a new level, level 5, labeled as ‘Cognitive (or User) Refinement’, which addresses cognitive issues and human computer interaction aspects.

(13)

Figure 2.2: Revised JDL model (1998). Redraw from (Steinberg, Bowman, and White, 1998-1999)

In this report I will refer to the levels (from level 0 to level 4) and terminology used in (Steinberg, Bowman, and White, 1998-1999). Following (Steinberg, Bowman, and White, 1998-1999),(McDaniel, 2001) and (Hall and Llinas, 2000, chap. 2 and 21) the levels are defined as follows:

• Level 0: Sub-Object Data Assessment

Estimation and prediction of signal or feature states. Examples of assignments in level 0 are signal processing (e.g. analog-to-digital converter) and feature extraction. • Level 1: Object Assessment

Level 1 processing seeks the detection, identification, location, characterization and tracking of entities. Key functions in level 1 include: data alignment (normalization of data with respect to time, space or other units), data correlation (determination of whether new data relate to existing entities), estimation of the entity state (position, velocity and attributes) and estimation of the entity identity (classification functions to identify emitters/receivers, low-level military units, etc.)

• Level 2: Situation Assessment

Level 2 processing seeks to understand the entities’ relationships with other entities and with their environment. Functions to achieve that include: object aggregation (temporal relationships, geometrical proximity, communications link and functional de-pendence among entities), event/activity aggregation (relationships in time to identify

(14)

activities or meaningful events), contextual interpretation (analysis of the data in con-text: weather, terrain, sea state, enemy doctrines and socio-political considerations) and multi-perspective assessment (analysis of the data with respect to the friendly, enemy and neutral forces).

• Level 3: Impact Assessment

Estimation and prediction of threats and potential opportunities. The situation is anal-ysed from a consequences point of view where alternative hypotheses are generated and projected into the future to determine courses of action. Key functions within level 3 include: capability estimation (like size of the enemy forces), predict enemy intent (enemy doctrine), identify threat opportunities (identification of potential enemy threat based on enemy actions, operation readiness analysis, environmental conditions, etc.), multi-perspective assessment (analyses of enemy, friendly and neutral forces), offen-sive/defensive analysis (prediction of hypothesized enemy engagements, enemy doctrine, weapon models, etc.)

• Level 4: Process Refinement

Level 4 (considered as a meta-process) seeks to monitor and optimize the overall data fusion process. Function examples within this level include: evaluation of the perfor-mance and effectiveness of the fusion process, source requirements and needs, mission management (recommendation for allocation and direction of resources) and data base management functions.

Level 5: Cognitive/User Refinement The need for a level 5 was first proposed in (Hall, Hall, and Tate, 2000) and labeled as Cognitive Refinement. The authors consider that extensive research in data fusion has focused on the data processing, from sensor data to a graphics display and little has been done in order to support a human decision-maker in the loop. Key functions within level 5 include cognitive aids and human computer interaction (Hall and McMullen, 2004, chap. 9)

(Blasch and Plano, 2002) re-labeled Level 5 as User Refinement. In the same line as the previous authors, Blasch and Plano claim that the JDL model is only for automatic processing of a machine and does not account for human processing. Issues like trust, workload, attention and situation awareness must be taken into account in the design of a fusion system which supports a user. Later publications, (Blasch and Plano, 2004), redefine the User Refinement Level and propose the JDL-User Model, (Blasch and Plano, 2003).

From my point of view, the functionality included in level 5 should be considered in all the other levels, so there is no need to create an isolated level to group these functions. Moreover, level 5 is at this stage just a proposal and it has not been widely accepted as a level of the JDL model.

2.1.2 OODA loop

One of the most commonly used model to describe the decision making process in information fusion is Boyd’s OODA (Observe-Orient-Decide-Act) loop, (Boyd, 1987). It has its origins in

(15)

the military domain, like the JDL model, but it focuses in the human (command and control group) decision process. Boyd considers four main activities in the decision process:

• Observe: the environment

• Orient: position yourself in the environment

• Decide: make a decision

• Act: perform the decision

The model illustrates the ultimate goal of a decision maker, taking the right decision within the minimum time, where speed is a condition for winning.

Figure 2.3: OODA loop. Redraw from (Brehmer, 2005)

In spite of being the dominant model for command and control, Boyd’s OODA loop has been criticized from two perspectives: it does not describe the decision making process in the military domain nor the decision making process in general (in (Brehmer, 2005) citing other authors: Bateman III and Bryant). In order to improve the OODA loop model by Boyd and emphasize its dynamic nature, Brehmer in (Brehmer, 2005) developed the DOODA (Dynamic-OODA) loop using cybernetics models for command and control. More details can be found in (Brehmer, 2005).

2.2 Other models in information fusion

Even though the JDL model and the OODA loop are widely used models in the informa-tion fusion community one can find alternative models in the literature. For example the Omnibus Model (Bedworth and O’Brien, 2000) which draws together each of the previous models (with their advantages and overcoming some of their disadvantages) and presents a general terminology of data fusion technology.

Dasarathy’s Functional Model defines a natural categorization of data fusion functions in regard to types of data or information processed (input) and types of results from the process (output) (Hall and McMullen, 2004, chap. 2). For example, neural networks or clus-ter algorithms tend to be patclus-tern classification algorithms that transform an input feature vector into an output feature vector (Hall and McMullen, 2004, chap. 2). An expanded view of Dasarathy’s model and its mapping to the JDL model, levels 0-4, is given in (Hall and

(16)

Llinas, 2000, chap. 2-15).

An example of hierarchical architecture often used by the data fusion community (Esteban, Starr, Willetts, Hannah, and Bryanston-Cross, 2005) is the Waterfall Model, described in (Harris, Bailey, and Dodd, 1998). The flow of data operates from the data level to the decision-making level where the sensor system module (level 1) is continuously updated with feedback from the decision- making module (level-3), see figure 2.4

A comparison among the different models is shown in table 2.1

Figure 2.4: Waterfall model. Redraw from (Esteban, Starr, Willetts, Hannah, and Bryanston-Cross, 2005)

2.3 HCI aspects in information fusion

The development of fully automated data systems has grown in the past decades. In almost any fairly complex systems, like nuclear reactors and aircrafts manual tasks are being replaced by automated functions (Matheus, Kokar, and Baclawski, 2003). It is, as well, a natural pro-cess, where the automatic part of the information fusion process is growing (for example, automatic target recognition applications). Nevertheless, many information fusion applica-tions are design for an human decision maker, and perhaps without enough consideration of its user. The lack of research in HCI related issues has been acknowledged by many authors in the information fusion Community. For example (Hall, Hall, and Tate, 2000), (Blasch and Plano, 2002) and (Hall and McMullen, 2004, chap. 9). The traditional approach, see JDL model, shows that data flows from sensors (source) toward the human (receiver). This could be a very simplistic interpretation, given that the human is actually involved in each each step of the fusion process. However, using this basic orientation, rich information from sensors is compressed for display on a two dimensional computer screen ((Hall and Llinas, 2000, chap. 19), referred as the “HCI bottleneck” problem by the authors).

(17)

Table 2.1: Comparison of data/information fusion models. Adapted from (Hall and Llinas, 2000, chap. 2-17)

Activity Waterfall JDL Boyd Intelligence being model model loop cycle undertaken

Command execution Act Disseminate Decision Decision level 4 Decide

making process making

Threat level 3 Orient Evaluate assessment

Situation Situation level 2 assessment assessment

Information Pattern level 1 Collate processing processing

Feature extraction

Signal Signal level 0 processing processing

Source/sensor Sensing Observe Collect acquisition

The effectiveness of a general and non-fully automatic information system highly depends on the human performance. More research is needed (Hall and Llinas, 2000, chap. 21) to understand information access preferences, how users perceive and process information, interact with the system and make decisions. Additionally (Waltz and Llinas, 1990) suggests that the overall effectiveness of a data fusion system is affected by the HCI efficacy. New advances should enhance the link between effective human cognition and the information fusion system, considering the human as the centre of the fusion process.

In order to overcome the HCI bottleneck in the information fusion process and account for functions for information representation and human machine interaction, Hall, Hall and Tate, (Hall, Hall, and Tate, 2000), proposed the introduction of a new level in the JDL model, Level 5: Cognitive Refinement. Level 5 processing involves developing functions to support a human decision-maker in the loop, users in collaborative environments and cognitive aids. Examples of functions for level 5 processing are (adapted from (Hall and Llinas, 2000, chap. 19-9)):

• Cognitive aids: functions to aid and assist human understanding and exploitation of data.

• Negative reasoning enhancement: humans have a tendency to seek for information which supports their hypothesis and ignore negative information. Techniques to over-come the tendency to seek confirmatory evidence could be developed.

(18)

• Uncertainty representation: methods and techniques to improve the representation of uncertainty.

• Time compression/expansion: time compression and time expansion replay techniques could assist the understanding of evolving tactical situations, on account of human capabilities to detect changes.

• Focus/defocus of attention: techniques to assist in directing the attention of an analyst to consider different aspects of data.

• Pattern morphing methods: methods to translate patterns of data into forms that are more easy for an human to interpret

Information fusion can certainly benefit from developments in information visualization research. At the same time information fusion applications provide a valuable source of case studies for researchers within the field of information visualization and human-computer interaction

2.4 The importance of information visualization in

informa-tion fusion

Many information fusion applications process and present huge quantities of data in order to enable an operator to make effective decisions. Commonly, the visual system is the core of the interface between the operator and the information system. Through the display the operator perceives data, process information and acquires knowledge (from data to knowledge and wisdom, (Shedroff, 2001)) in order to achieve some degree of situation awareness which will allow decision making. The the temporal limitations, the character uncertain of the information handled and how the information is presented clearly influences the decision process.

The following sections highlight the importance of information presentation on decision making.

2.4.1 The influence of information presentation on decision-making

Human decision-making can be seen as a complex information processing task. Regarding the level of complexity, decisions can be divided in three main groups (Rasmussen, 1987): skill-based sensor-motor behavior (automated or unconscious performance), rule-based behav-ior (simple procedural skills for well-practiced or simple tasks) and on the highest level of complexity, knowledge-based behavior. Knowledge-based behavior represents the most com-plex cognitive processing, used to solve unfamiliar problems or make decisions that require dealing with huge amounts of information and usually, with its associated uncertainty.

According to Endsley (Endsley, 1995), certain level of situation awareness must be reached in order to make a complex decision. Endsley defines situation awareness as the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning and the projection of their status in the near future. Hence, situation awareness

(19)

involves how to perceive, comprehend and project data/information. First, attributes and dynamics of the elements in the environment are perceived, then multiple pieces of informa-tion are integrated and their relevance to the decision maker’s goals is determined and at the projection level, future events are predicted. When a decision-maker faces a complex problem a mental model of the environment is built. In this process, humans seek for information that can help them to understand the situation.

In many real-world applications, the interface between an user and a computer or com-puter system includes a display/-s that connects the environment with the user who perceives, processes and makes a decision. Therefore a key element in the construction of the operator’s environmental mental model is the adequate presentation of the data that guides the decision-making processes efficiently. Among others, Card (Card, Mackinlay, and Shneiderman, 1999) and Tufte (Tufte, 2001) have highlighted the importance of information presentation on de-cision making.

Past research has investigated the role of information presentation on decision-making. Cognitive fit theory (Vessey, 1991) can be used as a theoretical framework for analyzing how the presentation of information affects decision making. Cognitive fit theory states that decision making is improved when the representation of the information matches the problem solving task, since decision makers develop a more accurate mental model of the problem.

2.4.2 Visualization and the OODA loop: observe and orient

The OODA loop (see section 2.1.2 for a detailed description) illustrates the ultimate goal of a decision maker, taking the right decision within the minimum time, where speed is a condition for winning. That includes both our own and our opponent’s decisions in the ground situation awareness (GSA) scenario. The minimization of the time consumed in the observe and orient phases of the OODA loop is essential in order to reduce the overall processing time. Here is where the visual system plays an extremely important role:

• Observe.

There are many sources which provide the user with new information: reports, audio, environment, interaction with other users (organization), etc. but it is the visual display one of the most valuable information sources, ”more information is acquired through the vision than through all of the other senses combined”, (Ware, 2000).

• Orient. As Boyd describes, the orientation phase is affected by factors like genetic heritage, cultural traditions and previous experience, as well as the mental processes of analysis and synthesis. Thereby the visual system should act as a cognitive tool in order to facilitate the mental processes of analysis and synthesis stage: presenting the infor-mation avoiding overload, overcoming human preceptive deficiencies and supporting the interaction human-machine.

• Decide

The displayed information will not only portray the topographical environment, condi-tion and state of friendly forces, enemy forces, neutral actors or weather condicondi-tions. It

(20)

should increment command and control ability to understand the battle flow of infor-mation and aid them to consider all the possible options and tactics and their impact and consequences in the battle space. Therefore the visualization system should be designed as an important part of the decision making support software. It is the bridge between data and seeing the picture which will allow the commanders to envision tac-tical alternatives, (Barnes, 2003).

The fig. 2.5 presents an interpretation of the OODA loop. The graphical interface, in the figure visual display, presents new information to the user and hence will influence the orient and the decide stage. The user interface should be designed taking into account how humans perceive, process, analyse information and make decisions. The ultimate goal of the user interface, including the graphical display, is the support of all these activities in order to speed the loop up.

Figure 2.5: My interpretation of the OODA loop. Orient and Decide takes place in the user/-s mind. Observe brings the new information to the user, through e.g. the user interface. Thus the user interface must be carefully designed to support decision making.

(21)

Table 2.2: Summary of HCI Enabling Technologies. Adapted from (Hall and McMullen, 2004, chap. 9, pp. 322-323)

HCI Description Current Technology

Compo- Technology Trends

nent

Devices for display -Large, high- fidelity -Proliferation of 3D of information to a graphic screens full immersion

human operator. -High density environments (e.g. Examples: television for collaborative Visual alphanumeric -High-end 3D devices data analysis) displays displays, icons and -Personal displays -Drive towards graphics devices; associated with increased reality

optical devices wearable computers via gaming

to monitor the industry

eye motion to -Interactive displays determine where the

user is looking

SW and HW -COTS voice -Increased to provide recognition SW interactivity either aural -Web search (voice recognition feedback to a engines using and voice feedback) Aural user; voice pseudo natural -Emulation of human interaction recognition and language facial expressions

natural language -Syntactic analysis and voice inflections

processing -3D sound

Devices that allow -Standard mechanical -Increased sensitivity a user to interfaces and feedback. provide commands -Trend towards

to a computer -Experimental haptic full-body haptic Haptic system (mouse, devices with interfaces devices joystick, touch emulation -Link between

touch screen) visual interface, acoustic and haptic -Wireless interfaces

(22)

Chapter 3

An introduction to information

visualization

3.1 Introduction

3.1.1 Visualization

The term visualization is frequently used ambiguously, referring to graphical representations that are normally carried out by a computer. Nevertheless its primary definition states that visualization is an activity carried out by a human being and not by a graphic engine. In order to exemplify, the following definitions are taken from different dictionaries:

1. visualize: v. to form a mental image of; imagine. (from The Concise Oxford English Dictionary)

2. visualize: v. to form a picture of someone or something in your mind, in order to imagine or remember them. (from Cambridge Advanced Learner’s Dictionary)

3. visualization: the display of data with the aim of maximizing comprehension rather than photographic realism.(from A Dictionary of Computing. Oxford Reference Online.)

In (MacEachren, 1995) and (Ware, 2000) visualization is referred to as a cognitive activity. An activity in which humans are engaged (Spence, 2001) as an internal construct of the mind and therefore cannot be printed on a paper or displayed on a screen.

However in The Visualization Handbook (Hansen and Johnson, 2005), Hansen and John-son use the definition described in the 1987 National Science Foundations Visualization in Scientific Computing Workshop report:

Visualization is a method of computing. It transforms the symbolic into the geometric, enabling researchers to observe their simulations and computations. Visualization offers a method for seeing the unseen. It enriches the process of scientific discovery and fosters profound and unexpected insights...The goal of visualization is to leverage existing scientific methods by providing new scientific insight through visual methods.

(23)

They treat visualization as the process of creating images that convey salient information about underlying data. Human perception is considered separately (part X of the book: Perceptual Issues in Visualization).

In (Card, Mackinlay, and Shneiderman, 1999), Card et al. define visualization as follows: ”the use of computer-supported, interactive, visual representations of data to amplify cogni-tion.” This definition contains both aspects, visualization as a graphical representation on the computer display and visualization as a cognitive activity.

3.1.2 Information visualization

Information visualization as a field of research is relatively new, with approximately ten years of history (Chen, 2004). It is also an interdisciplinary field, it combines disciplines such as computer graphics, HCI, communication theory, cognitive science and graphic design; and has influences from many domains, for example, World Wide Web, information retrieval, medical and bioinformatics applications, hypertext and virtual reality environments.

In order to reduce the scope of this ground, I will give some remarkable information visualization definitions.

In (Card, Mackinlay, and Shneiderman, 1999) Information Visualization is defined as follows: ”the use of computer-supported, interactive, visual representations of abstract data to amplify cognition”. The only difference between this definition and the above definition of visualization given by Card et al. in (Card, Mackinlay, and Shneiderman, 1999) is the use of ”abstract data” instead of ”data”.

In the following paragraph, (Ware, 2000), a more detailed definition of information visu-alization linking cognition and graphical engines is given:

What information visualization is really about is external cognition, that is, how resources outside the mind can be used to boost the cognitive capabilities of the mind. Hence the study of information visualization involves analysis of both the machine side and the human side. Almost any interesting task is too difficult to be done purely mentally. Information visualization enables mental operations with rapid access to large amounts of data outside the mind, enables substituting of perceptual relation detection for some cognitive inferencing, reduces demands on user working memory, and enables the machine to become a co-participant in a joint task, changing the visualizations dynamically as the work proceeds.

A similar definition is given by the User Interface Research Group in Palo Alto (Parc-Xerox):

Information Visualization is the use of computer-supported interactive visual representations of abstract data to amplify cognition. Whereas scientific visual-ization usually starts with a natural physical representation, Information Visual-ization applies visual processing to abstract information. This area arises because of trends in technology and information scale ... Information Visualization is a

(24)

form of external cognition, using resources in the world outside the mind to am-plify what the mind can do.

(http://www2.parc.com/istl/projects/uir/projects/ii.html)

Two major areas have been traditionally considered inside Visualization, Information Visualization and Scientific Visualization.

(Mackinlay, 2000) argues that scientific visualization focuses on physical data while infor-mation visualization does in abstract data. Examples of physical data are the human body, the earth or 3D medical pictures whereas information visualization treats non-physical data such as text, hierarchies or statistical data. Nevertheless the differences and the boundaries between information and scientific visualization are not clear. Examples of that are given by (Tory and M¨oller, 2002) and (Rhyne, 2003). Chen also shares this opinion in (Chen, 2004, Chapter 1, p.1): ”The boundary between information visualization and related fields such as scientific visualization and simulation modeling is becoming increasingly blurred.”

In (Tory and M¨oller, 2002), the authors suggested a new terminology arguing that the existing one is vague and ambiguous. They proposed a new classification based on character-istics of models of the data rather than on charactercharacter-istics of the data itself: continuous model visualization and discrete model visualization. Continuous model visualization refers to the visualization algorithms using continuous models of the data where the phenomenon being studied is continuous even though the data is discrete. On the other hand, discrete model visualization encompasses visualization algorithms which employ discrete data models.

In (Rhyne, 2003), ”Does the difference between Information and Scientific Visualization Really Matter?”, Rhyne questions the difference between scientific and information visual-ization. To demonstrate that information visualization is not unscientific and scientific visu-alization is not uninformative, as stated by (Munzner, 2002), the author uses two scenarios where this two major areas overlap: geographic information and bioinformatics visualization.

3.1.3 Visualization and cognition

Thinking is not something that goes on entirely, or even mostly, inside the users’ heads (Hutchins, 1995). Most knowledge is acquired or used as an interaction with cognitive tools and individuals and operating within social networks. More and more is the computer as an information system acting as a cognitive tool. The visual system, as a part of the information system functions as well as a cognitive tool (Ware, 2000) and its cognitive nature holds the difficulty of its study (Spence, 2001).

Visual displays provide the highest bandwidth channel from the computer to the human: more information is acquired through the vision than through all of the other senses combined (Ware, 2000).

But how can visualization amplify cognition? The work by (Larkin and Simon, 1987) is one of the seminal studies analysing why graphical representations are effective. Their results show that diagrams helped reducing the effort to some specific task in three ways:

1. The search is reduced because diagrams can group together information that is used together.

(25)

Figure 3.1: Diagram of the visualization process. Adapted from (Ware, 2000).

2. Search and working memory is also reduced by using location to group information about an element.

3. Graphical representations automatically support a large number of inferences that are easy for humans.

The ways in which graphical representations can amplify cognition is extended by (Card, Mackinlay, and Shneiderman, 1999) to six:

1. By increasing memory and processing resources available

2. By reducing the time to search for information

3. Enhancing the detections of patterns through visual representations

4. Enabling perceptual inference operations

5. By using perceptual attention mechanisms for monitoring

6. By encoding information in a manipulable medium

Table 3.1 shows different mechanisms that can enhance cognition. The table is taken from (Tory and M¨oller, 2004) but the basic content of the table is the same as in (Card, Mackinlay, and Shneiderman, 1999, table 1.3, p. 16).

3.2 Tools, systems and applications

Visualization provides an interface between two powerful information processing systems: the computer and the human mind (Card, Mackinlay, and Shneiderman, 1999). Effective visual

(26)

Figure 3.2: Reference model or framework for visualization. ”Visualization can be described as the mapping of data to visual form that supports human interaction in a workspace for visual sense making.” Adapted from (Card, Mackinlay, and Shneiderman, 1999, p. 17)

interfaces allow the interaction with large volumes of data and the discovery of patterns, hidden characteristics and trends among these data. This section briefly illustrates some interesting information visualization applications.

3.2.1 Visualization of large documents and software

Large bodies of text and great number of documents need to be analyzed in many applications, like for example in software development. Visualization methods can help the user to browse through the data in order to find interesting pieces or relations among the documents. An example of visualization tool in this group is SeeSoft. SeeSoft (Eick, Steffen, and Sumner, 1992) uses colored rows and columns to represent the frequency of use of certain lines of code (see figure 3.3).

Figure 3.3: SeeSoft visualizing software consisting of 38 files comprising 12037 lines of code. The newest lines are shown in red, the oldest in blue, with a rainbow color scale in between.

(27)

3.2.2 Visualization of hierarchies

Hierarchies or trees are abstractions of information structures (Chen, 2004). Visualizing trees is a mature branch in information visualization. A number of algorithms and approaches to represent hierarchies have concentrated on the focus vs context problem. Two well known examples in this category are tree maps and cone trees. Tree maps (TreeMaps) are defined by (Johnson and Shneiderman, 1991) as a space-filling approach to the visualization of hierarchi-cal information (see figure 3.4). A three dimensional representation of very large hierarchihierarchi-cal information is the cone tree, developed by (Robertson, J., and Card, 1991) (an example can be seen in figure 3.5).

Figure 3.4: SmartMoney: a market map of US stocks (generated on 17th September 1999), www.smartmoney.com/marketmap/

Figure 3.5: Cone Tree example (Robertson, J., and Card, 1991)

3.2.3 Focus + Context

In many occasions users need to access low level details and high level contextual information at the same time or in the same viewing area. The challenge is the compromise between detail and overview on a limited sized display. There are three approaches to overcome this tension (Chen, 2004):

• Overview + detail views: displaying overview and detailed information in multiple views.

• Zoomable views (or multiscale displays): displaying objects on multiple scales. A good example of multiscale displays is Pad++ (Bederson and Meyer, 1998).

• Focus + context views: displaying local detail and global context in integrated but geometrically distorted views (there is a large group of distortion techniques available, e.g fisheye or bifocal views). Examples of these techniques can be seen in (Furnas, 1986) (Furnas established the foundation of fisheye views) or (Spence and Apperley, 1982) (a good example of a bifocal view).

The potential solutions presented here have different strengths and weakness regarding easy of use, overall effectiveness and simplicity (Chen, 2004). Overview + detail solutions become

(28)

problematic when the user moves among the split displays of overview and detailed infor-mation. Zoomable views may lose the contextual information temporarily while the third approach, focus + context techniques, may lose the sense of continuity when the focal point is drastically changed.

3.2.4 Visualizing and Exploring the Web

The visualization of site maps or navigation trails is a challenging problem in information visualization. One of the most predominant applications of information visualization on the web are site maps and there is a great number of site map construction tools. WebForager (Card, Robertson, and York, 1996) (figure 3.6) and MAPA (developed by Dynamic Diagrams) are two of them. Another relevant tool is the WebBook, also presented in (Card, Robertson, and York, 1996). WebBook is a 3D interactive book of HTML pages that organizes pages and supports web exploration.

Figure 3.6: WebForager from (Card, Robert-son, and York, 1996).

Figure 3.7: WebBook (web pages organizer) from (Card, Robertson, and York, 1996).

3.2.5 Integrating information across multiple applications

Real life situations require the presentation of information from multiple sources while the application may require different visualization tools and techniques (Gershon, Eick, and Card, 1998). Researches in information visualization have developed a great number of visualization techniques for specific data. Nevertheless there is a lack of methods to integrate information across multiple applications (Gershon, Eick, and Card, 1998). An exception regarding this matter is the work of (Kolojejchick, Roth, and Lucas, 1997), where the authors describe a suite of basic tools into an integrated information workspace (see figures 3.8 and 3.9).

3.3 Challenges in information visualization

In this section I summarize the most important unresolved research problems in visualization, from both views, information and scientific visualization. A good orientation of issues that should be explored in the future is also given in (Rhyne, Hibbard, Johnson, Chen, and Eick,

(29)

Figure 3.8: The populated Daily Tag Sum-mary frame shows historical information for the tags arriving, leaving and remaining daily at a specified staging area. The data within the frame can be specified by dropping data on it from other frames (Kolojejchick, Roth, and Lucas, 1997).

Figure 3.9: Newly recomposed collections of shipments are dragged to the original Routes frame where they are shown at their destina-tion (Kolojejchick, Roth, and Lucas, 1997).

2004) (this panel, presented at the IEEE Visualization Conference, includes opinions from cited authors in Visualization).

1. Visualization as a Science

Frequently, application developers of visualization technology do not spend enough time trying to understand the underlying science (Johnson, 2004). Johnson suggests a direct collaboration with application scientists for working side by side with end users. Therese-Marie Rhyne in (Rhyne, Hibbard, Johnson, Chen, and Eick, 2004) suggests inter-disciplinary and intra-disciplinary collaboration as a key to solve challenges in visualization. Examples of these challenges are: ”How does our discipline effectively transfer its concepts and methods to specific domain scientists and experts who desire to apply visualization techniques? Does the Renaissance Team concept work as we extend visualization methods to hardware designed for computer games and mobile devices? Can there be an effective interchange between the information visualization and scien-tific visualization communities.”

2. Interaction

(30)

(Johnson, 2004). Many authors have addressed this challenge as essential in the future of visualization research, (Hibbard, 1999), (Johnson, 2004), (Tory and M¨oller, 2004). Other authors use interaction as a property to praise in their applications, e.g. SimVis described in (Hauser, 2005). Examples of interactive environments (virtual reality), their progress and future challenges can be found in (van Dam, Forsberg, Laidlaw, LaViola, and Simpson, 2000).

Hibbard in (Hibbard, 1999) points out the importance of user manipulation of graphical representations. He states that new ways of supporting interaction must be developed, for example the integration of gestures and speech parsing to the user interface. In future, direct manipulation will be essential for users in a immersing virtual world and there will be enormous complexity in the way that users manipulate visualization. In addition, Hibbard proposes the optimization of physical resources in order to underpin interaction. Solutions which allow interaction should include:

(a) parallel algorithms for common operations,

(b) strategies for data replication and movement in the memory hierarchy and on the network (interactions shared by multiple users).

The graphical representations should provide features to select, group and rearrange the information.

3. Collaborative and distributed visualization

Computers are mediators of human-to-human and human-to-data interaction. Many applications require team work and/or team decision making and support a certain common situational picture. In other occasions, domain experts are spread out in space (and even across time zones) and collaboration has to proceed remotely instead of face to face. Distributed teams need to collaborate and access shared data while some of their team members are outside of their office environments, presenting a new set of challenges in terms of data usage and share. Collaboration involves re-thinking interfaces in order to promote and facilitate joint work (Cartwright, Crampton, Gartner, Miller, Mitchell, Siekierska, and Wood, 2001). In military or near military settings, this aspects are of great interest due to the need of achieving a common situation awareness through a common picture.

4. Represent error and uncertainty

Johnson addresses this challenge in (Johnson and Sanderson, 2003), (Rhyne, Hibbard, Johnson, Chen, and Eick, 2004) and (Johnson, 2004). This unresolved problem is not new and many many articles have been written suggesting new methods for visualizing uncertainty or surveying existing uncertainty visualization techniques, e.g., the cited classification by Pang in (Pang, Wittenbrink, and Lodha, 1997). Other examples ad-dressing this problem in general are (Gershon, 1998), (Thomson, Hetzler, MacEachren, Gahegan, and Pavel, 2005) and (Griethe and Schumann, 2006); an example regarding the geo-spatial area can be found in (MacEachren, Robinson, Hopper, Gardner, Murray, Gahegan, and Hetzler, 2005)

(31)

The representation of error and uncertain information is crucial when large amounts of data have to be analyzed and evaluated in order to take a decision (Griethe and Schumann, 2005). In (Griethe and Schumann, 2005) the importance of uncertainty annotation is stated for supporting high level tasks, like decision making. A more detailed description regarding the representation of uncertainty can be read in chapter 5.

5. Perception and cognition-based design.

The effectiveness of a visualization depends on perception, cognition, and the user’s specific tasks and goals. (Tory and M¨oller, 2004)

(Tory and M¨oller, 2004) highlight the importance of human factors in the visualization process. They suggest that how people perceive and interact with a visualization tool should play a more important role in the design and posterior evaluation of a visual system.

In addition Mackinlay in (Mackinlay, 2000) emphasizes human perception as a future challenge for information visualization. From his point of view, visualization creates a feedback loop between perceptual stimuli and the user’s cognition but the existing knowledge about human perception and presentation design is still insufficient.

6. Empirical studies

Research in information visualization has been dominated by refined innovations, elab-orated visual representation, powerful methods and astonishing applications. However, empirical evaluations that validate their usefulness are often overlooked (Chen, 2004). Empirical evidence bring light on what fails, what works and what remains unknown (Chen, 2004). This matter has been acknowledged for many authors (Chen and Czer-winski, 2000), for example in a especial issue on empirical studies of information visu-alizations in Int. Journal of Human Computer Studies (e.g. (Morse, Lewis, and Olsen, 2000)). Established methodologies from HCI and psychology can be incorporated in information visualization.

From my point of view, empirical evaluations must be performed for newly developed techniques and cannot be overshadowed for what is possible rather than for what should be done.

(32)

Table 3.1: How information visualization amplifies cognition. (Tory and M¨oller, 2004) using (Card, Mackinlay, and Shneiderman, 1999, table 1.3, p.16). Adapted from (Tory and M¨oller, 2004).

Method Description

Increased Resources:

Parallel processing Parallel processing by the visual system can increase the bandwidth of information extraction from the data

Offload work to the With an appropriate visualization, some tasks can perceptual system be done using simple perceptual operations

External memory Visualizations are external data representations that can reduce demands on human memory

Increased storage Visualizations can store large amounts of information accessibility in an easily accessible form

Reduced Search:

Grouping Visualizations can group related information for easy search and access

High data density Visualizations can represent a large quantity of data in a small space

Structure Imposing structure on data and tasks can reduce task complexity

Enhanced Recognition:

Recognition instead Recognizing information presented visually can be of recall easier than recalling information

Abstraction and Selective omission and aggregation of data can allow aggregation higher level patterns to be recognized

Perceptual Monitoring: Using pre-attentive visual characteristics

allows monitoring of a large number of potential events Manipulable Medium: Visualizations can allow interactive exploration

through manipulation of parameter values

Organization Manipulating the structural organization of data can allow different patterns to be recognized

(33)

Chapter 4

The ground situation awareness

scenario

This chapter briefly describes the characteristics of the ground situation awareness scenario and presents general research questions in section 4.2.

4.1 Ground situation awareness scenario

Beside the study of generic aspects of information fusion, infrastructures, methods and algo-rithms, the Information Fusion Research Program at the University of Sk¨ovde is involved in multiple application oriented scenarios. Examples of these scenarios are precision agriculture, bioinformatics, simulation-based for manufacturing decision support and information fusion for rapid decision making in network-based systems (ground situation awareness, GSA).

The GSA scenario includes military applications in a network centric warfare, surveil-lance, national security and civilian operations, like catastrophe management. The goal of the information fusion research within GSA is to support decision making increasing the sit-uation awareness (SAW) by adding automatic and semi-automatic fusion processes (Andler, 2005). Before a decision maker can actually decide what to do, the relevant universe of dis-course needs to be observed and analyzed in order for the decision maker to become aware of how the observations relate to each other and influence potential decisions. The process of achieving SAW is called situation analysis (SA) (Matheus, Kokar, and Baclawski, 2003). SA is the process of examining a given situation, its elements and relations to provide a state of situation awareness for decision makers (Roy, Breton, and Paradis, 2001).

What is situation awareness? A general definition of SA is “the upto-the minute cog-nizance required to operate or to maintain a system” (Adams, Tenney, and Pew, 1995). (Endsley, 1995) focused more on the process and defines situation awareness as the percep-tion of elements in the environment, the comprehension of their meaning and the projecpercep-tion of their status into the near future. The term situation awareness is commonly used by the HCI community referring to a process that occurs in the mind of the operator:

(34)

It should be clearly noted, however, that technological systems do not provide SA in and of themselves. It takes a human operator to perceive information to make it useful. (Endsley and Garland, 2000)

SA is, however, a mental state and cannot be directly interacted with by use of technology. (Wallenius, 2004)

Manual tasks are more and more being replaced by automated functions. However, in many real-world applications human operators are still responsible for managing SAW. This raises new kinds of problems due to human limitations in maintaining SAW (Matheus, Kokar, and Baclawski, 2003) (SAW literature presents many examples of incidents and accidents which could have been avoided if operators had recognized the situation in time, see (Endsley and Garland, 2000)).

4.2 Research questions

The GSA scenario is characterized by huge amounts of information, very large number of objects/targets that are hard to identify and classify, hidden or camouflage entities, uncertain data and hidden patterns and relations. This information is not only handled and processed by the fusion system but it is typically visualized and presented to the decision maker who is normally under time pressure and overwhelmed by the overload of information.

In the described domain the information presented to the user should be easily assessed to effectively support high-level analytical tasks like decision making.

The general research question is how to visualize fused situation analysis information in circumstances characterized by information overload and time pressure. The problem domain is characterized by the need for rapid decision making and the presence of many different information sources. The aim within this work is to speed up and improve the decision pro-cess by means of propro-cessing and presenting the information: filtering information, enhancing human capabilities, supporting user-system interaction, etc.

These are some interesting research aspects that can constitute the focus of further re-search:

• Visualization of uncertainty, information reliability and quality of information. This question includes how to represent uncertain information to a user in such a way that he/she is aware of its nature. As example of previous work concerning representation of uncertainty can be found in (Bisantz, R. Finger, and Llinas, 1999). Preliminary results and uncertainty visualization methods can be found in chapter 5.

Two general research challenges in uncertainty visualization that are highly relevant to information fusion are: (1) the development of representation and evaluation methods for depicting multiple forms of uncertainty in the same display and (2) the development of methods and tools for interacting with uncertainty representations (MacEachren, Robinson, Hopper, Gardner, Murray, Gahegan, and Hetzler, 2005).

(35)

• Interactivity of visualization, depending on the user’s needs, his/hers experience and the level of trust in the system (involving the user in the fusion process).

• Past, present and future (predicted) information. The ultimate purpose of visualization aids to increase the commander’s ability to understand the battle dynamics, consider options and predict outcomes.” (Barnes, 2003). The system should provide with a time frame picture, showing the past, present and future state reflecting the impacts of the actions.

• Different levels of abstraction or granularity (in time and space).

• Collaborative visualization, enhancing group work (command and control)

The visualization system should be designed with an understanding of how users perceive and process information, interact with the system and make decisions. Additionally the visualization system should include the particularities of every task and reflect how users operate individually (role-based and user-centred design) and in collaborative environments.

(36)

Chapter 5

Uncertainty Visualization

As preliminary work, this chapter reviews general ways of representing uncertainty. The sur-vey also includes definitions of uncertainty and a general framework to identify its sources. Perceptual and cognitive theories from Tufte, Bertin and Chambers are described for the theoretical analysis and evaluation of uncertainty visualizations. These theories can provide insights into the weakness and strengths of existing and new developed uncertainty visual-ization techniques 1.

5.1 Introduction

Information visualization can provide valuable assistance to decision-makers. In many appli-cations the displayed data is imperfect and has some degree of associated uncertainty. The recognition of uncertainty and the awareness on the uncertain nature of the information is crucial in the decision-making process. Therefore, uncertainty should be appropriately repre-sented in order to avoid misinterpretations that may led to inaccurate conclusions. Neverthe-less, most of the developed techniques to represent uncertainty do not include a perceptual and cognitive evaluation that validates its usefulness.

Many applications process and present huge quantity of data in order to enable an op-erator to make effective decisions. Since there could be errors in the acquisition of the data (measuring devices), processing phases (data mining techniques), performance reasons, or even in the graphical display of the information (Griethe and Schumann, 2005), some degree of uncertainty is almost always associated (Cedilnik and Rheingans, 2000). If visualization is used to communicate the content of the data or to explore it, the uncertainty needs to be included (Griethe and Schumann, 2006). Moreover, the user should be aware of the nature and the degree of uncertainty of the displayed information, otherwise, there is a danger that data can be misinterpreted, potentially leading to inaccurate conclusions.

Even though the need for visualizing uncertainty associated with the data has now gen-eral acceptance (Zuk and Carpendale, 2006), most of the visualization research community has ignored or separated the presentation of the uncertainty from the data (Pang, Witten-brink, and Lodha, 1997). Part of the reasons are: it is not easy to include additional

un-1

(37)

certainty information into an existing visualization while maintaining comprehension ((Zuk and Carpendale, 2006),(Cedilnik and Rheingans, 2000)) and there is a lack of methods that present uncertainty along with data (Pang, Wittenbrink, and Lodha, 1997).

Johnson and Sanderson in (Johnson and Sanderson, 2003) suggested that a formal theo-retical framework for visualizing uncertainty and error should be also developed:

We see the need to create a formal, theoretical error and uncertainty visual-ization framework and to investigate and explore new visual representations for characterizing error and uncertainty.

This framework will be fundamental to a better understanding of the data with dubious origin or quality and as a result it will facilitate the decision making process. Nevertheless, it is worth examining available methods and techniques to visualize uncertainty from a perceptual and cognitive point of view.

In (Finger and Bisantz, 1997) the question of how to represent uncertainty is justified from two perspectives:

1. It is necessary to determine how different representations impact users and affect the decision-making process and actions

2. It is also necessary to find the best way of displaying uncertain information, particularly when there are a large number of objects associated with uncertainty.

5.1.1 Uncertainty definitions

There are many definitions of uncertainty in the literature. Normally, uncertainty covers a broad range of concepts like inconsistency, doubtfully, reliability, inaccuracy or error (un-known or not quantified error). Thus, it is difficult to give a generally accepted definition of uncertainty. From my point of view it depends on the context and the possible sources of uncertainty. The following are some well known definitions of uncertainty in the information visualization research field.

In (Pang, Wittenbrink, and Lodha, 1997) uncertainty includes statistical variations or spread, errors and differences, minimum-maximum range values and noisy or missing data. The authors consider three types of uncertainty in their discussion:

1. statistical : distribution of the data or estimated mean and standard deviation (confi-dence interval)

2. error : an absolute valued error among estimates or differences between a known correct datum and an estimate

3. range: an interval in which the data exists (and cannot be quantified into statistical either error definitions)

In ((MacEachren, Robinson, Hopper, Gardner, Murray, Gahegan, and Hetzler, 2005), citing other authors, Hunter and Goodchild, 1993) uncertainty is used meaning inaccuracy which is not known objectively (otherwise it would be considered as error ).

(38)

In (Foody and Atkinson, 2002, p. 26) it is simply defined as the ”quantitative statement about the probability of error”, where inaccurate measurements, estimates or predictions are associated with large uncertainty.

These definitions show that uncertainty has several concepts associated. Depending on which concept we refer to, there will be more than one way to quantify and represent uncer-tainty.

5.1.2 Sources of Uncertainty

One key point in the representation of uncertain information for a given application is iden-tifying the sources of uncertainty. A general model is described in (Pang, Wittenbrink, and Lodha, 1997). The visualization pipeline shows three major blocks as possible sources of uncertainty (see fig. 5.1):

1. Introduction of data uncertainty from models and measurements

2. Derived uncertainty from transformation processes

3. Visualization (representation) of uncertainty from the visualization process

Figure 5.1: “This visualization pipeline shows the introduction of data uncertainty from mod-els and measurements, derived uncertainty from transformation processes, and visualization uncertainty from the visualization process itself ” (Pang, Wittenbrink, and Lodha, 1997). Figure adapted from (Wittenbrink, Pang, and Lodha, 1996)

For example, one can identify the following possible sources of uncertainty in a generic information system:

• The sensors have limited resolution, its readings contain noise, their positions may be uncertain, sampling is sparse in time and space

(39)

• The process of converting the raw data into suitable input for numerical models may involve operations like averaging, interpolation, sampling, etc.

• The numerical models are also approximations, further, discrete computation introduces errors.

• The visualisation of the results introduces quantisation errors (data is interpolated, additional numerical integration may be used).

The level of uncertainty should be displayed in the visualization in order to help inter-preting the results. One should be sure that the operator or user has to do as few mental transformations as possible to understand the image (J¨a¨a-Aro, 2006). This is where theories of perception and cognitive sciences can be of great support in the evaluation of techniques used to depict uncertainty (see section 5.3).

5.2 Previous work on uncertainty visualization

Most of the previous work in uncertainty visualizations has been developed in the area of Geographic Information systems, GIS (for example, see (Hunter and Goodchild, 1993) for a survey of methods). In (Pang, Wittenbrink, and Lodha, 1997), the authors present a classifi-cation for uncertainty representation techniques. Seven categories are described: add glyphs, add geometry, modify geometry, modify attributes, animation, sonification and psycho-visual. In the following paragraph, examples of proposed techniques to display uncertainty are given (following the classification by (Griethe and Schumann, 2006)):

• Utilization of free graphical variables: colour, size, position, focus, clarity, fuzziness, saturation, transparency (e.g. fig. 5.7) and edge crispness. Figures 5.2 and 5.3 show examples of the use of colour to display uncertainty.

• Additional objects: labels, images or glyphs. For example, Wittenbrink et al. in (Wit-tenbrink, Pang, and Lodha, 1996), propose the use of glyphs to represent uncertainty in vector fields. Their approach is to include uncertainty in the magnitude, direction and length in glyphs (see figure 5.6 and 5.9). In (Pang, Wittenbrink, and Lodha, 1997) new ways of modifying glyphs in order to represent uncertainty are presented.

• Animation: the uncertainty is mapped to animation parameters such as speed or du-ration, motion blur, range or extent of motion.

• Interactive representation: e.g. uncertainty can be discovered by mouse interaction. An example can be found in (van der Wel, van der Gaag, and Gorte, 1998).

• Sonification and psycho-visual: incorporation of acoustics, changes in pitch, volume, rhythm, vibration, or flashing textual messages. See e.g. (Fisher, 1994).

Statistical properties can be plotted using a box-and-whisker plot (see figure 5.5). One interesting extension of the use of box-plots over 2D distributions can be found in (Kao, Luo,

Research proposal : Information Visualization for Information Fusion