Linköpings universitet SE–581 83 Linköping
Linköping University | Department of Computer and Information Science
Master’s thesis, 30 ECTS | Computer Science
2019 | LIU-IDA/LITH-EX-A--19/034--SE
Design and Evaluation of
a Visualization Method in a GIS
for Complex Electronic Warfare
Assessment with Regards to
Usability
Design och utvärdering med fokus på användbarhet av en
visualiseringsmetod i ett GIS för komplex telekrigsvärdering
Susanna Dahlgren
Supervisor : Anders Fröberg Examiner : Erik Berglund
Upphovsrätt
Detta dokument hålls tillgängligt på Internet - eller dess framtida ersättare - under 25 år från publicer-ingsdatum under förutsättning att inga extraordinära omständigheter uppstår.
Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka ko-pior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervis-ning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säker-heten och tillgängligsäker-heten finns lösningar av teknisk och administrativ art.
Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsman-nens litterära eller konstnärliga anseende eller egenart.
För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/.
Copyright
The publishers will keep this document online on the Internet - or its possible replacement - for a period of 25 years starting from the date of publication barring exceptional circumstances.
The online availability of the document implies permanent permission for anyone to read, to down-load, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility.
According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement.
For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: http://www.ep.liu.se/.
Abstract
This study presents a way to design a visualization method in a geographic information system for complex electronic warfare assessment. Enhanced usability was in focus and the work covers important factors and challenges when evaluating usability during the development process in a complex domain.
Electronic warfare is an important part of each country’s defence. Assessing Electronic Warfare capabilities in real war zones is normally impossible, making simulations and vi-sualizations important parts of the process. The field of visualization covers visual tech-niques for presenting large data sets in ways that provide insight to the user. In a visu-alization, an enhanced usability is achieved with a user-centered interface. Developing a visualization towards usability has its challenges, to develop it in a complex domain with few domain experts as users is even more challenging.
The study used an iterative method with three iterations of implementation and eval-uation, applying an approach adapted for experts working in a complex domain. When designing a visualization for a complex domain, the most important factors during the design process are to understand how the users work in the system and to make the visu-alization simple and over-intuitive.
The last user testing indicated that the users were able to draw conclusions about high-level conceptual goals based on the visual representation as well as to use the GUI to inter-act with the system. The result of this study is a visualization method for radar propagation with local refractive index, together with a process that can be used to design and evaluate a visualization method towards usability in a complex domain.
Acknowledgments
I would like to thank all the helpful people at the Department of Electronic Warfare Assess-ment at FOI with a special thanks to Petter Bivall, my supervisor at FOI, for all his help and support during the project. It has been invaluable! I would also like to thank all the people that participated in my user tests.
Contents
Abstract iii
Acknowledgments iv
Contents v
List of Figures vii
List of Tables viii
1 Introduction 1
1.1 Aim . . . 1
1.2 Research questions . . . 2
1.3 Delimitations . . . 2
2 Background 3 2.1 Swedish Defence Research Agency . . . 3
2.2 Electronic Warfare Assessment . . . 3
2.3 NetScene . . . 3
2.4 Local Refractive Index Model . . . 4
3 Theory 5 3.1 Geographic Information System . . . 5
3.2 Visualization . . . 6
3.3 Usability . . . 7
3.4 Usability Development . . . 9
3.5 Evaluation Process . . . 10
4 Method 15 4.1 Structure of the Study . . . 15
4.2 Experimental Design and Evaluation Method . . . 15
4.3 Pre-study . . . 17
4.4 Implementation of Paper Prototype . . . 17
4.5 Evaluation of Paper Prototype . . . 17
4.6 Implementation of High-Fidelity Prototype, Iteration 1 . . . 19
4.7 Evaluation of High-Fidelity Prototype, Iteration 1 . . . 19
4.8 Implementation of High Fidelity Prototype, Iteration 2 . . . 20
4.9 Evaluation of High Fidelity Prototype, Iteration 2 . . . 21
5 Results 23 5.1 Pre-study . . . 23
5.2 Implementation of Paper Prototype . . . 24
5.4 Implementation of High-Fidelity Prototype, Iteration 1 . . . 27
5.5 Evaluation of High-Fidelity Prototype, Iteration 1 . . . 28
5.6 Implementation of High-Fidelity Prototype, Iteration 2 . . . 31
5.7 Evaluation of High-Fidelity Prototype, Iteration 2 . . . 33
6 Discussion 35 6.1 Results . . . 35
6.2 Method . . . 38
6.3 The Work in a Wider Context . . . 41
7 Conclusion 42 7.1 Future Work . . . 43
Glossary 44
Bibliography 45
A Appendix: Paper Prototype 48 B Appendix: Test Plan, Paper Prototype 52 C Appendix: Test Results, Paper Prototype 54 D Appendix: High-Fidelity Prototype, Iteration 1 58 E Appendix: Test Plan, Iteration 1 61 F Appendix: Test Results, Iteration 1 63 G Appendix: High-Fidelity Prototype, Iteration 2 67 H Appendix: Test Plan, Iteration 2 70 I Appendix: Test Results, Iteration 2 72
List of Figures
2.1 Illustration of the four weather scenarios . . . 4
3.1 Models for collaboration with domain experts . . . 9
3.2 Comparison of different usability evaluation methods . . . 11
5.1 Scenario 1 . . . 23
5.2 Scenario 2 . . . 23
5.3 Scenario 1, Paper Prototype . . . 24
5.4 Diagram with wave propagation and diagram with threshold for discoverability. . 25
5.5 Alternatives for presenting the difference between calculations with and without LBM. . . 25 5.6 Scenario 1, Iteration 1 . . . 28 5.7 Scenario 3, Iteration 1 . . . 28 5.8 Scenario 1, Iteration 2 . . . 32 5.9 Scenario 3, Iteration 2 . . . 32 A.1 Scenario 1 . . . 48 A.2 Scenario 2 . . . 49
A.3 Diagram with wave propagation and diagram with threshold for discoverability . 49 A.4 Different alternatives of dialogs . . . 50
A.5 Alternatives for presenting the difference between calculation with and without LBM . . . 51
D.1 Scenario 1 . . . 58
D.2 Scenario 2 - Height Dialog . . . 59
D.3 Scenario 2 - LBM Dialog . . . 59
D.4 Scenario 2 . . . 59
D.5 Scenario 3 . . . 60
D.6 Scenario 3 - Controller . . . 60
G.1 Scenario 1 in Iteration 2 . . . 67
G.2 Scenario 1 - Height Dialog . . . 68
G.3 Scenario 1 - LBM Dialog . . . 68
G.4 Scenario 3 . . . 68
List of Tables
4.1 Participants in evaluation, paper prototype . . . 18 4.2 Participants in evaluation, iteration 1 . . . 19 4.3 Participants in evaluation, iteration 2 . . . 21
1
Introduction
The ways in which people communicate over long distances has been transformed dramati-cally over the years. In earlier times communication was achieved using couriers on horse-back or pigeon post. In today’s society most long distance communication uses some kind of telecommunication. The definition of telecommunication is the transmission of data, of-ten over a long distance by electric signals [1]. Since the introduction of telecommunications people have tried to disrupt, change or listen to other peoples’ communication. The tech-nique used by the military to secure the own communication and disrupt and complicate the enemies’ communication is called electronic warfare. [2] [3] [4]
To practice these actions, assessments have to be created in a real, or in a somewhat real, environment. It is more or less impossible to find a real war zone where systems can be tested and developed, therefore simulations are widely used. Realistic environments and situations are created for the equipment to be tested in, where multiple threats can be created to make it more realistic. The simulation process is an important aspect of electronic warfare assessment. [5]
To benefit from all data that is produced during simulations, the data has to be presented and visualized in a suitable manner. How to develop the visualizations is not an easy task, and the field of usability studies in visualizations and complex systems is under develop-ment. It is important that the visualization is easy to interact with and easy to understand, especially in a large complex system. There is not an obvious method to evaluate usability in these systems. The usability techniques normally used are often not optimal in these systems [6].
1.1
Aim
The aim of this study is to design and implement a visualization method in a geographic in-formation system (GIS) and evaluate this method with regards to usability during the devel-opment process. NetScene, a program created by FOI (Swedish Defense Research Agency), is used to visualize geographic information and electronic warfare scenarios. The data that is analyzed is retrieved from simulations of electronic warfare assessment situations. This study will lead to better understanding of how a visualization method can be designed in a GIS in a complex domain, such as a electronic warfare assessment, with focus on usability. It will also result in a high fidelity prototype of a visualization for radar propagation with
1.2. Research questions
LBM (Local Refractive Index Model), implemented in a GIS for analyzing electronic warfare assessment. LBM calculates the radar propagation with respect to weather data. It is desired to be able to present this data graphically in a GIS to get easier insight about how the weather affects the radar propagation.
1.2
Research questions
This study aims to answer the following research questions:
• How can a visualization method be designed in a GIS for complex electronic warfare assessment with regards to usability?
• How can a visualization method be evaluated with regards to usability during the de-velopment in a complex domain?
1.3
Delimitations
The study was performed in collaboration with FOI and the implementation was done in their program NetScene. No other GIS programs were considered. The visualization was designed in the specific context of complex electronic warfare assessment with a small target group of scientists and military people. Although the visualization and the structure of the work should be possible to use for designing different visualizations both the the same system but also in different complex systems where the target group is a small group of domain experts.
2
Background
This chapter presents the background to this study.
2.1
Swedish Defence Research Agency
Swedish Defence Research Agency is a defence research institute working under the Swedish Ministry of Defence. Approximately 900 employees work at the agency, doing research in for example cyber security, aeronautics and eletronic warfare. Commissions come from different authorities and companies, most commissions are given by the Swedish Armed Forces and the Swedish Defence Materiel Administration. [7]
2.2
Electronic Warfare Assessment
Electronic warfare is defined as when the electromagnetic spectrum is used by the military to complicate for the enemy to retrieve, treat and distribute information as well as to protect the own processes from being exposed to the same kind of attacks. The interference can happen anywhere on the electromagnetic spectrum, not only on the frequencies for radio communi-cation. It can affect the ability to see normal light as well as the use of infra-red radiation. It is often divided into three parts: Electronic support activities, which aims to detect and identify with for example radar warning systems, Electronic Attack, which aims to destroy the enemies ability to use the electromagnetic spectrum, for example with radar jammers and decoy generation and Electronic Protection, which aims to protect the own systems. [2] [4]
Electronic warfare assessment is to assess the effect of certain systems and situations re-lated to electronic warfare [8]. To evaluate the result of electronic warfare situations, simula-tions and visualizasimula-tions are an important part of the process.
2.3
NetScene
NetScene is a GIS tool for scenario planning, configuration and simulation. It is one of three parts of the framework EWSim (Electronic Warfare Simulation interface model). The other two parts are dynamic duel simulation and evaluation tools. EWSim is a framework devel-oped by FOI used to perform electronic warfare simulations. It can be used for training and
2.4. Local Refractive Index Model
to improve strategies for electronic warfare activities. NetScene is a tool where it is possi-ble to place a variation of entities with different equipment in a map. It is possipossi-ble to use NetScene by itself or to plan how the entities should behave during a simulation where all the three parts are used. NetScene uses a framework called Worldwind to present the map visualization. [9]
WorldWind is a GIS framework from NASA which can be used to visualize a three-dimensional (3D) globe and maps with different textures [10]. WorldWind is the second most used virtual globe system after Google Earth and it is easier to customize for different situa-tions [10]. Boschetti et al [11] present how a visualization using WorldWind was implemented to simulate both active fires and burned areas in the MODIS burned area project.
2.4
Local Refractive Index Model
Radar propagation is strongly affected by the weather. Changes of for example air temper-ature and air humidity changes how the radar waves behave. In this context, four different propagation scenarios are used to show how the propagation changes with different weather. The four scenarios are Normal propagation, Sub-refraction, Super-refraction and Duct. They are illustrated in fig. 2.1. [12]
In the Sub-refraction scenario the radar beams are deflected away from the surface more than normal. This results in a shorter reach for the radar than expected, which means that the ship can see shorter than expected. In the Super-refraction scenario the radar beams are bent more towards the surface than in the normal scenario. This results in a longer reach than expected of the radar. In the Duct scenario, the radar beams at low altitudes are bent down towards the surface even more. At sea the beams are repeatedly reflected in the water and refracted back towards the surface, thus going much longer than expected. [12]
The Duct results in the extremely long reach only for some radar frequencies and the radar need to be in the duct, at low altitude, to be able to get the extremely long reach. This can for example strongly affect a scenario with one small boat and one big boat or a boat and an airplane. At the ships this unforeseen changes in the radar reaches could not be explained, which motivated the development of the Local Refractive Index Model (LBM). LBM is a model developed by FOI and it uses weather data to calculate the atmospheric refractive index. This index is then used to calculate the radar wave propagation adjusted for the given weather. [12]
3
Theory
This chapter presents relevant theory for understanding the aim of this study. Geographic in-formation systems describes as well as the concepts visualization and usability. More specific subjects such as usability in visualizations and usability in complex domains are described. Finally there are theory related to the method about for example usability development, eval-uation and interviews.
3.1
Geographic Information System
A geographic information system is a tool for capturing and presenting spatial data and other forms of data given a spatial context. Spatial data is the same as positional data and means where things are, where things have been or where things will be. A GIS also provides the ability to store and maintain data as well as manipulate and analyze data. The use of GIS started to grow in the late 1970s and the tool is today used for many different purposes. Ev-erything from a natural hazard analyst who wants to find the areas where the annual mon-soon flooding will have the biggest impact, to an urban planner with the aim of analyzing the growth of suburbs. [13]
One of the challenges with using GIS is understanding the object to be studied. Since it will have different characteristics depending on the location and it will also change over time. Thus it is important how the data is converted into computer-readable format and how it is presented. The aim is for the treated data to be presented in the best possible manner. It is crucial to consider who the receiver is and what knowledge he/she is supposed to gain, as well as which techniques are available and most suitable to use. [13] Regarding the presenta-tion of data and the users’ interpretapresenta-tion of it, Michael F. Goodchild says that there is a need for developing more intelligent interfaces for GIS [14].
The use of GIS in the military
One of the first GIS applications was developed by the Canadian Navy and with the purpose of mapping and studying coastal areas. Today, GIS is commonly used by law enforcement and military forces and has five main areas of use within that context: to evaluate terrorist attacks and potential vulnerabilities that the enemy could exploit, to create possible scenarios involving terrorist attacks and stage reliable real-life exercises, for coordinating
communica-3.2. Visualization
tion during an actual incident involving emergency resources such as fire, police and hospital facilities, to coordinate recovery efforts needed after a real terrorist attack and to estimate the level of damage and lastly as a tool for planning counterattacks against terrorists. [15]
During wartime, military highway transportation is extremely important for transporting troops, supplies and in winning a war. The chosen roads might be attacked and sabotaged, which requires the army commander to be able to adapt and redirect the troops as quickly as possible. Information regarding the road network topology is necessary for planning a new route over remaining roads, or for exploring the possibility to build simple and temporary roads. In the article "GIS-based Simulation System for Wartime Military Highway Transporta-tion", the author Yin Xuri performs a study using a GIS and Dijkstras algorithm to simulate the shortest path, for military highway transportation, between two chosen positions. The resulting simulation succeeds in finding the shortest path. But there is need for further inves-tigation of the simulation method used, since this area is fairly uncharted. [16]
Another important factor during wartime is the possibility for communicating and send-ing information. Tactical radio stations are crucial and the most important tool for tactical communication. These systems uses electromagnetic waves, and these waves are affected by the terrain when they are transmitted. Usually the tactical radio systems are used by com-manders, and commanders are frequently re-located. Therefore, the systems’ location will often be changing as well as the conditions for radio wave propagation. Because of radio shade, the communication may often be interrupted. This requires tools for analyzing the terrain and how it could affect the tactical radio systems. In the article "Advanced GIS Func-tions for Tactical Radio Communication Planning", the author Ludek Lukas uses GIS and special functions called Automated Terrain Evaluation (ATE). The purpose is to evaluate the ter-rain and provide signal officers with information regarding locations where communication might be limited. [17]
3.2
Visualization
A general definition of visualization is the cognitive process done by humans when trying to create a mental image of a specific area of subject. In the context of computer and information science, it means trying to present large amounts of data in a visual, suitable manner. Either by using graphics, images or sounds to demonstrate systems, objects or processes. Visual-ization can also be defined as a technique used for gaining knowledge from simulations and computations. When talking about visualization, the main purpose usually is to gain insight. Insight can be described as retrieving wanted information regarding a specific problem, or getting new information that was not known before. [18]
In the article "A Taxonomy of Visualization Techniques using the Data State Reference Model", the author Ed H. Chi analyzes 36 visualization techniques with the purpose of categorizing them and creating a taxonomy. The taxonomy is supposed to extend previous research by also providing an implementer with the knowledge of how to apply these techniques in prac-tice. One of the categories in the taxonomy is called "Geographical-based Info Visualization" and includes a technique with the name "Profit Landscape", and another category contains a technique called "MapQuest". Both these techniques handle some sort of geographical data. [19]
In the article "Combining Metrics Data and the Structure of UML Models using GIS Visualiza-tion Approaches", Christian Lange and Michel Chaudron menVisualiza-tions a number of different visu-alization techniques in GIS. Every visuvisu-alization made in a GIS is based on a map and some of the techniques used to enrich the experience are the following: color (which could represent different characteristics such as an old building having one color and a young building hav-ing a different color), charts (which can illustrate changes over time, such as the population development in a city), lines and arrows (which represents different flows between elements,
3.3. Usability
such as the migration between countries) and lastly dot density (which basically illustrates the density of for example a disease in a specific area). [20]
3.3
Usability
The term usability was introduced in the 1980’s and was supposed to resolve the problems with the old vague term "user friendly" with a more clear definition [21]. According to the International Organization of Standardization (ISO), in ISO 9241-11, usability is defined as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” [22]. ISO defines the attributes effectiveness, efficiency and satisfaction but does not give any deeper explanations about how to use and how to measure the usability attributes [23].
• Effectiveness - is defined as "the accuracy and completeness with which users achieve specified goals" [22]. Which means that the product behaves as the user thinks it should, this is often measured by error rate [24] [25]
• Efficiency - is defined as "resources expended in relation to the accuracy and complete-ness with which users achieve specified goals" [22]. It refers to how fast a user can perform tasks in the program [24] [25]. Efficiency is often measured by time, for exam-ple time to perform a task correctly [24].
• Satisfaction - is defined as "freedom from discomfort and positive attitudes towards the use of the product" [22]. It can be explained as how the user feels about the product, if the user is satisfied with the product and is desirable for the user [24] [25]. Satisfaction is often measured by interviews and questionnaires [24].
Bevan, Kirakowski and Maissel define usability as how a user interacts with a system. And that the usability could be measured by three attributes: user performance, satisfaction and acceptability. Usability is not received as a fixed value for a system and it is not possible to say that a system has high usability or not. It is only possible to say that something is usable with a certain user, in a certain environment, given a certain task. [21]
Many authors have tried to define usability but according to Alonso-Ríos, Vázquez-García, Mosqueira-Rey and Moret-Bonillo [23] the definitions are often brief and do not in-clude any deeper explanations. There exists a multitude of definitions with varying attributes for usability, which may due to the complex nature of the term [23]. Despite its complex na-ture, usability is a well-used term and already in 2000 it was noted that usability testing was the most effective methodology for product design evaluation [26].
Usability in Visualizations
In the article "Evaluating the usability of visualization methods in an exploratory geovisualization environment", Koua, MacEachren and Kraak present a proposal for how usability could be used to evaluate a visualization environment. The evaluation method uses exploratory tasks and a taxonomy of visualization tasks and goals. Different tasks and goals are developed based on the taxonomy and during the tests, the users ability to finish and understand dif-ferent tasks are observed. They present ten conceptual visualization goals: Locate, Identify, Distinguish, Categorize, Cluster, Distribution, Rank, Compare, Associate and Correlate, The authors used the evaluation method to compare different representations, parallel coordinate plots and maps. They concluded that parallel coordinate plots resulted in a more effective representation while the map provides better visual attention. [27]
Ling and Chen want to improve the usability of geovisualization systems. They claim that the user interface is one of the most important concerns to focus on when developing for high usability. In the paper "Enhance the usability of cartographic visualization system by user-centered
3.3. Usability
interface design", they are using a user-centered design process to develop a user interface. The authors use an example to present how the process could be used to enhance the usability for different parts of the user interface, such as menu, toolbar and windows layout. [28]
Heidmann, Hermann and Peissner focus on the usability testing in their paper "Interactive maps on mobile, location-based systems: design solutions and usability testing". Usability testing is used during the design process. Users are given different scenarios and tasks during the test. During the test several measures are made, for example number of errors and number of tasks per time unit. After the test an interview was performed about satisfaction and suggestions. [29]
Freitas et al. studied how usability could be evaluated in information visualizations. They concluded two types of tests are needed, visual representation and interaction mechanisms. Visual representation includes for example displaying the relevant information and informa-tion mapping. Interacinforma-tion mechanisms includes for example selecinforma-tion of objects and control of levels of details. In their article "Evaluating Usability of Information Visualization Technique" definitions for all the proposed criterias are presented. [30]
Usability in Complex Domains
A complex domain explained by Mirel [6] and later used by Redish [31] is analysis of complex information - when domain experts try to solve open-ended complex problems. Some points that describe this kind of systems are: huge amount of information for the user, the users are often domain experts, time can be critical and visualizations are often used to present complex information [31].
According to Albers [32], the usability of complex systems is complex itself too. Useful-ness is an important part of usability testing in a complex domain. To properly focus on usefulness the designer must understand how the actual users work in the complex domain [33]. This is concretized in a pre-design usability study, which according to Redish[31] is directly necessary. This is performed before any design or development and the goal is to understand how the users work in the complex domain. Chilana [34] also mentions the ini-tial learning phase. The developer should get to know the system, get own knowledge about the complex system and get acquainted with the domain-specific terminology. The pre-study usability study is not enough and usability testing is still necessary during the development process [31].
One point mentioned by both Redish [31] and Chilana [34] is collaboration with domain experts. Redish [31] claims that it is important for the development to collaborate with do-main experts during the whole process. Chilana [34] presents three different models for col-laborations with domain experts. An overview of the models can be seen in fig. 3.1 and are explained below:
• Iterative elicitation - Contact with the domain experts back and forth during the whole development process. The most frequently used method.
• Persistent partnership - The domain experts are included in the team, they work in close collaboration during the process. The domain experts can be active during the development of the test sessions and can also help evaluating the results.
• Upfront investment - Much communication with the domain experts during the plan-ning phase. Help with understanding the system, the users, real scenarios and design requirements. No (or very little) collaboration later in the process.
Engaging the right participants for the test sessions is critical. They should represent the users, and in a complex domain the users are often domain experts and have little time for testing. If there are few domain experts available with time for testing and there is a need
3.4. Usability Development
Figure 3.1: Models for collaboration with domain experts. Adapted from Chilana [34]
evaluated on different groups of people with different level of knowledge it is, in a complex domain, necessary to give different information to the different groups. Also, the different level of knowledge makes it difficult to measure for example time to finish a task. [32] Mirel [6] explains that, in a complex system, it is not possible to measure the usability as the sum of individual components or tasks. It is more complex and needs higher level evaluation. According to Redish [31] another critical part of usability testing is applying good scenarios. Since the scenarios need to be realistic and are of complex nature, it is a good idea to get help from domain experts for the scenario design.
Some points that Redish [31] recommends to take into account in usability testing is: try-ing to understand the domain, ustry-ing the think aloud protocol and ustry-ing multiple participants in the tests to study how different people work and discuss during the tasks. According to Re-dish a traditional method for usability testing is not optimal for complex systems. Stantchev [35] concluded that, in a complex domain, usability should be a part of all the different phases of the development process and that the iterative process is a good alternative when devel-oping in a complex domain.
3.4
Usability Development
A product life cycle includes phases of design, implementation and evaluation. Before the design phase and the prototyping even start the designers should think about usability of the system. If this is done before any prototyping there is less risk that the system will need large restructuring after the first evaluation. [36]
When developing a product with usability in mind, it is efficient to use an iterative pro-cess, where the product life cycle is repeated a number of times. According to Rubin the best chance to end up with a usable product is to work in an iterative process. The iterative process lets the developers get input from the users quickly in the process, allowing for adjustments of the development to what the users actually want. [24]
Nielsen [37] describes an iterative design cycle, by using this design it will be possible to identify usability problems early in the process. The iterative process lets the designer change the prototype and then re-evaluate the design to see if the problems are solved. The designer learns much about the system, the design and the usage by studying the user during the tests. Nielsen [37] recommends doing a number of different simple prototypes and testing them early during the design phase. By doing this early and getting rid of the designs that are not meeting the requirements, a lot of time and money will be saved. It should be done early as low-fidelity prototypes, such as paper prototypes, and often as in several times during the development process. He mentions that the early low-fidelity prototypes gives much insight about what the user wants and need. [37]
There are many different recommendations, lists and guidelines for usability develop-ment. One of those guidelines is Shneiderman and Plaisant’s, presented in "Designing the User Interface". Their guidelines are called the Eight Golden Rules and are presented here: [38]
3.5. Evaluation Process
1. Strive for consistency 2. Cater to universal usability 3. Offer informative feedback 4. Design dialogs to yield closure 5. Prevent errors
6. Permit easy reversal of actions 7. Support internal focus of control 8. Reduce short-term memory load
3.5
Evaluation Process
How to do a proper evaluation of a visualization is difficult, but important for the reliability and validity of the study [39]. Forsell presents a set of guidelines for this in the paper "A Guide to Scientific Evaluation in Information Visualization". Except the plan of the evaluation design, Forsell presents a couple of points to remember during the preparations of the evaluation. [39]
• Background information - Relevant background information should be collected before the evaluation.
• Color blindness - Is color blindness relevant for the study?
• Written protocol - Prepare a written protocol for what information is to be given to the participants, so all participants are given the same information.
• Written instructions - Use written instructions for the evaluation, let the participants read the information before the tests and make sure there are no questions.
• Training Scenario - Use a training scenario before the tasks, make sure that the partici-pant has understood what to do.
• Pilot Study - Perform a pilot study before the evaluation to make sure that the tasks are relevant and possible to perform for the participants.
The evaluation design should be selected so that the results from the study can be gener-alized. One factor in the choice of method is the availability of participants in the evaluation. This can be particularly hard with information visualization since they are often developed for a small group of expert users. Some evaluation designs require a higher number of partic-ipants, this can be an important factor in the choice of design. Another factor is the aim of the study, if the evaluation are supposed to compare how different groups work with a visual-ization you need a larger number of participants to be able to get any statistical valid results about the performance in the comparison. If there are any confounding factors, such as that the results for the second visualization would be affected of that the participant already have tried the first visualization, then there are need for to different test groups. [39]
Rubin [24] mentions four factors to take into account when deciding how many partici-pants the study should have. The first factor is what degree of confidence the study should have. The second is available resources, the third is the availability of test participants that match the user profile and the fourth is how long time each test session will take. According to Rubin it also depends on the evaluation method. An evaluation that should result in statis-tically valid results need many participants. If the evaluation are supposed to detect usability
3.5. Evaluation Process
possible, the use of at least eight participants. If there are several iterations of tests of identi-fying usability issues there can be less participants in each iteration since the total number of testers will be higher in this way. [24] Dumas [40] claims that 6-12 participants are the typical number of users in user testing.
When it comes to the number of participants needed to detect usability issues there are several studies that claims different things. There are studies from the beginning of 1990s have claimed that five participants is enough, and this number has been used in many us-ability studies during the years [41]. Nielsen’s book "Usus-ability engineering" [42] has been cited over 19 000 times according to Google Scholar and Virzi’s paper "Refining the test phase of usability evaluation: How many subjects is enough?" has been cited over 1000 times, both are from the beginning of 1990s and claimed that approximate 80% of all usability issues were de-tected with five participants. There are papers published in later years criticizing that claim. Faulkner [41] studied this and published the paper "Beyond the five-user assumption: Benefits of increased sample sizes in usability testing" in 2003. Faulkner claimed that the use of 5 random participants detected a mean of 85% of the problems, but some random groups only found 55% of the problems. 10 random participants detected a mean of 95% of the problems and a minimum of 82% of the problems.
Usability Evaluation
There are several different methods to evaluate the usability of a product. Holzinger [36] tried in his paper "Usability engineering methods for software developers" to do a systematic overview of the different methods and when each method is a suitable choice. He divides the methods into two different types of methods: Inspection methods and Test methods. Inspection meth-ods use usability experts and contains the methmeth-ods: Heuristic Evaluation, Cognitive Walk-through and Action Analysis. Test methods use end users and contain the methods: Thinking Aloud, Field Observation and Questionnaires. Holzinger’s comparison [36] is presented in fig. 3.5.
Figure 3.2: Comparison of different usability evaluation methods by Holzinger [36]
Thinking Aloud
Thinking aloud is according to Holzinger [36] a very important usability evaluation method. The method is a user testing method where the end-users are testing the system. During the test, the test participant should talk out loud what are on their minds. This method helps
3.5. Evaluation Process
the developer to understand what the users think about the system. By doing this several times during the development important usability issues can be detected and fixed. The test participant should talk continuously during the test, not after the test, since it is the users direct thoughts that is desired. [36]
Formative Study and Summative Study
Usability testing is often divided into different types, one division used by Barnum [25], Du-mas [43], Nielsen [37] and Rubin [24] is a division in two different types. One diagnostic type, often called formative testing, and one benchmark test, often called summative testing. For-mative testing is a type of test that explore how the user uses the product. They are supposed to detect issues and to get alternatives. There are more interaction between the participant and the moderator in this type of test. [43] It is not important how good or how much wrong the prototype performs, but rather understand why it is wrong [43] [37]. To understand the "why", a qualitative method is preferred rather than a quantitative method [37].
Summative testing is a benchmark test that measures how good the product performs. This type of testing is often performed when the product is finished or almost finished and are used to measure how well the product meet the requirements [25] [37]. A quantitative evaluation method is typically used, and there are not so much interaction between the par-ticipant and the moderator [43].
Test Planning
One important step to do when performing a test is to create a test plan. According to Rubin and Chisnell [24] the test plan should be started at the very beginning of the project, and that the plan then should be changed and updated during the project. When it is time for testing, the test plan is refined and detailed. The test plan can be informal or formal, but it should be written and explain how the testing will be performed [25]. A detailed test plan will give a better replicability of the research work since it presents an exact description of how to do the test. Rubin and Chisnell [24] mention that it is important to focus the test on the user and the relationship between the user and the product rather than the product itself during usability testing. There are many different examples of test plans presented in the usability testing literature. It could include the following sections, which is inspired by Barnum [25] and Rubin & Chisnell [24].
• Purpose and goals - Presents the high level goals of the test and the motivation to why do the test now. [24] [25]
• Research questions - Presents the questions and issues that the test should try to answer. According to Rubin & Chisnell it is important to have clear questions that is possible to measure or observe, and they claim that this is important also in early exploratory testing. [24]
• Participant characteristics - Describes the user profile of the participants. This part should also describe how many participants are going to be used as well as particular characteristics of the users. If there are different groups in the test, each group should have their characteristics presented. [24] [25]
• Test design / Methodology - A short explanation of the test design, based on the pur-pose of the study. [24]
• Task list - Presents the tasks that will be performed during the test session. [24]
• Environment - Presents the environment that the test will be conducted in. It should be as similar as the environment for the end-user as possible. [24]
3.5. Evaluation Process
• Test moderator role - Presents how the moderator should act during the test session. Only observe, act in role playing or something in between. [24]
• Collected Data - Presents what information will be collected during the tests. This de-pends on if it is a qualitative study or a quantitative study. For example it could be different metrics, such as number of errors or time to finish a task. [24]
• Deliverables - Presents how the results will be summarized and presented. [24]
Domain Experts
Usability experts and UX designers are often used as participants in usability testing [44]. According to Virzi [45] the most important knowledge of the participants is the knowledge about usability. But there are problems with this approach, these persons do not have any deeper insight about the domain-specific system [44]. Følstad [44] did investigate this further and concluded that the use of domain experts as participants in an usability evaluation gave a smaller number of identified problems and suggestions. But the items were classified as more important or severe than the items found by the usability experts. He concluded that the use of domain experts could result in more important improvements for the usability.
The use of domain experts is even more important in complex domains where the system itself can be difficult to understand for someone other than the domain experts [34]. Chilana [34] concluded in the paper "Understanding usability practices in complex domains" that even if usability experts tried to understand the complex domain they did not perform as good in usability evaluations. Chilana expressed the importance of collaboration with domain experts both in the evaluations but also during the whole development process.
Test Moderator
How to be a good test moderator in usability testing is difficult and there are not much liter-ature in the subject. Dumas [43] wanted to improve this with his book "Moderating Usability Tests". It presents ten golden rules and information about all the steps from initial contact and recruiting to using participants from diverse populations and remote test sessions. Ac-cording to Dumas it is important that the moderator know how to behave during the tests, and that this is important for both the results themselves and for the validity of the results. The ten golden rules presented by Dumas are:
• Rule 1: Decide how to interact based on the purpose of the test • Rule 2: Respect the participants’ rights
• Rule 3: Remember your responsibility to future users
• Rule 4: Respect the participants as experts, but remain in charge • Rule 5: Be professional, which includes being genuine
• Rule 6: Let the participants speak!
• Rule 7: Remember that your intuition can hurt and help you • Rule 8: Be unbiased
• Rule 9: Don’t give away information inadvertently • Rule 10: Watch yourself to keep sharp
3.5. Evaluation Process
Interviews
There are three different types of interviews used in research. Structured interviews, unstruc-tured interviews and semi-strucunstruc-tured interviews [46] [47]. A strucunstruc-tured interview exactly fol-lows a manuscript with questions, the interview should not add any follow up-questions or take any different directions. An unstructured interview does not have a written manuscript with questions, the interview is more like a regular conversation - only with follow-up ques-tions when needed. A semi-structured interview is a combination of the two other types. It is based on a manuscript, often called interview guide, with questions. But the questions might be asked in a different order, and the moderator could ask other follow-up questions during the interview. In qualitative studies, unstructured and semi-structured is the most common used types. The two types are often called qualitative interviews together. [46]
During the preparation of an interview an interview guide should be developed. The interview should be prepared and performed with as few preconceptions as possible. The questions should be formulated so the participant is encouraged to present alternative solu-tions. Bryman [46] presents some points to remember:
• Remember to ask about background information before the interview. • Sort the different questions and themes in a good logical order. • Formulate interview questions based on the research question. • Use a language well-adapted for the interview participants. • Do not formulate the questions as leading questions.
• Record the audio from the interview, it is important to be able to analyze all the details from the interview later.
• Do the interview in a quite place. • Be prepared.
4
Method
This chapter presents the method used in this study. First, general information about the structure and methodology are presented and then specific information about each iteration.
4.1
Structure of the Study
The study began with a pre-study divided in two parts. The first part was to understand the complex system and the second part was to develop interesting scenarios. After the pre-study, paper prototypes were developed and evaluated with user testing. Results from the evaluation of the paper prototypes were used to implement and evaluate a high-fidelity pro-totype in NetScene. The implemented propro-totype was developed in two iterations. As pre-sented in section 3.4, an iterative process is preferable when developing with usability in mind. Stantchev [35] concluded that, also in a complex domain, an iterative process is a good alternative. Beginning the development process with simple and quick paper prototypes, to be able to test early and several times during the process, was recommended by Nielsen, see section 3.4.
4.2
Experimental Design and Evaluation Method
This section presents the experimental design used during the evaluation sessions. The eval-uation method was user testing with thinking aloud protocol in a semi-structured form. This was chosen based on Holzinger’s comparison presented in sec. 3.5. There were no available usability experts, that is why one of the test methods had to be chosen. Both Field Observa-tion and QuesObserva-tionnaires required more participants than were available for this study. Think-ing aloud was good for evaluation of the design and the number of users required for this type of test was 3+.
Participants
The participants were mostly persons with much experience of the system. According to Forsell [39] and Redish [31], complex systems are often developed for a small group of expert users. This was the case in this system, which led to a limited number of persons for the user testing. The four persons with most experience of the system and the context were
4.2. Experimental Design and Evaluation Method
therefore used during all of the evaluations. The domain experts were both end-users as well as developers of the system. They were the participants with best knowledge about the system and radar propagation.
Beside the four participants used in all the evaluations, the second and third evaluation included four new test participants for each evaluation session too. The new participants were four persons with experience of the domain as well as four persons in their last year of a master’s degree in software development and computer science. The domain experts are scientists and developers but have little experience in usability and interaction design. As presented in the theory, sec. 3.3 and sec 3.5, a development project in a complex domain needs both domain experts and usability designers as test participants in the evaluation. The students were chosen based on their extra courses in design, usability and interaction design in their masters. The eight new participants were divided in two groups, one for iteration 1 with two domain experts and two students and one for iteration 2 with two domain experts and two students. The participants were divided to get as equal as possible test groups in the different iterations.
Procedure
First, a test plan was developed. It was inspired by Barnum [25] and Rubin & Chisnell’s [24] explanation of a test plan, the theory can be found in section 3.5 Test Planning. It included information about, for example, the purpose of the test and the evaluation method. Based on the test plan a manuscript with written instructions were constructed. Written instructions should be used according to Forsell [39] to ensure that all the test participants receive the same information. To increase the validity of the test results, the ten golden rules by Dumas, see section 3.5, were studied before the evaluations. The evaluation was conducted as a number of test sessions. Each session was performed individually with one participant at a time. In section 3.5 it is mentioned that a pilot study is critical for the evaluation. A pilot study was conducted before each evaluation. The results from the pilot studies were not included in the results.
The test session began with general information about and the purpose of the test. A consent agreement was reviewed and signed by the participant. The consent agreement con-tained information that interactions will be observed and recorded during the test, as well as information about that it is okay to ask questions at any time and that it is fine to leave the test at any time. Background information was collected and a training scenario was presented for the participant before the test begun, to ensure that the participant had understood what to do.
When the general information and the training scenario were done, the test begun and the test participant was encouraged to think aloud during the test and share the first impressions of the prototypes. The test itself was structured as a structured interview. The semi-structured type, presented in section 3.5, was chosen because of the possibility to be flexible during the interview. But the prepared questions gives possibility to summarize and compare the answers from the different interviews. Different prototypes were presented, tasks were performed and questions were answered.
The tasks and questions were developed based on the theory about usability testing in complex domains and in geo-visualizations. The test was developed to both test the visual representation as well as the user interaction with the interface, this method were inspired by Freitas et al. [30]. During usability testing in complex domains it can be difficult to use tasks and questions normally used during usability testing. It should include some higher level questions about the understanding of the visualization. Koua [27] presented an ap-proach for usability testing in geo-visualization with exploratory tasks based on conceptual visualization goals, for example identify or relate something in the visualization to a given problem. The questions were also inspired by Knapp, Zeratsky and Kowitz and their book
4.3. Pre-study
“Sprint: How to solve big problems and test new ideas in just five days” [48] and Bryman’s recommendations [46] about how to prepare an interview.
After each test session the audio record was used to summarize the results from the test. The results were considered before the next iteration. Based on the result from all the test sessions in the iteration, the plan for the prototype in the next iteration was constructed.
4.3
Pre-study
The first period of the study was a pre-study phase without any designing or development. The first part of this phase was understanding the complex system and the context and the second part was developing scenarios.
Understand the Complex System
Understanding the complex system before any designing or development is directly nec-essary according to Redish [31]. The first part was to understand the context of electronic warfare assessment, which was done by reading literature about the context. Literature was also read about relevant subjects, such as, visualization, GIS, usability in visualizations and about how to develop and evaluate in a complex system. After the period of literature study the pre-study focused on the system. According to Mirel [33] it is important to understand the system and how the users work. To understand how the system works and how the users work in the system, time was spent to test the system and watch different available scenarios and presentations. During this phase there was a close collaboration with the experts of the system to understand how it actually was used. Different diagrams were discussed and sev-eral informal meetings were held during this period. It is not enough to only understand the users work, according to Chilana [34] developer should get their own knowledge about the system too. This was done by going through the code with a developer and try to implement small functions and visualizations - to get a general understanding about the structure of the system.
Develop Scenarios
Before the prototyping, relevant scenarios were developed in collaboration with a domain expert. The development of good scenarios is critical when developing in a complex domain, and it is a good idea to get help from domain experts [31]. The scenarios were the foundation for the development. The goal with the implementation was to visualize the scenarios in the system. The scenarios were also used during the usability testing.
4.4
Implementation of Paper Prototype
The first design phase was to create paper prototypes. According to Nielsen [37] early low-fidelity prototypes gives much insight about what the user wants and needs. Several paper prototypes were developed early in the process. The goals with the paper prototypes were both to understand what the user wants as well as first design suggestions to evaluate the usability.
During the sketching of the paper prototypes, knowledge gained during the pre-study about the system was used. To develop towards usability the eight guidelines presented in section 3.4 were taken into account.
4.5
Evaluation of Paper Prototype
The evaluation was conducted as four user test sessions with the procedure presented above in section 4.2. The four test participants were domain experts with between 6.5 and 35 years
4.5. Evaluation of Paper Prototype
User Title Domain Experience
1 Scientist, Domain Expert 6.5 years 2 Scientist, Domain Expert 9 years 3 Scientist, Domain Expert 17 years 4 Scientist, Domain Expert 35 years Table 4.1: Participants in evaluation, paper prototype
of experience in the domain, the median was 13 years of experience. The participants can be seen in table 4.1. They were the persons with best knowledge about the system and the radar propagation models. There were no changes of the test structure after the pilot study.
During the test different paper prototypes were presented. After each prototype different questions were asked, see the questions below. The questions were made inspired by Knapp, Zeratsky and Kowitz [48]. The evaluation of the paper prototypes was shorter and focused on not only to evaluate the usability but also to understand what the users want and gain more in-depth understanding about how the users use the system. The paper prototypes were simple and were mostly a base for the discussions.
The scenarios, tasks and question asked are presented here:
Scenario 1 - Two ships
• What do you like about this prototype? Why? • What do you like less about this prototype? Why?
• What would you like to improve with this prototype? Why?
Scenario 2 - One Ship and One Airplane
• What do you like about this prototype? Why? • What do you like less about this prototype? Why
• What would you like to improve with this prototype? Why?
• What do you think would be most interesting and intuitive, change target height of the opponent or height of the own ship? Why?
Radar Wave Propagation vs Circle of Threshold for Discoverability
• How would you compare the different alternatives? • What alternative do you prefer? Why?
Dialogs
• How would you compare the different alternatives? • What alternative(s) do you prefer? Why?
Scenario 3 - Difference of Calculation With and Without LBM
• How would you compare the different alternatives? • What alternative(s) do you prefer? Why?
4.6. Implementation of High-Fidelity Prototype, Iteration 1
4.6
Implementation of High-Fidelity Prototype, Iteration 1
After the evaluation of the paper prototypes, the first prototype was implemented in NetScene. The implementation was done in Java in NetBeans IDE 1, Swing 2 were used to set up the GUI components. The implemented prototype was developed with usability in mind. To develop towards usability, the eight guidelines presented in section 3.4 were taken into account as well as the results from the evaluation of the paper prototype. To implement the GUI was prioritized higher than to implement all the correct calculations behind the in-terface. This method was chosen to be able to test as much of the user interface as possible in the evaluation of this prototype. According to Ling and Chen [28], the user interface is one of the most important thing to focus on when developing for usability in a geo-visualization.
The color scheme was chosen based on the color scheme normally used in the domain. Green is used for allied and red for enemies. The combination with red and green can be problematic for a person with colorblindness. The controls were checked in a color blindness simulator3 during the implementation. The visualization would appear different but the controls should still be possible to use for a person with color blindness.
4.7
Evaluation of High-Fidelity Prototype, Iteration 1
User Title Domain Experience Earlier involved
1 Scientist, Domain Expert 6.5 years Yes
2 Scientist, Domain Expert 9 years Yes
3 Scientist, Domain Expert 17 years Yes
4 Scientist, Domain Expert 35 years Yes
5 Scientist, Domain Expert 20 years No
6 Scientist, Domain Expert 25 years No
7 Software Engineer student No
8 Software Engineer student No
Table 4.2: Participants in evaluation, iteration 1
The evaluation was conducted as eight user test sessions with the procedure presented above in section 4.2. Six test participants were domain experts with between 6.5 and 35 years of experience in the domain, the median was 18.5 years of experience. Two test participants were master students in their final year in software and computer science with competence in interaction design and usability development. The participants can be seen in table 4.2.
During the test, different scenarios were presented and different tasks were given. After each task, questions were asked, the tasks and questions were developed based on the the-ory about usability testing in complex domains and in geovisualizations. It included some higher level questions about the understanding of the visualization. Koua’s approach [27] with exploratory tasks based on different conceptual visualization goals were used. For ex-ample Compare, Locate and Identify something in the visualization which all were recom-mended tasks for evaluating maps. Some basic questions were inspired by Knapp, Zeratsky and Kowitz [48], such as "What would you like to improve with this prototype? Why?" and "How would you compare the different alternatives?".
1Netbeans IDE - https://netbeans.org/features/java/index.html
2Swing - https://docs.oracle.com/javase/7/docs/api/javax/swing/package-summary.html 3Color blindness simulator - https://www.color-blindness.com/coblis-color-blindness-simulator/
4.8. Implementation of High Fidelity Prototype, Iteration 2
The scenarios, tasks and question asked are presented here:
Scenario 1 - Two Ships
• Rank the different LBM cases according to how long ship1 can see. • Compare how long ship1 sees at different target heights.
• What do you like about this prototype? Why? • What do you like less about this prototype? Why?
• What would you like to improve with this prototype? Why?
Scenario 2 - One Ship and One Airplane
• Compare how long flight1 sees at different target heights. • What do you like about this prototype? Why?
• What do you like less about this prototype? Why
• What would you like to improve with this prototype? Why?
Scenario 3 - Difference of Calculation With and Without LBM
• What conclusions can you make from the situation?
• How much longer can you see according to the LBM calculation compared to the calcu-lation without LBM?
• Rank the different LBM cases according to how close they are to the normal radar prop-agation calculation.
• What do you like about this prototype? Why? • What do you like less about this prototype? Why
• What would you like to improve with this prototype? Why?
After the Different Scenarios
• What do you think could be innovative and valuable to bring with you when creating a new visualization?
4.8
Implementation of High Fidelity Prototype, Iteration 2
In the second iteration of implementation, changes were done in the prototype based on the results from the evaluation in the previous iteration. To develop towards usability the eight guidelines presented in section 3.4 were taken into account. The implementation was done in Java in NetBeans IDE. Swing was used to set up the GUI components. The implementation of the graphic interface was prioritized higher than the implementation of all the correct calculations behind the interface. This method was chosen to be able to test as much of the user interface as possible in the evaluation of this prototype.
The color scheme was chosen based on the color scheme normally used in the domain. Green is used for sensor’s ranges of allied and red for sensor’s ranges of enemies. The combi-nation with red and green can be problematic for a person with colorblindness. The controls were checked in a color blindness simulator during the implementation. The visualization would appear different but the controls should still be possible to use for a person with color
4.9. Evaluation of High Fidelity Prototype, Iteration 2
4.9
Evaluation of High Fidelity Prototype, Iteration 2
The evaluation was conducted as eight user test sessions with the procedure presented above in section 4.2. Six test participants were domain experts with between 6.5 and 35 years of experience in the domain, the median was 20 years of experience. The other two test partici-pants were master students in their final year in software and computer science with compe-tence in interaction design and usability development. The participants can be seen in table 4.3. There was no changes of the test structure after the pilot study, the results from the pilot study were not included in the results of the study.
User Title Domain Experience Earlier involved
1 Scientist, Domain Expert 6.5 years Yes
2 Scientist, Domain Expert 9 years Yes
3 Scientist, Domain Expert 17 years Yes
4 Scientist, Domain Expert 35 years Yes
5 Scientist, Domain Expert 23 years No
6 Scientist, Domain Expert 30 years No
7 Software engineer student No
8 Software engineer student No
Table 4.3: Participants in evaluation, iteration 2
During the test, different scenarios were presented and different tasks were given. A new scenario was added in this evaluation, Scenario 4 with three ships. According to Bryman [46] the questions should be sorted in a logical order. The new fourth scenario used the same visualization as scenario 1 and 2. To get a logical order scenario 4 was performed before Scenario 3. The tasks and questions were developed based on the theory about usability testing in complex domains and in geo-visualizations. This last evaluation was a verifying test to see how the users perceive and work with the visualization. It included some higher level questions about the understanding of the visualization. Koua’s approach [27] with exploratory tasks based on different conceptual visualization goals was used. For example to identify or relate something in the visualization to a given problem.
The scenarios, tasks and question asked are presented here:
Scenario 1 - Two Ships
• What conclusions can you make from the situation?
• Identify the combination of own radar height and LBM case that gives ship1 the longest and the shortest range.
• Compare how long ship2 sees at different target heights given own target height 15m and Duct.
Scenario 2 - One Ship and One Airplane
• What conclusions can you make from the situation?
• Compare how far plane1 sees at different altitudes given a target height of 20m.
Scenario 4 - Three Ships
• Identify and move ship1 on a position where ship1 sees ship2 but is difficult to be de-tected by ship3, given Super-refraction weather case.
4.9. Evaluation of High Fidelity Prototype, Iteration 2
Scenario 3 - Difference of Calculation With and Without LBM
• What conclusions can you make from the situation?
• Identify the highest target height where there the continuous difference is at least 50km. • Rank the different LBM cases according to how close they are to the normal radar
prop-agation calculation.
• Describe how the pattern changes at different heights between 0 and 50 meters, for LBM case Normal and Duct.
After the Different Scenarios
• What do you think could be innovative and valuable to bring with you when designing a new visualization?
5
Results
5.1
Pre-study
This section presents the results from the pre-study.
Scenarios
Three different scenarios were developed together with the domain experts. The scenarios correspond to the different identified situations for which visualizations were wanted, see section4.3.
• Scenario 1: Two ships, one small ship with the radar below the duct and one big ship with the radar above the duct. The scenario is presented in an illustrative figure in fig. 5.1.
Figure 5.1: Scenario 1
• Scenario 2: One ship and one airplane. The scenario is presented in an illustrative figure in fig. 5.2.
5.2. Implementation of Paper Prototype
• Scenario 3: In some scenarios, under special weather conditions, the normal radar propagation model provides incorrect data compared to a model based on radar prop-agation with local refractive index model (LBM). In this scenario the goal is to visualize this difference in a way that is easy to understand.
5.2
Implementation of Paper Prototype
This section presents the paper prototypes developed during the first phase of the project. They were based on the three scenarios set during the pre-study. For each prototype, the in-tended scenarios is sketched in the top right corner of the paper prototypes. The small sketch was not implemented in the software, but illustrated the scenario to the test participants. The paper prototypes can be seen in Appendix A.
Figure 5.3: Scenario 1, Paper Prototype
The prototypes for scenario 1 and 2 were similar to each other. The first scenario had two ships marked with different colors and can be seen in fig. 5.3. The visualization had two coverage diagrams one for each ship. A coverage diagram is a box in NetScene where something can be calculated, for example radar propagation for a certain ship and radar. The brown ship had a brown color scheme. It connected the diagram and the dialog to each other. The blue ship had a blue color scheme. The color scheme connected the diagram and the dialog. In the dialog it was possible to change the target height and the height of the own radar. Scenario 2 was similar, but had a ship and an airplane. The diagram and the dialog were connected by colors but the parameter "radar height" was changed to "height".
The next prototype presented the data in two different ways and can be seen in fig. 5.4. One coverage diagram presented the information as radar wave propagation. The other cov-erage diagram showed only the threshold for discovery ability. The wave propagation di-agram presented more information from the calculations in the LBM model, the threshold
