Linköpings universitet SE–581 83 Linköping
Linköping University | Department of Computer and Information Science
Bachelor’s thesis, 16 ECTS | Datateknik
2019 | LIU-IDA/LITH-EX-G--19/058--SE
Efficiency of a Scenario Editor
for Connectivity Between Virtual
Military Combat Systems
Effektivitet av en scenarioredigerare för kontakt mellan virtuella
militära stridsledningssystem
Liam Vilhelmsson
Tobias Mellberg
Supervisor : Jonas Wallgren Examiner : Mikael Asplund
Upphovsrätt
Detta dokument hålls tillgängligt på Internet - eller dess framtida ersättare - under 25 år från publicer-ingsdatum under förutsättning att inga extraordinära omständigheter uppstår.
Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka ko-pior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervis-ning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säker-heten och tillgängligsäker-heten finns lösningar av teknisk och administrativ art.
Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsman-nens litterära eller konstnärliga anseende eller egenart.
För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/.
Copyright
The publishers will keep this document online on the Internet - or its possible replacement - for a period of 25 years starting from the date of publication barring exceptional circumstances.
The online availability of the document implies permanent permission for anyone to read, to down-load, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility.
According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement.
For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: http://www.ep.liu.se/.
Abstract
Today people spend a lot of time trying to complete software-related assignments. Poorly designed software can waste both time and resources needed to complete a task. It is therefore important to have efficient ways to complete these tasks. The Swedish Defence Research Agency, also known as FOI has developed software to calculate whether simulated units in the terrain have radio contact or not. In the current approach the employees manu-ally writes scenario files which contains information about the contact. However, these sce-nario files quickly become very large and difficult to work with. A possible solution to this issue is creating a scenario editor where the user can use an interface to create information about the contact between the units. This thesis suggests a Military Combat System Scenario Editor (MCSSE) which is compared to writing scenario files manually. The comparison is made by first performing a number of tasks with both applications. The applications there-after evaluated using a metric called Minimal Actions Performed (MAP) which is defined for this comparison. The thesis also suggests appropriate tasks for evaluating the applications using an iterative method consisting of meetings with an expert with specified questions. By using the MAP metric, the application can be evaluated and the results show that the MCSSE is on average 66% more efficient than the current approach.
Acknowledgments
We would like to thank our examiner Mikael Asplund and our supervisor Jonas Wallgren for the feedback given during the thesis work. A special thanks to our supervisor at FOI, Per-Anders Albinsson, for the input on the proof of concept and the time spent when selecting tasks. Finally, we would like to thank Jacob Nilsson and William Tyrsing for commenting on our work.
Contents
Abstract iii
Acknowledgments iv
Contents v
List of Figures viii
List of Tables ix 1 Introduction 1 1.1 Background . . . 1 1.2 Motivation . . . 3 1.3 Aim . . . 3 1.4 Research questions . . . 3 1.5 Method Overview . . . 4 1.6 Report Overview . . . 4 1.7 Terminology . . . 4 2 Theory 5 2.1 Scenario Editor . . . 5 2.2 Usability . . . 6 2.3 ISO 9241-11 . . . 7 2.4 Purpose of Usability . . . 7 2.5 Usability Evaluation . . . 8 2.5.1 Contextual Inquiry . . . 9 2.5.2 Usability Inspection . . . 9 2.5.3 Usability testing . . . 9
2.5.4 Earlier Comparison of Usability Evaluation Methods . . . 10
2.6 Measuring Efficiency . . . 11
2.7 Message broker . . . 12
2.8 GUI Class Libraries . . . 13
2.8.1 Windows Forms . . . 13
2.8.2 Windows Presentation Foundation . . . 13
2.8.3 QT . . . 13
2.9 Manual Scenario Editing Application . . . 13
2.9.1 Overview . . . 13
2.9.2 Scenario File Structure . . . 13
3 Creating the Scenario Editor - Proof of Concept 15 3.1 GUI Class Library Selection . . . 15
3.3 MCSSE Features . . . 16
3.3.1 Military Unit Hierarchy . . . 16
3.3.2 Scenario Creation . . . 17
3.3.3 Scenario Library . . . 17
3.3.4 Scenario Saving/Loading and Loading a Configuration File . . . 18
3.3.5 Connection Representation . . . 19
4 Evaluating the Scenario Editor 21 4.1 Efficiency Measurement . . . 21
4.1.1 Work Prerequisites . . . 21
4.1.2 Choice of Metric . . . 21
4.1.3 Defining an Action . . . 22
4.2 Task Selection Method . . . 22
4.3 Choosing an Usability Evaluation Method . . . 24
4.3.1 Earlier Comparison . . . 24
4.3.2 Requirements . . . 24
4.3.3 Selection . . . 25
4.4 Comparison . . . 25
5 The Chosen Tasks 26 5.1 Iteration One . . . 26
5.1.1 Primary Set of Tasks . . . 26
5.1.2 Review Tasks . . . 27
5.2 Iteration Two . . . 28
5.2.1 Edited Set of Tasks . . . 28
5.2.2 Review Tasks . . . 28
5.3 Selected Tasks . . . 28
6 Efficiency Result 31 6.1 Actions Required for Performing Tasks . . . 31
6.1.1 Task 1.1 . . . 31 6.1.2 Task 1.2 . . . 32 6.1.3 Task 2.1 . . . 32 6.1.4 Task 2.2 . . . 33 6.1.5 Task 3.1 . . . 33 6.1.6 Task 3.2 . . . 35
6.2 Efficiency Improvement Summary . . . 36
7 Discussion 37 7.1 Results . . . 37 7.1.1 Tasks . . . 37 7.1.2 Efficiency Result . . . 37 7.2 Method . . . 38 7.2.1 Usability Evaluation . . . 38 7.2.2 Tasks . . . 39 7.3 Sources . . . 39
7.4 The work in a wider context . . . 39
8 Conclusion 41 8.1 Result . . . 41
8.2 Future Work . . . 42
List of Figures
1.1 A scenario with two groups with three units each. In this scenario every single
unit have contact with each other. . . 2
1.2 A scenario with two groups with three units each. In this scenario the two groups loses contact to each other. . . 2
2.1 Simplified scenario editor that shows connectivity between nodes. . . 6
2.2 A sequence model of a contextual inquiry method. . . 9
2.3 A JSON file structure of two network events where the radio component with the name vf1.radio1.si1.bms.mil can contact vf1.radio2.si2.bms.mil and vf2.radio2.si2.bms.mil can contact vf2.radio1.si1.bms.mil. . . 14
3.1 An overview visualization of the MCSSE. The numbers in the figure follows the same order as the subsections of Section 3.3. . . 17
3.2 An enlargement of a military unit hierarchy. . . 18
3.3 A scenario file that has been saved in to a text file with a JSON format. . . 19
4.1 The iterative method for designing the tasks for the MCSSE. . . 23
5.1 The military unit hierarchy for group two. The dotted circles represents branches in the hierarchy and the quadrilaterals represent one or multiple units. . . 29
5.2 The military unit hierarchy for group three. The dotted circles represents branches in the hierarchy and the quadrilaterals represent one or multiple units. . . 30
6.1 An illustration of the amount of actions in relation to the amount of military units. The x-axis shows the amount of military units and the y-axis shows the amount of actions. The bars are presented with all military units having two-way contact with each other on one waveform. The red bar represents the MSEA and the black bar represents the MCSSE. . . 36
List of Tables
2.1 A summary of different usability evaluation methods. Table adopted from work made by A. Seffrah and E. Metzker’s [Seffah2009]. . . . 8 2.2 A summary of the usage area for the usability evaluation methods and what
ad-vantages they posses. . . 11 3.1 An overview the GUI class libraries’ properties which was presented in Section 2.8 16 4.1 An overview of what properties the usability evaluation methods has. . . 24 5.1 A reviewing of the first iteration tasks based on the question established in Section
4.2 . . . 27 6.1 A summary for the presented tasks with an efficiency comparison between them.
The comparison formula is as follows: 1 ´(MCSSE/MSEA). Where MSEA and MCSSE is represented by their MAP value. A positive value indicates that the MCSSE is more efficient and likewise for the MSEA with a negative value. . . 36 A.1 The tasks found for iteration one. . . 46 B.1 The final tasks found. . . 49
1
Introduction
This report concludes a bachelor thesis work within the bachelor of science in computer engi-neering at Linköping University. The work has been conducted at the Swedish Defence Research Agency, also known as FOI, in Linköping. FOI is a public authority that works towards a safer world and is Sweden’s primary strategic research institution for defence and security.
1.1
Background
A department at FOI in Linköping has developed software to calculate whether units in the terrain have radio contact or not. In this case, a unit represents a simulated military unit such as a vehicle or a person, carrying components such as GPS or a radio. These simulated units can have radio contact with each other, provided that they both have radio components. To get a better understanding why and when different units gain or lose radio contact to each other, simulations are done. These simulations are a way to test large and complex scenarios before applying them in the real world with actual people and equipment. So instead, these simulations consists of different kinds of units represented in a virtual clus-ter which then can test radio contact between different units. The simulations consist of a scenario which is explained by one or multiple events. An event explains a single contact between two simulated units and a set of multiple events will therefore results in a scenario explaining many different simulated units and their contact to each other. For instance, imagine several units which belongs to two different groups where all units have contact with each other as seen in Figure 1.1. A scenario could in this case be that the two groups loses contact to each other as seen in Figure 1.2. The events contained in the scenario would in this case explain the connection between two units. Since there are multiple units losing contact with each other, the scenario has to contain multiple, but similar, events explaining how the unit no longer have contact to each other.
The simulations result in a file which includes the connection status between the units every second. The files can be used by other researchers that focus a lot on testing how, especially military, IT-systems behave when the contact between the units vary in quality. Today it exists software to repeatedly test different scenarios loaded from a scenario file. These scenario files consist of multiple events such as one unit losing contact with another.
1.1. Background
Figure 1.1: A scenario with two groups with three units each. In this scenario every single unit have contact with each other.
Figure 1.2: A scenario with two groups with three units each. In this scenario the two groups loses contact to each other.
1.2. Motivation
1.2
Motivation
Today employees spend time trying to complete different software-related tasks, such as writing a document. This can be very time consuming depending on what software is used. Poorly designed software can waste both time and resources needed to complete a task. The resources needed to complete a task in relation to the achieved result can also be described as efficiency, which is an important aspect to think of. Efficiency also closely relates to the usability of a system since a more usable system will make a user feel satisfied and motivated to use the software. A more usable and efficient system could potentially lead to businesses saving a lot of time and resources when a system is used. Furthermore, for systems being sold, a more usable system can have a competitive advantage over other systems [1].
It is currently difficult and inefficient to test different kinds of events that can partake in a scenario. To see how the connection status is changed when, for example, one unit loses contact to another, a text files have to manually be loaded into the program. These text files are called scenario files and consists of multiple events each represented by one line of text. These events could, for example, be one unit losing contact to another. This could result in er-rors and a considerably amount of time being wasted to test the quality of a single connection since the whole scenario has to be resimulated. Furthermore, the scenario files might have to be rewritten even though only one of the events in the scenario is changed for test purposes. The scenario files become very large when many connections are required or when there are a lot of units involved. It is very time-consuming to manually write the scenario files and when they become extremely large it is unfeasible to rewrite them. Users also require a lot of knowledge about the scenario file structure, which at the time not many employees at FOI have. To solve this issue, the efficiency of a scenario editor is to be evaluated and compared to the current approach used by FOI. This to make it more efficient to create scenarios, thus giving FOI the ability to more quickly create and test these scenarios.
1.3
Aim
The purpose of the work is to see how efficiency can be affected with a scenario editor. In order to evaluate the efficiency, a scenario editor is created to quickly and repeatedly be able to test different simulated scenarios without having to create different scenario files. The scenario editor is then compared to the current approach of manually writing text files. Fur-thermore, the aim of the scenario editor is also to streamline the creation of different scenarios and thus making it more efficient and user-friendly to test different events that can occur in a scenario. The scenario editor consists of a control panel where the user can load and save scenarios. It also supports a visual presentation of the simulated units, where the user can make changes to the radio contact between them.
1.4
Research questions
This study investigates to what extent a scenario editor can improve the efficiency of scenario creation. This study also looks into what tasks could be used to evaluate the efficiency of the scenario editor. The result of this study is presented in chapter 5 and 6, which will be answering the two following research questions respectively:
1. What set of tasks can appropriately be performed by a user to evaluate the efficiency of a scenario editor?
2. What is the change in efficiency when creating, saving, editing and playing text based events with the suggested scenario editor, compared to doing it manually, measured by the presented minimal actions performed metric?
1.5. Method Overview
1.5
Method Overview
The method chapters introduces the proof of concept called Military Combat System Scenario Editor (MCSSE) which is used for a comparison in efficiency with the current approach. The design choices for the MCSSE is presented, motivated and its features are explained in de-tail. To measure the efficiency in the applications a efficiency metric MAP is defined which is based on the Minimal actions metric. An action in the efficiency metric is then defined and motivated, an action could, for example, be playing a scenario file, etc. To realistically evalu-ate the amount of actions between the applications a task selection method was introduced. The method goal was to present a set of tasks that are common in the everyday work, to achieve this an iterative method was chosen. The iterative method consists of creating and reviewing tasks with an expert in the area. Then an usability evaluation method is chosen to perform the selected tasks. Finally, to extract the most efficient application both of the tools were compared directly to each other.
1.6
Report Overview
The report has the following structure: Chapter 2 presents the theory needed to answer the research questions which includes terms such as usability and efficiency as well as different methods to evaluate them. Furthermore, the chapter presents an overview of FOI:s current approach to create scenarios. Chapter 3 contains the method that was used to create the scenario editor, it also presents an overview of the created scenario editor for the military combat system. Thereafter the method for evaluating the efficiency as well as a method for choosing tasks is presented in Chapter 4. Chapter 5 contains the result of the selection of tasks using the iterative method described in Chapter 4. Chapter 6 shows the efficiency result when comparing the two applications. 7 and 8 shows discusses the work and concludes the study.
1.7
Terminology
Unit: A simulated military unit in a virtual cluster which can contain different kinds of components. These components could, for example, be a GPS or a radio. These simulated units can have radio contact with each other, provided that they both have radio components.
Event: Some action that will be executed on the simulated units. This could, for in-stance, be one unit losing radio contact to another. These are represented by one line of text consisting of an event type and different kinds of parameters.
Scenario: A scenario consists of a set of multiple events that will be executed on simu-lated units. It can describe all the radio contacts for a set of simusimu-lated units.
Task: A task consists of multiple steps where each step can be calculated as actions with the MAP metric.
2
Theory
In this chapter key concepts and terms are presented in order to give a broader understanding of the subject that is being analyzed in this report. This includes basic concepts such as usabil-ity and efficiency, coupled with different methods and metrics to measure them. This chapter also contains prior and related work to give a better overview and a deeper understanding of the topic.
2.1
Scenario Editor
A scenario editor is an application where the user can define different scenarios to easily test them. A scenario editor can be used to create scenarios, for example, flight traffic, network connectives and circuit connectivity, etc. A simplified example of how a scenario editor works can be seen in Figure 2.1. In this study, the scenario editor is used to create scenarios for connectivity between military combat systems.
There are some previous studies that explore this type of application, however, the ben-efits of creating scenario editors are barely described in previous work. Most of the previous work only contains the method of implementing one or confirming that a scenario language can be used. S. Jafer et al. [10] presents a flight scenario editor using Aviation Scenario Defini-tion Language (ASDL). The only conclusion the author drew from that study was that normal flight landing scenario could easily be created using their proof of concept with the ASDL framework. Moreover, the report mostly contains information about the development of the scenario editor.
A scenario editor is also commonly used in reports to assist other research questions. For instance, in D. Lafond and M. D. DuCharme [15] work where a scenario editor is used to create a counter-insurgency scenario which then can be used by complex decision-making platform. Their study draws a conclusion about the decision-making platform while only using the scenario editor for creating scenarios. Likewise, the scenario editor in this study is used to draw conclusions about usability aspects and how scenario editors can be made to potentially increase efficiency.
2.2. Usability
Figure 2.1: Simplified scenario editor that shows connectivity between nodes.
2.2
Usability
Efficiency is a component of usability and the method used in this study to measure efficiency is based on methods used to measure usability. The following sections 2.2 – 2.5 will therefore present usability, why it is important and ways to measure it.
The word usability can be referred to as "ease of use", however, the term can not merely be trimmed down to that. Many definitions of usability have been declared not to mention the different ways of measuring it [24]. These definitions of usability can be used broadly with different meanings, for example, one definition made by J. Nielsen [16] defines the usability by five components:
• Learnability: How easy is it for users to accomplish basic tasks the first time they en-counter the design?
• Efficiency: Once users have learned the design, how quickly can they perform tasks? • Memorability: When users return to the design after a period of not using it, how easily
can they re-establish proficiency?
• Errors: How many errors do users make, how severe are these errors and how easily can they recover from the errors?
• Satisfaction: How pleasant is it to use the design?
Many definitions are similar but with small differences. The usability definition literature survey made by A. Rana [21] shows that the most used component of usability definition is learnability shortly followed by satisfaction. Yet, many components had almost as many mentions as learnability. Even though there are many definitions, many of them lean towards the same concept, that the user should feel satisfied and feel motivated to use the application. As the concept matured, one ISO-standard is commonly referred to, which is the ISO 9241-11 [1]. This standard will be used as the primary source for key concepts and definitions of usability and is described more briefly in Section 2.3.
2.3. ISO 9241-11
models are assessed. As previously mentioned there are many methods to measure the usability. Seffah et al. [24] argue that this can cause confusion when usability is evaluated. For this reason, the authors present a consolidated model called Quality in Use Integrated Measurement (QUIM) which aims to be a complete model of usability definitions and mea-surement. The model strives to give clearer communication between developers and experts when it comes to usability, but also a ground for comparing different methods and metrics. The QUIM model is a hierarchical model which includes 10 different properties decomposed into 26 different criteria where each property is mapped to one or more criteria. For instance, the efficiency property can be measured with the criterion time behavior, which measures how well appropriate times can be achieved when performing tasks [24]. Each of these criteria can then be calculated with several metrics. The model includes efficiency as a property as well as the criteria mapped to the property, these are explained further in Section 2.6.
2.3
ISO 9241-11
The most updated version which is referred to in this study is the ISO 9241-11:2018 [1] which replaces the previous and similar version ISO 9241-11:1998. This standard is commonly referred to when it comes to usability and it also acts as a baseline for models such as the previously mentioned QUIM model [24]. Usability, as explained by the latest ISO 9241-11 standard [1], is the degree of which a system can be used to solve and complete tasks while also considering user satisfaction as well as the effectiveness and efficiency of the task solv-ing. This standard describe usability as an outcome of using a system and the usability can differentiate depending on the context of use. The user’s background and knowledge, as well as the intended goal of the task, will all influence the usability of the system [1]. Therefore the result of usability can vary a lot for the same system since it all depends on what environment the system is used in.
Usability consists of properties such as effectiveness, efficiency, and satisfaction. These properties are all independent and do not affect each other. This means that a user, for instance, can use some type of software efficiently without being satisfied using it. The properties are according to the ISO-9241 standard [1] defined as following:
• Effectiveness: To what accuracy and completeness a task can be performed on a system. This could be seen as the fraction of tasks the user is able to complete but also how well the intended outcome match the actual outcome.
• Efficiency: The amount of resources needed from the user to be able to complete the task in relation to the achieved results. This could be things such as time or effort spent on completing the task as well as the amount of material resources used. Therefore, using less resources to complete a task gives higher efficiency.
• Satisfaction: Satisfaction represents the response that the user leaves when using the system. This could for instance be things like having preferences or feeling discomfort when using a system, as well as other physical or emotional responses.
These definitions focus on the user and look at usability as a result of using the system and the same principles can be applied when looking at the usability of a system. The better prerequisites that exist for a system to be usable, the better experience the user will have using it.
2.4
Purpose of Usability
The main purpose of usability is often to reduce the cost, more specifically the maintenance and training cost [25]. Furthermore, it is also used to improve the performance and accuracy
2.5. Usability Evaluation
of system tasks. The improvement gained by usability also affects the human perception of the program which will lead to an increase in the human productivity [25].
B. W. Boehm [4] explores the risk when creating and maintaining software. Boehm explains that the exposure of risk as a relation between the probability of an insufficient outcome and the loss if it occurs. The author identifies some items that are at risk of being exposed and among those are creating wrong user interfaces as well as wrong functions and properties. Further, the author conducts an experiment on software handling satellites which shows that there is a high-risk exposure of operation being made wrongfully due to bad user interfaces [4]. Poorly designed systems will make for a higher probability of insufficient usage, not to mention the potential losses that comes with it [4].
C. Karat [13] illustrates the trade-offs from using techniques to improve usability. The author conducts an experiment to estimate the savings that can be made using usability improving techniques. The experiment includes two internal applications, one small and one large with 27 respectively 69 test participants. Each application has three iterations of usability test, where the usability improvement happens between the iterations. The time difference between each iteration together with the end user population gave an estimate of the dollar savings. For smaller projects of around 23 000 users, the potential savings were $41.700 and the savings from using techniques to improve usability will be two times higher than the cost [13]. For larger projects with around 240 000 users, these numbers were a lot higher, with savings of $6.800.000 and with savings being 100 times higher than the cost of it [13].
Increasing usability in systems does not only lower the probability of insufficient use as earlier explained by Boehm [4], but it also saves a lot of resources for companies in terms of work hours spent on using applications [13]. A more efficient application would essentially lead to less work being spent on performing tasks and would for this reason benefit the user.
2.5
Usability Evaluation
Involving end users is essential when creating a usable and efficient system. There are many different ways to evaluate the usability of an application, J. Nielsen [17] presents four ap-proaches: automatic, empirical, formal and informal. Automatic is measured by executing a user interface through a program. Empirical is testing the program with real end users. For-mal uses formulas to calculate the usability. Finally, inforFor-mal which evaluates based on the experience of evaluators. The author also states that empirical methods are the most common way of evaluating usability, but that the different evaluation methods combined will identify the most usability issues. This report choose three different empirical evaluation methods from A. Seffrah and E. Metzker’s [25] work which are concluded in Table 2.1. Contextual in-quiry and usability inspection requires user participants where the user tests the application, whereas the inspection methods only requires the user to inspect the application.
Table 2.1: A summary of different usability evaluation methods. Table adopted from work made by A. Seffrah and E. Metzker’s [25].
Name Evaluation
Contextual Inquiry Communicating or observing users that uses the system. The users may also answer forms verbally or in text.
Usability Inspection Users or experts examines the usability in an interface. Usability Testing Observes and analyzes users that works on predefined tasks.
2.5. Usability Evaluation
2.5.1
Contextual Inquiry
Contextual inquiry is an informal data extracting method, as shown in Table 2.1 it is done by observing and questioning the participant users. Contextual inquiry testing typically consists of interviews, focus groups and surveys. The focus is on extracting information from the users which can be conducted by the sequential steps as illustrated in Figure 2.2. The observation allows the evaluators to gain an ongoing user-experience and a full view of the program rather than an experience within a certain restraint.
The methodology is based on three principles [22]. Firstly, "Data gathering must take place in the context of the users’ work". Secondly, "The data gatherer and the user form a partnership to explore issues together". Lastly, "The inquiry is based on a focus; that is, the inquiry is based on a clearly defined set of concerns, rather than on a list of specific questions (as in a survey)". The purpose of using a contextual inquiry is to see different perspectives from the same information set. Moreover, the use of this method is based on that the evaluators and users are equal in their different interpretations of the program. In order to assert equality, the interviewer does not lead the conversation with topics and instead asks questions like "why did you do that?". Likewise for the conducted surveys, as mentioned in the third principle.
2.5.2
Usability Inspection
Usability inspection is a common name for methods that evaluates a user interface. The inspection methods can differ and some are used to find usability issues in an interface while some also acquire information about issues from a whole design [19]. The inspection methods are used in the same manner as a developer uses a debugger. The debugger helps the developer locate the bug and improve the code whereas the inspection method helps the developer locate and improve the usability problems.
Inspection methods are relatively cheap in relation to usability testing methods and are often used while a program is being developed [17]. An example of an inspection method is a pluralistic walkthrough. A pluralistic walkthrough is a brainstorming meeting where a scenario is reviewed successively while discussing each sequence [17]. Even though usability inspection methods are cheaper than usability testing it can also be superior. R. Jeffries et al. [11] presents that when comparing four usability evaluation methods (heuristic evaluation, usability testing, guidelines, cognitive walkthrough) the heuristic evaluation found most problems.
2.5.3
Usability testing
Usability testing is an evaluation method where the estimation is based on user testing and gives direct input on user experience. The user testing consists of different tasks for a prod-uct’s intended function, the actual testing is conducted within a controlled environment. The different tasks are carefully planned for the product’s intended purpose during the execution
2.5. Usability Evaluation
of this scenario, the user participant is being observed and evaluated. For an improvement in terms of evaluation, an interview is often conducted after the scenario execution in order to better understand the participants’ motivation for their choices.
Usability testing is similar to contextual inquiry with regards to user involvement, the main difference is that usability testing is executed within a controlled environment with specific tasks for the product while contextual inquiry observes more of a sandbox envi-ronment where the participant uses the product freely. This makes for a good method for detecting learnability issues for the product.
A positive side of usability tests is that it does not require many participants to identify most of the usability issues. J. Nielsen [18] claims that it is not necessary to have more than 5 users, in fact, it is considered a waste of resources to have more. The author make this claim on the grounds that one single user will discover about a third of the usability problems when the next user makes a discovery it may already have been discovered by the first user. Therefore, by the fifth user most of the usability issues will already be discovered. However, more recently this statement has become disputed and in terms of web testing many does not agree with this [27] [29] [3]. J. Spool and W. Schroeder [27] mean that the context of the application dictates how many participants are needed for usability testing. Moreover, the study says that five participants are not enough for web testing.
2.5.4
Earlier Comparison of Usability Evaluation Methods
The different approaches to evaluate usability can cover different development stages and it does not have a standard comparison operator. Unfortunately, this makes it difficult to make a direct comparison. W. D. Gray and M. C. Salzman [7] review papers about usabil-ity evaluation method comparison and the authors reach the conclusion that the reviewed material shows small problems in terms of the conclusions of the papers’ trustworthiness. Nevertheless, a paper by C. Karat et al. [14] is reviewed by Gray and Salzman [7]. The paper in question is summarized with "none fatal flaw" and therefore considered applicable in this study.
R. W. Bailey et al. [2] present issues with a usability inspection method such as heuristic evaluation and presses the importance of usability testing. The issues found by the authors associated with usability is that heuristic evaluation found 43 potential changes for usability improvement whereas usability testing found two possible improvements. Usability testing can also be used with fewer participants if following J. Nielsen’s standard "five users are enough" [18]. As earlier mentioned usability inspection can be used during the develop-ment phase since it does not require any users to test the program. Moreover, the usability inspection provides a cheaper alternative compared to usability testing [17]. C. Karat et al. [14] discover that a walkthrough which is a usability inspection method misses severe issues with usability that usability testing found. The authors also found that usability testing provides a more cost-effective method, meaning that usability testing takes the same or less time than the walkthrough method. L. Kantner et al. [12] present directions on when to use a condensed contextual inquiry and when to use field usability testing, these terms are fairly similar to contextual inquiry and usability testing. Condensed contextual inquiry is a more constrained method with fewer issues to examine while field usability testing provides more personal tasks where the user can access their own files and is not conducted in a controlled environment. The author draw the conclusion that contextual inquiry should be used for exploring a "continued use", for example, as how users use the product and identify new features. Field usability testing should on the other hand be used when the "ease of use" is examined.
2.6. Measuring Efficiency
Table 2.2: A summary of the usage area for the usability evaluation methods and what ad-vantages they posses.
Name Usage Area Advantage
Contextual Inquiry
Exploring "continued use" since the participants are using the product in everyday situations.
Gathers direct input from the participants information as well as discussing different solutions to the problems during inter-views
Usability In-spection
Create a usable design during the development phase.
Cheaper than usability testing and in some cases can find more usability problems.
Usability Testing
Find usability problems and learnability issues.
More cost-effective than usabil-ity inspection and finds severe problems that the inspection method missed.
The comparison between the usability evaluation methods will lay the ground for the evaluation decision together with identified related categories for this work, this is presented in Section 4.3. The comparison is summarized in the Table 2.2.
2.6
Measuring Efficiency
There are different methods to measure usability as illustrated in Section 2.5. The methods only explain ways to systematically gather data about users integrating with an application, but there is also a need for metrics to measure the data gathered. Different metrics to measure usability has been presented and the same goes for measuring efficiency.
As previously mentioned in Section 2.3, efficiency is a measure of how much resources a user needs in relation to the achieved results. By this definition it is therefore better to have a low efficiency because it means less resources. Efficiency metrics are said to be used when comparing different products against each other to see which is more efficient [9]. Some met-rics to measure this are presented in the ISO/IEC 25022:2016 [9] standard which replaces the previous ISO/IEC TR 9126-4:2004 standard. The metrics presented in ISO/IEC 25022:2016 [9] standard are as follows:
• Task time: This metric measures the time that it takes for a user to complete a single task. If the time that it takes to complete task is called T then the task time would be X is explained by the expression X=T. A lower X indicates higher efficiency.
• Time efficiency: Time efficiency is similar to the task time metric but includes multiple objectives to reach a goal. If O is the number of objectives completed and T is the time it took to complete the objectives then the time efficiency E= O
T.
• Cost-effectiveness: Cost-effectiveness explains the cost per objective. The total cost C for achieving a goal could, for example, be hours spent by the user, cost for materi-als, computing resources etc. O is again the number of objectives achieved. The cost-effectiveness CE would in this case be CE= C
O. A lower cost-effectiveness will in turn indicate higher efficiency.
• Productive time ratio: This is the fraction of the time that the user actually per-form productive actions of the total time spent on a task. The productive time Tp = Time to complete the task ´ (Hel p or assistance time + Error recovering time +
2.7. Message broker
Ine f f ectual searching time). The total time Ttis given by the total time that it takes to complete a task. This gives the productive time ratio PTR = Tp
Tt. A higher PTR is therefore desirable since it indicates higher efficiency.
• Unnecessary actions: Another way to measure efficiency is to look at what fraction of actions that did not have to be made to complete the task. This is useful when, for example, there are multiple options and the user have to select different things. The number of unnecessary actions can be denoted as Auand the total number of actions as At. The fraction of unnecessary actions can then be calculated with U A= Au
At. A lower value of U A will give higher efficiency since less unnecessary actions are performed. • Consequences of fatigue: Lastly the efficiency can also be measured as a consequence
of fatigue, meaning that a users current performance Pc can be measured against the initial performance Piof the user. Performance would in this case mean some measure of effectiveness or efficiency. The consequences of fatigue would thus be explained from the expression COF = 1 ´Pc
Pi. Lower consequences of fatigue therefore gives higher efficiency.
As previously mentioned in Section 2.2 the QUIM model presents different properties with different criteria to measure usability [24]. There are a total of 26 different criteria for usability and eight of these can be used when measuring efficiency. These are explained by Seffah et al. [24] to be as follows:
• Time behavior: The ability to perform tasks within an appropriate task time.
• Resource utilization: The ability to perform tasks while using the appropriate amount of resources.
• Minimal action: How well a software product can be used to complete a task with the least amount of steps.
• Minimal memory load: A measure of how much information a user needs to memorize when completing a task.
• Operability: How much effort that is needed to operate and use the software. • Feedback: How well the software responds to user input or other kinds of events. • Navigability: A measure of how efficient a user can move around in the application. • Load time: Time for the application to load, in other words how fast the application
responds to user actions.
There are many ways to measure efficiency and there are many ways the scenario editor, that will be evaluated in this study, can be measured. The specific criterion used for the evaluation in this study is more deeply covered in the method chapter 4. The specific metric to measure this is also presented.
2.7
Message broker
To send events to the virtual cluster, a message broker has to be used. The message broker makes it possible to communicate with the virtual cluster that simulates the units. The mes-sage broker used in for this study’s proof of concept is an open source mesmes-sage broker called RabbitMQ [5]. RabbitMQ has support for languages such as Python, Java, Ruby, PHP, C#, JavaScript. The RabbitMQ broker was used for the fact that it was already implemented on the receiving side, that is the virtual cluster.
2.8. GUI Class Libraries
2.8
GUI Class Libraries
A GUI class library provides a user interface for producing rich client platforms. A GUI class library provides features such as drag and drop functionality with buttons, labels, etc. This section describes three different GUI class libraries, the logic behind the choice as well as the selection of library.
2.8.1
Windows Forms
Windows Forms (WinForms) is based on .NET Core Framework and uses the C# program-ming language [28]. WinForms offers elements with attributes such as size, font, color, etc. Moreover, the elements offers a drag and drop functionality which can be used to drag ele-ments into the screen.
2.8.2
Windows Presentation Foundation
Windows Presentation Foundation (WPF) is based on .NET Core Framework and C# pro-gramming language [8]. It uses DirectX and a markup language called XAML. XAML is used to define the rendering of the user interface like elements and animations. The new approach to render user interfaces makes WPF the informal successor to WinForms.
2.8.3
QT
QT is a cross-platform application framework and uses the programming language C++ to create graphical interfaces [20]. QT supports various functionality such as drag and drop, data storage, network connectivity, 2D and 3D graphics, etc.
2.9
Manual Scenario Editing Application
This section describes the current approach for manually testing different scenarios. It de-scribes an overview of the application as well as the structure of the text file. This is pre-sented for the reader to understand the current approach and provide knowledge of how the efficiency will be measured in the application.
2.9.1
Overview
The Manual Scenario Editing Application (MSEA) can simply load scenario files which contain text-based events, transform it and output it to a virtual cluster. However, the scenario files need to be manually written, the text files contain events where each line represents one event. Since the application does not provide a usable interface for scenario editing, testing different scenarios requires manually manipulating the text file. This can cost a lot of time because of mistyping and other usability issues.
2.9.2
Scenario File Structure
The structure of the scenario file is fairly complicated and previous knowledge is required before writing these files. The scenario files are in JSON format which is a commonly used standard for communication. The JSON format has a mapping system that maps a key to a value. An example is "firstname" as key with a first name as the value, "firstname" is then an identifier for the name. A scenario file consists of one or multiple events, the events can only be of the type network_quality.
A network_quality event has the following JSON structure with the keys declared to the left of the ":" and value to the right, an example of a scenario is demonstrated in Figure 2.3.
2.9. Manual Scenario Editing Application
Figure 2.3: A JSON file structure of two network events where the radio component with the name vf1.radio1.si1.bms.mil can contact vf1.radio2.si2.bms.mil and vf2.radio2.si2.bms.mil can contact vf2.radio1.si1.bms.mil.
• event: The event type, in this case network_quality. • from: The military radio which sends data.
• to: The military radio which receives data.
• max_bw: The maximum bandwidth in which the sender can send information. • speed_rate: The fraction of bandwidth that is being used.
• packet_loss_rate: The fraction packet loss when sending packets. • delay: The delay of the data being transmitted.
3
Creating the Scenario Editor
- Proof of Concept
This chapter contains an overview of the Military Combat System Scenario Editor (MCSSE). This chapter includes the decision of the selection of graphical user interface (GUI) class library, how it was implemented and how it was used. This will give the reader insight into how the editor was used and evaluated in terms of efficiency.
3.1
GUI Class Library Selection
In order to select a GUI class library some requirements and preferred properties of the library had to be established. Countless libraries exist, the libraries that were selected was based on good reputation and personal preferences of the authors of this paper. The GUI:s are fairly similar to each other, however, the language in which they are programmed differ as well as ease of use. The properties that the GUI class library should posses are reported in the list below.
Required properties:
• The programming language must be one of the following ones: Python, Java, Ruby, PHP, C#, JavaScript, Go, Elixir, Objective-C, Swift, Spring or AMQP
Preferred properties:
• Has the drag and drop functionality • Has a fast learning curve
The programming language requirements are set because the message broker, RabbitMQ has thoroughly documented support for these languages. The preferred properties were established to determine the GUI class library’s ease of use as well as its learning curve. Since the MCSSE was used to evaluate the efficiency of creating scenarios and because of a limited time frame, the time to learn new technologies and to implement the tool were deprioritized. The drag and drop, meaning the functionality of dragging elements and dropping them to create the GUI, provided a lower time for developing a GUI for the MCSSE. The easy to learn aspect provides less time and effort for learning the GUI class library.
3.2. Overview of the MCSSE
Table 3.1: An overview the GUI class libraries’ properties which was presented in Section 2.8
.
Library Approved programming language Drag and drop Easy to learn
WinForms
WPF
-QT
-WinForms and its successor WPF uses C# as a programming language which is approved in terms of RabbitMQ documentation. Even though WPF is the successor to WinForms it can be more difficult to learn. This is due to the XAML markup language that is used to describe the elements, while WinForms uses a simpler drag and drop. XAML has several advantages such as that its functionality is commonly used, however, this is out of the scope for this thesis and will therefore not be described any further. QT does use a drag and drop and is also considered easy to learn, on the other hand, it uses C++ which is not an approved programming language because of the lack of documentation. WinForms uses drag and drop functionality and is considered to be an easy library to learn. WinForms is the only library that has all the required and preferred properties, the result of the comparison between the GUI class libraries’ can be seen in Table 3.1. WPF and WinForms also supports the choice in terms of the learning curve and WPF is said to provide a more flexible solution although it can require more work [30].
3.2
Overview of the MCSSE
The MCSSE contributes to a usable interface for making scenarios. The MCSSE has the same functionality as the earlier approach, the MSEA. Additionaly, the MCSSE has a new function-ality to save scenarios as well as a visual representation of the scenarios and the military unit hierarchy in which the radios are presented. The saving feature was implemented to make it more efficient when the user wants to redesign an already created scenario. The visual repre-sentation of the military unit hierarchy allows the user to use drag and drop for the choosing of radios. For instance, a group of units could be dragged and dropped which would get all of the radios that are contained within the group. The functionality of the MCSSE makes the user able to not worry about the structure of the scenario files shown in Section 2.9.2. This is because the MCSSE provides a usable interface. A visual illustration of MCSSE can be seen in Figure 3.1.
3.3
MCSSE Features
This section explains the different features that the MCSSE provides in details. The features explained is a military unit hierarchy, scenario creation, scenario library, scenario saving, and connection representation. The goal of the features is to provide an application that can produce the same result as the MSEA, yet in a more efficient way. The following subsection shows the features of the tool with the subsections being in the same order as the numbers in Figure 3.1.
3.3.1
Military Unit Hierarchy
The military unit hierarchy feature provides a visual representation of the hierarchy in the form of a tree structure. An example of a military unit hierarchy can be seen in Figure 3.2 which is an enlargement of number one in Figure 3.1. The example shows that a platoon has two units, in this case vehicles. These units in turn have two radios each. The radios can transmit at different waveforms (wf ). This feature makes it possible for a user to use drag and
3.3. MCSSE Features
drop to select a branch in the hierarchy, this can be used to select multiple vehicles with one drag and drop. For example, in Figure 3.2 the user can drag vehicle1 which will choose radio1 with waveforms wf1 and wf2 as well as radio2 with waveform wf1. The feature also makes it possible for the user to visualize the structure of the hierarchy. The hierarchy is loaded into the editor with a configuration file which is further explained in Section 3.3.4.
3.3.2
Scenario Creation
Creation of scenarios consists of two steps, making events followed by saving or playing them. In the editor a scenario is named as a "playcard", this is due to the terminology used at FOI. To create a scenario the following steps are required:
1. Click the button "Create Playcard" to show the playcard menu. 2. Use drag and drop at the military unit hierarchy to select radios.
3. Choose the appropriate parameters for the link between them, the check box called double edged can make the transmission link go two ways. That is to say that the link "from" to "to" will also be set for "to" to "from".
4. Now the user can choose whether to add more events, remove events, play or save the playcard. If the user chose to save the playcard a name is required and it will be saved in the scenario library which is described in Section 3.3.3.
3.3.3
Scenario Library
The scenario library consists of saved scenarios and folders, this is visualized using a tree structure. As earlier mentioned the scenario can the possibility to be saved and will in that case be put it in the scenario library, the scenario will be saved in a selected folder, which is selected by the user. Note that a scenario can only be saved in a folder and not in another scenario. In order to make folders, another folder has to be right-clicked which will produce a menu strip with add folder and remove folder as alternatives. If add folder was selected a popup menu will appear where a name can be entered. To indicate that it is a folder a "*" is
Figure 3.1: An overview visualization of the MCSSE. The numbers in the figure follows the same order as the subsections of Section 3.3.
3.3. MCSSE Features
Figure 3.2: An enlargement of a military unit hierarchy.
placed in front of it. This creates a delimitation on the scenario name since it cannot consist of a "*" character. When a scenario is selected it will appear in the scenario selection window where it can be played.
3.3.4
Scenario Saving/Loading and Loading a Configuration File
This subsection is divided into two parts. The loading of a configuration file as well as the saving and loading of a scenario.
The configuration file consists of all information about the military unit hierarchy. When a configuration file is loaded, it will produce the military unit hierarchy in a tree structure, as mentioned earlier. The configuration files are built in another application which is in JSON format.
If the user wishes to change an already created scenario then a feature which supports saving and loading are useful in terms of efficiency. The saving feature saves the current state of the scenario library to a file. The file is formatted in a JSON structure and the required knowledge needed for the saving is a type and name if its a folder or playcard. The playcard then consists of parameters for the scenario. Figure 3.3 illustrates how a saved JSON file can be viewed. In the example, a playcard is created with one network event that is double-edged with the connection from vf1.radio1.si1.bms.mil to vf1.radio1.si2.bms.mil.
When the scenario file is loaded it is recursively read and the scenario library is filled. If the user wants to play the loaded scenario it can, as earlier mentioned, be played through selecting it at the scenario library.
3.3. MCSSE Features
Figure 3.3: A scenario file that has been saved in to a text file with a JSON format.
3.3.5
Connection Representation
The visual representation of the connectivity between the units is displayed as a matrix. Figure 3.2 illustrates that the hierarchy consists of two units, vehicle1 and vehicle2. In the overview shown in Figure 3.1 four elements exist, two black and two red. If followed left to right the figure illustrates the connections between unit zero and unit zero, unit one and unit zero, unit zero and unit one, etc. The connections are color coded and the following colors exist:
• Black: A black color means that the connection is between itself, in other words, the connection is unit zero and unit zero and the vehicle can not have connection with itself.
• Red: Red means that no connection exist between the respective unit. • Green: Green implies that there is a connection between the units.
To identify which radios that are connected and on what waveform, a tooltip is presented when the user hovers the mouse over an element. The tooltip presents the appropriate infor-mation about the connection, the same as the params variable in Figure 3.3. In the current example, there are only two units which make for four elements in the matrix. However, there can be as many as 140 units, this produces 140x140 elements. Capturing all the connection with a human eye would be impossible in this situation. B. B. Bedersom and B. Shneiderman
3.3. MCSSE Features
[26] present a mantra for visualizing large data sets called Shneiderman’s visual information seeking mantra, the mantra is presented like the following: "Overview first, zoom and filter, then details-on-demand". The mantra was followed in this work and therefore a zooming feature was implemented in the tool.
4
Evaluating the Scenario Editor
This chapter describes the method for evaluating the usability. This includes what measure-ment used for the efficiency as well as which tasks that were chosen. Firstly, an appropriate measurement for efficiency was picked in consideration of the prerequisites of the work. Sec-ondly, a set of tasks was selected to evaluate the MCSSE. Lastly, a set of requirements was established to choose one of the usability evaluation methods.
4.1
Efficiency Measurement
There are many ways to measure efficiency as presented in Section 2.6. This section handles the metric that was chosen for this study. A metric to calculate the efficiency is also presented which was inspired by prior work.
4.1.1
Work Prerequisites
Since there were some prerequisites when evaluating the tools, the method of measuring efficiency had to be thought through carefully. There were only two people who could par-ticipate if the evaluation method required users to test the application. That is users in some way interacting with the tool with a criterion such as time behavior, which uses some kind of metric to calculate some sort of time while using the application. Since this kind of metric often requires multiple users the two that could participate was not enough since there had to be at least five users to make a usability test of this kind [18]. Therefore, a metric that was not reliant on many users was chosen.
4.1.2
Choice of Metric
The appropriate metric should contain efficiency in terms of how fast the user can create scenarios. However, time could not be measured since the prerequisites of the thesis. The cri-terion Minimal actions described by Seffah et al. [24] looked promising and a similar approach was used in this study. Minimal actions only require one user to test it to observe how many actions were required for the task, this assumes that the users use the least amount of steps required for the given task. Other than the definition described in Section 2.6, Minimal actions is missing a clear description of what task that should be measured as well as the relation to
4.2. Task Selection Method
efficiency. Inspired by the criterion found by Seffah et al. [24], we present the metric Minimal Actions Performed (MAP). MAP is the minimal number of actions that has to be performed to complete any given task. The lower the MAP the higher the efficiency.
4.1.3
Defining an Action
The metric that was used to measure efficiency is the minimal number of actions that had to be performed. An action may seem relatively unclear and can have several different meanings. This subsection describes what was chosen to be seen as an action and why. The MCSSE was compared to the manually creating a scenario file which was then used in the MSEA. Since the compared tools do similar things but with different approaches, the actions performed will as a consequence also be different.
First, let’s look at manually loading and creating scenario files. To be able to create a scenario and play a scenario using the MSEA, different actions had to be performed. A single action could be any type in the following list:
• Writing an event inside a scenario file. • Loading a scenario file.
• Playing a scenario file.
Then using the proof of concept, the goal is similar, but with a different approach. With the MCSSE the user is able to create, save and play a scenario. A single action could be any type in the following list:
• Load a configuration file which includes a given hierarchy of military units. • Creating a set of events as explained in Section 3.3.2.
• Removing a set of events from a scenario. • Playing a scenario consisting of multiple events. • Saving a scenario consisting of multiple events.
• Save a scenario library consisting of multiple scenarios. • Load a scenario library consisting of multiple scenarios.
Large amount of actions can be too large to be manually written in the MSEA and therefore a formula was constructed, the formula is as follows:
A=2 ˚ n ÿ m=1
(n ´ m)
Where n represents the amount of vehicles. Note that the formula can only calculate actions of the events that yields "all units in XX has contact with each other".
4.2
Task Selection Method
The efficiency of the MCSSE is evaluated with tasks. To realistically evaluate the efficiency and compare the applications, a set of tasks that are common in the everyday work was defined. This section explains the method that was used for defining the tasks.
The most appropriate tasks type for collecting quantitative data is the assessment task [23]. The assessment task prioritizes the functionality of the program and not the users’
4.2. Task Selection Method
thoughts and experience. Assessment tasks can be used together with a comparable task, the comparable task is designed to compare different design choices. These design choices can be as simple as using a button or not as well as comparing two different applications [23]. There was no finding of any existing techniques for developing tasks for usability testing, however, some guidelines were found [23]:
• The tasks should be in a common usage area for users. • The tasks should be realistic.
• The tasks should not be completed in a stressed environment.
The most common usage areas for the MCSSE was difficult to realize because it was a newly created application. To the find appropriate tasks an iterative approach was used. The iterative approach is commonly used when the usage areas can not easily be defined. For instance, G. A. Carole [6] presents an iterative method for usability testing. The iterative approach is used to test different stages in the development since design is a hard concept to understand fully at first sight. Likewise is the concept of finding tasks for an application. The iterative method that was used in this thesis can be seen in Figure 4.1. The method consisted of meetings with an expert, which in this case also is an end-user. Some primary tasks were defined, reviewed and thereafter reconstructed for the next iteration. This method was followed until a pleasing set of tasks were defined, in other words, until the tasks could appropriately define common usage for the application. The reviewing of tasks is basically to determine if the task is in the usage area for the application, however, the scenario compli-cated and consists of several parameters. The reviewing can be broken down to the following questions:
1. Is the military unit hierarchy realistic? What can be improved?
4.3. Choosing an Usability Evaluation Method
Table 4.1: An overview of what properties the usability evaluation methods has.
Method User Testing Task Based Controlled Environment
Contextual Inquiry -
-Usability Inspection - -
-Usability Testing
2. Is the amount of events within the range of common usages?
3. Do the events represent a realistic scenario? If not what can be changed? 4. Is there anything missing in this task?
It is also paramount that the tasks utilize different features and that different amounts of events are used to draw conclusions of the efficiency effect in different usage areas.
4.3
Choosing an Usability Evaluation Method
This section describes the selection of the usability evaluation method based on the methods presented in Section 2.5.4. The presented evaluation methods compared were contextual inquiry, usability inspection and usability testing. This section also establishes requirements that the selection is based upon and finally the selected evaluation method is presented.
4.3.1
Earlier Comparison
As earlier mentioned an evaluation comparison was made by a literature study. The eval-uation is summarized in Table 2.2. Furthermore, the table consists of the usage area and advantages for the respective methods. The usage area was used for picking an appropriate method for measuring efficiency by tasks meanwhile the advantages was used to select the best suited method for the prerequisites of the work.
4.3.2
Requirements
The requirements are based on the aspects of the tasks. The different usability evaluation methods are all used for the same purpose, measuring usability, however the approaches for doing this differ. For instance, the evaluation method needs to be able to handle tasks since the method uses tasks for comparing the applications and cost is not of importance. Note that no interviews can be done since the considerable low amount of participants. The chosen requirements were:
• User Testing • Task Based
• Controlled Environment
User testing was required because to evaluate efficiency a user to test the application is needed. Task based was a requirement since the method to measure the efficiency consists of typical tasks for the tool. Finally, to measure the efficiency of specific tasks, the tasks are required to be executed in a controlled environment. This was to ensure that efficiency is measured and followed correctly.
4.4. Comparison
4.3.3
Selection
This subsection presents the choice of the usability evaluation method in regards to the requirements shown in Table 4.1. Usability testing had all the required properties and was the chosen usability evaluation method in this thesis. As earlier mentioned, usability testing requires at least five participants [18], however because of the efficiency metric MAP used in this study, this is not a problem. This is because to MAP only requires one user to test it since it only counts the minimal actions that are performed to accomplish the task. This results in a special case for the usability testing where one participant is used and is assumed to only make the least amount of steps to accomplish the task.
Contextual inquiry also includes user testing. The difference is that contextual inquiry is used for the evaluation of users normal activities. That is to say, it does not use specific tasks for the user which is vital in this thesis.
The inspection method does not use any user participants in the evaluation, provided that the method was used for evaluating efficiency made the technique unsuitable to use. Table 2.2 explains that usability testing can find severe problems that the usability inspection missed as well as contributes with a more cost-effective approach. This furthers expresses the importance of user testing since time is a limiting factor in the thesis.
4.4
Comparison
To extract the most efficient application both of the tools were compared directly to each other. The usability inspection method was applied to both of the tools with the tasks selected with the iterative method earlier displayed in Section 4.2. The efficiency from each of the tasks was counted with the Minimal Actions Performed (MAP) metric. The measurements were then compared between the two applications. MAP was used since the prerequisites of the thesis made it impossible for measurement in time and the metric was analyzed to be the best alternative approach. An action was defined as using one of the features of the MCSSE, for example loading a configuration file. Furthermore, in the manual scenario editing approach, one event in the scenario file was counted as one action. The choice for making one event counted as an action was because the it was approximated that it would require as much time as using one of the presented features. The full description of the action definition is presented in Section 4.1.3.
5
The Chosen Tasks
This chapter contains the choice of tasks. The choice was based on the iterative method described in Chapter 4. The chapter has two iterations that follow the iterative method and finally, the selected tasks are presented.
5.1
Iteration One
This section describes the first iteration in the iterative method. For the first set of tasks, we established a set of tasks based on our knowledge of the problem and application.
5.1.1
Primary Set of Tasks
The tasks are divided into three different groups where each group has a different military unit hierarchy. The first contains only 10 units, the second 50 and the third 100. The groups each have two different tasks. The three groups and their military unit hierarchy are pre-sented below:
1. A platoon containing 10 units. The platoon is called PV with the units U0...U9.
2. One battalion (BN) which consists of two companies (CO1, CO2). Each company con-sists of three (PV1.1...PV1.3) respectively two platoons (PV2.1 and PV2.2). Each platoon contains 10 units ((U1.1.0...U1.1.9)...(U2.2.0...U2.2.9)).
3. One brigade (BDE) which contains two platoons (PV1 and PV2), one company (CO1) and one battalion (BN1). The two platoons consists of 10 units each ((U1.1...U1.9)...(U2.0...U2.9)). The company consists of two platoons (PV1.1 and PV1.2) with 10 units each ((U1.1.0...U1.1.9)...(U2.2.0...U2.2.9)). The battalion (BN1) contains three companies (CO1.1...CO1.3), each with two platoons. Each of the platoons con-tains 10 units ((U1.1.1.0...U1.1.1.9)...(U1.3.2.0...U1.3.2.9)).
