accuracy from context
and sensor data
A study of how to perform data mining and knowledge
discovery in the target shooting domain
PAPER WITHIN: Computer Science AUTHORS: Viktor Jansson, Max Pettersson TUTOR: Niklas Lavesson
of Science in Engineering programme. The authors take full responsibility for opinions, conclusions and findings presented.
Examiner: Tuwe Löfström Supervisor: Niklas Lavesson
Scope: 15 hp
Date: 2019-07-24
Mailing address: Visiting address: Phone:
Box 1026 Gjuterigatan 5 036-10 10 00 (vx)
Abstract
The purpose of this thesis is to develop an interpretable model that gives predictions for what factors impacted a shooter’s results.
Experiment is our chosen research method. Our three independent variables are weapon movement, trigger pull force and heart rate. Our dependent variable is shooting accuracy. A random forest regression model is trained with the experiment data to produce predictions of shooting accuracy and to show correlation between independent and dependent variables.
Our method shows that an increase in weapon movement, trigger pull force and heart rate decrease the predicted accuracy score. Weapon movement impacted shooting results the most with 53.61%, while trigger pull force and heart rate impacted shooting results 22.20% and 24.18% respectively. We have also shown that LIME can be a viable method to give explanations on how the measured factors impacted shooting results.
The results from this thesis lay the groundwork for better training tools for target shooting using explainable prediction models with sensors.
Keywords:
Acknowledgments
We would like to thank Niklas Lavesson for the guidance we received on our thesis. And for the opportunity to do our thesis within MAPPE with support from the Knowledge foundation. We would also like to thank Andreas Månsson from SAAB AB Training & simulations (SAAB) for helping us with equipment and contact with SAAB. We also thank Anton Berkman and Gustav Andersson who helped us with our experiment.
Contents
1 Introduction 1 1.1 Background . . . 2 1.2 Problem description . . . 5 1.3 Research question . . . 6 2 Method 7 2.1 Experiment design . . . 7 2.2 Independent variables . . . 9 2.2.1 Heart rate . . . 92.2.2 Trigger pull force . . . 9
2.2.3 Rifle movement . . . 11
2.3 Dependent variable . . . 12
2.3.1 Target hits . . . 12
2.4 Experiment execution . . . 14
2.5 Data processing . . . 15
2.6 Random forest regression . . . 16
3 Results 17 3.1 Analysis . . . 20
4 Discussion 22 4.1 Results . . . 22
4.2 Validity and limitations . . . 23 4.3 Future work . . . 24 4.4 Conclusions . . . 25 References 26 A Appendix 29 B Appendix 30 C Appendix 32
1
Introduction
The aim of this thesis is to further study how to perform data mining and knowledge discovery in the target shooting domain with the goal to generate interpretable models or model explanations. This thesis specifically tries to determine a method to give predictions for what factors impacts a shooter’s results. The independent variables are rifle movement, trigger pull force and heart rate. These variables will be studied to determine the correlation with the dependent variable shooting accuracy.
Target shooting is a broad activity that has use in sports shooting and weapons training both in law enforcement and in the military sector. Multiple studies have been conducted on target shooting that shows that it is a good avenue to apply statistical prediction models to find ways to improve shooting accuracy (i.e Deng, Liu, and Hsieh, 2011; Lin and Wu, 2012 and Maier, Meister, Trösch, and Wehrlin, 2018). The use of machine learning could potentially introduce new ways to conduct training within target shooting. This could be beneficial to a large audience of sports shooting practitioners and hunters as well as police and military personnel.
Machine learning is the study of statistical models and algorithms within the context of computer systems. One of the tools of machine learning are learning algorithms that take data and trains a predictive model that can then predict future outcomes. It is a fast-growing field and learning algorithms can outperform humans in specific areas (Doshi-Velez & Kim, 2017). It has been successfully used to analyze, for example, performance in sports (Novatchkov & Baca, 2013). However, a problem with this growth in popularity is algorithms where the user does not know why the algorithm makes the prediction it does. According to Doshi-Velez and Kim (2017) is it important for a machine learning model to be explained in cases where you might want a deeper knowledge of the problem that the model is trying to solve.
This thesis is part of a larger research project called MAPPE (Mining Actionable Patterns from complex Physical Environments) which explores how to design and evaluate predictive models that can provide explanations for their predictions. MAPPE is grounded in the model LIME (Local Interpretable Model-agnostic Explanations). The goal of this thesis is partly in support of the goals of that
project.
1.1
Background
A black-box algorithm is an algorithm where the inner workings are unknown to the user, which means that the user does not know how the algorithm made its prediction (Guidotti et al., 2018). For some areas, this is not an issue, e.g. ad servers. However, as machine learning grows more popular and is used in more areas, factors other than task performance become more important, such as security and nondiscrimination (Doshi-Velez & Kim, 2017). This means that underlying structural or societal patterns can incorrectly train a predictive model, resulting in incorrect predictions and even discrimination against groups of people. Guidotti et al. (2018) gives an example of such a problem where a military application of image recognition of tanks seemingly made correct predictions of enemy versus friendly tanks. It turns out, however, that the algorithm was actually trained to look at whether the picture was taken on a clear or overcast day, not the actual features of the tanks. This problem of interpretability of algorithms has been recognized by the European union with the new legislation that essentially creates a "right to explanation" rule, where an algorithmic decision needs to be able to provide an explanation for how it came to a decision (Goodman & Flaxman, 2017). Interpretability within machine learning, or machine learning explainers, is a fairly new sub-field originating from the need for interpretable algorithms (Gilpin et al., 2018).
To address the question of interpretability within machine learning several methods have recently been developed, Gilpin et al. (2018) and Guidotti et al. (2018) lists and categorize some of these. One recent example is LIME, which was developed to interpret and give explanations for any machine learning algorithm (Ribeiro, Singh, & Guestrin, 2016). LIME is categorized as a linear proxy model. This means that LIME will slightly change the inputs for a black-box model and see what happens with the output. From this relationship between in- and output, LIME can then construct its own proxy model for the black-box within the scope of the inputs. This means that LIME is locally interpretable, i.e. it creates a proxy model for the instance of the inputs, but not for the entire black-box model. This method also makes it model-agnostic. This means that LIME does not have to know anything about the inner workings of the black-box model since it only
looks at in- and outputs. The result is that LIME can show which of the inputs had the largest impact in how the black-box model made its prediction (Ribeiro et al., 2016). LIME has been used in different areas of research, for example to aid with human decision making for a model trained on radar signal recognition (W. Zhang, Ge, Jin, & Guo, 2018) and in an expanded version for the use in music content analysis (Mishra, Sturm, & Dixon, 2017). Because LIME is a relatively new method, there are several areas in which it has not yet been applied, for instance in target shooting.
There are many factors to determine shooting accuracy to explain certain shooting results. Ball, Best, and Wrigley (2003) looked at how body sway and aim point fluctuation impacted performance. Era, Konttinen, Mehto, Saarela, and Lyytinen (1996) studied how posture control between experienced and inexperienced shooters impacted performance. Z. S. Zhang, Qiu, Liu, Huang, and Wang (2013) and Lin and Wu (2012) looked at aiming accuracy, hitting stability and trigger control. In a handbook about pistol marksmanship (USMC, 2001) it is stated that aiming, trigger control and breath control are the most important factors. All these studies point to weapon movement and weapon control to be the main factors in determining shooting results and this will form the basis of what this thesis looks at.
There are numerous studies which have used statistical and learning algorithms to draw conclusions on shooting accuracy. Deng et al. (2011) used three different prediction models to look at how a shooter grips a pistol and pulls the trigger. In addition to concluding that weapon stability and correct trigger pull are the main factors for shooting accuracy. Lin and Wu (2012) studied how to design an intelligent evaluation system for shooting, by measuring factors of distance of the hit to the bull’s eye, weapon holding stability and shaking intensity during the shot. The study provides explanations in the form of text through "data-mining" and image recognition. However, a limitation with this study is that the models used are not described very well. Maier et al. (2018) used various machine learning models in biathlon shooting to predict performance and found that previous hit rate was the dominant predictor. However, one of the limitations of this study is that they only trained predictive models with target hit results from competitions and did not measure bio-mechanical or physiological factors, e.g. weapon movement, heart rate etc. A common sentiment from all these studies is that further work on expanding the use of predictive models for shooting is of interest.
Previous master theses have been conducted on related subjects. In a study, SAAB’s simulator GC-IDT (Ground Combat Indoor Trainer) was used with heart rate sensors and an eye-tracker to investigate if it is possible to measure physiological factors such as heart rate to accelerate learning for a novice shooter (Hansson, 2017; Karlsson, 2017). GC-IDT has a suite of sensors that can measure "trigger pulling, canting and pressure against the shoulder" (p. 5), as well as how the bullet hits are grouped on the target, so called shot grouping. The main results from those studies were that eye-movement, above all other measured physiological factors, had the most significant impact on shooting score. Yates (2004) studied how well indoor simulated marksmanship trainers can train the fundamentals of marksmanship. His findings show that there is a positive correlation between performance in the simulator and live fire performance, but also that "repetitive practice in the ISMT [Indoor simulated Marksmanship Trainer] without coaching does not produce steady improvement from untrained subjects." (p. 67), suggesting that sole usage of a simulator is not sufficient for improvement. Instead, simulator usage has to be complemented with coaching.
GC-IDT is the simulator used for this thesis. It uses a projector and screen to project a scene with a target that the user shoots at with a replica rifle. This replica rifle is equipped with a laser which can be used to record both hit impacts and weapon movement using a high speed camera that records how the laser has traveled on the projector screen. The rifle also simulates recoil using CO2 (carbon dioxide) cartridges within the rifle magazine, this CO2 cartridge powers a piston that pushes against the shoulder at the butt stock. The data from the sensors are stored to a database within a computer connected to the sensors. The data from the sensors are expressed in SI units, meaning X and Y coordinates are in metres, time is in milliseconds. The data recording is event based, meaning data gets recorded when sensor data changes and not on a set time.
Random forests is a supervised learning algorithm that is categorized as a so called model ensemble, which means that it is a combination of different predictive models (Flach, 2012). In this case it is a combination of decision tree predictors where the decision trees vote on what classification the output will be. In a decision tree there are nodes that represent a feature that is going to be classified. Each branch is a potential value for the connected node. The deeper in the tree you go the more homogeneous the features in the nodes become until you have leafs with completely homogeneous features (Kotsiantis, 2007). Random forests can be used for both
classification and regression tasks (Breiman, 2001). A regression task involves using a so called regressor to estimate a continuous function of the measured data. This is in contrast to classification where a label space is a set of discreet classes. For a regression task, an estimate for the continuous function of the data can be done by fitting polynomials of different degrees on the data points (Flach, 2012). Regression can be used to find relationships between independent and dependent variables and is therefore commonly used for this purpose in statistical modeling. A method of evaluating trained learning algorithms is using k-fold cross-validation. This method produces a cross-validation accuracy score that is based on comparing the prediction of the classification with the observed classification. It does this by randomly taking a proportion of the data, train it with the chosen learning algorithm and then testing it on another proportion of the data. Essentially it divides all the data D into k number of subsets D1, D2, ..., Dk. The chosen learning algorithm is then trained and evaluated on these subsets k times for each subset. This means that t ∈ 1, 2, ..., k and that DtD portion of the data is trained on. The predictions are then tested on Dt (Kohavi, 1995). For this thesis, the metrics that is used as the cross-validation score will be mean-absolute-error (MAE) and mean-root-squared-error (RMSE). RMSE is a measure of the standard deviation of the residuals, where a residual is how far the data points deviate from the regression line. This means that a residual tells us the deviation of the prediction errors, or how concentrated the data is around the regression line. RMSE is more sensitive to errors that deviate farther from the regression line, which makes it a good evaluation measure for models where larger errors have a higher impact (Barnston, 1992). Mean absolute error is the mean of how far the observed data point deviate from the predicted data point.
1.2
Problem description
Tools for visualizing and explaining target shooting performance does exist today in the form of simulators and instructors, but often these can only look at a limited set of factors to explain shooting results. And while a simulator, like GC-IDT, may record a lot of data, it is mostly used for visualization purposes and some analysis. But the data is then mostly left to either an instructor, or in the worst case, left for an inexperienced shooter to interpret. There can also be big difference in how well different instructors can observe how a shooter performs. Learning algorithms may
be a way to address this problem, if an automated method of analysing measured data could be developed then explaining shooting performance could become more standardized. These explanations would then also be empirically more rigorous than human instruction since it would be less prone to human interpretations. While Deng et al. (2011) applied machine learning on how a shooter grips the pistol and trigger, to our knowledge, learning algorithms have not yet been applied to the combination of factors that we will look at to predict shooting accuracy. These factors are rifle movement, trigger pull force and heart rate. The intention of this thesis is to explore which sensors can be used to measure these factors and to investigate how they affect shooting accuracy with help of supervised learning algorithms. And also creating an interpretable method that will explain how much these factors impacted the shooting result. This thesis is not focused on doing a comprehensive study on what impacts shooting accuracy, instead the focus is on exploring a method to find a relationship on the three factors stated, tied to shooting accuracy. However, some conclusions on accuracy may still be drawn from the results.
There is also an interest from sports shooting practitioners and SAAB to study how this method can be use to make target practice and weapons training even more efficient and possibly find new ways to conduct training.
1.3
Research question
The aim with this study is twofold: The first goal is to explore if sensors measuring hear rate, trigger pull force and rifle movement can be used to show a relationship with shooting accuracy. To reach this goal, demonstrating a relationship between our independent variables and shooting accuracy is enough. The second goal is to train a machine learning model with the data from the measured factors to make predictions. In addition to finding a relationship between the factors and training a machine learning model, LIME will be used on the model assess if a explanation can be produced with LIME. These goals can be achieved independently of each other, but without proving the correlation of the measured factors and shooting accuracy, the validity of these predictions may not be accurate.
Can we develop an interpretable method to make predictions on shooting accuracy, based on sensor information about the shooter?
Our hypothesis is that rifle movement, trigger pull force and heart rate can be used to predict shooting accuracy, in the context of standard precision shooting. This hypothesis will be tested quantitatively through an experiment.
If we can demonstrate a relationship between shooting accuracy and the three independent variables measured, the hypothesis would be considered proven and the first goal would be completed. Hereby, a method to predict shooting accuracy, satisfying the second goal would have been demonstrated.
2
Method
Following the Creswell (1996) classification, this thesis is quantitative in nature. Since the data collection is done quantitatively, experimentation is the chosen data collection method. The experimental data will be used to strengthen or reject the given research question.
2.1
Experiment design
The independent variables are chosen based on three factors which we deem impact shooting accuracy. These factors are: rifle movement, trigger pull force and heart rate are tested for correlation with the dependent variable shooting accuracy. The dependent variable is collected in a simulator while the independent variables are collected with external sensors. The experiment is conducted on SAAB’s simulator GC-IDT by the authors of this thesis.
The independent variables are controlled by testing each variable in a low and high state in different permutations, essentially measuring the best case scenario and the worst case scenario for each variable. This is done to isolate the impact of each variable and combination of variables as much as possible. This results in eight combinations of Low and High states (see Figure 1) of the independent variables that is tested. In each of these cases, the variables will be controlled like shown in figure 1 and a shooting exercise will be conducted in the GC-IDT. This shooting
exercise consists of shooting five shots at a target 25m away. The reason for this distance is that it reduces the number of shots that miss the target completely. This will be done five times for each case. A regression model is then trained with the data from these independent variables paired with the shooting results. The regression model will then make predictions on the shooting accuracy.
Figure 1. Visualization of the different variations of the variables that will be
tested. White circle means low, black circle means high. For example, case two will have high rifle movement, low trigger pull force and low heart rate.
For low heart-rate the participant is in a relaxed state and tries to lower the heart rate as much as possible and for high heart-rate the participant will use a spinning bicycle until their pulse is above 80% of their max pulse before the shooting exercise. For low rifle movement the rifle is handled normally with two hands during the shooting exercise. For high rifle movement the rifle is handled without a supporting hand on the handguard, so that the rifle is only in contact with the shoulder and the hand holding the pistolgrip during the shooting exercise. For low trigger pull force the participant is focused on pulling the trigger as slow as possible during the shooting exercise, and for high trigger pull force the participant is focused on pulling the trigger as fast as possible during the shooting exercise.
2.2
Independent variables
2.2.1 Heart rate
Heart rate is measured with a Polar H10 heart rate sensor. This sensor is attached to a chest strap that is worn over the chest. The chest strap has built in electrodes that, when worn, are situated close to the heart. The H10 sensor is connected through bluetooth to a smartphone with the accompanying software Polar beat installed. The data from this sensor gets uploaded to Polar’s servers and can be downloaded through their website. During a measurement the data from the H10 sensor is logged each second.
To build this into a feature for the learning algorithm the heart rate is converted to a percentage of the shooter’s theoretical maximum heart rate. The maximum theoretical hear rate is calculated with the following equation (Wohlfart & Farazdaghi, 2003):
Heart ratemax=
203, 7
1 + e0,033·(age−104,3), for men
Heart ratemax=
190, 2
1 + e0,045·(age−107,5), for women.
A percentage of the participant’s theoretical max pulse is used because of the fact that the resting and high pulse may differ between people. The feature is calculated with the following equation:
Heart rate = 100Heart ratemeasured Heart ratemax
.
Heart rate will in the end be represented as a value between 0 and 100.
2.2.2 Trigger pull force
Trigger pull force is measured using an iCubeX TouchGlove v2.3 and the accompanying sofware EditorX. This glove was chosen based on the findings from the study by Deng et al. (2011). The glove has six force sensitive resistors in its fingertips that can be used to measure pressure on the fingertips, see Figure 2. The pressure sensitivity for these resistors are 10-1000g. These sensors outputs an analogue
signal that gets converted to a 14-bit digital value through a Analogue-to-digital converter, or ADC. This ADC signal is sent to a computer and recorded using EditorX. For this experiment only the force sensor on the index finger is used.
Figure 2. Image of iCubeX TouchGlove v2.3 and where the senseros are placed.
Pressure sensor five and the bend sensor is used in the experiment.
The recorded data from the trigger pull force is represented by a graph with the ADC value for force on the y-axis and time on the x-axis. The difference between squeezing the trigger and jerking the trigger can be explained as how long force is exerted on the trigger. Building from this explanation, a feature from this sensor is built from the positive mean derivative of the force graph. A higher mean derivative means the force is exerted on a shorter time frame, which means a higher spike of force. This represents jerkin the trigger. A lower derivative means the opposite. We only look at the positive derivative because we are only interested in the force up to and including the shot, see Figure 3 for a visual representation. Any force changes on the trigger after the shot will not impact the result. The feature is calculated with the following equation:
∆y
Figure 3. Graph of the pressure on the trigger over time during a five shot series.
The top graph is from jerking the trigger and the graph below is from squeezing the trigger.
2.2.3 Rifle movement
Rifle movement is represented by an iCubeX Orient3D v3.2 with the sofware EditorX and data from GC-IDT on muzzle movement. The Orient3D component uses a Microelectromechanical sensor that can be used to measure acceleration in x,y and z axes. The Orient3D component is fastened close to the muzzle. Since the muzzle is the furthest from the rifle’s point of rotation, which is located at the point of contact at the shoulder. This is the area of the rifle that will experience the highest movement.
GC-IDT saves the muzzle position on the projector screen up to 3 seconds before the shot. This is done using a high speed camera that detects where a laser from the rifle hits the screen. The position of this laser is recorded to the database.
The data from Orient3D is represented as a 14-bit value on the Y-axis and time for each data-point on the X-axis, see Figure 4 for a visual representation. A higher number on the Y-axis represents a higher instantaneous acceleration. From this the summation of the hypotenuse between all points one second before the shot is calculated. A higher sum represents higher acceleration during the one second leading up to the shot. The data from GC-IDT is represented as coordinates relative to the target bull’s eye over time. A circle is imposed to encompass all the positions one second before the shot and then the radius is calculated of this circle. A higher radius of the circle represents a larger area of rifle movement before the shot.
Based on this, rifle movement is modeled as a combination of the summation of the acceleration from Orient3D and the radius of the imposed circle based on the muzzle coordinates from GC-IDT.
Figure 4. Graph of the data from Orient3D, acceleration over time. Spikes depict
shot moment.
2.3
Dependent variable
2.3.1 Target hits
Accuracy will be evaluated similarly to Deng et al. (2011) which is based on the ISO5725-2 standard where precision and trueness determines the accuracy.
Trueness is calculated based on the angle between the group shot center point and the horizontal plane and the distance from bull’s eye. Precision is calculated based on the radius of the group shot. These two factors will be synthesized into an accuracy score where shot group radius is scored 0-50 and shot group center position is scored 0-50. When added together the accuracy score have a total range of 0-100. This allows for cases where a shot group is tight but is not close to the bull’s eye, in which case you have good precision but not good trueness. This accuracy score will be calculated as follows:
f (r, d, α, n) =
2(50 − r − d) ∗ |cos(α)| − 20(5 − n) when |sin(α)| < |cos(α)| and d ≥ 5, 2(50 − r − d) ∗ |sin(α)| − 20(5 − n) when |sin(α)| ≥ |cos(α)| and d ≥ 5, 2(50 − r) − 20(5 − n) when d < 5 and r = r when r ≥ 5, 0 when r < 5.
where r is the radius of the shot group, d is the distance between bull’s eye and the shot group center, α is the angle between the horizontal axis and the shot group center and n is the number of shots who has hit the target, see figure 5 (a) for a visual representation. If the radius is smaller than 5cm then the radius is set to 0. Calculating the shooting accuracy with this equation promotes hitting the target inline of the center of the target. The reason for this is that hits along the axes have good trueness for one axis, while hits 45 degrees from both axes have worse trueness for both axes, see figure 5 (b) for a heatmap of the score.
The data for the hits is gathered from GC-IDT which provides coordinates for each target hit. The longest distance between two hits in the group is used to calculate the diameter of the shot group. This diameter is then divided in half to get the radius. To get the distance and angle of the group, the group center must first be calculated. The true center of the group is the point midway between of the two points that are furthest away from each other. The true center does not represent the whole group in a good way because it is only based on the two hits that is furthest away from each other and not all the hits. Instead, the center
point is calculated by taking the mean of the coordinates for all the hits in the group. The distance is calculated with Pythagorean theorem from the bull’s eye to the group center and the angle is calculated with the two-argument arctangent with the group center’s x- and y-coordinates as arguments.
(a) A visual representation of the
parameters in the accuracy score
equation.
(b) A visual representation of the score heatmap when the radius of the shot group is 25mm. Green means a higher score.
Figure 5. Visual explanation for shot accuracy
2.4
Experiment execution
Before starting the experiment all sensors are configured and placed. The H10 heart rate sensor and chest belt are placed on the chest of the participant. The TouchGlove v2.3 is worn on the hand as a glove so that the force sensor on the fingertip is placed on the trigger. The Orient3D sensor is attached to the end of the barrel of the replica-rifle and set to measure acceleration. Calibration data is recorded where the rifle is completely stationary on a table with the correct orientation to record a base line zero acceleration and zero trigger force. Before starting each case we first focus on getting the required high or low pulse. Low pulse is maintained by resting between series, and high pulse is maintained by physical activities between series. Using the replica-rifle, five shot series are shot standing infron of the GC-IDT projector screen. With each five-shot series a new measurement is started with all the sensors. This is done five times for each
case. Each participant shoots 200 simulated shots, and thus 200 data points per participant is recorded with the three measured factors and the corresponding target hit data. Three runs of the experiment is done, so a total of 600 simulated shots is recorded.
A Wi-microDig v6.4 is used as a link to receive data from the TouchGLove. This device can send data throguh a wireless link to a computer in real-time over Bluetooth. The microDig is powered with a 9V battery. The data from the Polar H10 sensor is recorded to an android app over Bluetooth. See Figure 6 for an overview of the equipment used in the experiment and see Appendix A for detailed images on the TouchGlove setup.
(a) Visual representation of how heart rate is collected.
(b) Visual representation
of how GC-IDT looks
like, where movement and accuracy score is collected.
(c) Visual representation of how trigger pull force and movement is collected.
Figure 6. Images of the three groups of equipment.
2.5
Data processing
Since the different sensors and GC-IDT represents data in different ways, some processing is done to standardize the data before training the learning algorithm. The accelerometer data from Orient3D not only contains the acceleration of the rifle but also the acceleration of gravity. Because of this the acceleration data is offset on the axis parallel to the acceleration of gravity. The data has an additional offset because the base value of the acceleration, as in zero acceleration, is nonzero. This is due to fact that negative acceleration is represented as a value below the base value and not a negative number. To simplify and standardize the acceleration
data, it is shifted to remove these offsets so that zero acceleration has a value close to zero. This is done by taking the minimum acceleration from the calibration measurement and subtracting this from the data.
Shots can be detected using the acceleration data, and can then be used as the basis for identifying shot moment for trigger and rifle movement. The recoil from the simulator rifle gives short spikes in the acceleration value that are significantly larger than the rifle movement. If the acceleration goes above a set threshold a shot is detected. These detected shots are also examined manually to eliminate any false positives. Since the accelerometer and trigger force data share timestamps, the shot detection from the acceleration is used to infer the shot moment for trigger force data. When shot moment has been identified, rate of change for trigger force and the length of the acceleration curve is calculated between the shot moment and one second before the shot moment.
Data for muzzle position is collected from the GC-IDT database. The data from the muzzle position is already automatically measured from one second before up to the shot moment. The muzzle position is represented as X and Y coordinates over time. The two points that are furthest from each other are found and the radius for the imposed circle of movement is calculated by taking the distance between these two points and dividing it by two.
2.6
Random forest regression
Random forest regression is chosen as the learning algorithm to use based on a few factors. The data measured from the different sensors have a difference of scale between them, e.g. acceleration is represented as a 14-bit value, while heart rate is measured roughly between 60 and 180 bpm. Learning algorithms such as linear regression are unsuited for such data because the higher magnitude data points will be weighted much higher in the prediction than the smaller magnitude data points. We want to avoid such an outcome since we are looking at how much each of the inputs affect the prediction. Random forest regression does not need the input data to be normalized or scaled the same. An additional factor to use random forest regression is that feature importance can be extracted from the trained algorithm. It is also not sensitive towards unbalanced data. Overall it is a user-friendly method and suits our data. (Liu, Wang, Wang, & Li, 2013)
Before training the random forest regression model, hyperparameter optimization is done using a grid search. For this thesis GridsearchCV in sci-kit is used. This is done to optimize the parameters the algorithm uses to learn. A better optimization means predictions with less error, i.e. the predictions deviate less from the actual observed result. See Table 1 for values from the grid search. After the model has been trained it is evaluated with cross-validation to find out the performance of the trained model expressed in mean-absolute-error and mean-root-squared-error. Table 1
Results from grid search.
Bootstrap Max depth Max features Min samples leaf Min samples split Number of estimators Yes 80 2 5 12 100
3
Results
A total of 120 data points has been extracted from the raw data that was collected during the experiment. This data consists of data from three participants firing 600 shots. All the results in this sections are presented with a predicted accuracy score. These predictions are the resulting predictions from the random forest regression model trained on our experiment data. The accuracy score is defined in section 2.3.1.
Figure 7 show how the predicted accuracy score from the random forest regression model change when all independent variables increases linearly between the their minimum and maximum measured value. Figure 8, 9 and 10 show the predicted accuracy score of two independent variables fixed to their minimum or maximum value and the third independent variable linearly increasing. For example when isolating the heart rate, trigger pull and movement is fixed to their minimum/maximum value.
Figure 7. Graph of predicted accuracy score when all three features are linearly
increasing between the minimum and maximum measured values. The three x-axes represents the corresponding feature.
Figure 8. Graph of predicted accuracy score for linearly increasing rifle movement.
Trigger pull force and heart rate are fixed at their minimum (top graph) and maximum (bottom graph) values. The x-axis represents the feature heart rate .
Figure 9. Graph of predicted accuracy score for linearly increasing trigger pull
force. Rifle movement and heart rate are fixed at their minimum (top graph) and maximum (bottom graph) values. The x-axis represents the trigger pull force.
Figure 10. Graph of predicted accuracy score for linearly increasing heart rate.
Rifle movement and trigger pull force are fixed at their minimum (top graph) and maximum (bottom graph) values. The x-axis represents the feature heart rate .
3.1
Analysis
To validate the performance of the machine learning model, cross-validation was used with a k-value of 5. The results from the cross validation can be seen in Table 2. The mean absolute error measures the magnitude of the errors and because the maximum accuracy score is 100 the mean absolute error is about 28% ((13.77*2)/100). The feature importance (see Table 3) from the model points to that movement affects the accuracy score the most and that heart rate and trigger pull almost affects the accuracy score similarly.
Table 2
Results from cross-validation.
Metric Score Standard deviation
Mean absolute error 13.77 0.74
Table 3
Feature importance from the
trained random forest regression model.
Feature Feature importance (%)
Heart rate 24.18
Trigger pull 22.20
Movement 53.61
Figure 11 shows LIME applied to eight different predictions from our random forest regression model. The eight predictions are chosen to correspond to our experiment, see figure 1. This means that for Case 2 a prediction with high movement, low trigger pull force and low heart rate was chosen. A positive number in the histogram means that the feature contributed positively towards the prediction, and a negative number means the feature contributed negatively towards the prediction.
Figure 11. Histogram of how the features are affecting the predicted accuracy
score based on output from LIME. The cases is directly corresponding to the cases in Figure 1.
4
Discussion
4.1
Results
The predictions from the random forest regression model show that increased weapon movement, trigger pull force and heart rate decreases the accuracy score predictions, see figure 7. When looking at the features individually (figure 8, 9 and 10) you also see a decrease in predicted accuracy score, in particular for weapon movement. From figure 8 you see that the predicted accuracy score start to decrease for weapon movement of around 4500 and up, which is the equivalent of a stable grip with some movement up to a lot of movement. This suggests that an unstable grip will decrease the accuracy score significantly. From figure 9 you see that only really slow trigger pull (0 to ∼ 100) gives high predicted accuracy score, suggesting that focusing on very slow trigger pull is important for score. The graph also shows that the accuracy score drops faster when the other two factors, heart rate and weapon movement, are high. This indicates that trigger pull force have a higher impact when the other factors are high rather than low. From figure 10 we see that heart rate does not affect the predicted accuracy score significantly until the heart rate goes over 80% of the participants max heart rate. The drop in score is higher when the other two factors, weapon movement and trigger pull, are low. This indicates that heart rate has less of an impact when the other two factors increase. We can see that both trigger pull force and heart rate have less of an drop in predicted accuracy score when they increase compared to weapon movement.
The feature importance from the random forest regression model show that weapon movement is the most important, at 53.6%, influence in the predictions. Heart rate and trigger pull have an importance of 24.2% and 22.2% respectively. The results from LIME points to a similar conclusion as the feature importance, where movement has the most impact and trigger pull and heart rate have similar impact to each other, see Figure 11. The explanations from LIME follow each factor from figure 1 as we would expect. This means that when a feature is low, e.g. low heart rate, LIME shows that it contributes positively to the prediction. When a factor is high, e.g. high heart rate, LIME shows that it contributes negatively to the prediction. If you compare Case 1 and Case 8 in figure 11, you see that all factors contribute positively for Case 1 and negatively for Case 8. Which corresponds
to Case 1 and Case 8 in figure 1, where in Case 1 all factors are low, and Case
8 all factors are high. This shows that LIME can be used as an alternative tool
to explain which factors impacted a prediction. One explanation for why weapon movement has such a large influence may be that heart rate and trigger pull may affect how much the rifle moves.
The results show that the predictions from the random forest regression model has a root mean square error of 17.22(±7.2). Since the accuracy score is between 0 and 100, this represents a devation of 34.44% for predictions.
4.2
Validity and limitations
Although we see a relatively high error in our predictions this does not change the fact that our method is able to explain which features impacted the predictions and how much. The focus on this thesis is towards being able to interpret the predictions and give explanations over accurate predictions. As such, the method is still seen as valid even with our evaluation scores.
There are some threats to the validity of the results. One of the experiment runs did not have the same case order as seen in figure 1. Although the experiment was designed so it should not matter which order the cases were done, we noticed that the order of the cases impacted shooting performance slightly. The further the experiment went, the more physical work was exerted and this increased the measured low heart rate. We could see especially for cases where the participants had to use only one hand with the rifle that the heart rate would climb just because of that (see Appendix B). This means that the low heart rate is not entirely consistent. Some variations in trigger pull may also occur due to the TouchGlove force sensor. The sensor would sometimes move around in the glove resulting in small differences in the max value of the trigger pull. The experiment was conducted by two people, which means that the random forest regression model was trained with data from two people. The experiment was designed to minimize the impact of the shooting ability of different people but it is still possible that the amount of participants affects the learning algorithm.
By the nature of how weapon movement was measured, by using one hand to hold the rifle to simulate high weapon movement, we cannot directly say weapon movement decreases predicted shooting accuracy. But rather that shooting with
one hand decreases predicted shooting accuracy. We can, however, infer that using one hand increases weapon movement and an increase in weapon movement decreases predicted shooting accuracy.
The implementation of the learning algorithm is currently not generalizable to a shooting scenario other than the one done in the experiment, using a simulator and a shooting distance of 25 meters. Although the sensors used to measure the independent variables would allow generalization to other scenarios, the current implementation is dependent on accurate hit detection. To develop a generalized method that could be used outside a simulator and on any distance would be to use a hit detection method such as a bullet radar. It is important that bullets that does not hit the target is detected. You would need to train the learning algorithm for different distances, since the distance to target significantly affects where the bullets hit the target. A model trained on 25 meters shots will predict higher scores than one trained on a longer distance, since it is easier to hit high scoring hits if you are closer to the target.
4.3
Future work
Some future work that could be done includes expanding the experiment from this thesis. Training the random forest regression model with more experiment data with different people would only improve the method. It would be interesting to see if the method proposed in this thesis could be applied to target shooting outside the simulator on a real rifle. Some other interesting avenues would be to look at another combination of variables, a key factor for Hansson (2017) and Karlsson (2017) was eye movement. It would be interesting to add an eye tracker as an independent variable. It would also be interesting to compare different learning algorithms with the method from this thesis. While doing the experiment we identified some possible improvements that could be done to the sensors for the experiments. A custom-built system where we would implement hardware and software for reading the sensors could be beneficial. We would then have a greater control in how the data is collected, and we could reduce some of the overhead required with the current solution. A problem we encountered with the TouchGlove was that the force sensitive resistor would move around in the glove. A better solution might be to use the force sensitive resistors seperately from the glove, or even build our own glove with the sensors placed better.
More future work could be done testing different kinds of learning algorithms. For example, Deng et al. (2011) found that support vector machines worked best in his study to predict shooting accuracy. If an automatic system to explain the shooting results were to be created, it would be interesting to compare different learning algorithms and how a shooter responds to the explanations.
4.4
Conclusions
The aim of this thesis was to identify combination of sensors that could be used to show a relationship between our three chosen factors and shooting accuracy. And to develop an interpretable prediction method that predicts shooting accuracy and explains which factors impacted the shooting results. We have demonstrated that the combination of a force sensitive resistor, an accelerometer and a heart rate sensor in conjunction with the sensors from GC-IDT can show a relationship between the three measured factors and shooting accuracy. We have developed a method that gives explanations for how much each factor impacted the shooting result. We have also shown that LIME can be used as a method to explain which factors impacted the shooting results from the random forest regression model. Our results show that increased weapon movement, trigger pull force and heart rate decreases the predicted accuracy score. We found that weapon movement had the greatest impact on shooting accuracy, while trigger pull force and heart rate had similar impact.
The results of this thesis are relevant in multiple areas. For target shooting it lays the foundation for automated training tools based on interpretable predictive models. This could help target shooting practitioners conduct more empirically based training. The results from this thesis further emphasise the importance of interpretable prediction models. Not only is interpretability important for fairness and transparency when it comes to learning algorithms, but it can also be used as a tool for solving problems.
References
Ball, K. A., Best, R. J., & Wrigley, T. V. (2003). Body sway, aim point fluctuation and performances in rifle shooters: Inter- and intra-individual analysis. Journal of Sports Sciences, 21 , 559–556. doi: 10.1080/ 0264041031000101881
Barnston, A. G. (1992). Correspondence among the correlation, rmse, and heidke forecast verification measures; refinement of the heidke score. Weather and Forecasting, 7 (4), 699-709. doi: 10.1175/1520-0434(1992)007<0699: CATCRA>2.0.CO;2
Breiman, L. (2001). Random forests. Machine Learning, 45 (1), 5-32. (cited By 28071) doi: 10.1023/A:1010933404324
Creswell, J. W. (1996). Research design: qualitative and quantitative approach.
Thousand Oaks: SagePublications.
Deng, S., Liu, D. M., & Hsieh, S. L. (2011). Applying machine learning methods to the shooting accuracy prediction: A case study of t-75 pistol shooting.
Information Technology Journal, 10 (8), 1508–1517. doi: 10.3923/itj.2011
.1508.1517
Doshi-Velez, F., & Kim, B. (2017, Feb). Towards a rigorous science of interpretable machine learning. arXiv e-prints, arXiv:1702.08608. doi: 10.1016/j.intell .2013.05.008
Era, P., Konttinen, N., Mehto, P., Saarela, P., & Lyytinen, H. (1996). Postural stability and skilled performance-a study on top-level and naive rifle shooters.
Journal of Biomechanics, 29 (3), 301–306.
Flach, P. (2012). Machine learning: The art and science of algorithms that make
sense of data. United Kingdom: Cambridge University Press. doi: 10.1017/
CBO9780511973000
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018). Explaining explanations: An approach to evaluating interpretability of machine learning. arXiv e-prints, arXiv:1806.00069. doi: 10.1039/ C39930001188
Goodman, B., & Flaxman, S. (2017, fall). European union regulations on algorithmic decision-making and a "right to explanation". AI Magazine, 38 (3), 50–57. doi: 10.1609/aimag.v38i3.2741
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2018). A survey of methods for explaining black box models. ACM
Computing Surveys (CSUR), 51 (5), 93.
Hansson, A. (2017). Accelerating learning during rifle shooting training focusing on
the shooter (Unpublished master’s thesis). Linköping University, Linköping.
Karlsson, J. (2017). Identifying patterns in physi-ological parameters of expert
and novice marksmen in sim-ulation environment related to performance outcomes (Unpublished master’s thesis). Linköping University, Linköping.
Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the 14th international
joint conference on artificial intelligence - volume 2 (pp. 1137–1143). San
Francisco, CA, USA: Morgan Kaufmann Publishers Inc.
Kotsiantis, S. B. (2007). Supervised machine learning: A review of classification techniques. In Proceedings of the 2007 conference on emerging artificial
intelligence applications in computer engineering: Real word ai systems with applications in ehealth, hci, information retrieval and pervasive technologies
(pp. 3–24). Amsterdam, The Netherlands, The Netherlands: IOS Press. Lin, Z., & Wu, S. (2012). The design and implementation of shooting training and
intelligent evaluation system. In E. Mao, L. Xu, & W. Tian (Eds.), Emerging
computation and information technologies for education (pp. 107–115).
Berlin, Heidelberg: Springer Berlin Heidelberg.
Liu, M., Wang, M., Wang, J., & Li, D. (2013). Comparison of random forest, support vector machine and back propagation neural network for electronic tongue data classification: Application to the recognition of orange beverage and chinese vinegar. Sensors and Actuators B: Chemical, 177 , 970 - 980. doi: https://doi.org/10.1016/j.snb.2012.11.071
Maier, T., Meister, D., Trösch, S., & Wehrlin, J. P. (2018). Predicting biathlon shooting performance using machine learning. Journal of Sports Sciences,
36 (20), 2333-2339. (PMID: 29565223) doi: 10.1080/02640414.2018.1455261
Mishra, S., Sturm, B. L., & Dixon, S. (2017). Local interpretable model-agnostic explanations for music content analysis. In Ismir.
Novatchkov, H., & Baca, A. (2013). Artificial intelligence in sports on the example of weight training. Journal of Sports Science and Medicine, 12 , 27–37. Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). "why should i trust you?":
Explaining the predictions of any classifier. In Proceedings of the 22nd acm
sigkdd international conference on knowledge discovery and data mining -kdd ’16. doi: 10.1145/2939672.2939778
USMC. (2001). Rifle marksmanship-mcrp 3-01b (1st ed.). Washington, D.C.: Author.
Wohlfart, B., & Farazdaghi, G. R. (2003). Reference values for the physical work capacity on a bicycle ergometer for men – a comparison with a previous study on women. Clinical Physiology and Functional Imaging, 23 (3), 166-170. doi: 10.1046/j.1475-097X.2003.00491.x
Yates, W. W. (2004). A training transfer study of the indoor simulated marksmanship trainer (Unpublished master’s thesis). Naval Postgraduate
School Monterey CA.
Zhang, W., Ge, P., Jin, W., & Guo, J. (2018). Radar Signal Recognition Based on TPOT and LIME. In Chinese control conference, ccc. doi: 10.23919/ ChiCC.2018.8483165
Zhang, Z. S., Qiu, D. S., Liu, G., Huang, C. J., & Wang, J. J. (2013). Outside light based shooting training evaluation system design and implementation.
Appendix A
(a) Image of all equipment used to collect trigger pull force.
(b) Image on when the TouchGlove is equipped on a participant.
Appendix B
Figure B.1. Histogram of measured heart rate collected from three experiments.
The red numbers and boxes indicates separate experiments. Note that the experiment order of experiment one and two is the same and the third is different.
Figure B.2. Histogram of measured trigger pull force collected from three experiments. The red numbers and boxes indicates separate experiments. Note that the experiment order of experiment one and two is the same and the third is different.
Figure B.3. Histogram of measured rifle movement collected from three experiments. The red numbers and boxes indicates separate experiments. Note that the experiment order of experiment one and two is the same and the third is different.
Appendix C
All code used in this thesis can be accessed through the following link: https:// github.com/vija96/PSACS.git
