A tool to facilitate the development of Mocap applications using IMUs - Design and Prototyping

(1)

Department of Computer Science and Media Technology

Master Thesis Project 15p, Spring 2018

A tool to facilitate the development of Mocap

applications using IMUs

Design and Prototyping

By

Pablo Cano and

Mahamed Hassan Mahamed

Supervisors:

José Font Javier Marín

Examiner:

(2)

2

Contact information

Author:

Pablo Cano

pablo.cano@outlook.com

Mahamed Hassan Mahamed mhmohamed54@hotmail.com

Supervisors:

José Font

E-mail: jose.font@mah.se

Malmö University, Department of Computer Science and Media Technology.

Javier Marín

E-mail: javiermar1993@gmail.com

University of Zaragoza, Aragon Institute of Engineering Research.

Examiner:

Johan Holmgren

E-mail: johan.holmgren@mau.se

(3)

3

Abstract

This paper presents to find a solution, by designing a tool that could be used as a framework for the development of Motion Capture applications (Mocap) which is using with inertial Measurement Units (IMUs). Our main objective is how to find a tool that would be extremely fascinated and improved the accessibility to the development of Motion Capture applications. We applied for the literature review,and we described and presented the factors about relevance of the technology and a different area of use and application, as well as to discover characteristics that need to be considered for the creation of a tool that could be utilized in the development of Mocap Applications. The second part of our thesis is certainly based on the design science research methodology; an artifact is planned through a designing a tool and prototype. We created a framework known as IMOTRAF which stands for Inertial Motion Tracking Framework that targeted more platforms than aREST framework, is to facilitate The Mocap application using by IMUs. The IMOTRAF framework should only focus in MCU/IMU platform. Throughout the collection data for the empirical analysis, we have learnt two protocols commonly used in wireless communication such as MQTT and Firmata. In this thesis we have implemented and tested to a new artifact that is made possible to connect many devices simultaneously via WIFI using by Firmata protocol.

(4)

4

Popular science summary

In recent years, the use and adoption of different types of sensors to collect all sorts of different data have experienced a considerable increment; especially within the consumer market, in which this technology has found a wide spectrum of uses and applications. All this, mainly because of the relatively recent development and evolution in technologies such as the Internet of Things, Ubiquitous Computing, Wearable Devices, Cloud Computing, etc. As well as the advances in miniaturization and cost reduction of hardware (e.g. sensors, microcontrollers, etc.), it has contributed to the discovery of new opportunities, uses and applications, especially when these technologies are interlaced with one another.

One of the new opportunities is in the area of Motion Capture using Inertial Measurement Units. Therefore, an open question that we address in this thesis is which factors or characteristics need to be considered for designing a tool for developing Motion Capture applications with the use of Inertial Measurement Units. We argue that although there are some systems and solutions in existence, developers have no access to the internals, and such systems and solutions can also prove to be expensive since they commonly required licensing. We also argue that, in spite of the fact that now more than ever the technology has widespread and has become cheaper, thus resulting in many open hardware options and open software solutions there is no framework that focuses is the development of Motion Capture applications with the use of Inertial Measurement Units.

By answering our second research question, how a tool can be designed and implemented for developing Motion Capture applications with the use of Inertial Measurement Units we have contributed a framework that will facilitate the development of Motion Capture applications with the use of Inertial Measurement Units.

This framework will provide a layer of abstraction between the different open hardware (e.g. microcontroller and sensors) available in the market and the software (e.g. different

(5)

5

applications). Consequently, this will help developers to develop quick and cheaper solutions. At the same time, the area where the framework can have an impact is vast, since Inertial Measurement Units can be found almost everywhere, as it is the use of Motion Capture applications. For instance, in sports, healthcare, the film industry, the music industry, etc. just to mention a few.

Lastly, it is worth mentioning that we expect a more specific and concrete contribution as a result of this thesis work by providing a framework that will be used by the University of Zaragoza for the development of Motion Capture applications to help in the rehabilitation of patients that suffered an injury caused by a job-related activity.

(6)

6

Acknowledgement

Firstly, we would like to thank God Almighty for providing us with the strength and capacity to complete our studies and finalizing this work. All glory to God!

Secondly, we would like to thank our supervisors José Font and Javier Marín for all the valuable support and guidance, thoughts, time and feedback which make it possible for us to accomplish this thesis work. We are also delighted to thank our examiner Johan Holmgren for his valuable comments, critiques and useful suggestions.

Finally, we must express our profound gratitude to our beloved wives for their unfailing support and continuous encouragement throughout our studies and through the process of researching and writing this thesis. I huge amount of time was invested in this research, thank you for your comprehension and sacrifices.

This research would have never been accomplished without the help and support of all these people. We are very grateful!

(7)

7

Table of contents

1. Introduction

1.1 Topic

The use of Inertial Measurement Units (IMUs) sensors has widely spread, particularly in the field of human motion assessment. Inertial Measurement Units (IMUs) typically including 3D accelerometers, gyroscopes and magnetometers are commonly used where body motion capture (Mocap) data is required [1]. As it is the case for Inertial Measurement Units (IMUs), Human motion capture (Mocap) is widely used and recognized for it uses and applications within so many different fields, such as health, sports, leisure, etc. [8].

The use of IMU goes back in time until 1930, where it was used in aircraft navigation and large devices. However, because of the size, cost and power consumption of the units they were so unpopular at that time and it was not possible to use them in smaller size devices and consumer applications. It is until recently, when the micro-electromechanical system (MEMS) IMU was introduced, that it started to gain popularity with a very attractive low cost, small size and low processing power. The demand for these sensors and their applications is so high that the competition among the different manufacturers has also increased and so, as mentioned before, the areas of applications for the IMUs has also widespread to areas such manufacturing, navigation, robotics, etc. [24]

1.2 Motion Capture Overview

Motion capture (Mocap) is the process of recording the movement of human or objects with the help of inertial measurements [2]. Mocap is used within different fields such as sports, healthcare, and music, etc. motion capture data is used today in many different forms. The most obvious is the entertainment field. Most animated movies and virtually all video games use and adapt motion capture information to provide human or human-like characters with plausible motion. Because there are so many realistic details that are difficult to coordinate with manually or procedurally defined motion paths, this motion

(13)

13

seems more real to observers. Most animated advertisements also use motion capture data to provide realistic motion. Motion capture systems are commercially available, and the two main types of systems are optical and magnetic. Now neither has a clear advantage over the other, but magnetic systems are significantly cheaper [50]. In our motion capture units are required to read data from a Mocap Unit, this will be shown in Mocap application, which it is possible to request and collect Mocap data simultaneously from the total number off the Mocap units are being connected

1.3 Motivation

During the awareness phase of the Design Science methodology used for this research, three key factors have been identified, through a literature review, which play an important role in motivating this research and contribute with an artifact in the form of a tool that could help improving the accessibility for developing Mocap applications. Each of these factors are described in the following sections.

1.3.1 Advances in IMUs

The literature analysis presented in section 2, provides a short panorama of the technological advances and improvements that IMUs have experienced. It introduces the miniaturization and portability of some of these IMUs in the form of wearable devices but mostly, it briefly describes the different areas where this technology is currently being used and the constant development into other areas. Not to mention, the possibility of the creation of new markets in the form of new products and services as described in the next indent. This gives an indication of how relevant this technology is and will be.

1.3.2 New Opportunities

There is a point of intersection between the different technologies described in section 2. The mixture of the Inertial Measurement Units (IMUs) technology and the Human Motion Capture (Mocap) uses and applications, together with the inclusion of current wearable’s, that include, both IMUs and Mocap applications, plus the constant growing of the context awareness of intelligent objects which are interconnected with each other and to the cloud in what is called the Internet of Things (IoT) could provide new ways to

(14)

14

improve and explore new capabilities for different needs in so many different areas. For example, with the miniaturization of technology, some IMUs are now found inside wearable devices [10] [11], this together with the Internet of Things (IoT) paradigm, where extensive amount of data is collected and analyzed [12], remote monitoring of people for health-related issues [2] is a possibility, and opens new opportunities as these technologies merge and advance in other areas where IMUs are already being used such as Sports [2], Music [4], etc. A final example of new type of services and products, that resulted from the intersection between these technologies, is Mocap systems marketed and branded as System as a Service (SaaS) [8].

1.3.3 Other Considerations

There are some important aspects that we need to consider from some of the systems that are already in existence. In one hand the use of IMUs for Motion Analysis, represent an inexpensive [18] [21] and viable alternative in comparison to other types of systems such as the optical ones which are expensive and only allow measurements in a restricted volume [17], on the other hand, commercially available, IMU systems for Motion Analysis come with a variety of packages, which are usually expensive and not customizable. In order to customize them, one is required to add more packages or modules which mean additional costs. Additionally, the data processing inside the IMU is unknown and restricted [22] and in some other cases, unfortunately, some of the protocols that some solutions use is proprietary [8] which means that the access to communicate to the sensors is straightened.

Other systems serve very specific needs, as the wearable gait measurement system for control of an exoskeleton robot [19] or the wireless evaluation system for in-home physical rehabilitation [20] and the Micro-IMU-Based Motion tracking system for virtual training in the field of military simulation [21]. Although these systems are “in-house” made systems and probably anyone could replicate them, they can be considered custom made and they are not meant to be used as a framework or interface for developing MoCap applications.

(15)

15

There are, however, some ready to use IMUs that provide an option to develop software, however, in some cases, the software is not open source, which means, the processing of the data is sealed and in some other instances, the firmware is not open source, whence one must rely on its pre-processed output [22].

Finally, there are already proposals to build inexpensive and reliable open source IMUs that can be used for motion capture data reading [22] [23] but the software part is missing. There is a need of a tool that could use these inexpensive IMUs to provide a framework or interface that could be used for the development of motion caption applications.

When considering what was previously described on each of these three main factors, finding a solution for improving the accessibility to the development of applications for anyone who wants to create a Motion Capture (MoCap) application could prove to be beneficial within many of the different fields where the Inertial Measurement Unit (IMU) technology is in use. Wherever IMUs are used with the purpose of collecting motion capture data, there is a need for a MoCap application. As previously stated, there is a need of a tool that could use these inexpensive IMUs to provide a framework or interface that could be used for the development of MoCap applications therefore improving the accessibility to the development of this kind of applications. And by improving accessibility we mean to improve the access to the development of such applications. That anyone could have access to this tool and use it to develop applications without any restrictions and at the same time they could modify this tool in accordance to their specific needs.

(16)

16

1.4 Research Goals

The purpose of this research is to find a solution, by designing a tool that could be used as a framework for the development of Motion Capture (MoCap) applications using Inertial Measurement Units (IMUs). The aim of this tool is to facilitate and improve the accessibility to the development of MoCap applications. Firstly, a literature review is conducted to describe and present factors about the relevance of the technology and the different areas of use and application, as well as to find characteristics or factors that need to be considered for the creation of a tool that could be used in the development of MoCap Applications. Secondly, based on the design science research methodology, an artifact is created through designing a tool and a prototype. All these in an attempt to address the following research questions:

RQ1. Which factors or characteristics need to be considered for designing a tool for developing Mocap applications with the use of IMUs?

RQ2. How the tool could be designed and implemented for developing Mocap

(17)

17

2. Related Work

In this section, we present an overview of the recent literature related to our research. We present some information about the relevance, widespread adoption and popularity of the IMUs and MoCap. Then we attempt to describe other technologies such as Wearables as well as Internet of Things (IoT) and how they can be related and relevant to the use of sensors and Motion Capture (MoCap) applications. Proving how all these technologies interlace to each other.

2.1 A Wireless Micro Inertial Measurement Unit (IMU)

When we studied a literature review, we found some articles that defined how the wireless Micro Inertial Measurement Units (IMUs) encountered the size, format and power use limits for embedded mobile application like the integration into clothes and shoes. By using Wearable sensors with huge scale, this is integrated with inertial IMUs. In their Micro-IMU, they applied modern accelerometers and gyroscopes which feature three axis technology and integrated analog to digital conversation with automatic temperature compensation in one-chip design.

2.1.1 IMUs

IMUs are frequently used in devices to measure velocity, orientation, gravitational force, etc. The use of these sensors has widely been spread, not only they can be found almost everywhere but they have many different areas of application. Ahmad et al., has summarized and exemplifies just a few of the applications of IMU in industries and human daily lives. For instance, some of the applications of IMUs are in Industrial Quality Control, Medical Rehabilitation, Robotics, Navigation Systems, Sports Learning and Augmented Reality Systems [24]. Next, some examples of how IMUs are been used in some of these areas:

(18)

18

2.1.2 Sports

The extraction of data and key performance metrics that will provide the coach and athlete with valuable information which will allow them to improve their performance can play an important role within this highly competitive area [2]. Thence the introduction of IMUs within disciplines such as swimming, where IMUs are used to perform a technical evaluation of swimming performance. Novel methods of analysis together with the incorporation of body worn inertial sensors have received much attention during recent years as a way to use this technology to improve the analysis of the adequate extracted metrics to enable efficient competitive and quantitative coaching among other benefits [3]. In addition to swimming, IMUs are been used, as well, in other sports such as baseball and volleyball [2].

2.2 Tracking Limb Motion Using Inertial Measurement Units

The healthcare department, it utilizes with IMUs that would help a variety of patients, for instance, tracking limp used inertial measurement units (IMUs) that helped to accomplish the development of low cost wireless real-time inertial limp tracking system for virtual training, which will be made an artificial limb to obtain the movement as a normal leg motion. In health care facilities, the reapplication of the motion in limp muscle plays a role in inertial measurement units. This system is designed to provide very accurate human body motion capture and interactive three dimensional by combining low cost MEMS Inertial Measurement Units.

2.2.1 Health

As noted by Roetenberg et al., the use of inertial sensors has become a common practice in ambulatory motion analysis [2] more specifically within individuals that required remote monitoring of their health status [7]. A specific area where researchers and clinicians, which address health issues, are being an ageing population, focusing their attention is towards maintaining or improving quality of life on patients that need constant monitoring of their ability to function but also identifying the loss of functional abilities in order to intervene as needed. The use of IMUs could help address this

(19)

19

problem, as proposed by Zijlstra et al. [5]. And in some instances, some wearable systems are already being used to monitor remotely Parkinson disease patients [6].

2.3 Music

The popularity among music researchers, composers and performers for motion capture (MoCap) has increased. Inertial sensors offer an alternative for collecting body motion data. These sensors attached to key areas of the human body allow the acquisition of relevant data for musical interaction [4].

2.4 Internet of Things (IoT)

In the same way as the sensor technology advances and progresses, so the number of sensors deployed everywhere. Sensors play an important role in the development of the Internet of Things (IoT). It predicts an era where billions of sensors are connected to the Internet. To process all the data collected by these sensors and make sense of such data, context-aware computing is proposed in order to solve this issue, as it has proven to tackle this challenge in previous paradigms such as the mobile and pervasive [12].

2.

5 A Wearable, Self-Calibrating, Wireless Sensor Network for Body

Motion Processing

In related work of motion tracking technology allows many applications that distance from medicine to the arts. While the large number of applications that require motion tracking, there is no single technology can solve the problems associated with every application. Using fused sensors, they developed a complete motion processing system that consists of a gateway where the human kinematics modeling is embedded.

(20)

20

2.5.1 Wearable’s

As mentioned before, IMUs are being used in different areas. IMU sensors have been used mainly for monitoring, controlling and tracking several human activities. These sensors can also be found in different wearable devices. Wearable devices can be applied in several domains, ranging from entertainment to medicine and safety of critical systems. For instance, wearable devices observe the vital signs of patients, expand human capabilities, monitoring replaced organs, or even alerting in case of a medical emergency [10]. This monitoring can not only be achieved at one specific place, but it can also be done remotely. Remote monitoring, by using wearable sensors, can allow people (users) to remain at home and lower their medical expenses [7].

There are some attempts to make this technology, not only portable and always available, but to find ways to improve the designs such as the improvement of the materials, for instance, the elasticity of the garment [11] as well as to improve the comfort ability. IMUs sensors are very beneficial to integrate with dynamic wear-ability that will allow the monitoring of the human body motion by designing a wearable that will prove to be comfortable and will allow the freedom of movement that is required by some specific users [9].

(21)

21

3. Research Method

The abstracts of the retrieved papers were read to verify whether a human motion capture system was used in the work. We focused on papers that use measurement systems such as Sports, music, healthcare environments. We carried out a literature search and studied various articles related to our research question. We collected and gathered a lot of resources.

In order to carry out our research and to answer our research questions, different research methods. The following figure presents a general overview of the research:

(22)

22

Firstly, we began with a general overview to the research approach, design science, followed by the specific research methods such as the framework (method artifact) and prototype (instantiate prototype) related to our research questions.

3.1 Design Science

According to March and Smith, in a widely cited paper contrasting design science research with natural science research, Design Science consists of two basic activities: build and evaluate in which building is concerned with the creation of artifacts, or, as they propose, four general outputs for design science research: constructs, models, methods, and instantiations [13]. In a similar proposal, Rossi and Sein [14] and Purao [15], have set their own list of design science research outputs, in which all but one of these, better theories, can be mapped to March and Smith’s list (Table 1).

Table 1 - Outputs of Design Science Research

Output Description

1 Constructs The conceptual vocabulary of a domain

2 Models A set of propositions or statements expressing relationships between constructs

3 Methods A set of steps used to perform a task – how-to knowledge

4 Instantiati

ons

The operationalization of constructs, models, and methods

5 Better

theories

Artifact construction as analogous to experimental natural science, coupled with reflection and abstraction

Our research is concerned with the creation of two artifacts’: a method and an instantiation. The method is the design and implementation of an open source framework which will work as an abstraction layer for the IMUs and the different functionalities

(23)

23

related to MoCap systems. The instantiation is a prototype application demonstrating the use of the method (framework). For these reasons and for what was expressed previously in this section, we decided to follow a Design Science Research approach for our research since it serves to the needs of this research project.

3.2 Evaluating the Artifacts’

Once the artifact has been developed, it must be evaluated [16]. Since the Design Science process is an iterative one, evaluation, in a sense, takes place continuously. However, in this section we describe the evaluation as a general approach to the artifacts, that evaluation consists of different criteria based on connectivity (how many IMUs can be connected to the system), functionality and usefulness of the artifacts’. The objective of this research is to show “proof of concept” via a prototype and demonstrate the feasibility of an open source solution for MoCap applications using IMUs. Next, we describe the research methods that are used to evaluate the artifacts.

3.2.1 Quantitative Evaluation

For this research, to prove and evaluate the relevance of the artifacts’, a set of metrics need to be collected and analyzed.

▪ Experiment

The quantitative evaluation consists of an experiment. The experiment will evaluate the method (framework), which is one of the artifacts, with the help of a second artifact, which is the instantiation (a prototype application). This experiment will serve as data collection method to acquire all the relevant metrics related to connectivity; see section 7.1 for a wider description. Since we are only interested in the data generated by the artifacts, a black box testing approach to run the experiment and collect the data will suffice.

(24)

24

3.2.2 Qualitative Evaluation

This evaluation aims to prove and evaluate the functionality and usefulness of the artifacts. The evaluation consists of an experiment which is briefly described as follows:

▪ Experiment

The experiment will evaluate the method (framework), which is one of the artifacts, with the help of a second artifact, which is the instantiation (a prototype application). This experiment will serve as data collection method to acquire all the relevant metrics related to connectivity; see section 7.2 for a wider description. Since we are only interested in the data generated by the artifacts, a black box testing approach to run the experiment and collect the data will suffice.

(25)

25

4. Empirical Analysis

The following analysis aims to give an answer to our first research question: RQ1 Which factors or characteristics need to be considered for designing a tool for developing MoCap applications with the use of IMUs?. We determined the appropriate framework library to use our tool. We found two frameworks which are aREST and Go framework more detail about aREST framework (see figure 2).

4.1 Descriptive Analysis

The compilation of the necessary information for this analysis was carried out through the search on the internet. To filter the content and target only information relevant to this research, the following keywords were used: “IMU”, “Open Source”, “Arduino”, “MoCap”. Thereafter, from the results, only platforms that offered wireless (Wi-Fi) connectivity were considered.

The purpose of this analysis is to motivate the need for the creation of an artifact in the form of a specialized framework that can be used for the development of MoCap applications with the use of IMUs; by presenting what is currently available in the market, specifically in the open hardware-software market, and at the same time by finding some of the characteristics and limitations that these platforms present.

4.1.1 IMUs Platforms Analysis

In this section we present an analysis of some of the platforms, with WIFI capabilities, currently available in the market for the development of MoCap applications with the use of IMUs.

(26)

26

▪ SparkFun

SparkFun is an open hardware platform that allows anyone to use it or modify it. The Read Board is their flagship board that can be programmed over a USB Mini-B cable using Arduino IDE (Integrated Development Environment) [25]. Their IMU platform is composed of a microcontroller board and a separate IMU sensor that can be integrated. The platform includes the following characteristics:

The microcontroller uses the Arduino IDE and the ESP32 Arduino Core third party library [37] for the development of onboard applications and because of the pair of Tensilica Xtensa-based cores, the ESP32 board requires its own GNU compiler as result of their non-standard ARM or AVR cores. There is a third-party framework called ESP-IDF [38] which can be used as well for the development of onboard applications which allows the use of an IDE such as Eclipse [39].

Microcontroller

• Model: DEV-13907 ROHS

• Integrated 802.11 BGN Wi-Fi transceiver

• Integrated dual-mode Bluetooth (classic and BLE) • ESP32 based

• Price: USD $19.95

Sensor

• SparkFun 9DoF Sensor Strick • Operating Voltage: 3.3V

• 3-axis accelorometer, 3-axis gyroscope and 3-axis magnetometer • Price: USD $14.95

• SparkFun 9DoF IMU Breakout - LSM9DS1 • Operating Voltage: 3.3V

• 9 Degrees of Freedom IMU

• 3-axis accelerometer, 3-axis gyroscope, and 3-axis magnetometer • Price: USD $14.95

(27)

27

Both of the sensors use the same in-house open source SparkFun LSM9DS1 Arduino Library [40] together with the Arduino IDE to ease the interfacing with the IMU for the development of applications.

▪ Adafruit

Adafruit is an open source hardware that can be programmed with the Arduino IDE [26]. Their IMU open platform consists of two different microcontrollers to choose from and one IMU sensor

(28)

28

Microcontroller

• Model: Adafruit - HUZZAH ESP8266 • Wi-Fi: ESP8266 Wi-Fi

• Espressif ESP8366 processor

• Pre-programmed with NodeMCU (Lua) • Price: USD $16.95

• Model: Adafruit Feather M0 WiFi-ATSAMD21 + ATWINC1500 • Wi-Fi: 802.11 bgn

• Operating Voltage: 3.3V

• WEP, WPA and WPA2 encryption

• ATSAMD21G18 ARM Cortex M0 Processor • Price: USD $34.95

Sensor

• Adafruit BNO055 Sensor

• Adafruit 9-DOF Absolute orientation IMU fusion Breakout - BNO055 • Accelerometer, Gyroscope and Magenetometer

• Fusion Algorithms

• High Speed ARM Cortex-M0 based processor

• Absolute Orientation (Euler Vector, 100Hz) Three axis orientation data based on a 360° sphere

• Absolute Orientation (Quaternions, 100Hz) Four-point quaternion output for more accurate data manipulation

• Angular Velocity Vector (100Hz) Three axis of 'rotation speed' in rad/s • Acceleration Vector (100Hz) Three axis of acceleration (gravity + linear

motion) in m/s^2

• Magnetic Field Strength Vector (20Hz) Three axis of magnetic field sensing in micro Tesla (uT)

• Linear Acceleration Vector (100Hz) Three axis of linear acceleration data (acceleration minus gravity) in m/s^2

• Gravity Vector (100Hz) Three axis of gravitational acceleration (minus any movement) in m/s^2

• Temperature (1Hz) Ambient temperature in degrees Celsius • Price: USD $34.95

• Adafruit Precision NXP 9-DOF Breakout Board

• 9 IMU (FXOS8700 3-axis accelerometer and magnetometer and FXAS21002 3-axis gyroscope)

(29)

29

For the development of applications, the microcontroller Adafruit – HUZZAH ESP8266 uses a third-party open source set of libraries, called Arduino Core for ESP8266 [41], together with the Arduino IDE. The Arduino Core provides support for the ESP8266 chip embedded on the board and it has implemented libraries to communicate over Wi-Fi using TCP and UDP, set up HTTP, mDNS, SSDP, and DNS servers, do OTA updates, use a file system in flash memory, work with SD cards, servos, SPI and I2C peripherals.

The Adafruit Feather M0 Wi-Fi board requires the use of the Arduino IDE as well as its own in-house set of libraries contained in a package called Adafruit SAMD Boards which includes support for the feather M0 board for the development of on-board applications. There is no way to contribute to the development of these libraries.

For the development of applications, in order to obtain raw data from the sensor BNO055, the in-house open source Adafruit BN0055 driver [42] is need it. If the data needs to be collected in standard format, the in-house open source Adafruit Unified Sensor Driver [43] can be used instead.

The Arduino IDE is used for the development of onboard applications, for the sensor Adafruit Precision NXP 9-DOF Breakout Board, together with the following in-house open source libraries: Adafruit FXOS8700 [44], Adafruit FXAS21002C [45], Adafruit Unified Sensor [43] and to generate orientation data from raw sensor outputs the Adafruit AHRS library [46] can be used.

▪ Arduino

Arduino is an open hardware platform. The flagship IMU platform from Arduino is the Arduino Genuine 101 which is a classic board with the latest technologies. The board recognizes gestures and features a six-axis accelerometer and gyroscope [27]. Everything is integrated in one single board with the following characteristics:

(30)

30

This board can be programmed with the Arduino IDE and its own set of libraries.

▪ X-IMU

X_IMU, although not an open hardware platform and their software and algorithm are license based [28], it is a platform that uses the Open Sound Control (OSC) message-based protocol for communication [29], which allows developers to make use of such protocol in order to communicate their applications with the IMU and collect the data.

Microcontroller & Sensor

• Model: Arduino/Genuino 101 CurieIMU • Operating Voltage: 3.3V (5V tolerant I/O) • Bluetooth LE

• 6-axis accelerometer/gyro

• No Wi-Fi (An extra Wi-Fi module can be integrated) • Price: USD $30.00

Microcontroller & Sensor

• Model: IMU (NGIMU) • Operating Voltage: 3.3V

• On-board sensors - embedded data processing algorithm • Triple axis gyroscope, accelerometer, magnetometer

• AHRS fusion algortihm: Quaternion, Rotation Matrix, Euler, Angles, Linear Acceleration, Earth Acceleration

• Barometric Pressure, humidity and 3 temperature sensors • High performance IMU with USB, Wi-Fi, serial RS-232 • SD card logging

• Price: USD $389.31 • Model: X-OSC

• Ob-board sensors

• Gyroscope, Accelerometer, and Magnetometer • High performance Wi-Fi (802.11b/g, 54 Mbps) • Price: USD $389.31

(31)

31

For the model NGIMU, the vendor provides and Open-Source GUI and API (C#) for windows for the development of applications. However, the device cannot be programmed or directly accessed. Data can only be accessed through the provided API or trough OSC protocol.

For the development of applications for the X-OSC board, the vendor does not provide any API or library. The devices cannot be programmed and access to the data it is only allowed through the use of OSC protocol over Wi-Fi.

▪ FreeIMU

FreeIMU is an open hardware framework for Orientation and Motion Sensing [30]. The FreeIMU project aims to design Open Hardware 9/10 DOM/DOF IMUs together with easy to use Orientation and Motion Sensing libraries. The framework is built on top of the Arduino platform and the latest model has the following characteristics:

Unfortunately, the project, at the moment of this analysis, was abandoned with no new developments since 2012 when the latest FreeIMU library was released with the support of the following boards:

Sensor

• Model: FreeIMU v0.4 Sensor

• FreeIMU v0.4 connects to Arduino Board

• FreeIMU v0.4r3, features the MPU6050 accelerometer and gyroscope, the HMC5883L magnetometer and the MS5611-01BA03 pressure meter altimeter module.

(32)

32

Table 2 - List of boards supported by FreeIMU

4.1.2 Open Frameworks Analysis

In the same way, as the descriptive analysis previously described, the compilation of the necessary information for this analysis was carried out through the search on the internet. To filter the content and target only information relevant to this research, the following keywords were used: “IMU”, “Open Source”, “Arduino”, “MoCap” “Framework”, “IoT”.

The following is an analysis of the different open source frameworks currently available for the development of different type of applications that make use of some of the hardware described in the previous section. These frameworks simplify the development of applications that required access to sensors generated data.

Arduino

-Sparkfun IMU Digital Combo Board - 6 Degrees of Freedom ITG3200/ADXL345

SEN-10121

-Sparkfun 9 Degrees of Freedom - Razor IMU

SEN-10736

-Sparkfun 9 Degrees of Freedom - Sensor Stick

SEN-10724

-Sparkfun 9 Degrees of Freedom - Sensor Stick

SEN-10183

-DIYDrones ArduIMU+ V3

-Generic MPU6050 Breakout boards (e.g.: GY-521, SEN-11028 and

other MPU6050 which have the MPU6050 AD0

(33)

33

▪ aREST API

First and foremost, the aREST APIs was developed with home automation applications in mind, but it’s now a complete solution used for internet of things projects such as mobile robots and drones, as well as healthcare applications, etc. The aREST framework provides a RESTful interface to several embedded boards and platforms. It allows the sending of commands as well as the reception of data within a JSON container [31].

Example:

In order to set digital pin 3 on an Arduino board running aREST to LOW and that it is connected to the network with an IP address 192.168.1.20, the following command should be performed:

192.168.1.20/digital/3/0

The answer from the board will be contained in a JSON message as shown next:

{“message”: “Pin D3 set to 0”, “id”: “1”, “name”: “arduino”, “connected”: true}

The aREST framework has the following characteristics:

a. The aREST framework supports the following: Arduino, Raspberry Pi, and the ESP8266 Wi-Fi chip.

b. The aREST API can be used to build web-based applications, to control and communicate with the different boards and platforms.

c. aREST allows the control of the projects from anywhere in the world. d. The aREST API facilitates the development of mobile applications.

(34)

34

e. The aREST library handles different types of communications, such as Serial, Wi-Fi, Ethernet, etc.

The aREST library is simple to use and it can be easily loaded into any of the supported boards. The aREST Server component is a Node.js module that can be deployed on a central server to control several boards. The following figure illustrates the different components of the aREST framework.

Figure 2 - aREST Framework

▪ GOBOT

Gobot is a framework written in the open source programming language Go [32], developed by Google, and it can be used to simplify the creation of solutions in the area of robotics, physical computing and the Internet of Things (IoT). Gobot provides

(35)

35

support for multiple drivers and adapters for controlling a wide variety of physical devices (see Appendix A and Appendix B).

Gobot’s implementation is heavily oriented on robotics; therefore, the “Robot” is the main software abstraction of the framework. The “Go Programs” or applications are called Robots, and they can run on a host machine communicating to connected devices, or they can run directly on a single-board Linux computer, or anywhere in between [33].

Gobot includes a RESTful API to allow the programs to query the status and control the different robot-devices shared on a network. The API is implemented using JSON-over-HTTP. Gobot has its own Command Line Interface (CLI) but Gobot can also be used in conjunction with the CLI Gort [34], to access important features right from the command line. Thus, allowing scanning, connecting, updating device firmware, etc. right from the command line.

Gobot provides an adaptor for microcontrollers that support the Firmata protocol [35] for communicating with microcontrollers from software on a computer (or Smartphone, etc.). It also implements and adaptor/driver to connect to MQTT servers and communicate by using the MQTT protocol [36], which it is an Internet of Things connectivity protocol designed as a lightweight publish/subscribe messaging transport protocol.

4.2 Critical Analysis

4.2.1 IMUs Platforms

The following table shows a comparison between all the relevant platforms found during the descriptive analysis. Each platform is composed by a microcontroller unit (MCU)

(36)

36

plus a sensor unit (IMU), except on those cases marked with an (*), where all the components are embedded in one single unit.

Table 3 - Comparison Table of IMUs Platforms

Platfor m Communications We ar-abi lity D o c Acces sible Cost Frame work (APIs) SparkFu

n YES (Wi-Fi + Bluetooth) NO

Y E S

YES NO

Adafruit YES (Wi-Fi) NO

Y E S YES NO *Arduin o (CurieIM U) YES (Bluetooth) Requires Extra Wi-Fi

Module NO Y E S YES NO *X-IMU

(NGIMU) YES (Wi-Fi + Bluetooth) YES

Y E S

NO YES

*X-IMU

(XOSC) YES (Wi-Fi + Bluetooth) NO

Y E S NO NO Arduino + FreeIMU

YES (Wi-Fi) YES

F e w

(37)

37

*Single Board Unit

The table (see table 2) considers the following characteristics:

a. Communications

A way to connect to the platform other than wired, specially through Wi-Fi. For the development of MoCap applications, it is a feature that allows the user to freely move around without the restrictions of wired technology.

b. Wear-ability

Was the design of the platform taking the wear-ability aspect into account? For the use of IMUs it is important that the sensors are placed in specific areas of the body therefore the need to consider wear-ability aspects when designing the devices. Firstly, to provide as much comfort as possible, secondly, to adequate the shape of the device to the shape of the body part where the device is attached and finally, to place the device in such a way that fail data can be avoided.

c. Documentation

Is the platform well documented so it will facilitate the development of applications?

d. Cost

Is the cost a low price? e. Frameworks (APIs)

Has the platform developed a framework or API to facilitate the development of applications and more specifically MoCap applications?

From the information gathered in descriptive analysis of the IMUs platforms and by taking into consideration the characteristics of the different platforms summarized in the table (see table 2) we can derive the following critical analysis:

(38)

38

1. All of the platforms comply with the communication aspect, all of them provide Wi-Fi technology.

2. However, in the wear-ability area, only Adafruit, X-IMU (NGIMU) take this aspect into consideration, this is because this is a commercial product aimed at the market of applications for IMUs. No so, the rest of the platforms that although they are not focused on a specific market, the MCU and IMUs could be used inside a wearable customized enclosure.

3. All of the platforms provide good documentation to ease the development of applications except for FreeIMU.

4. Only SparkFun, Adafruit and Arduino provide accessible prices. Apparently FreeIMU is no longer for sale.

5. Only one platform, X-IMU (NGIMU), provides an API for the development of advanced applications with IMUs. However, the API is proprietary and

requires licensing which could prove to be and expensive platform besides the high cost of their MCU/IMU.

6. The rest of the platforms, with the exception of X-IMU (X-OSC) which only allows the creation of applications that interact with the unit through OSC protocol, required their own set of libraries.

One of the most important aspects to note in this critical analysis is the fact that each platform requires its own set of libraries and some of those libraries are third party libraries. There is an opportunity to unify all these different libraries into one framework. It is also worth to note, that these libraries are APIs that, although they ease the development of applications, they are only useful to create applications for the MCU/IMU board. These are not APIs to ease the development of applications for PC or Mobile, even less for the development of specialized applications such as MoCap applications.

(39)

39

4.3 Empirical Analysis Conclusion

One of the most important aspects to note from the critical analysis of the IMUs platforms is the fact that each platform requires its own set of libraries and some of those libraries are third party libraries. There is an opportunity to unify all these different libraries into one framework. It is also worth to note, that these libraries are APIs that, although they ease the development of applications, they are only useful for developing applications that are to be load on the MCU/IMU hardware devices.

From the analysis of the Open Frameworks, we can note that aREST is an attempt to create a framework on the side of the application development for the MCU/IMU hardware device whereas Gobot, on the other hand, is an attempt to create a framework for the development of applications on the PC side. Both frameworks were created for a very specific use, and from there they have been trying to become more general, but none of the frameworks have the objective of facilitating the development of MoCap applications.

We can then conclude the following:

1. There is a need for a framework that will target more platforms than the aREST framework but at the same time, since the objective is to facilitate the development of MoCap applications, the framework should specialize in MCU/IMU platforms only.

2. There is a need of a well-designed framework such as Gobot but in a more popular, well documented programming language and platform.

3. Throughout the collection of data for the empirical analysis, two communication protocols commonly used in wireless communication stood out: MQTT and Firmata.

These characteristics and factors need to be considered when designing a framework that can be used to ease the development of MoCap applications.

(40)

40

5.Framework

IMOTRAF, which stands for Inertial Motion Tracking Framework, represents the second contribution of this research and provides an answer to the second research question: How a tool can be designed and implemented for developing MoCap applications with the use of IMUs?

IMOTRAF, is a Windows Runtime Component library, this allows the developers to use the library in their projects and write their MoCap applications in any of the Windows Runtime Languages (C++/CX, C# and JavaScript). The library enables the integration of an MCU + IMU into a Windows (UWA) MoCap application by controlling these units via Wi-Fi, using firmata [35] as a communication protocol. The framework, based on the open source Windows Remote Arduino library [49], allows the connection to several numbers of units and it provides an interface to interact with their GPIO, Analog and PWM capabilities.

As part of the scope of this research, the designing of the IMOTRAF framework consisted of providing support and to integrate a microcontroller unit such as Arduino [27]. And because the framework makes use of the firmata protocol to communicate to an MCU, it doesn’t only support Arduino boards but any other board that can run Standard Firmata [35] or any equivalent.

The following figure shows a list of the Arduino functionalities that IMOTRAF is capable of controlling:

(41)

41

Figure 3 - List of Arduino Functionalities

5.1 Framework Architecture

The framework consists of three layers: Interface, Protocol, and Transport (see Figure 4). The physical communication layer (Transport), is the bottom layer, which is necessary to allow raw data to be exchanged between Windows and the MCU + IMU. In the middle, the communication layer (Protocol), decodes the raw incoming data into meaningful messages. And, lastly, at the top, the Interface, which abstracts away all protocol messages and allows the control of the MCU.

• digitalWrite - Digital Write • digitalRead - Digital Read • analogWrite - Analog Write • analogRead - Analog Read

• pinMode - Sets the mode for any pins

• Eventing - Receive events when values change GPIO - Analog and Digital I/O

• Send/Receive data to and from other devices over I2C I2C/TwoWire

• Firmata SysEx command Custom Protocols

(42)

42

Figure 4 - IMOTRAF Layers

Each layer can be used as an entry-point. The topmost layer (Device) exposes an interface to all the basic Arduino instructions like GPIO and communication to other devices through I2C. The protocol layer provides an interface with the Firmata protocol, which allows the creation of advanced functionalities.

IMOTRAF s architecture (See figure 5) was designed considering that each layer could grow in the future by implementing new functionalities and increasing the support of new devices (MCUs) and/or protocols (e.g. MQTT) and/or serial communication devices (e.g. Bluetooth). The top layer is the highest level of abstraction, it represents what we have called a Motion Caption Unit, or MoCapU for short. It is composed by an MCU/Device (e.g. Arduino, NodeMCU, etc.) plus an IMU.

MoCapU (Interface)

Firmata (Protocol)

WiFi

(43)

43

(44)

44

6.Prototype

As previously described, within the Design Science research approach, the development of a prototype, serves as an instrument to this research to evaluate an artifact; in this case, the IMOTRAF framework and to show “proof of concept”.

6.1 Design

6.1.1 Backend

Since the IMOTRAF framework was developed in C++ using the Windows Runtime Component library, there are several programming languages to choose from when developing a prototype. For this prototype, C# was chosen among those programming languages. The backend of the prototype interacts directly with the IMOTRAF framework by using its public API.

6.1.2 User Interface (UI)

The user interface consists of a screen which allows the user to connect simultaneously to a fixed number of devices (Motion Capture Units) and display the data from each of IMUs. As shown in Figure 6, there are two sets of buttons available to the user. the first set, in red, allows the user to connect to a specific device via Wi-Fi. The second set, in grey, allows the user to send a request to the Mocap Unit in order to start receiving data from the IMU.

(45)

45

Figure 6 - UI with no connections

Once a connection is established with a specific MoCap Unit, the button representing the connection to that unit becomes green. At the same time, the button for requesting motion capture data from that specific unit becomes available as well (see Figure 7).

(46)

46

As request to read motion captures data from a MoCap Unit, this will be displayed in its respective data field, as shown in Figure 8.

(47)

47

7.Evaluation and Results

The highest level of abstraction of the IMOTRAF framework represents a Motion Capture Unit, which is a hardware unit composed by an MCU with Wi-Fi capabilities and an IMU (see Figure 5). Since one of the requirements for the framework is the capability to support different open hardware manufacturers (i.e. Arduino, NodeMCU, Adafruit, etc.) and different hardware configurations, different manufactures and configurations were tested and tried out in order to validate its functionality.

Before describing each of the evaluations and their respective results, following there is a brief description of each of the hardware configuration used in our evaluations. This, in an attempt to provide a classification for each of the different configurations, as well as to present the information in a more organized and synthesized manner.

● Configuration

This Mocap Unit configuration consists of an Arduino Uno board, an ESP8266 Wi-Fi microchip and an MPU-9250 IMU breakout.

7.1 Qualitative Evaluation and Results

The purpose of this evaluation is to evaluate different functionalities of a certain hardware configuration together with the IMOTRAF framework.

7.1.1 Wi-Fi Connection

This is a simple evaluation that consisted of validating the functionality of the IMOTRAF framework by proving that a Wi-Fi connection can be established. We are going to test a single button and numerous buttons to see whether IP-Address connections are being established or not through Wi-Fi. Hence, we should be expected two results based on the following questions:

a) When using the prototype, can a Wi-Fi connection be established to one MoCap Unit (Y/N)?

(48)

48

b) When using the prototype, can a Wi-Fi connection be established to all the available Mocap Units (Y/N)?

7.1.1.1 Experiments

● Experiment – Test

For this experiment Mocap Units of Configuration A type were used. The units were connected to a Wi-Fi network as well as the PC running the prototype.

a) The first test consisted of clicking one connect button from the prototype application to connect to one Mocap Unit.

Two possible outcomes could be expected:

- Passed, with a “Connection to – IP - address - is ready” message on the prototype’s display or,

- Failed, with a “Connection Failed” message on the prototype’s display.

b) The second test consisted of clicking 6 of the connect buttons from the prototype application to connect to all the Mocap Units.

- Passed, with a “Connection to - ip address - is ready” message on the prototype’s display for each specific connection between the prototype and the Mocap Unit or,

- Failed, with a “Connection Failed” message on the prototype’s display for each specific connection between the prototype and the Mocap Unit.

7.1.1.2 Results

● Test

a) The first test passed. The question was answered with a Yes. b) The second test passed. The question was answered with a Yes.

(49)

49

7.1.2 Data Collection

This is a simple evaluation that consisted of validating the functionality of the IMOTRAF framework by validating a specific Mocap unit that it is possible to collect data from single Mocap unit. The second test is based on how to collect data various Mocap units simultaneously. Therefore, we are expecting two results that is based on the following questions:

a) When using the prototype, is it possible to request and collect Motion Caption data from one of the Mocap Units (Y/N)?

b) When using the prototype, is it possible to request and collect Motion Caption data from all the available Mocap Units simultaneously (Y/N)?

7.1.2.1 Experiment

For this experiment Mocap Units of Configuration A type were used. The units were connected to a Wi-Fi network as well as the PC running the prototype. As a precondition, a connection to a Mocap Unit should be established.

a) The first test consisted of clicking the correspondent Read MPU button from the prototype application to request and collect Motion Caption data from the Mocap Unit.

- Passed, the prototype would display the collected data on screen.

- Failed, the prototype would not display any data on the screen.

b) The second test consisted of clicking 6 of the correspondents Read MPU buttons from the prototype application to request and collect Motion Caption data from the Mocap Units.

(50)

50

- Passed, the prototype would display the collected data on screen for each of the connected Mocap Units.

- Failed, the prototype would not display any data on the screen for any of the connected Mocap Units.

7.1.2.2 Results

● Test

c) The first test passed. The question was answered with a Yes. d) The second test passed. The question was answered with a Yes.

7.2 Quantitative Evaluation and Results

The purpose of this evaluation is to test different functionalities of a certain hardware configuration together with the IMOTRAF framework.

7.2.1 Wi-Fi Connection

This is a simple evaluation that consisted of validating the functionality of the IMOTRAF framework by proving its capability to connect to several Mocap Units simultaneously. As mentioned before, one of the limitations of the existing frameworks is the number of devices that can be connected simultaneously. This limitation is related to the type of connection used, so to say, Bluetooth. By choosing Wi-Fi as a connection means, it is possible to allow a mayor number of connected devices that Bluetooth allows. For this specific evaluation we focus on the number of devices connected simultaneously by answering the following question:

a) When using the prototype, how many connections can be established based on the total number of Mocap units available?

7.2.1.1 Experiments

(51)

51

For this experiment Mocap Units of Configuration A type were used. The units were connected to a Wi-Fi network as well as the PC running the prototype.

a) The test consisted of clicking as many as possible of the connect buttons from the prototype application to connect to their correspondents Mocap Unit.

- A “Connection to - ip address - is ready” message on the prototype’s display for each of the Mocap Units or,

- A “Connection Failed” message on the prototype’s display for a specific failed connection with a Mocap Unit.

7.2.1.2 Results

● Test

Six MoCap Units were successfully connected. This is the total of units available for testing for that specific configuration.

7.2.2 Data Collection

This is a simple evaluation that consisted of validating the functionality of the IMOTRAF framework by proving that it is possible to request and collect Motion Caption data simultaneously from the total number of the Mocap Units connected. This, in order to prove one of the main requirements collected from the literature review and the empirical analysis, which is, the possibility to use one of the widely used communication protocols, namely, firmata. For this evaluation we test the capability of the protocol and the framework to collect data from a specific number of devices. We test the capability by answering the following question:

(52)

52

7.2.2.1 Experiment

For this experiment Mocap Units of Configuration type were used. The units were connected to a Wi-Fi network as well as the PC running the prototype. As a precondition, a connection to a Mocap Unit should be established.

a) The test consisted of clicking as many as possible of the Read MPU buttons from the prototype application to request and collect data from their correspondent Mocap Unit.

- The prototype would display the collected data on screen. - The prototype would fail to display any data on the screen.

7.2.2.2 Results

● Test

(53)

53

8. Limitations & Threads to Validity

Although our objective is to produce results that are as generalize as possible, still there are some limitations to the conclusions we will be able to draw from our research. Since our studies and validations are to be limited to a certain type of MoCap application, we do not claim to cover all the different areas where MoCap applications are in use. However, the main purpose of the resulted artifact is that it can be used and expanded by anyone, as needed, thus providing some flexibility for the artifact to cover other areas where MoCap applications are used. In other words, our artifact can be used as a foundation and could be a direct contribution to the development of IMUs and MoCap applications in general.

One of the scopes of the IMOTRAF framework is that of supporting different hardware manufacturers, architectures and configurations. However, because of the lack of resources, this research was limited to a couple of hardware configurations (see section 7). We only counted with a total of two MCUs with a working IMU plus six MCUs without a working IMU.

(54)

54

9. Discussion, Conclusion and Future Work

With the help of the conducted a literature review, we could observe how important the use of motion capture technologies have become and how they have widespread into different areas. With an analysis of the technology that is currently available in the market we were able to find some characteristics or factors that are to be considered when creating a tool that could improve the accessibility to the development of MoCap applications using IMUs. Through this, we were able to provide an answer to our first research question. From our results shown in section 4.3, we could conclude that:

1) There is a need for a framework that specifically targeted low cost – open hardware architectures. The framework should facilitate the development of MoCap applications using IMUs and had to be implemented using at least one of the most common used software architectures: aREST or Client-Server.

2) There is also needed of a well-designed framework, well documented and able to offer the option to develop MoCap applications with the help of common-well known programming languages and platforms.

3) In our research, we were also able to identify that the framework should use at least one of the two most common communication protocols used in wireless communication, either MQTT protocol or Firmata protocol.

4) Lastly, we were able to identify that most of the commercially available frameworks have a limitation on the number of devices that can be connected, wirelessly, to a central pc. This, as part of the limitation bound in the Bluetooth protocol, which is commonly and broadly used among different frameworks. Therefore, the need of a framework that would allow the connection, wirelessly, of a multiple number of devices via Wi-Fi.

This collected information, was then used as input data to come up with the designing of an artefact, which it encompassed the second part of our research. The result was the implementation of the IMOTRAF framework, our first artifact, thus providing an answer