Institutionen för datavetenskap
Department of Computer and Information Science
Final thesis
Andrei Buhaiu
by
Radio Frequency Remote
Control Unit with Gyro and
Accelerometer
LIU-IDA/LITH-EX-A--13/051--SE
2013-11-07
Linköpings universitet
Linköping University
Department of Computer and Information Science
Final Thesis
Radio Frequency Remote
Control Unit with Gyro and
Accelerometer
by
Andrei Buhaiu
LIU-IDA/LITH-EX-A--13/051--SE
2013-11-07
Supervisor: Ivan Ukhov
Examiner: Petru Ion Eles
Abstract
Today digital TV receivers (Set-Top-Boxes) are mainly controlled by IR-based remote control units (RCUs). With new services emerging in the receivers where better browsing and pointing abilities are desirable (VOD services, Web services, games etc), one solution is a new type of RF remote control. An RF RCU has a number of advantages, e.g. when in range, it has 100% reliable transmission, it is not sensitive to direction, and it does not require a free way to the receiver (i.e. it allows the receiver to be hidden behind the TV-set or in a cabinet).
Contents
Abstract i 1 Introduction 1 1.1 Background . . . 1 1.2 Overview . . . 2 2 Goals 4 2.1 Goals . . . 4 2.2 Challenges . . . 52.3 State of the Art . . . 6
3 Data Acquisition 7 3.1 Development Choices . . . 7
3.2 Communications Protocol . . . 8
3.3 Remote control . . . 9
3.3.1 Gyroscope . . . 9
3.3.2 Low Power Accelerometer . . . 10
3.3.3 Radio Transmitter . . . 10 3.3.4 Keyboard . . . 11 3.4 Receiver . . . 12 3.5 Set-top box . . . 12 4 Heuristics 13 4.1 Determining Orientation . . . 13
4.1.1 Stable Remote Control . . . 15
4.1.2 Unstable Remote Control . . . 17
4.1.3 Using the orientation . . . 18
4.2 Confidence Mechanic . . . 19
4.3 Cursor Reset . . . 21
4.4 Key Presses . . . 21
CONTENTS CONTENTS
5 Development 22
5.1 Compiler . . . 22
5.2 Receiver . . . 23
5.3 Set-top Box Driver . . . 23
5.4 UI Integration . . . 24 5.5 Opera Integration . . . 24 5.6 Hardware Updates . . . 25 6 Conclusion 27 6.1 Results . . . 27 6.2 Future Work . . . 28 6.2.1 Z-axis movement . . . 28 6.2.2 Heuristics . . . 28 6.2.3 Communications Protocol . . . 28 6.2.4 Gesture Movements . . . 28 6.2.5 Pairing . . . 29 6.2.6 Keyboard . . . 30 Bibliography 31 iii
List of Figures
1.1 Overview of the components . . . 3
3.1 Structure of a mouse/keyboard packet . . . 8
3.2 Schematic of the keyboard . . . 11
4.1 Acceleration and Rotation Orientation . . . 15
4.2 Division into quadrants for the angle between the acceleration on the X and Z axes . . . 17
4.3 Confidence values for different bases . . . 20
5.1 Example of complete communication between remote control and STB . . . 23
5.2 Keyboard Schematic with removed component circled(C27) . . 25
5.3 Remote control with removed component circled(R14) . . . 26
Chapter 1
Introduction
1.1
Background
Modern entertainment systems are seeing a rapid development of more advanced input technologies. At the center of this wave of new technologies are gesture recognition and pointing through the use of accelerometers and optical sensors.
Most of these techniques have been used with gaming consoles. However, other entertainment systems are now picking up on the trend. The great advantage of using gesture recognition systems is that it allows users to interact with a system in a natural and intuitive way. An example of such a project is described in Boulos et al.[7]. The caveat is that translating the data obtained from the device into actual input is difficult.
One of the ways to integrate the data is by transforming it into traditional mouse movement. This allows users to interact with a new input system while expecting a traditional reaction from the user interface. While this type of integration is easily accepted in most cases it does not use the technology to its full extent.
The other way is to integrate the data as three dimensional movement. This can offer a smoother and more intuitive experience, however, it greatly depends on how the application manages to integrate the movement. Three dimensional input has been currently used mostly for gaming as most motion controllers are deployed within gaming consoles. Bowman et al.[8] analyses the relevance of such interfaces in great depth.
This project follows the full integration of a remote control using a gyro-scope and accelerometer with a set-top box. This includes the code for the remote control data acquisition, the code for the receiver and the mouse-like driver for the set-top box.
Master Thesis - Andrei Buhaiu 2
1.2
Overview
The motivation for the project is to offer the users of digital television a more intuitive interface. Even though such interfaces are not new they have not been used in other entertainment systems except for gaming con-soles. This project aims to prove that developing such interfaces for digital television is both easy and worthwhile for the user.
The starting point is a design by Nordic Semiconductor that uses a re-mote control with a trackpad and keyboard that is communicating with a USB dongle that acts as the receiver. The major change is the fact that the receiver is now inside the set-top box. Another big hurdle was moving the development environment to a Linux based one instead of the one on Windows.
The main components of the project are: • Remote Control Code
• Receiver Code • Set-top Box Driver
• Integration Code with the Set-top Box UI
Master Thesis - Andrei Buhaiu 3
Chapter 2
Goals
2.1
Goals
The goal of the project is to deploy an innovative input technology for set-top boxes. This means full development, from the remote control and receiver code to integration with the current user interface on the set-top box.
The initial goals of the project were:
• Fluid communications between the remote control and the set-top box • To use the remote control as a mouse pointer in the native applications
running on the box
• To use the remote control as a mouse pointer in the browser running on the box
• Develop an application to showcase the capabilities of the remote con-trol
The first item on the list was a prerequisite for the others in fact. The mouse pointer for the native applications and browser have proven to be quite similar due to the fact that the native application is run in the browser user interface right now.
The last item on the list was dropped because it should really be the subject a different thesis.
A more business oriented goal was to be able to have this technology demonstrated at different exhibitions. This has also been achieved.
Master Thesis - Andrei Buhaiu 5
2.2
Challenges
One of the biggest challenges was implementing the communication be-tween the remote control and the set-top box. Designing an efficient protocol is key in order to offer a good response time and low power consumption.
The most obvious challenge of the project is translating the physical mo-tion into cursor movement. This is because the large swings of acceleramo-tion do not translate into traditional mouse movement easily. It is complicated to have both rapid and slow motions that allow precise movement. This can be divided into two separate issues, the noisy nature of the sensors and the unpredictable nature of human hand movement. Both of these are alleviated by implementing a confidence mechanic that focuses the movement into the particular direction that the previous movement seems to be following.
Determining the actual orientation of the remote control is also a relevant topic. This is due to the fact that the acceleration and rotation of the remote control are relative. Therefore the raw data received from the gyroscope has to be processed to ensure that it is translated into an absolute coordinate system.
Master Thesis - Andrei Buhaiu 6
2.3
State of the Art
The innovative part of the project lies in demonstrating that this rather new technology can be deployed on a wide scale. The technologies used in this project can be deployed for any type of computerized system that uses a pointer interface.
Even though the technology itself has been used before it was only de-ployed in gaming consoles, dedicated systems that showcase the technology, as can be seen in Cheema et al.[9]. This project aims to demonstrate that it is worthwhile to use such technologies in any mass produced entertainment system. The reason why this is an interesting proposition is that most of the new digital television systems offer extended functionality and thus benefit greatly from this very intuitive and intriguing user input technology.
Another innovative concept is the radio frequency channel used instead of the traditional infrared remote controls. This allows faster transmission of large amounts of data while still using considerably less power than the traditional wireless communication.
Chapter 3
Data Acquisition
3.1
Development Choices
During the development process a number of alternatives have arisen and in this section we will motivate our choices.
One of the first obvious choices we made was to move the development environment from a Microsoft Windows based one to a Linux based one. This was done because most of the people at the company are using such an environment and my personal familiarity with the Linux environment would speed up the development.
The initial design had the radio receiver process the raw data and send cursor movements. We chose to remove the processing from the radio receiver and move it into the set-top box driver. There are a number of reasons for doing this:
• It is far easier to change the processing of the raw data in the set-top box driver
• The processing power of the set-top box is much greater than that of the receiver, thus allowing more advanced heuristics to be deployed • Reduced code size for the receiver, which is relevant as the program
memory is restrictive
• Better response times as the receiver only takes care of relaying data from the remote control to the set-top box
Master Thesis - Andrei Buhaiu 8
3.2
Communications Protocol
The sensors are represented by the gyroscope and accelerometer. The data is packaged according to a simple protocol and then it is sent to the receiver over radio. From there it is sent to the set-top box over I2C. More
details regarding the I2C protocol can be found in [1].
The same protocol is used for structuring packets between the remote control and receiver and between receiver and set-top box. This works be-cause the receiver simply relays the data it received on radio to the set-top box over I2C.
The protocol is rather simple. Before any packet of actual data is sent a header packet is sent. The header packet is currently 5 bytes, right now only 3 of them are used.
Header Magic Value Flags Length Reserved Reserved
• Header Magic Value - the magic value that identifies this packet as a header packet, right now the value is 0x77
• Flags - specifies what kind of packet is going to follow the header packet. A packet can have multiple type flags set. There are two flags available currently:
– Mouse – Keyboard
• Length - specifies the size of the following packet. The keyboard packet can vary in size, the mouse packet is of fixed size. If a packet contains both mouse and keyboard data, the mouse data comes in first, due to the fixed size, and the keyboard data follows.
Figure 3.1: Structure of a mouse/keyboard packet ACCX refers to the acceleration on X axis.
Master Thesis - Andrei Buhaiu 9
3.3
Remote control
The remote control has a number of components that are of interest: • gyroscope + accelerometer
• low power accelerometer • radio transmitter
• keyboard
The remote control model is based on the nRFready 2.4GHz RF Smart Remote[5].
The interaction and configuration of the other peripheral devices is done through I2C. The remote control acts as the master and all the other
periph-erals are configured as slave devices.
3.3.1
Gyroscope
The device actually contains a 3-axis gyroscope and a 3-axis accelerom-eter. We will refer to the device as the gyroscope further for simplicity and to avoid confusion with the second low power accelerometer.
The gyroscope[3] is the essential component as it measures the acceler-ation and rotacceler-ation of the remote control. From these readings the mouse driver running on the set-top box will generate the actual movement.
The gyroscope was set at the highest sampling rate and largest range. This is because finesse does not benefit us and we want to be able to process faster movement.
Master Thesis - Andrei Buhaiu 10 Byte Description 1 acceleration X MSB 2 acceleration X LSB 3 acceleration Y MSB 4 acceleration Y LSB 5 acceleration Z MSB 6 acceleration Z LSB 7 temperature MSB 8 temperature LSB 9 rotation X MSB 10 rotation X LSB 11 rotation Y MSB 12 rotation Y LSB 13 rotation Z MSB 14 rotation Z LSB
The temperature data is not relevant for us so it is simply ignored. The acceleration and rotation data is packaged by the remote control into a 12 byte packet and sent to the receiver.
3.3.2
Low Power Accelerometer
The low power accelerometer is used for power management. If the gyro-scope does not detect any movement for 5 seconds it will go into sleep mode and the keyboard and low power accelerometer interrupts will wake it up from that state.
Power mode Electric Current Battery lifetime Running 10.5mA 5 days
Sleep 0.08mA 625 days
The battery lifetime is estimated for a AAA battery with 1200 mAh electric charge. The power consumption for the running mode could be reduced by setting some of the parameters of the remote control to less demanding values. The power consumption in running mode has not been relevant for this project, as performance was what the project focuses on.
3.3.3
Radio Transmitter
The radio transmitter works on 2.4GHz frequency. The output power is set to -18dBm. Radio transmission works well over short distances of around 3 meters.
Master Thesis - Andrei Buhaiu 11
The radio transmitter is very relevant from a power consumption point of view. The relevant issue is having the radio transmitter working only when we actually have data to send.
All the parameters of the radio transmitter can be changed, the output power can be 0, -6, -12 and -18dBm. The frequency can be modified in 1MHz increments up to 512MHz. We simply chose the parameters that offer the lowest power consumption.
3.3.4
Keyboard
The keyboard is actually a standard television remote control with some extra buttons. The letters are structured just like in a mobile phone keyboard with letters on the number keys.
Because there are no actual mouse buttons on the remote control some buttons will be configured to represent the different mouse clicks.
A future improvement would be to have a traditional keyboard layout for the remote control instead of the mobile phone layout currently available. The current layout makes it hard to browse easily.
The schematic of the keyboard can be seen in Figure 3.2.
Master Thesis - Andrei Buhaiu 12
3.4
Receiver
The receiver’s only goal is passing the data received from the remote control and relaying it to the set-top box. It is configured as an I2C slave.
Because of this, when a message is received from the remote control an interrupt is triggered so that the set-top box can issue the I2C master read.
In the initial design the receiver CPU was used for some processing of the data received from the remote control. Our implementation simplifies the receiver greatly and uses it to simply pass the packets from the remote control to the set-top box. This was done because the set-top box has much more processing power than the receiver.
3.5
Set-top box
The set-top box has a driver that registers a callback that will be called when an interrupt is triggered from the receiver device. The data will then be processed by the heuristics into actual mouse movement.
In our implementation all the processing is done on the actual set-top box, in the driver. This is because the set-top box has more processing power than the dongle. Also code size is an issue because of the compiler used.
The heuristics are completely separated from the other processing. This is done to allow easy improvement of the heuristics later on. The driver is responsible for taking the accelerometer data that is structured as acceler-ation and rotacceler-ation on each one of the three axes and then turning it into a mouse coordinate movement that is structured as delta movement on the three axes.
Chapter 4
Heuristics
After the data has been collected it must be processed in order to be transformed into actual mouse movement. There are a number of heuristics implemented: • Determining Orientation • Confidence Mechanic • Cursor Reset • Key Presses
4.1
Determining Orientation
The orientation of the remote control is critical to moving the mouse cursor in a fluid fashion. Because of this the heuristics that determine the orientation of the remote control are the most important ones. These issues are outlined in more depth in Spano[12].
The heuristic used to determine the orientation of the remote control is in fact formed out of two very different heuristics, one for the case that the remote control is stable and another for the remote control in motion.
Determining whether the remote control is currently in motion or not is a critical part of the heuristics. The problem is that if the stability threshold is too high the remote control can get into a situation in which it is hard to determine the orientation of the remote control because of continuous movement. If the threshold is too low it will think it is stable when it is still in motion and the orientation it will interpret will not be the actual one.
Another issue with triggering the stable threshold too early is that it resets the confidence. This translates into slowing down the movement in
Master Thesis - Andrei Buhaiu 14
that direction; however, this is a good thing if the user is trying to move more precisely.
The reason why we need to have two separate states is that using the stable heuristic when the remote control is not stable the acceleration of the actual movement will interfere with gravity.
The issues with the unstable heuristic are related to the fact that it is very hard to determine for how long the rotation movement lasted. This is usually good enough for short periods of time. However, if the remote control is in continuous motion for long periods of time it will have some sampling errors and one particular direction will become biased thus shifting orientation into that particular direction. This is also the reason why whatever the unstable heuristic has previously determined is completely ignored once the remote control is stable. The stable heuristic is always superior to the unstable one.
Master Thesis - Andrei Buhaiu 15
4.1.1
Stable Remote Control
In this situation the rotation movement on the remote control is below a particular threshold and thus the remote control is considered not to be in motion, or at least it is performing a slow movement.
For this scenario the acceleration readings can be used to determine the orientation of the remote control. Figure 4.1 shows the acceleration and rotation on a 3-axes coordinate system.
Figure 4.1: Acceleration and Rotation Orientation
The following notations are used throughout this chapter:
accX, accY and accZ represent the acceleration read by the sensors on the
X, Y and, respectively, Z axes.
rotX, rotY and rotZ represent the rotation read by the sensors on the X,
Y and, respectively, Z axes.
∠XZ is the angle between the force pushing on the remote and the XZ
plane.
compensationZ is the compensation for the Z axis.
ΔROT is the change in rotation around the Y axis.
ΔX is the change to the X axis.
Master Thesis - Andrei Buhaiu 16
∠min threshold and ∠max threshold are the threshold angles used to
approxi-mate the angle to a straight horizontal or vertical position.
The only relevant angle is the one between the X and Z axes. This is because the Y rotation is not used for mouse cursor movement. The formula for this angle is:
accZ = accZ+ compensationZ
accXZ=
� acc2
X+ acc2Z
∠XZ = asin(accX/accXZ)
The Z axis sensor seems to be biased towards one direction and that is why a compensation is added to it in order to get a realistic reading.
Master Thesis - Andrei Buhaiu 17
4.1.2
Unstable Remote Control
If the remote control is currently moving over the stability rotation thresh-old. The acceleration will not provide relevant data. This is because the movement of the remote control also affects the acceleration and, thus, it is harder to determine gravity reading.
The Y rotation is used to determine how the angle between X and Z changes. The difficult part is how to determine for how long the remote control moved at that speed. Because of this, sometimes, the rotation is not entirely accurate.
The formulae regarding the unstable rotation are focused mostly on de-termining in which quadrant the remote control currently is:
∠XZ =
�
∠XZ− ΔROT, if sign(accX)�= sign(accZ).
∠XZ+ ΔROT, otherwise. (4.1) ∠XZ = � Π− ∠XZ, if∠XZ > Π/2. −∠XZ, if∠XZ < 0. (4.2)
When a quadrant switch occurs due to the situations in formula (4.2) the sign for the X or Z acceleration is updated artificially. We did not get another stable reading, but we do know that now we should be in a quadrant where the acceleration has changed direction.
Figure 4.2: Division into quadrants for the angle between the acceleration on the X and Z axes
Master Thesis - Andrei Buhaiu 18
4.1.3
Using the orientation
After we have determined the orientation of the remote control we have to decompose the acceleration on the X and Z axes into cursor movement. This is done by the following formulae:
ΔX =
−rotX ∗ sign(accX), if∠XZ <∠min threshold.
−rotZ ∗ sign(accZ), if∠XZ >∠max threshold.
sign(accX)∗ −rotX∗ cos(∠XZ)+
sign(accZ)∗ −rotZ ∗ sin(∠XZ)), otherwise.
(4.3) ΔY =
rotZ ∗ sign(accX), if∠XZ <∠min threshold.
−rotX∗ sign(accZ), if∠XZ >∠max threshold.
sign(accZ)∗ −rotX∗ sin(∠XZ)+
sign(accX)∗ rotZ ∗ cos(∠XZ)), otherwise.
(4.4)
The ∠max threshold and∠min threshold are used in order to force the remote
control to behave as if it is completely vertical or horizontal. This is done to remove any small movement on the axis perpendicular to the one where the main movement is performed, meaning that the user can actually draw a straight line movement with the remote control.
Master Thesis - Andrei Buhaiu 19
4.2
Confidence Mechanic
The confidence heuristic is used to prevent shaky mouse movement. The remote control is very sensitive and it can detect very small movements, however, most of the times those movements are undesired and just a side effect of another movement.
A confidence of -1 means full confidence in moving left or down, depending on the axis, a confidence of 1 means full confidence in moving right or up. Full confidence means that the movement will not be scaled down at all. A confidence of 0 means that there is no bias, whatever the next movement is it will be treated the same.
The algorithm that updates the confidence is rather simple. Each move-ment adds confidence on the X and Y axes depending on how much it moved and in what direction. Thus, if we have continuous movement in one particu-lar direction, confidence will increase and the mouse will move faster in that direction. If the movement switches to the opposite direction then confidence is drastically reduced, as it is unlikely the user switched direction that fast. Thus, we will require more packets requesting to move into that direction before we gain confidence in it.
The confidence gets reset to 0 when the remote control is stable. We consider that if the remote control is stable whatever movement we had previously has ended and we are not biased towards any direction right now. The initial formula used to update the movement in regards with the confidence on that direction was this:
Δ
X=
2base∗(confidenceX+1)−1
22∗base−1
∗ Δ
X,
where base is the sensitivy factor, this is configurable depending on the de-sired user experience and conf idenceX is the confidence on the X axis.
This ensures an exponential increase, thus confidence acts as a major factor. The downside is that small motions with the remote control will pretty much be ignored. This makes it particularly difficult to navigate so we moved to a much simpler formula:
ΔX =
�
ΔX∗ confidenceX, if conf idenceX > 0.
0, otherwise. (4.5) The new formula completely nullifies movement in the opposite direction of the confidence and it scales the movement linearly for when the movement matches the direction of the confidence. This allows both slow and fast movement to be performed and makes sure the cursor does not shake because it will not move in the opposite direction.
Master Thesis - Andrei Buhaiu 20
Figure 4.3 shows how different sensitivity base values impact the move-ment adjustmove-ment ratio in relation to the confidence.
Figure 4.3: Confidence values for different bases
Another addition to the confidence mechanism is the movement bias check. When we generate movement for the mouse we check to see if one of the axes has considerably more movement than the other, if this is the case we assume that the movement on the other axis is involuntary and nullify it. This could further be improved using techniques described in Gallo and Minutolo[10].
Master Thesis - Andrei Buhaiu 21
4.3
Cursor Reset
The cursor reset is not really a heuristic, it simply resets the mouse posi-tion to the center of the screen. This is done by shaking the remote control. This movement is never considered to be useful so it can be used for the reset of the cursor position.
The way that the shaky movement is detected is by sampling a number of movement packets and checking if the direction on any of the two axes has changed, if that happens we increase an internal counter for that axes. If at any point one of the two internal counters surpasses the threshold value we decided that we need to reset the position and the cursor is repositioned to the center of the screen. In addition to, this we also ignore the next few movement packets in order to not move the cursor by mistake after just reseting and to make sure the user sees that the cursor is now centered.
4.4
Key Presses
The key presses do not have any particular heuristic integrated in them. However, similar to the cursor reset, they block mouse movement for a short duration. This has been done to prevent sliding the mouse when pushing a button. This is particularly unpleasant when clicking with the remote control as it would perform a drag instead of a click.
It is very important that the number of ignored packets is large enough to make sure that no movement occurs due to the pushing of buttons and small enough so that it does not impede the mouse movement.
Chapter 5
Development
5.1
Compiler
One of the initial steps of the development process was porting the code to a different compiler. The example code received from Nordic RF was written for the C51 Keil compiler[4]. The project was ported to the SDCC[6] due to a number of reasons:
• the SDCC compiler is an open source compiler • C51 Keil compiler only runs on Windows machines
Porting did involve rewriting some of the code in order to compile. Some components were unnecessary for our design and were, thus, removed.
One of the major challenges of the porting process has been the size of the binary generated. The sdcc compiler does generate a binary with a larger footprint. This meant that the code had to be specifically written with size optimization in mind. This also meant that all of the extra functionality in the code base was removed. As the project stands right now both the remote control and receiver have 32kB flash memories and the current binary footprint is 18-20kB. This means that there is still some room for extra functionality.
I did not manage to port to bootloader and kept on using the uVision Studio that only runs on Windows. Because of this the compiled image is shared over samba with a Windows machine that loads the image on the board.
Master Thesis - Andrei Buhaiu 23
5.2
Receiver
The receiver retransmits data from the remote control to the set-top box. The interesting part is that the receiver is configured as a I2C slave, while it
needs to send data when it receives it from the remote control. As a slave, it can never initiate communications and that would normally mean that the master would have to poll for data. This was solved by using an interrupt pin for the data from the receiver. When data is received an interrupt is sent to the set-top box and the set-top box starts a read over I2C while the
receiver does a write with the data over I2C.
Figure 5.1: Example of complete communication between remote control and STB
5.3
Set-top Box Driver
The driver for the mouse/keyboard device is the first mouse-like device used for the particular project. Integration has not been very difficult because there was frame for the mouse events in place. The keyboard part was already there, being used by the traditional infrared remote control.
Master Thesis - Andrei Buhaiu 24
The structure that describes the cursor movement contains the following fields:
• X - position on the X axis • Y - position on the Y axis • Z - position on the Z axis • dX - delta on the X axis • dY - delta on the Y axis • dZ - delta on the Z axis
• buttonKeycodes - the keycodes for the buttons pressed • timestamp - the timestamp when the data was received
At this moment, the only fields that are actually used by the UI are the X, Y and buttonKeycodes. The other information is available, not used by the UI. This is because the system only accepts two dimensional movement. This is not a problem at the moment as we do not have any applications using three axes.
5.4
UI Integration
The data provided by the driver will be passed to the UI as mouse or keyboard events and from there normal processing ensues. In order not to flood the UI event queue before a mouse event is added to the queue, we check if another mouse event is already within the queue. If we find a mouse event already in the queue we save the event and aggregate new events in one event and when the mouse event in the queue gets resolved we insert the aggregated one.
The current UI can work with the mouse events and will trigger the events properly.
5.5
Opera Integration
Integration with the Opera browser was rather easy because it reads mouse data from the UI event queue and the driver already inserts the data in the UI event queue. Integration is not complete. This is due to the fact
Master Thesis - Andrei Buhaiu 25
that the browser code does not support all the features. Before any fur-ther integration is done we require a browser version that integrates all the missing features.
5.6
Hardware Updates
During the development process a number of hardware bugs were discov-ered and the schematics for the remote control was updated accordingly.
Figure 5.2: Keyboard Schematic with removed component circled(C27)
C27 was removed because it was charged when pushing a button and the charging took too long so the voltage on the KEYIN signal never got high enough to be detected in the time that the kEYOUT signal was high.
Master Thesis - Andrei Buhaiu 26
Figure 5.3: Remote control with removed component circled(R14)
R14 was removed because the INT GYRO signal was low when the gyro was in sleep mode and that meant that there would be a current through R14.
Chapter 6
Conclusion
6.1
Results
The cursor movement at the moment is quite fluid. The heuristics in place are highly customizable. An easy example is mouse sensitivity which simply divides the movement by a particular factor. The settings menu could have this configurable so that different users can adjust the sensitivity according to their desire.
It is also very easy to integrate new heuristics as the entire process is very well divided into distinct steps. It is also easy to link multiple heuristics together to get better responses.
The project is currently used as a demonstrative product. There is cur-rently no specific application that uses the accelerometer specifically. At the moment it is only relevant as a pointing device. In the current setup the mouse pointer works well with both the UI the set-top box menus and the opera browser.
It is hard to quantify the actual performance of the project as it is tightly related to the user experience. The project has been updated several times for different demonstrations. This has proven that porting it to different platforms is quite easy. Future exhibitions and user level testing will offer more feedback on the subject.
Master Thesis - Andrei Buhaiu 28
6.2
Future Work
6.2.1
Z-axis movement
One of the most important additions to the project would be an appli-cation that uses the Z axis movement as well. At the moment no processing is done on the Z axis as it would only be a waste of resources. However, enabling it would be very easy as it is similar to the X and Y axes. All the data structures contain a Z position and Z delta and the data necessary to generate the movement is transmitted to the set-top box already.
6.2.2
Heuristics
Further improvement of the heuristics is always an option. At the moment the two complicated issues are determining the position of the remote control correctly and slow movement. Determining the position can be enhanced by mostly filtering out noise from the acceleration and rotation readings. The problem with slow movements is that some of them need to be nullified because they are not desired and yet we need to allow the user to make a slow movement.
6.2.3
Communications Protocol
Another improvement would be fine tuning the communication protocol. Right now the protocol is a very simple best effort protocol. This works fine with our current needs. However, if a header packet is lost, the system might go into an inconsistent state. The communications did not go through thorough stress testing, the results would be highly relevant.
6.2.4
Gesture Movements
Right now the UI only processes mouse positions and not the deltas. Using the deltas along with a collection of the positions could be used for gestures. Invense[2] the company that produces the gyroscope suggests that it is particularly useful to be used in gesture recognition systems. This is rather easy to do in an entertainment system. As an example:
• Moving the remote control up or down when watching television would turn the volume up or down.
• Moving the remote control sideways would change the channel up or down.
Master Thesis - Andrei Buhaiu 29
• Rotating the remote control 180 degrees would shut the television down. A more detailed look into this can be found in Ni et al.[11]. The article describes hand movement gestures. However, most of them still apply for the remote control, except for finger gestures, which are the least reliable of all to begin with.
6.2.5
Pairing
Another relevant subject is pairing the remote control with the specific receiver on a set-top box. The demonstration product by NordicSemiconduc-tor performs this operation. In our project, however, this has been removed. The reasoning behind this was that the code base to perform encryption and decryption is quite large and due to our more stringent size restrictions it would not fit using the current compiler. It also introduces more complexity that would affect the relevance of the project as a proof of concept.
The pairing process basically refers to the association between one par-ticular remote control and one receiver. This is done in order to ensure that a receiver does not get commands from two different remote controls at the same time as this would confuse the driver and in order to allow some sort of authentication. The authentication is relevant due to security reasons. Keep-ing in mind that the remote control is meant to enable traditional internet access some encryption is required in order to ensure sensitive data cannot simply be intercepted. Most of the traditional issues that come up with wireless communications are relevant here as well. This is one of the major differences between the traditional accelerometer systems and this project.
The way the pairing process should work is that the remote control and the receiver each have a product identifier that they will exchange using a three way handshake mechanism. These messages would also need to be encrypted using a preshared symmetric key or a pair of asymmetric keys. After this initial handshaking is complete encrypted communication can be done by using a shared key mechanism. This should be done because it is faster than asymmetric encryption.
Another important aspect would be to only encrypt the security sensitive information. This can speed up traffic greatly as encryption would have to be performed on the remote control as well and this would greatly affect both speed and battery lifetime.
Master Thesis - Andrei Buhaiu 30
6.2.6
Keyboard
The keyboard used at the moment is a mobile phone style one. This makes it hard to use, especially when using the browser. The problem is that the remote control does not offer much physical space for a decent sized keyboard. The keyboard that the NordicSemiconductor product uses is rather small and still does not alleviate the problem.
In order to solve this, the design of the remote control should probably be wider so that the buttons can be further apart.
Bibliography
[1] I2C Specification. http://www.nxp.com/documents/user_manual/ UM10204.pdf. Accessed: 2013-04-29.
[2] Invensense homepage. http://www.invensense.com/index.html. Ac-cessed: 2013-05-06.
[3] Invensense MPU-6050 Datasheet. http://www.invensense.com/mems/ gyro/documents/PS-MPU-6000A.pdf. Accessed: 2013-04-29.
[4] Keil C51 Compiler. http://www.keil.com/c51/. Accessed: 2013-05-06.
[5] nRFready 2.4GHz RF Smart Remote Product Page. http: //www.nordicsemi.com/eng/Products/2.4GHz-RF/nRFready-2. 4GHz-RF-Smart-Remote. Accessed: 2013-04-25.
[6] SDCC Compiler. http://sdcc.sourceforge.net/. Accessed: 2013-05-06.
[7] M. Boulos, B. Blanchard, C. Walker J. Montero, A. Tripathy, and R. Gutierrez-Osuna. Web gis in practice x: a microsoft kinect natu-ral user interface for google earth navigation. International Journal of Health Geographics, 10, 2011.
[8] D. Bowman, R. McMahan, and R. Ragan. Questioning naturalism in 3d user interfaces. Communications of the ACM, 55(9):78 – 88, 2012. [9] S. Cheema, M. Hoffman, and J. LaViola. 3d gesture classification with
linear acceleration and angular velocity sensing devices for video games. Entertainment Computing, 4:11 – 24, 2013.
[10] L. Gallo and A. Minutolo. Design and comparative evaluation of smoothed pointing: A velocity-oriented remote pointing enhancement technique. International Journal of Human - Computer Studies, 70:287 – 300, 2011.
Master Thesis - Andrei Buhaiu 32
[11] T. Ni, D. Bowman, C. North, and R. McMahan. Design and evalua-tion of freehand menu selecevalua-tion interfaces using tilt and pinch gestures. International Journal of Human - Computer Studies, 69:551 – 562, 2011. [12] Lucio Davide Spano. Design of a 3d mouse using accelerometers. 2009.
