Linköpings universitet SE–581 83 Linköping
Linköping University | Department of Computer and Information Science
Master’s thesis, 30 ECTS | Datateknik
21 | LIU-IDA/LITH-EX-A--21/001--SE
Managing objects and events in a
distributed user interface
–
by further extending a framework that can be used when
de-veloping a multiplayer game
Hantering av objekt och event i ett distribuerat
användargrän-snitt
Wallentin Rosén
Supervisor : Anders Fröberg Examiner : Erik Berglund
Upphovsrätt
Detta dokument hålls tillgängligt på Internet - eller dess framtida ersättare - under 25 år från publicer-ingsdatum under förutsättning att inga extraordinära omständigheter uppstår.
Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka ko-pior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervis-ning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säker-heten och tillgängligsäker-heten finns lösningar av teknisk och administrativ art.
Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsman-nens litterära eller konstnärliga anseende eller egenart.
För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/.
Copyright
The publishers will keep this document online on the Internet - or its possible replacement - for a period of 25 years starting from the date of publication barring exceptional circumstances.
The online availability of the document implies permanent permission for anyone to read, to down-load, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility.
According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement.
For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: http://www.ep.liu.se/.
Abstract
As more and more research are carried out in the field of distributed user interface some aspects of the it are still missing. In this report I will propose a method to how we can manage distributed objects and events in a distributed user interface environment that is suitable for a developer. An implementation with focus on the developers creativity, understandability and maintainability has been made on an existing framework. The re-sults show how the solution can be applied to create a multiplayer game in a distributed environment, however a more thorough investigation on how this solution affects the de-velopment of distributed user interfaces is still needed.
Acknowledgments
First and foremost I would like to thank my family for all the help they have given me during my studies, without you I would never have come this far. I would also like to thank my examiner Erik Berglund for being a big help in keeping this thesis going in the right direction. As well as my supervisor Anders Fröberg for helping me understand when I got stuck during the process of the thesis.
Contents
Abstract iii
Acknowledgments iv
Contents v
List of Figures vii
List of Tables ix 1 Introduction 1 1.1 Motivation . . . 1 1.2 Aim . . . 2 1.3 Research questions . . . 2 1.4 Delimitations . . . 2 2 Theory 3 2.1 Environments . . . 3
2.2 Inter device communication . . . 5
2.3 Distributed user interface . . . 5
2.4 Distributed objects . . . 6
2.5 Event management . . . 7
2.6 Event management in distributed environments . . . 7
2.7 Synchronization . . . 9
2.8 Marve.js . . . 9
2.8.1 Marve Backend . . . 10
2.8.2 Distribution manager . . . 10
2.8.3 Distribution Objects in Marve . . . 11
2.8.4 Ownership . . . 11
3 Method 12 3.1 Finding demands through project implementation . . . 12
3.2 A more general distribution of functionality . . . 13
3.2.1 Options for events . . . 14
3.3 Handling of distributed objects . . . 16
3.3.1 Positioning of Object . . . 16
3.3.2 State synchronization . . . 17
4 Results 19 4.1 Demands for the framework . . . 19
4.1.1 Demands for an event system . . . 19
4.1.2 Demands for the object handling . . . 19
4.2 The Multiplayer Game . . . 19
4.2.1 A more general distribution of functionality . . . 20
4.2.2 Handling of distributed objects . . . 21
4.2.3 State synchronization . . . 22 4.3 Documentation . . . 23 5 Discussion 24 5.1 Related work . . . 24 5.2 Results . . . 24 5.3 Method . . . 24
5.4 The work in a wider context . . . 25
6 Conclusion 26 6.1 Research Questions . . . 26
6.2 Future Work . . . 26
List of Figures
1.1 The finished multiplayer game. . . 2
2.1 An example of a Dom-tree model. . . 4
2.2 A button which is a child element of a container with personal id c_1 defined in the html, designed in the css and made functional in the javascript . . . 4
2.3 A server-client model [server-client-wiki] . . . . 5
2.4 Example of a DUI application where the play button of a media player on Device 1 have been distributed to Device 2 and Device 3 . . . 6
2.5 Each element has their own event pool which calls functionality on triggers. These events are all defined by the the developer in the javascript and html code. . . 7
2.6 The button on device 2 has been distributed from device 1. When a callback event system is in place each event will be sent back to device 1 for handling. . . 8
2.7 The widget approach will let each widget keep its own functions which are called when the element is triggered. . . 8
2.8 Example of a server event manager where all functionality is handled by the server. 8 2.9 The application communicates through the distribution manager which commu-nicates with other devices through the backend. . . 10
2.10 The association between the data in the Marve backend . . . 10
3.1 The welcome screen of the application containing the map of all turfs. . . 12
3.2 The moccup of the turf view in the multiplayer game. . . 13
3.3 Flowchart of the all trigger event functionality. This flowchart starts at the object owners device when a user has triggered the element. the counter refers to the container which keeps track of the devices which has triggered the element already. 14 3.4 Example of a event where server callback is set to false and the distribution manager calls the handler on the same device as the trigger happened. . . 15
3.5 Example of a server callback were Button 1 has been mirrored and a event with server callback set to true has been put on it. When button is triggered the distri-bution manager on device 2 tells the server that device 1 should run the handler with the matching handler name. . . 15
3.6 When distribution object 1 is distributed the object which are not defined as dis-tribution objects will become child objects with their own unique id . . . 16
3.7 The flow chart of how Marve inserts a received Node into the DOM-tree . . . 17
3.8 Each Distributed object will have their own observer. The observer will observe until next distributed object in the hierarchy . . . 18
4.1 Screenshot from the game. (1) is the end turn button, you can see the connected devices in (2). The grid is seen in (3), a player character in (4), the inventory in (5) and the information box in (6) . . . 20
4.2 The program presents the resulting path of a A star algorithm to the user . . . 21
4.3 A turf is filled with cells which has a cosmetic child box and either a character, tree or nothing. The blue circles indicate distributed objects and yellow ones child objects. . . 22
4.4 When hovering the enemy the application gets the health from the white knights session variables and the damage taken from attacks. Then it displays it in the information box together with various information from the enemy. . . 23
List of Tables
1
Introduction
Collaboration has always been a great part of any kind of task humans do. It does not matter if it is work or playing games, it is common knowledge that for most tasks a collaboratory method is the most efficient and enjoyable. That is why we are trying to find new ways to bring this collaboration into our digital applications. One field of research that even though relatively new is showing this kind of collaboratory thinking is the research in distributed user interfaces or DUI for short.
DUI is as it sounds when the user interface of an application is not bound to one device but can be sent between devices. By letting users share parts or all of the application they can collaborate on the tasks and work together.
In this paper we will investigate this research field from the developers side of things and see if there is a way to handle the problems that are occurring when interfaces are shared across devices. I will bring forward a method for handling distributed objects and the events they bring with them that is focused on making the development of DUI as intuitive and creative as possible. I have then applied this method in the implementation of a multiplayer game which can be seen in figure 1.1.
1.1
Motivation
Researchers in the field of DUI applications are despite their differences all agreeing on sur-passing the single device experience.[2] Distributed user interface is a relatively new research area and is in need of new ways to improve the experience while keeping the development process manageable. A complex problem with multi device systems in contrast to a single device system is to keep the program synchronized. A single device system has access to everything at every point in time. Insert another device in the program and suddenly we need a communication system that needs to continuously send over information about the users interactions to make both of the devices have the exact same state. And this is where a lot of research has been made, trying to mimic the single user applications by keeping the experience the same, just on more devices. What many of these researchers does not realise is that sometimes the developer can benefit from being able individualize the application for each device. By allowing local and global states we let the developer themselves decide what will be communicated and shared between devices. There is also a need for a more general event management system. A system that lets the developer decide where the functionalities
1.2. Aim
Figure 1.1: The finished multiplayer game.
of the program will exist during the application lifetime, in contrast with past research where each functionality is bound to the interface objects. A solution to these needs would make for a richer and more creative experience working with DUI applications which I believe would drive this field of research forward.
1.2
Aim
The aim of this thesis is to contribute into the research of DUI frameworks by presenting a new way of thinking about managing distributed interface objects. I will propose an event management and an object handling system that will make development of DUI applications more dynamic and test this by developing a multiplayer game along side the framework.
1.3
Research questions
Q1 How can a event system be implemented in a DUI framework so it works in a multi-player game.
Q2 How can management of distributed objects be generalized while still understandable and manageable for a developer.
1.4
Delimitations
This research is focused on the developers side of distributed user interface and will not take end user experience into account when analyzing the results.
2
Theory
In this chapter I will present the theory needed to understand why I chose the methodology I did and why the results are interesting. I will start by describing how the interface of a website is built and go on to relate it to the research of distributed user interfaces. Then I will go on to talk about some previous work done on event handling for distributed environments and how components are handled when distributed across devices.
2.1
Environments
The web interface is a common environment when working with distributed user interfaces and are used in many research projects [8, 10, 1, 15] First reason being that you do not have to limit yourself for a specific device, it is supported by a wide range of devices by default. The web interface is also already build up by visual components, called DOM elements. These DOM-elements make up the DOM-tree body, see figure 2.1.
The DOM tree has the document root in the top, and one step below that is the Head where the definitions of the websites will be placed. This could for example be the title or references to the scripts that are sent with the page. The Body element and its children are what makes the interface of the website.
Each DOM element are created by three building stones, html, css, and javascript, which are seen in figure 2.2.
2.1. Environments
Figure 2.1: An example of a Dom-tree model.
c
Figure 2.2: A button which is a child element of a container with personal id c_1 defined in the html, designed in the css and made functional in the javascript
The html can be explained by being the definition of the object, telling what will exist in the DOM-tree and how they relate to each other. You then design the objects by writing css code for them. The last building stone is the javascript which defines the functionality of the application. Javascript lets you tell the application what will happen when you interact with the elements.
2.2. Inter device communication
This simple representation makes the DOM elements suitable for distribution.
The framework which is developed further in this thesis is built for the web interface and is partly implemented in the programming language javascript which is the standard language when implementing functionality for web applications.
2.2
Inter device communication
The communication between devices in a dui environment can be done in several ways. Fis-cher et comp argues that a peer to peer communication is to prefer by pointing towards the pros with scalability, robustness, security and trust in comparison with a server-client com-munication [5]. Peer to Peer is a comcom-munication method where no device is working as a server but each device can be both server and client depending on the current application state. And even though some research in the field have been using peer to peer like Melchoir and comp. [12] most of the research we have found are using a server-client communication [8] [15] [9]. This comunication method can be seen in figure 2.3 and works by having a device which acts as a server and often holds some kind of data about the devices which are con-nected and the state of them. Each communication method also have to go through the server. This communication method is also used by the framework Marve which is the framework that is developed upon in this research paper [6]. The biggest reason to use a server-client communication is to have fully control over the data that is passed between devices. You can also have full control over the state of the application so you access the data that someone might lose when disconnecting for instance.
Figure 2.3: A server-client model [14]
2.3
Distributed user interface
Distributed user interface or DUI for short is a type of application where the interface ele-ments are not just handled by one user on one device but by one or many users on one or many devices. Or to use a definition from Elmqvist in his paper about the subject
A distributed user interface is a interface whose components are distributed across one or more of the dimensions input, output, platform, space and time [4]
2.4. Distributed objects
Which basically means its an interface which functionality and visuals are not bound to one device but can be used by many.
Figure 2.4: Example of a DUI application where the play button of a media player on Device 1 have been distributed to Device 2 and Device 3
Using the distribution methods found in the framework Marve [6] there are three types of distributing an object and these are Mirror, Transfer and Clone. Mirror is when two or more devices are sharing an DOM-element. The DOM-element will be updated on all devices that share it. When an DOM-element is Transferred it will seize to exist on the sending device and will appear on the receiving device. Clone is when an object will be copied and sent to one or many devices, every device will have their own instance of the object.
2.4
Distributed objects
We define a distributed object as an object that can be distributed from one device to another. But like Vanderdonckt discusses in his paper it is the graphical elements that are the inter-esting parts of the research in DUI[13]. Otherwise it would just be transferring of data and not interfaces across devices. However each framework will have to decide what data will be part of each object. And that is why there are many ways of implementing these objects. One common way of representing these objects are as widgets [8] [10] [9]. Using a widget based approach is reliable in many ways because you can keep all functionality for each widget separately. Each little package includes all code and design which can then be distributed and kept intact. The problem with this approach is that each object or widget needs to be im-plemented in a similar way, often using a third party component representation like Googles Polymer[8]. This gives overhead and special knowledge needs to be acquired for the devel-opers before implementing these components. Another problem is that each widget needs to be pre-configured and the application wont be very dynamic.
A system called proxywork was proposed by Villanueva and comp. [15]. This system works as a proxy which will let the user perform operations such as hide or delegate elements of the requested web application to other browsers that are connected to the proxy. Their focus was to create a more general system that will work on any device and on any web page
2.5. Event management
without any need for extra implementation. This type of system will be very restricted in regard to varied functionality between devices but will work well on static web pages were the functionality is limited.
2.5
Event management
An event always starts with some kind of web object for the user to interact with, this could be a button, a input field or any kind of element. The developers then defines what kind of triggers the elements are supposed to have. To decide what actually happens when an element is triggered the developer need to define the callback functions. These functions will be called by the web applications when the user triggers the elements. And that is how a single device experience can look on web applications, a series of user inputs and application reactions. [7]
Figure 2.5: Each element has their own event pool which calls functionality on triggers. These events are all defined by the the developer in the javascript and html code.
2.6
Event management in distributed environments
Event management in DUI applications are more complex than in single device systems. You have to always consider where the event should be handled to keep the application intact.
One solution is to give the device who accessed the element first full control over every event, see figure 2.6. This is a method currently used by the framework Marve [6]. Every trigger will have to communicate to the device which sent the element that the element has been triggered. The problem with this approach is that no functionality can happen on the receiving device which limits the developer a lot when creating the application. There will also be a lot of traffic over the network because every event will have to be communicated to the device where the handler should be called.
Another approach is to distribute the full functionality and the state of every element with widgets. Widgets are closed packets which include all functionality, design and definitions that the element needs to work on any device. This is the solution in frameworks like Liquid or XD-mvc [8], [9]. Every device will be self sufficient but the functionality for each compo-nent will be statically defined. The developer also need special knowledge in the specified type of widget development. This method works very well when you know the
applica-2.6. Event management in distributed environments
Figure 2.6: The button on device 2 has been distributed from device 1. When a callback event system is in place each event will be sent back to device 1 for handling.
tion will run on different types of devices because you can configure each widget to handle different environments.
Figure 2.7: The widget approach will let each widget keep its own functions which are called when the element is triggered.
Another way of handling each event is by letting each element be a visual trigger and having a global event manager on the server as seen in Polychrome [1]. The state will be synchronized at all time which makes this a safe approach. All though it will create a lot of traffic on the server and each elements functionality will be the same no matter on which device it will exist.
Figure 2.8: Example of a server event manager where all functionality is handled by the server.
2.7. Synchronization
2016 Krug presented a extension to a widget based approach were components can act as subscribers or/and publishers for key words in text format. These key words will be stored on the server together with the subscribers and publishers linked to them. The server will then make sure every subscriber to a key word will access the data sent when it is pub-lished[11]. This would not work in our program because working with a multiplayer game the data needed for a device can vary a lot depending on the current state of the application. It would be a lot of work for a developer to make sure you are subscribing to the correct data at all time. It is better for static web application were you at every point in time know what data goes were.
2.7
Synchronization
Fisher and comp. presents three case studies on building distributed user interfaces, where they shine light on problems when designing a DUI system like asymmetric functionality be-tween clients or what happens with the clients data on a sudden disconnection[5]. The paper talked about the importance of keeping a synchronized state for all clients. Their solution to this was to have a shared memory and a robust notification system which manages the mem-ory. The problem with this method as for many others are that there will be a lot of traffic between the devices.
Using a widget based approach is a good way of keeping the application synchronized because all widgets have their own state and together they make up the application[10].
None of the frameworks I looked at have even considered not synchronizing every part of the distributed object. The DireWolf framework is not letting the widgets be shared on several devices [10]. And a framework like Liquid is constantly synchronizing the Polymer component upon change [8].
The framework Which is built upon in this thesis are using the MutationObserver [3]. This observer are placed on DOM elements and will observe for changes in attributes, children etc. What specific things that this observer will look for needs to be configured.
In this paper I will propose a implementation which let the developer define what is syn-chronized across devices. This way a distributed object can have a local and a global state which will give the developer more creativity when implementing the application.
2.8
Marve.js
Marve is a javascript framework for developing DUI applications. This is also the framework the project in this thesis builds upon which makes it necessary to understand how it works.
If Marve is included in the application it will upon application start set up a connection to the Marve backend. If a connection is established users can access and create rooms by calling methods in the distribution manager, see section 2.8.2. The distribution manager will be the communication link between the connected devices and the backend, the link can be seen in figure 2.9.
2.8. Marve.js
Figure 2.9: The application communicates through the distribution manager which commu-nicates with other devices through the backend.
2.8.1
Marve Backend
Marve saves data on its backend about the different rooms, which devices that are connected and which objects that are possessed by which devices, see a model over the database in figure 2.10.
Figure 2.10: The association between the data in the Marve backend
2.8.2
Distribution manager
When you include marve in your project you get an instance of a distribution manager. Through this you can connect to the marve backend and join a room. The distribution man-ager works by sending back the data to handlers when the data arrives. You will as a devel-oper need to set these handlers to define what will happen with the data.
An example of these handlers are the object reference which is called to set the parent ele-ment of a incoming distribution object. So when a eleele-ment is distributed to the device this is where it will be placed.
2.8. Marve.js
2.8.3
Distribution Objects in Marve
Distribution objects are a special type of objects in Marve which besides the element node includes data about the event listeners, observers, etc. You can create a distribution object of any element in your DOM-tree. These elements are then the ones which are distributed across devices. Each object have an MutationObserver which alerts the distribution manager when a element have been altered in any of the following ways, either the attributes of the node is updated or a child element has been added or removed from it. The other devices in possession of the object are then notified about the change.
Each distributed object has their own personal id assigned to them. With this id the objects can be identified across devices.
It is through the distribution objects you access the distribution methods mirror and trans-fer. When an object is mirrored it will be packaged together with information about the re-ceiving devices and sent to the server. The server will distribute the object to the specified device or all devices if there is no specified one. On the receiving device the object will be reconstructed and saved in the applications list of distributed objects, the DOM element will also be reconstructed and put into the DOM-tree as a child of the object reference. Transfer works as Mirror with the exception that it hides the object on the sending device. When a distribution object is distributed Marve will create distribution objects of all its children elements. This is because they are transferred as well and needs to be identifiable.
2.8.4
Ownership
In Marve there is the concept of ownership. When we are talking about the owner of an object or an event we are talking about the device from which the object is distributed. This means when a device is redistributing an object they will claim the ownership over that object on the receiving devices. The ownership comes into use when there is a server callback for an event. The server callback will always travel back to the device which is the owner of the object on the device where the event was triggered.
3
Method
This section presents the method used to first find needs in a DUI framework. Then it goes into my proposed implementation to handling events and objects in a distributed environ-ment.
3.1
Finding demands through project implementation
The multiplayer game which was implemented to analyze the need of a DUI application was made in html, css and javascript and did not use any third party frameworks. What we came up with was a multiplayer strategy game based on 2D grids were the users work together against a computer opponent. The map consists of 4 grids, see figure 3.1, which all consists of enemies placed by the program at the start.
Figure 3.1: The welcome screen of the application containing the map of all turfs.
The game is played in turns were you can move characters from your inventory into the boards, move the character around the board and attack the enemies on the board. Each
char-3.2. A more general distribution of functionality
acter has health, damage, movement range and attack range. Some characters have special effects which takes action when they are placed on the board. Some of these effects include raise the damage of all friendly characters or lowering health in a radius around it. When the characters are moved between cells on the board the program runs a A* algorithm to calculate the shortest path.
The users is all playing at the same time during their turn and when they feel ready they can press the end turn button. When all users pressed the end turn button it is the computers turn.
The game starts with each user having characters in their inventory and the first connected user has possession of all grids on the map. The users can then distribute the grids they control. They can distribute the grids by mirroring them or transferring them. They also have the choice of transferring the characters in their inventory to any other user.
Figure 3.2: The moccup of the turf view in the multiplayer game.
3.2
A more general distribution of functionality
To give the developer more creativity when setting up an event system in their application several new options was made for them at each event creation. The trigger type of the event was updated with the choice of several new types of triggers called all triggers . These triggers works in such a way that each device that has possession of the node needs to trigger the node using any kind of trigger the developer wants. When all the instances of the node has been triggered the handler set at the event creation will be called at the node owner.
To make this functionality I first had to make sure that each distribution object kept track of the number of devices that are in possession of this object. I did this by sending this number together with all other data whenever someone accessed the objects. As well as updating the number on the device sending the object depending on how many devices it was sent to.
So with each object keeping track of the number of devices that is in possession of it all it needs to do is creating a container which is keeping track of each device that has triggered the all trigger event.
3.2. A more general distribution of functionality
Each device which acquire the object with the all trigger event attached to it will set a callback event to it. On each callback the device which owns the object will now check if the event has the correct trigger type and see if the device id is not in the container with the devices that has already triggered the element. If it is new and the trigger type matches it will check if all devices has triggered it by matching the number of device ids in the container and the number of devices in possession of the object. When this occurs the handler which the developer defined when creating the all trigger event will be called on this device. The full flow of this function can be seen in figure 3.3.
Figure 3.3: Flowchart of the all trigger event functionality. This flowchart starts at the object owners device when a user has triggered the element. the counter refers to the container which keeps track of the devices which has triggered the element already.
3.2.1
Options for events
If not a all trigger event the developer has more options to form the event as they want. These new options can be seen in table 3.1.
name type default value
———— ————
————-local event boolean true
server callback boolean true
distribution type list of strings ["mirror", "transfer"] Table 3.1: Event options
The first option which can be specified is if the event is a local event. This tells Marve that this event will be created on the owner as well. To be more specific, by enabling this options Marve will create a event listener on the node with the specified trigger and handle according to javascript standards as described in section 2.5, not using marve event system.
The next option tells Marve if this event will be handled as a server callback when dis-tributed to other devices. With this options the developer have two ways of handling this event on the device that receives the object. Either the event will be handled as a local event and will run the handler on the receiving device when triggered, seen on figure 3.4.
3.2. A more general distribution of functionality
Figure 3.4: Example of a event where server callback is set to false and the distribution manager calls the handler on the same device as the trigger happened.
Or you can tell the receiving device to do a server callback back into the owner device when the event is triggered, see figure 3.5. In this case the receiving device will send back the name of the handler and the event data. The sending device will then call the function using its name with the event as parameter.
Figure 3.5: Example of a server callback were Button 1 has been mirrored and a event with server callback set to true has been put on it. When button is triggered the distribution manager on device 2 tells the server that device 1 should run the handler with the matching handler name.
One last options for the developer is defining for which distribution type(s) the event will be sent with. The developer can tell Marve that this event will only be handled if its sent with a specific kind of distribution type. Each event created will be saved inside the events container matching the correct distribution type as
e v e n t s = { mi rror : { t r i g g e r _ 1 : [ handler_1 , handler_2 , . . . ] , t r i g g e r _ 2 : [ handler_1 , handler_2 , . . . ] , . . . } , t r a n s f e r : { . . . } } .
The receiving device will then only handle the events which are matching the distribution type of which it received the object. Meaning if it received the object by a mirror it will handle the events in the mirror object and same for transfer.
3.3. Handling of distributed objects
3.3
Handling of distributed objects
I divided the distributed objects into two parts. The distributed objects which are defined by the developer to be just that. These objects have data such as variables, listeners etc. that are sent together with the object on distribution. Then we have the child objects which are just nodes with a personal child id. This way any object which is defined as a distributed object by the developer will stay as one while the rest of the objects that are distributed together as children elements will be child objects as can be seen in figure 3.6.
Figure 3.6: When distribution object 1 is distributed the object which are not defined as dis-tribution objects will become child objects with their own unique id
The child objects are nodes that will be distributed together with the distributed object because they are children of it in the DOM-tree. Because of the personal child id we can keep track on them even though they exist on different devices. This way we can synchronize all object without needing to fill each child with a bunch of unnecessary data like for example empty variable containers. If a child node is changed and we need to synchronize it across devices we send the child id together with the update. then we update the child object on the receiving devices by finding the object in the DOM-tree with the same child id and updating that node.
Worth noting is that a new child to a distributed object which has been created through javascripts appendChild function will become a child node directly and be updated on the devices needed. But only if the observer of the distributed object are configured to observe children, see chapter 3.3.2.
3.3.1
Positioning of Object
The distribution methods have been updated with a new option which is called parent. With this option a developer can choose to put in an id. The distribution manager will upon re-ceiving an object try to match the given parent id first to a elements distribution id. If there is
3.3. Handling of distributed objects
no element with that distribution id it will try it as a normal element id. If that does not work either it will use the object reference defined by the developer. If there is no object reference the received object will be thrown away. The flow of the position decision can be seen in figure 3.7.
Figure 3.7: The flow chart of how Marve inserts a received Node into the DOM-tree
3.3.2
State synchronization
The method which is implemented divides a distributed objects state into two parts, visual which are handled by observers and underlying state which are defined by session variables.
Session variables
The underlying state will be handled by the session variables. Each distribution object owns its own session variables which will be updated on each device in possession of the object on each declaration or update. The variables can then be received locally without any server calls. These variables are also distributed together with the object so it will always be syn-chronized across devices.
When we set a variable to a node value it will first be serialized and the keyword node will be attached to it. On the receiving side we will always check for the keyword node. If it is found the node will be parsed back into a element node.
Observers
To handle the visual state I have decided to let the developers decide what will be observed by a observer on each distributed objects node. Each distributed object will have their own instance of observer configurations which will let them observe different things.
The observer options are in line with the options in javascripts standard observer, the MutationObserver.
With these new functionality each observer callback will have to be monitored. If the owner of the observer is not the distribution object which is the closest parent relative to the
3.3. Handling of distributed objects
target the observation will be thrown away. This means a distributed object will only observe children which are not handled by another distributed object which are closer in relation, see figure 3.8.
Figure 3.8: Each Distributed object will have their own observer. The observer will observe until next distributed object in the hierarchy
Because of these new functionalities the developer have the choice to let different devices have different local states on node. If they want to sync a visual state from one device to another for a chosen object they can do that by just calling the sync node which will set the html and css to be same on all instances of that object.
4
Results
4.1
Demands for the framework
When analyzing the multiplayer game, as described in section 3.1, demands for the frame-work occured that needed to be implemented for the game to be realized.
4.1.1
Demands for an event system
• A developer needs to be able to distribute the functionality across devices in a general way.
• There should be a specialized event which only triggers the handler when all devices in possession of the element has triggered it.
4.1.2
Demands for the object handling
• The developers should be able to have control over what is observed on each object. • There should be a better way for the developers to keep track the distributed objects in
the running application.
• Elements should be able to be distributed by being added into a existing distributed DOM-tree structure.
4.1.3
Other demands
• There should be a clear documentation about the API for the developer to learn the framework from.
4.2
The Multiplayer Game
4.2. The Multiplayer Game
Figure 4.1: Screenshot from the game. (1) is the end turn button, you can see the connected devices in (2). The grid is seen in (3), a player character in (4), the inventory in (5) and the information box in (6)
4.2.1
A more general distribution of functionality
Using the new event management system we could now distribute the functionality of the application.
The game has an event which triggers when a cell on the map is hovered and a character is chosen. The event starts a A star algorithm from the character to the cell and then present the resulting path to the user. Because the A star algorithm is demanding for the device processor we wanted the algorithm to only run on the device were it is currently used. By setting the options local event to true and server callback to false we could create this effect. The algorithm will only run on the user who pressed the character and intends to move it. The functionality can be seen in figure 4.2.
4.2. The Multiplayer Game
Figure 4.2: The program presents the resulting path of a A star algorithm to the user
The end turn button is created on the first device and mirrored to every device that con-nects to the room. Using the all_click event we created an event which will only trigger when every device in possession of the object has pressed the element. Another event is also ap-plied to the end turn button. This event changes the buttons color and text to waiting for other players. Because this event is not a server callback we could give the button a local state that tells the user it has been pressed.
4.2.2
Handling of distributed objects
One example of how the game takes advantage of the new observe system can be seen in section 4.2.1. Here we talked about how one user can do the A algorithm on their device and the path will only be seen to them. This is an example of the observer system where the cells have their own configuration of their observer where the design attributes are not observed. This way the path will not be shown to anyone else.
In the game you can distribute the full grid and its content from one device to another. With the new child object distinction the receiving device will get a distributed object which has a hierarchy as the one seen in figure 4.3.
4.2. The Multiplayer Game
Figure 4.3: A turf is filled with cells which has a cosmetic child box and either a character, tree or nothing. The blue circles indicate distributed objects and yellow ones child objects.
4.2.3
State synchronization
Going back to the example with the end button referenced before. Here we can use the imple-mented node synchronization method to give the instances of the buttons the same look as before being pressed when the all event has been triggered. This works by having the owner of the end turn button change back the color and text of it and then call the synchronization method. Each instance of the button will be updated to look like the new version.
The session variables allows us to keep non visual data about the objects like health or damage which are essential for a good multiplayer game. In figure 4.4 you can see that the user decides to damage the enemy which sets the health of it to be its current health minus the attacking characters damage. In this case both the health and the damage are session variables which can be set throughout the application lifetime.
4.3. Documentation
Figure 4.4: When hovering the enemy the application gets the health from the white knights session variables and the damage taken from attacks. Then it displays it in the information box together with various information from the enemy.
4.3
Documentation
A documentation web page was created to help the developers find information about Marve and the API. This was done so the knowledge and functionality implemented would not go to waste in further implementations.
5
Discussion
5.1
Related work
When it comes to related work in the field of distributed user interface it was hard to find research which had the same focus area as the ones in this thesis. Instead they tend to focus on cross device development or user experience. However the application state, object handling and event management are something that needs to be in place for DUI to work. That is why we were able to pick out the parts of their research that was relevant for us and see the pros and cons with their method. All though we should take into consideration this was not their area of focus.
5.2
Results
As a game developer you often wants a flexible tool to create your games with. You should not limit the developer but rather let them use their creativity. When I created the game I too focused on not limiting myself to what I though a DUI framework would be able to do. But rather imagine I had no limits to what I could do, but time off course. And this was why the demands for the framework and more precisely the event system are very focused on generalizing the framework for the developer.
An aspect of the pre-existing framework I found lacking was the object handling. As a developer I want total control over what is a distributed object and what is not and that was very present in the demands for the framework.
The finished game is proof of that the proposed solution lets the developer be creative. The functionalities I wanted from the beginning was realised using the new event and object management systems. What the result does not show is the user experience of the framework and how easy it actually is to develop a multiplayer game using it. Further studies should be made on the framework were more developers try to use it. This would give a better idea of how useful the new implementations are.
5.3
Method
I found the needs for the distribution framework by first designing a multiplayer game and then understand what was needed by the framework to realize it. Off course I could have
5.4. The work in a wider context
chosen another project idea and that would generate other needs for the framework. You could argue that this method is very dependent on choosing a project type that matches the use of a DUI application as much as possible. And looking at some other research with frameworks like liquid [8] or mvc-xd [9] they often choose a simpler project that matches some kind of every day use like controlling a media player [10]. A multiplayer game is however very demanding in itself and a lot of functionality needs to be in place for it to work. Furthermore, a multiplayer game is often larger in scale which generates demands on optimization for the game to work. Demands that a simple media player controller might not even think about, because the small delay is not as important.
When it comes to the different method strategies for implementing these demands you can see that the following aspects were important to me, letting the developer be creative with its implementation while still giving them the control they need over the application.
The event system is a good example of when my methods could be a bit overwhelming for the developers. The functions has too be named and globally defined. This can be annoying for a developer if for example it only is a small function that changes a single variable. It also contributes to large code bases which in many cases lead to bad software design. The variable system can also be hard to manage for a developer because the framework gives them the full responsibility to keep track of the synchronization of the project. Because you as a developer can choose when to use the synchronized session variables it gives them the opportunity to keep a local state as well as a global state. However if not treated carefully the application can be unsynchronized across devices to the point of malfunction.
The biggest issue with my methods might be that it gives the developers to much con-trol over the application. I wanted to let the developer create their applications completely free from third party limitations like using Polymer for widget implementations[8]. But be-cause of this they are gonna have to think about all the possible outcomes when the interface are distributed across devices. Some data will not be available on some devices and some functionality might not even be possible when a button have switched place in the environ-ment. However, for some developers this might be acceptable if some sort of documentation is included in the framework.
Because of these issues I find that the written API documentation is highly important for a developer getting into the framework. Using the documentation I do believe any developer with knowledge of Marve could implement the multiplayer game presented for this research. When you start to understand how the global and local state of any object exists you can play and be creative with the framework.
5.4
The work in a wider context
The research of distributed user interface needs to start focusing on the security aspect. When the users are sharing the content of the application the control is completely lost. Most en-vironment such as the browser also gives the users full control over the interface. For DUI application to do the big leap into the hands of actual consumers there is a need for keeping the control without limiting the applications. If this is done I could see a interesting future for this area.
6
Conclusion
This report has given a proposition to how a framework for distributed user interface ap-plication can be implemented. Future tests should be made to validate the method used in this paper, but I do believe it is an interesting method that do have a place in the future of distributed user interface.
6.1
Research Questions
Q1 How can a event system be implemented in a DUI framework so it works in a multi-player game.
Q2 How can management of distributed objects be generalized while still understandable and manageable for a developer.
The answer to Q1 are answered with the fully functional multiplayer game that was done in this project. With the method taken to realise the event management system I can in full confidence say that it is a system that works in the demanding nature of a multiplayer game. Q2 could have had a more definite answer if a study of the understandability was made. However I think I could give a maybe somewhat low validity answer by saying that for me the new child object gives the developer more control over what will be distributed objects and what is not. The system is however fairly generalized where the developer have a lot of freedom in implementing the distributed objects.
6.2
Future Work
Future work for this framework should focus on security as mentioned in section 5.4. The multiplayer game is not really making full use of the DUI functionality. That is why a study has to be done on how these functionalities are best used for the game to be fun and inter-esting. We need to find ways to make full use of what a DUI application can do and find the places it works better than the single device applications.
Bibliography
[1] Sriram Karthik Badam and Niklas Elmqvist. “PolyChrome: A Cross-Device Framework for Collaborative Web Visualization”. In: (2014).
[2] Frederik Brudy, Christian Holz, Roman Rädle, Chi-Jui Wu, Steven Houben, Clemens Nylandsted Klokmose, and Nicolai Marquardt. “Cross-Device Taxonomy:Survey, Op-portunities and Challenges ofInteractions Spanning Across Multiple Devices”. In: (2019).
[3] MDN contributors. MutationObserver. n.d.URL: https : / / developer . mozilla . org/en-US/docs/Web/API/MutationObserver.
[4] Niklas Elmqvist. “Distributed user interfaces: State of the art”. In: Distributed User In-terfaces. Springer, 2011, pp. 1–12.
[5] Eli Raymond Fisher, Sriram Karthik Badam, and Niklas Elmqvist. “Designing peer-to-peer distributed user interfaces: Case studies on building distributed applications”. In: Int. J. Human-Computer Studies 72 (2014).
[6] Anders Fröberg, Henrik Eriksson, and Erik Berglund. “Developing a DUI Based Oper-ator Control Station”. In: J.A. Gallud et al. (eds.) (2011).
[7] Keheliya Gallaba, Ali Mesbah, and Ivan Beschastnikh. “Don’t call us, we’ll call you: Characterizing callbacks in JavaScript”. In: 2015 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE. 2015, pp. 1–10.
[8] Andrea Gallidabino and Cesare Pautasso. “The Liquid.js Framework for Migrating and CloningStateful Web Components across Multiple Devices”. In: (2016).
[9] Maria Husmann. “Investigating Tool Support for Cross-Device Development”. In: (2017).
[10] Dejan Kovachev, Dominik Renzel, Petru Nicolaescu, and Ralf Klamma. “DireWolf-distributing and migrating user interfaces for widget-based web applications”. In: In-ternational Conference on Web Engineering. Springer. 2013, pp. 99–113.
[11] Michael Krug. “Distributed event-based communication for web components”. In: Pro-ceedings of Studierendensymposium Informatik (2016), pp. 133–136.
[12] Jérémie Melchior, Donatien Grolaux, Peter Van Roy, and Jean Vanderdonckt. “A toolkit for peer-to-peer distributed user interfaces: Concepts, implementation, and applica-tions”. In: (2009).
Bibliography
[13] Jean Vanderdonckt et al. “Distributed user interfaces: how to distribute user interface elements across users, platforms, and environments”. In: Proc. of XI Interacción 20 (2010). [14] David Vignoni. Client–server model. 2011.URL: https://en.wikipedia.org/wiki/
File:Client-server-model.svg.
[15] Pedro G. Villanueva, Ricardo Tesoriero, and Jose A. Gallud. “Distributing Web Compo-nents in a Display Ecosystem Using Proxywork”. In: (2013).
