Linköping University | Department of Computer and Information Science Bachelor’s thesis, 16 ECTS | Computer Engineering Autumn 2019 | LIU-IDA/LITH-EX-G--20/010--SE
Layout management in distributed user interfaces
Alexander Eriksson, Gustav Ljungberg
Supervisor:
Anders Fröberg
Upphovsrätt
Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare – under 25 år från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns lösningar av teknisk och administrativ art.
Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/.
Copyright
The publishers will keep this document online on the Internet – or its possible replacement – for a period of 25 years starting from the date of publication barring exceptional circumstances. The online availability of the document implies permanent permission for anyone to read, to download, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: http://www.ep.liu.se/.
Abstract
Human computer interaction is a topic which is quickly progressing forwards, with users looking for new ways to interact with digital content. Graphical user interfaces are all around us, featured in smartphones, tablets, and personal computers. The next step from a graphical user interface, a
distributed user interface is becoming increasingly popular, and is offered by many applications, such as Spotify for example. This bachelor's thesis carried out at the University of Linköping at the
institution for computer science discusses the history of human computer interaction, user interaction and interfaces, and how to manage the layout of a disturbed user interface system. A framework for managing the layout is developed and tested in a prototype.
Acknowledgment
The authors would like to thank Anders Fröberg and Erik Berglund for guiding us through this thesis, and always being available for answering questions. We would also like to extend a thanks to Alex Tanskanen, and Viktor Rotting Tjädermo for providing feedback on our report.
Layout management in distributed user interfaces 0
Abstract 3
Figures and tables 7
Abbreviations 8 1. Introduction 9 1.1 Background 9 1.1.1 Python 9 1.1.2 JavaScript 9 1.2 Motivation 9 1.3 Aim 10 1.4 Problem statement 10 1.5 Summary of method 10 1.6 Delimitations 11 2. Theory 11
2.1 The history of Human Computer Interaction 11
2.2 Distributed user interface 12
2.3 Interaction openness 13 2.4 Callback 13 2.5 Types of component 13 2.6 jQuery 13 2.7 Flask 14 2.8 Socket.IO 14 2.9 Base DUI-system 14 2.10 Related work 15 3. Method 16
3.1 Limitations with the current system 16
3.2 Multiple session handling 16
3.3 Saving and importing data 16
3.3.1 Save layout 17
3.3.2 Import layout 17
3.3.3 Saved information 18
3.4 Refreshing of the page 18
3.5 Disconnect resilience 19 3.6 Components 19 3.6.1 Mirrored 19 3.6.2 Cloned 19 3.6.3 Default 19 3.7 Prototype 20
4. Result 20 4.1 Save 20 4.2 Refresh 21 4.3 Prototype 22 4.3.1 Startup 23 4.3.2 Control Station 23 4.3.3 Blank 25
4.4 Check with mandatory requirements 26
4.4.1 Mandatory requirements for save and import 26
4.4.2 General requirements 27 5. Discussion 27 5.1 Result 27 5.2 Replicability 28 5.3 Future work 28 6. Conclusion 28 References 29
Figures and tables
Table 1: Lists all mandatory requirements for save and import.
Table 2: Lists all general mandatory requirements.
Figure 1: The creation of a folder via the command line.
Figure 2: A music application is running on a PC and a smartphone, the computer can view the
album, and the phone shows the current song playing, selected from the PC. The UI has been distributed over two computing platforms.
Figure 3: Two users connected, user 1 has created a form and colorpicker component and shared
them to user 2 by dragging and dropping. The components are connected, and user 1 can see the changes user 2 makes, and vice versa.
Figure 4: The flowchart describing how the save feature operates. A client sends a request to the
server and the server responds with asking all the clients in the corresponding room to send their components to the server.
Figure 5: A flowchart for the import feature. A client asks the server to import an old layout for the
current room, if it exists, the server will send out the saved data to corresponding client in the room.
Figure 6: An example of a JSON-save file.
Figure 7: This figure shows how cookies are set on the client.
Figure 8: This figure shows how local storage is used to save components on disconnect.
Figure 9: The startup page of the application.
Figure 10: A default room.
Figure 11: Before save.
Figure 12: After importing.
Figure 13: Shows how CSS can be included when sending a component between Control Station and
Abbreviations
1. Distributed User Interfaces (DUI) 2. Graphical User Interface (GUI) 3. User Interface (UI)
4. Human Computer Interaction (HCI) 5. Identifier (ID)
1. Introduction
This bachelor's thesis has been performed at the university of Linköping, at the institution of computer science. The report discusses what a distributed user interface (DUI) is, the history of human
computer interaction, and how to manage the layout of a DUI-system.
1.1 Background
In recent years, the use of DUI has become increasingly popular. For example, the platform Spotify allows the users to change the current song playing on the TV via a smartphone. This is done by having the application running on two devices, and then allowing the user to distribute the content between the two instances of the application.
The path of evolution for human interaction with computers has gone from using punch-cards and switches, to a command line, followed by the first graphical user interface (GUI) to now having DUI [1]. A common denominator between GUI and DUI is that it will be used by people with vastly varying knowledge on computers, and therefore it needs to be simple and intuitive to use. This imposes a challenge on to develop these types of systems.
1.1.1 Python
Python is a high-level programming language which uses the object-oriented approach released in 1990 . Python is a garbage collected language. It focuses on readability while using the off-side rule, 1 meaning that blocks are separated by indentation. Python is used in both large and small projects, and is used in many different scenarios, because of its ability to manage software packages. Flask, which is explained in the section 2.7, is one of these software packages. Python is today one of the most used programming languages and is still updated and worked upon in 2019, with its latest stable release on the 14th of October 2019 . 2
1.1.2 JavaScript
JavaScript is a scripting/programming language, with its main usage in web development . JavaScript, 3 together with HTML and CSS are the fundamental components of the World Wide Web. JavaScript was released in 1995 . 4
1.2 Motivation
Human computer interaction (HCI) is a topic which is constantly moving forward with users looking for new ways to interact with content where DUI is one of the later trends. As DUI-systems become increasingly complex, and offers more features, it is inconvenient for the user not being able to manage their layout. It is time consuming to manually recreate a previous session and position all the
1 Python https://www.python.org/
2 TIOBE Index for December 2019 https://www.tiobe.com/tiobe-index/ 3 Javascript https://www.javascript.com/
UI-elements to where they previously were. This thesis therefore develops a framework with the functionality to save and import the state of a DUI-system. The state in this context means which UI-elements exist and which clients who has them.
1.3 Aim
The aim of this thesis is to develop a framework with the functionality to save and import the state of a system using DUI, improve and extend a basic DUI-system, and analyzing the added features.
1.4 Problem statement
The problems this thesis is aimed to solve are the following:
1. How to develop a framework which can save and recreate the current state of a DUI-system? 2. How to develop such a framework in an effective and intuitive way?
Mandatory requirements for Save and Import: 1. All existing components shall be saved
2. Information about which client owns what components shall be saved
3. The connection between components (if such a connection exists) shall be saved 4. The internal state of a component shall not be saved
5. The state of the DUI-system shall be able to be recreated accordingly to a previous session Table 1 lists all mandatory requirements for save and import.
General requirements:
1. Multiple sessions shall be able to exist simultaneously, and independently 2. The page shall be able to refresh without losing the state of components 3. A default component shall be able to be set
Table 2 lists all general mandatory requirements.
1.5 Summary of method
The first stage of the method is to analyze the base DUI-system and find out the limitations and areas that need extending. The second stage of work is the develop the framework for save and import, as well as extending the base DUI-system. This work was added upon a previous DUI-system, which in its current form did not support a save or load features. The DUI-system on which the new features are developed on is written with HTML, CSS, JavaScript and Python. The initial phase of this stage consisted of developing, and the latter parts more on testing the system, improvements, and design in the prototype. The third and final stage was documenting the results, writing of the report, and presenting the solution.
1.6 Delimitations
Below follows a list of the delimitations for this work:
1. Since the framework is built in a previous existing DUI-system, this implementation is limited to the languages and libraries it used. (JavaScript, CSS, HTML and jQuery).
2. Theory
The section below explains the most crucial steps in human computer interaction, what a distributed user interface is, and how to manage the layout in a distributed user interface.
2.1 The history of Human Computer Interaction
Human interaction with a computer is a relationship of inputting and outputting information [2]. The human inputs information, it is processed by the computer, and data is given in return. This principle still holds today, however how it is done has changed dramatically. At the start of using computers, the information was fed to the computer by moving physical bits, like punch-cards and switches. At this point, the individuals using computers had to be educated on how to use them. They were therefore mostly, if not exclusively, used by scientists. The next stage was command-line interface (CLI). With this type of interface, lines of text are inputted into a terminal, each line of text is a command, telling the computer what to do. This technique took shape in the mid-1960s and was used extensively for the coming 20 years.
The next significant step in HCI was the graphical user interface. With a GUI, a human interacts with a computer via a graphical interface, instead of directly inputting commands. The main reason for the development of GUI was to make computers easier to use. Before the GUI, learning how to use a computer was quite difficult, with many things to learn in order to use it properly, and could require reading documentations about the commands. A GUI made computers more accessible for the average user. Today, GUI’s appear all around us, to our smartphones, music players and even some household appliances as refrigerators features a GUI. Figure 1 below illustrates the creation of a folder via a command line. The user, in Windows 10 in this example, would have to open the CMD and type the command mkdir, and then the name of the folder. The command needs to be typed correct, otherwise it would result in an error. This is not very intuitive. A. Feizi and C. Wong [3] performed a study to compare CLI and GUI. They came to the following conclusion: “CLI were found to be more difficult to learn and less ease of use, even for developers as well as designers.”. Meanwhile the process of creating a folder via a GUI for example in Windows 10 simply requires a right click followed by the selection of new and then folder.
Figure 1: The creation of a folder via the command line.
2.2 Distributed user interface
DUI is defined by J. Melchior et al [4] as “Any application User Interface (UI) whose components can be distributed across different displays of different computing platforms that are used by different users, whether they are working at the same place (co-located) or not (remote collaboration)”. An important clarification here is that in a DUI-system it can be the same user distributing their content over several computing platforms, where as in the definition above, the authors stated that it is used by different users which does not always have to be the case for a DUI. Neither does it have to be different computing platforms, it can be the same platform with e.g. separate monitors. Figure 2 below illustrates a scenario where a user has distributed their content over two computing platforms.
Figure 2: A music application is running on a PC and a smartphone, the computer can view the album, and the phone shows the current song playing, selected from the PC. The UI has been
distributed over two computing platforms. Below follows a list of keywords within the topic of DUI
1. Multi-monitor usage. Example: one user at a home computer connects a second monitor to their system and can now distribute UI-components across the same platform, on two separate screens. [5]
2. Multi-device usage. Example: one user uses their smartphone as a remote to control an application on their personal computer. [6]
3. Multi-user. Example: a scenario where several devices and/or monitors could be connected, and several users can interact and move UI-components across them. [7]
2.3 Interaction openness
When a user interacts with a UI, different events can be started. For example, when a user clicks a button on a website, the button being clicked is an event. The event could trigger a section of code to be executed on the server side. The concept of interaction openness is that the type of event should not matter [8]. The event could be a button being pressed; however, it should also be able to be changed into voice control, and the functionality of the system should not be changed. Furthermore, changing the event to another type of event should be easy. In essence, the section of code that executes when an event happens, should be decoupled from what type of event it is.
2.4 Callback
A callback is a function that gets executed after another function completes such as when a shared text field gets updated [8]. A user edits the text field which triggers a callback to fire to another function with the changed state. A component can contain several events, e.g., buttons. This means that a component can have several actions which results in a callback.
2.5 Types of component
The DUI-system worked on features three types of components;
Mirrored component: A mirrored component in this context is a component which has been copied, and then moved to present an identical object in a second place. The two components are now coupled in a way that if one is modified, the second one responds equally. [8]
Cloned-component: A cloned component can be seen as a copy-paste function. An identical copy to a given component is generated, and moved elsewhere, however they are not coupled. They can be modified independently. [8]
Atomic component: A component which only appears in one place. [8]
2.6 jQuery
jQuery is a JavaScript library which goal is to be fast, easy to learn, light weight and offer additional 5
features to developers. jQuery was started in 2006 as a side project by John Resig and is today one of the most used JavaScript libraries [9]. jQuery is constantly updated by the 6 The jQuery Team. jQuery is a powerful selector, animation handler and event handler. It seeks to simplify complicated
JavaScript programming. jQuery was made and is updated to be compatible and to work consistently across vast majority of browsers and devices.
5 jQuery https://jquery.com/
6 Historical trends of JavaScript libraries
2.7 Flask
Flask is a 7 Web Server Gateway Interface (WSGI) web application framework written in Python. The WSGI handles the communication between a web server and web application. Besides this, it also describes how web applications can be chained together. Flask also has an extension called Flask Socket.IO . The extension can be used to communicate with the client side. Flask was officially 8 released in 2010.
2.8 Socket.IO
Socket.IO is a JavaScript library, commonly used to implement features such as real-time analytics, 9 binary streaming, instant messaging, and document collaboration. The latest stable release was September 20, 2019.
2.9 Base DUI-system
The work in this thesis extends a previous DUI-system, which is referred to in this report as base DUI-system. The base DUI-system was developed and documented by Anton Stevensson [12]. For a full understanding of the base DUI-system it is recommended to read the report by Anton Stevensson, however a short explanation is done here. It is programmed with HTML, CSS, JavaScript and Python. It uses jQuery for most of its component functionality and uses Python Flask for the communication between clients. The two core files behind the functionality is a server written in Python and a client side JavaScript file called DistributionManager.js. When the server is running the system allows several users to be connected with their web browsers, and the distribution manager does most of the underlying work. There are different demo scenarios that can be ran, however they work in a similar way. The demo scenario illustrates the base DUI-system's functionality well. demo running is illustrated in the figure below.
Figure 3: Two users connected, user 1 has created a form and colorpicker component and shared them to user2 by dragging and dropping. The components are connected and user1 can see the
changes user2 makes, and vice versa.
On the left-hand side there is a menu, showing which users are connected. A “me”-text is displayed to illustrate which user is yourself. Below the list of users there is list of components which can be created. Creating an object is simply done by clicking the corresponding text box, after which it will appear on the screen as a local (atomic) component. Sharing the component is done by clicking and dragging, a feature given by jQuery, and then releasing over a connected user in the left-hand side menu. The user the component was shared with will now be seen in their browser. In demo2,
components are mirrored when shared, i.e., the components are connected. If the component is altered by one user, all users with the component would see the change. The limitations with this base
DUI-system is explained in heading 3.1 in Method.
2.10 Related work
The report by A. Demeure et al [11] introduces a reference model to define a DUI-system, in order to understand and classify current DUI-systems. The reference model is divided into four parts:
computation, communication, coordination and configuration. The four parts are defined by the authors as what is distributed, when is it distributed, who is it distributed to, and from where and to where is the distribution operated. The authors concluded that with this new reference model, many different types of DUI-system can be characterized. The authors also identify the most common situations for DUI-systems. Multiple monitors situation, multi-platform situation, detachable user interface and fully distributable situation. The authors did not consider any scenarios with multiple users, they limited their reference model to only DUIs with one user.
The paper by K. Luyten and K. Coninx et al [10] shows different approaches on interactions in a distribution user interface. They take advantage of the information a device shares about itself when it
is interacting with other devices on a network. Using this approach allows them to automatically decide if the targeted device is suited for the task. They built 2 prototypes and used them to compare against each other in a case study. One was called peephole and it focused on 3D tracking. The display would change depending on where your device was and in what direction it was pointing. The other one was a distributed web interfaces and it was a HTTP based daemon and its job was to distribute an interactive webpage between different kind of devices. A benefit of using this type of system is that it supports co-location and remote users. They also conclude how easy it was for a programmer to add collaboration features to a DUI system like theirs.
3. Method
The following section describes limitations with the base DUI-system, what extensions were made to it, how they were implemented, and advantages and disadvantages with the selected approaches, together with other possible solutions.
3.1 Limitations with the current system
The base DUI-system was a working product; however, it had several limitations. The first one being that there was no support to manage the layout, i.e. no way to save nor load the state. The state in this case being defined as which components that exist, and how and if they are shared and distributed between users. In practice this means that every user would have to create and share UI-elements every time they started the base DUI-system. This is time consuming and unnecessary work. The second issue noticed is that refresh effectively worked as a disconnect, meaning that if a user were to refresh, they would then re-enter as a new user. This meant that all components owned by the user would be lost. The third issue identified was the lack of handling multiple sessions. The base
DUI-system could only handle one session at the time, meaning all users were one group. In practice this meant that when the server was running, only one group of people could use the system. The fourth limitation was that there was no feature to set a default component.
3.2 Multiple session handling
In order to handle multiple sessions simultaneously, users must be divided into subsets. Flask offers support for this with the Socket.IO extension, which contains the join_room and leave_room functions. The Flask functions send and emit now takes an extra parameter, room. The room parameter is used to specify which room the information is to be sent to; thus, users are divided into subsets. When the application is started, the user is prompted to enter a string to join a room. If the inputted string matches the name of an existing room, the room in question is entered. If the name of the room does not match the name of an existing room, a new room is created with that name.
3.3 Saving and importing data
The three subheadings below describe how the save and import framework is implemented, and the structure of the save files.
3.3.1 Save layout
Figure 2 below is a flowchart describing how the save feature works. The task of the save feature is to save the current state of a DUI-system. The client which is requesting a save sends a save request to the server, the server then proceeds to tell all other clients connected to the same room that a save has been requested. All N clients responds back to the server with their data. The data in this case is the name and the id of all their components, i.e., any component that exist on the client. The owner of the component, and the connections with that component is determined by the server who tracks this information during runtime. The server converts the data it receives from the clients to json and saves it locally with the room’s name as the name of the file.
Figure 4, the flowchart describing how the save feature operates. A client sends a request to the server and the server responds with asking all the clients in the corresponding room to send their
components to the server.
A drawback to this solution in its current form is that some unnecessary data is being sent and processed by the server. For example, if multiple clients have the same component, they will all send the same information to the server. The server is tracking components by their ID’s and which users share the component, but the server has no information of the component itself. One solution to this would be to save a descriptive name for each component or only having the owner of the component sending its information to the server. An advantage of having all users sending their information to the server is error handling. If the information sent by the users about the component is not the same this could indicate there being something wrong in the system.
3.3.2 Import layout
Figure 3 below illustrates how the import feature works. The task of the import feature is to meet the following criteria, as mentioned in the mandatory requirements “The state of the DUI-system shall be able to be recreated accordingly to a previous session”. It does so accordingly to the flowchart displayed in figure 3. Client 1 requests an import, by clicking the import button. An import request is then sent to the server. The server upon receiving this request reads from a save file, and the server then sends out the component data to all N clients.
Figure 5, a flowchart for the import feature. A client asks the server to import an old layout for the current room, if it exists, the server will send out the saved data to corresponding client in the room.
3.3.3 Saved information
The information about the components is saved in a JSON-format. The benefit of using a JSON-save is that a user could write their own save file, and then import it when the system is running. If there is a component that the user wants set as a default, or an administrator wants to change the owners of a component, it does not necessarily have to be done when the system is running, it can simply be modified in the save file.
3.4 Refreshing of the page
By default, refresh is regarded as a disconnect, this causes a few problems. If a user were to refresh their page, they would be thrown out of the room, and if they enter the same room again, the components will not be there, since they are now considered a new user. To solve this, the browser cookies and local storage is used. The user’s identifier (ID) is saved in cookies when a user enters a room. When a refresh then occurs, a check is made to see if a user ID exists in cookies. If it does, the user will receive the ID of the one saved in cookies. The user is now regarded as the same user as it was before the refresh occurred.
Just as the disconnect happens, the components which the user currently has are saved in local storage. All the information about the component is saved, including its state. When the user then rejoins the rooms automatically after a refresh, the components saved in local storage are loaded in. When a user leaves a room manually, their ID is removed from cookies, and local storage is cleared. Another approach to solve this could have been to do an auto-save feature, which would make a save every time a change occurred. However, this was deemed too inefficient. Saving every time a
component is moved a pixel or every time the internal state changes would result in a large amount of information being saved constantly. Most of the saves would also be redundant. Because of this, the approach to only save when a user disconnects was chosen.
3.5 Disconnect resilience
If a user were to lose internet connection, they would be disconnected immediately, which is inconvenient since an internet connection may have minor fluctuations. If the connection were to be lost for even a millisecond, the user would be disconnected. This is fixed by adding some resilience to the connection. If the communication between the server and a user is lost, the server waits a few seconds and sees if a connection can be reestablished. If a connection cannot re-establish, the user is removed by the server. In our implementation, the socket tries to send information to the client for 60 seconds, if the user does not respond, they are considered disconnected and removed. If the system is large enough, and there are a large amount of users connecting and disconnecting, a resilience of 60 seconds could be too long, and resulting in a loss of performance, since sockets are trying to send information to users that could be disconnected, resulting in unnecessary sending of data. However, in a phone application, a 60 second resilience is quite convenient, for example, if a user is walking between two Wi-Fi-hotspots and needs to reconnect.
3.6 Components
When a component is first created it exists in an atomic state. This state can be changed to cloned and mirrored using functionality from the distribution manager (DistributionManager.js).
3.6.1 Mirrored
Mirrored components work the same as in the base DUI-system but with a few exceptions. The user experience is the same. When a component is mirrored it will send information to the server to set it apart from the new components and functionality that was added. The new information that was added includes if the object should be hidden such as it is when a component is cloned, if the state should be shared and there is also an option to send the initiation script as a string instead of the name of the script. These parts were added to make the function that handles transfer more generalized.
3.6.2 Cloned
A cloned component works independently from other components with the same ID. Cloned
components do not share its state. This was implemented in the base DUI-system. This is still the case in the new system, but new functionality has been added to create a shared atomic component. This component acts like an atomic component but it is mirroring the state of the component and it is instead hiding it on the original user. This allows a component only to be edited by and shown on one client at the time. A hidden component is hidden until the other client is done with the component. At this point it will disappear from the second client and reappear on the first client with the same state it had when it disappeared from the second client. This was added because it was deemed useful in a lot of scenarios in the real world.
3.6.3 Default
A default component is a component every client gets assigned when they join a room with a default component set in it. This means default components is limited to existing on a room level. The way the system is built allows for two different kind of default components. One where the components
state is shared between all users, mirrored, and one where the state is not shared between users, cloned.
There are two ways to create a default component, one way is by importing a save file with a default component set. If this is the case, User 1 will be assigned to be the owner of the component. The second way is to ask the server, during runtime, to set a local component as a default component for the room. The second way to create a default component assumes that the user will always be there to handle any updates to the component, this requirement is only needed if the component is in a
mirrored state.
The two ways of creating default components has its own problems associated with them. Setting a default object in a save file assumes that the first user has the required functionality to be the owner of a component. This can be combated in different ways but below are two solutions presented, both solutions can be implemented to work together or by their own. First solution, a check with the client to make sure the required functionality to handle being the owner exists on the client. Second solution, functionality can be added to server to specify which user the component originally belonged to or specifically set to be assigned to a user.
3.7 Prototype
To test and showcase the developed framework, a prototype was designed, as shown in 4.3. The prototype is divided up in two parts: Control Station and Blank.
4. Result
The following section showcases the functionality of the developed system.
4.1 Save
Figure 6 illustrates an example of how a save file could look like. Each block is one component, and the information saved about all components is their ID, name and which devices the components are shared with. If the component is set as a default component, devices are set to all, to indicated that all connected clients shall have them. If a default component is set, a Boolean is also added to the saved information, to indicated whether it is a cloned or mirrored component. Since the system supports multiple rooms, there needs to be a way to distinguish which components are in which rooms. Therefore, each room gets its own save file, and the name of the save file is the name of the room.
Figure 6, An example of a JSON save file.
4.2 Refresh
In the base DUI-system, a refresh of a client’s web browser would disconnect the client and destroy all components. To combat this, cookies and local storage was used. Also, server changes had to be made to support this feature. The moment a client leaves the page, such as a refresh of the page, its session ID would get destroyed and there was no way for the server to know it was the same user. This problem was solved by saving a client’s session ID as a cookie on the client itself. The cookie would be sent to the server on the next page load, if the server recognizes the old session ID as an old user it will port all components with ties to the old session ID to the new session ID. This would allow a user to leave a page and come back later and have all its components calls and callbacks transferred to the new user. Even though this solves the problem of connecting the old components to the new session id this would not solve the problem of a user losing its components. How a cookie is set on the client is shown in figure 7.
On a page refresh all the HTML code alone with saved JavaScript data gets destroyed. To save the component a local save must be made. This save is executed when the JavaScript function called window.onunload() is called. An example of how the saved data can look like is shown in figure 8. This function does not execute reliably and will not execute on instances where a client’s web browser closes unexpectedly.
Figure 8, This figure shows how local storage is used to save components on disconnect.
4.3 Prototype
In the first part of the prototype, Control Station, users can enter and exit rooms, create components, share components, and use the save and import framework. The prototype was designed in order to test and verify all the functionality thoroughly. There are five components that can be created, named: JavaScript, Form, checkboxes, Range, and Select. Each of these five components can be mirrored or cloned to other users connected. The components can also be set as default components. To share a component, it needs to be clicked and dragged and dropped upon a connected user in the user list on the left-hand side. The drag and drop functionality were supported previously, and not something developed now.
The new functionality added is that if the component is dropped on the left-hand side of the user’s hitbox, the component is mirrored, if dropped on the right-hand side, the component is cloned. Once a user clone one of its components it will be hidden and only shown on the receiving clients end. When the receiving client is done with the component it will reappear on the first client’s device.
In order to make a component into a default component, the component needs to be created, and then dropped on the corresponding element. There are two elements that can make turn a component into a default component. If a component is dropped upon the element called “All users” the default
component that is created will act as a clone and will not share any changes made to it. If its instead dropped on the element called “All users (shared)” the default component that is created will mirror any changes made to it.
The second part of the prototype, Blank, is a blank slate with no functionality to create components, share components, nor save and import.
4.3.1 Startup
When the application is started, the figure 9 below shows what a user is greeted with. In the text box, the name of a desired room is to be entered. If the name of an existing room is entered, the user is connected to that room. If a name is entered which does not correspond to an existing room, a room is created with that name. How a room looks like is described in the heading 4.3.2 below.
Figure 9: The startup page of the application
4.3.2 Control Station
The figure below is what happens after Enter is clicked, the user is entered into a room. Similar to the base DUI-system, there is a menu on the left-hand side. The menu shows connected users, and components that can be created. In this prototype, if a component is dragged and dropped on the left-hand side of a connected user, the component will be mirrored to the second user. If the
component is dropped on the right-hand side of a user, the component becomes cloned. In order to set a default component, the desired component is dragged and dropped on the “All users (shared)” text. The same logic applies here, if the component is dropped on the left-hand side of “All users (shared)”, it becomes a default mirrored component, and if dropped on the right-hand side it becomes a default cloned component. In the room, there is also the possibility to save and import a layout, via the save and import buttons, also found in the left-hand side menu.
Figure 10: A default room.
The two figures below illustrate before, and after a save, in a room with two users. User 1 has created a Form component, and mirrored it to user number two, and inputted some text. User 1 has created an atomic component Range, user 2 has created an atomic component Select.
Figure 12: After importing
As visible in figure 12, all components which existed before save, have been saved, and has successfully been imported. The state of each component has not been saved, accordingly to the mandatory requirements. The Range component has been reset to its default value in the middle, and the text boxes in the Form component has also been reset to corresponding default values. The connection of the Form has been saved, and changes made by user 1 into the Form are visible for user 2, and vice versa. Furthermore, the Range component, and Select component are not connected, they were successfully imported as atomic.
4.3.3 Blank
The functionality to handle connections with the server is a part of a JavaScript file called
distributionManager.js. This file is crucial to include on the client side. Other than that, Blank was made to require as few lines of code as possible and to be flexible. By comparing the two HTML files we can see a major size difference. The main HTML file is 19 kB while Blank is about 3 kB.
Blank also does not need any CSS more than the basic for its body and component list. The required CSS is sent when sharing a component. Blank does not need know anything more than the bare minimum to function. This is shown in the figure 13 below which explains how each component can send its own CSS by separating its CSS into its own file.
Figure 13: shows how CSS can be included when sending a component between Control Station and Blank.
To make Blank as lightweight and flexible as possible, it was tested with JavaScript’s eval() function. When a user shares its component with another user it will also send a text version of the initialization function to the user. This is where eval is useful. The receiving will execute the function as a string and get the same result as if the initiation function was locally stored. Using eval() is a huge security risk and should never be used in production or anywhere near unknown parties. The use of eval in this paper were for testing purposes only.
4.4 Check with mandatory requirements
This section presents the result of the mandatory requirements.4.4.1 Mandatory requirements for save and import
1. All existing components shall be savedThe prototype can save atomic, mirrored and cloned components, every component created by a user is saved.
2. Information about which client owns what components shall be saved When a component is saved, it has a list of users the component is owned by.
3. The connection between components (if such a connection exists) shall be saved
If a component is mirrored to another user, and then a save occurs, when the import is made and the component is recreated, it remembers that connection to another user. In the save file, any connection that a component has to other users is saved.
4. The internal state of a component shall not be saved
As showcased in the figure 11 and 12 under Result, when the import occurred, the components were recreated and set in their default state, the changes made to them previously has not been saved. In the save file, the information saved is just their existence, and not their state.
5. The state of the DUI-system shall be able to be recreated accordingly to a previous session Also shown in figure 11 and 12 under Result, the state of the system has been successfully recreated, accordingly to the previous session.
4.4.2 General requirements
1. Multiple sessions shall be able to exist simultaneously, and independently
The use of Socket.IO and rooms made it possible to divide users into subsets and work independently. Several sessions can exist simultaneously without interfering with each other, as demonstrated in Result.
2. The page shall be able to refresh without losing the state of components
As shown in figure 8 in result, when a refresh occurs, the data is saved in the browser, all components remain, the state of the component is not changed, and any connection between components still exists.
3. A default component shall be able to be set
As demonstrated in result, a default component can be set. Therefor all the above requirements have been met.
5. Discussion
In the three headings below, the result, the replicability and future work is discussed.
5.1 Result
The first aim of this thesis was to implement a framework for managing the layout of a DUI-system and fulfilling all the mandatory requirements. A framework was developed that could save and load the layout, and all the mandatory requirements has been fulfilled. One thing we would do differently if we were to start over, is to have a clearer picture of exactly what is to be implemented, and how to do it, before starting to implement. We do not think this greatly impacted the result; however, it would likely have sped up the working process a bit. Quite a few times during the implementation, we got stuck simply because we had not exactly agreed on how a given function was supposed to work. In essence, a more detailed work plan from the beginning would have sped up the working process. The framework for saving and importing the layout can be activated via a GUI, instead of a command via the command line. We deemed it important to use a GUI since it is shown that a GUI is easier to use than a CLI by A. Feizi and C. Wong [3].
The framework itself is easily accessible and beginner friendly for developers because it requires the developer to do minimal configuration to get started. The paper by M. Mattsson et al [13] describes how important it is with abstractions to make it usable for the developer. The authors also explain how mixing two frameworks can cause problems if they are overlapping and how this can be solved by including wrappers and design patterns. Our framework was made to be the only distribution manager in the system, but it does not limit the use of other frameworks.
We used the local storage on the client side to store the user data, instead of storing it on the server. According to M. Jemel and A. Serhrouchni [14] web applications are moving towards client side storage. The authors proposed an encryption of data on the client side, to increase the data security. Data encryption is something that could be added to our work..
The eval() function itself is a very useful tool for testing functionality, it takes a string as a parameter and will evaluate it as JavaScript code. This is where the problem arises, a malicious user can insert its own code into the system. The executed code will be in the same scope as the system. Apart from the malicious use it is also very slow to execute the eval() function.
5.2 Replicability
If someone were to replicate the work in this thesis, we believe the end product would have similar functionality. However, how it was implemented may vary. Many of the features implemented, could have been made in several different ways, which is discussed throughout the report. The different approaches impose both advantages and disadvantages, however they mostly end up offering the same types of functionality.
5.3 Future work
There are a few areas that can be extended upon. The first of which being more features and specifications for sharing components. For example, a component could be specified such that it is automatically given to the user with the largest monitor. Secondly, more research could be put into the use of eval, and possible alternatives to creating a Blank without using eval(), i.e. alternatives to eval. Thirdly, a login-system of sorts could be convenient. Users could be limited as to which rooms they can access, different users can have different privileges, such as what components can be created. A user can be set to administrator of a room, etc.
6. Conclusion
In conclusion, a framework was developed to manage the layout of a DUI-system and various improvements were made such as disconnect resilience, refresh support etc. A prototype was developed, where the functionality of the framework and improvements were demonstrated. The prototype fulfilled all the mandatory requirements.
References
[1] Brad A. Myers. 1998. A brief history of human-computer interaction technology. interactions 5, 2 (March 1998), 44–54.
DOI: https://doi.org/10.1145/274430.274436
[2] G. Chao, "Human-Computer Interaction: Process and Principles of Human-Computer Interface Design", 2009 International Conference on Computer and Automation Engineering, Bangkok, 2009, pp. 230-233.
DOI: https://doi.org/10.1109/ICCAE.2009.23
[3] A. Feizi and Chui Yin Wong, "Usability of user interface styles for learning a graphical software application," 2012 International Conference on Computer & Information Science (ICCIS), Kuala Lumpeu, 2012, pp. 1089-1094.
DOI: https://doi.org/10.1109/ICCISci.2012.6297188
[4] J. Melchior, J. Vanderdonckt and P. Van Roy, "Modelling and developing distributed user interfaces based on distribution graph", 2012 Sixth International Conference on Research Challenges in Information Science (RCIS), Valencia, 2012, pp. 1-10.
DOI: https://doi.org/10.1109/RCIS.2012.6240450
[5] Grudin, J. “Partitioning digital worlds: focal and peripheral awareness in multiple monitor use”, Proc. of ACM Conf. on Human Aspects in Computing Systems CHI’01 (Seattle, March 2001), ACM Press, New York, 2001, pp. 458-46.
DOI: http://dx.doi.org/10.1145/365024.365312
[6] Sjölund, M., Larsson, A., and Berglund, E. “Smartphone Views: Building Multi-device Distributed User Interfaces”, Proc. of Conf. on Mobile Human-Computer Interaction MobileHCI'2004, Lecture Notes in Computer Science, Volume 3160, Springer-Verlag, 2004, pp. 127-140.
DOI: http://dx.doi.org/10.1007/978-3-540-28637-0_66
[7] Berglund, E. and Bång, M. “Requirements for distributed user interface in ubiquitous
computing networks”, Proc. of 1st Int. Conf. on Mobile and ubiquitous multimedia MUM'2002. ACM Press, New York, 2002.
DOI: https://doi.org/10.1016/j.ijmedinf.2005.05.013
[8] A. Fröberg, “Models and Tools for Distributed User Interface Development”, Licentiate dissertation, Linköping, 2012.
[9] C. Severance, "John Resig: Building JQuery", in Computer, vol. 48, no. 5, pp. 7-8, May 2015. DOI: https://doi.org/10.1109/MC.2015.135
[10] K. Luyten and K. Coninx, "Distributed user interface elements to support smart interaction spaces," Seventh IEEE International Symposium on Multimedia (ISM'05), Irvine, CA, 2005, pp. 8 pp.-.
DOI: https://doi.org/10.1109/ISM.2005.52
[11] A. Demeure, J. Sottet, G. Calvary, J. Coutaz, V. Ganneau and J. Vanderdonckt, "The 4C
Reference Model for Distributed User Interfaces," Fourth International Conference on Autonomic and Autonomous Systems (ICAS'08), Gosier, 2008, pp. 61-69.
DOI: https://doi.org/10.1109/ICAS.2008.34
[12] A. Stevensson. (2017). Distributed User Interfaces for the Web. OAI: oai:DiVA.org:liu-140995
[13] M. Mattsson & J. Bosch & M. Fayad. (1999). Framework Integration Problems, Causes, Solutions.. Commun. ACM. 42. 80-87.
DOI: https://dl.acm.org/doi/10.1145/317665.317679
[14] M. Jemel and A. Serhrouchni, "Security enhancement of HTML5 Local Data Storage," 2014 International Conference and Workshop on the Network of the Future (NOF), Paris, 2014, pp. 1-2. DOI: https://doi.org/10.1109/NOF.2014.7119784
