RELIABLE ACCESS TO SYNCHRONIZED WORLD STATE INFORMATION IN PEER TO PEER NETWORKS

(1)

Thesis no: MCS-2005:20 08 2005

RELIABLE ACCESS TO SYNCHRONIZED WORLD STATE INFORMATION IN PEER TO PEER

NETWORKS

André Borgqvist

Department of

Interaction and System Design School ofEngineering

Blekinge Institute of Technology Box 520

SE – 372 25 Ronneby Sweden

(2)

This thesis is submitted to the Department of Interaction and System Design, School of Engineering at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Master of Science in Computer Science. The thesis is equivalent to 20 weeks of full time studies.

Contact Information:

Author:

André Borgqvist

Email: anbb01@student.bth.se

University advisor:

Martin Fredriksson Societies of Computation

Department of Interaction and System Design Email: martin.fredriksson@bth.se

Department of

Interaction and System Design Blekinge Institute of Technology Box 520

SE – 372 25 Ronneby Sweden

Internet : www.bth.se/tek Phone : +46 457 38 50 00 Fax : +46 457 102 45

(3)

Abstract

Virtual environments where users can interact with each other as well as with the environment are today used in many application areas ranging from military simulations to massive multiplayer online games. But no matter the application area, as soon as the number of users reaches a certain threshold, hosting a virtual environment on a single machine can become problematic. Speed and quality of the network connection will limit the number of concurrently connected users in terms of acceptable visual quality and hardware requirements of the server will be strict. With a single point of failure, system reliability could easily be compromised by means of network or host failure.

Distribution of the virtual environment therefore seems a reasonable approach in order to address this problem.

Hardware and network requirements would not be so critical and it would increase reliability by having no single point of failure. Unfortunately distribution introduces new problems dealing with synchronization of the world state within the distribution network.

A possible solution to these problems with the focus on reliability will be presented in this thesis. The solution uses a peer to peer platform that is able to adapt to changes in the network infrastructure as a base for all communication.

To improve synchronization efficiency the network will be dynamically divided into multicast groups based on synchronization needs. The solution will be tested for performance with the network fully functioning and in a number of more of less broken states to determine the reliability. The results from the tests conclude that the system is able to perform with what must be seen as acceptable performance levels even in very problematic network environments. The scalability of the system did also meet the expectations but the results would have benefited from more experimentation with larger user populations.

Keywords: Distributed systems, peer to peer, virtual worlds

(4)

(5)

Chapter 1 Introduction

Virtual environments where users can interact with each other as well as with the environment are today used in many application areas, ranging from military simulations to massive multiplayer online games. But no matter the application area, as soon as the number of users reaches a certain threshold, hosting a virtual environment on a single machine can become problematic. Speed and quality of the network connection will limit the number of concurrently connected users in terms of acceptable visual quality and hardware requirements of the server will be strict. With a single point of failure, system reliability could easily be compromised by means of network or host failure resulting in loss of accessibility, the utmost important variable in virtual environments to address.

Distribution of the virtual environment therefore seems a reasonable approach in order to address this problem. Hardware and network requirements would not be so critical and it would increase reliability by having no single point of failure.

Unfortunately distribution introduces new problems regarding synchronization of the world state within the distribution network. The aim of this thesis is to provide insight into some of the problems that anyone face when constructing a reliable distributed interactive simulation, or more specifically a distributed virtual world. The problems that will be addressed deals with how to provide reliable access to world state information stored and distributed in a peer-to-peer network. Due to the nature of virtual environments this information is very time-dependant; it is massive both in size and in update frequency and the user experience is very much dependant on the perceived responsiveness of the network.

A possible solution to these problems with the focus on performance and reliability will be presented and tested in this thesis. The solution uses a peer to peer platform that is able to adapt to changes in the network infrastructure as a base for all communication.

To improve synchronization efficiency a synchronization algorithm will be applied and made content aware and the network will be dynamically divided into multicast groups.

The solution will be tested for performance with the network fully functioning and in a number of more of less broken states to determine the solution’s reliability.

This first chapter of the thesis is an introduction to the topics that will be addressed later on in this thesis. Section 1.1 gives a background description of interactive simulations and the traditional techniques used for hosting such systems. The following section 1.2 emphasizes certain problems associated with these techniques, whereas section 1.3 presents the hypothesis that this thesis is based upon; which aims to solve the mentioned problems. Section 1.5 continues to describe the research approach and contributions introduced herein and the last section is an outline of the rest of the thesis.

1.1 Background

During the last decade interactive simulations and virtual worlds have become more and more used in many application areas; they are categorized below based upon the user service or experience they try to give.

1

(8)

Background

Figure 1.1: The left figure illustrates a client-server architecture and the right a peer to peer architecture.

• Togetherness – This category includes application areas where the virtual world serves as an extension of the real world to provide a sense of togetherness for the users involved. The actual purpose of the virtual world can range from a company with world-wide employees that wish to improve their communication to virtual conferencing, virtual diplomacy and social chat rooms.

• Information sharing – The virtual word is here used for sharing of information between the users or between the users and some authoritative actor. Examples of this application category include e-learning, virtual campus, museums.

• Service sharing – Exchange of services is here the prime use for the virtual world.

Just as in the information sharing category the exchange can occur between the users or the users and some authoritative actor. Examples include: Groupware, telemedicine, virtual shopping malls.

• Simulation – In areas where real world experience is dangerous or expensive to obtain, virtual worlds that simulate reality are used for training and educational purposes. The military uses virtual worlds to simulate battle situations and pilots, both civilian and military, can practice their skills in flight simulators.

• Entertainment – This category includes many types of games, some similar to the simulations in the previous category but here used for entertainment, ranging from multiplayer board games to massive multiplayer online games.

The outstandingly most active and broadly used category of those mentioned above is that of entertainment and in particular the application domain of massive multiplayer online games (MMOG) [11]..

Consequently, this thesis will focus on the MMOG area which has requirements on many qualities but foremost on the sense of reality and experience there of;

encompassing good responsiveness, high interactivity and visual quality. But what really sets MMOG apart from other application areas is the high number of concurrently connected users and the associated scalability problems.

2

(9)

1.2 Problem description

An interactive virtual world that aims to support a large number of users hosted on a single machine will require a lot from the hardware on that machine and even more of its network connection. The traditional solution to scalability problems is to somehow distribute the system and thereby spreading the load of a single host to multiple machines. But distributing an interactive virtual world will also require that the world is kept in a consistent state on all involved machines - as inconsistencies, delays and anything that draws from the sense of reality will lessen the user’s experience. This introduces very high requirements on the synchronization protocols and mechanisms used in such a system. Not only must they be quick in order to prevent delays but the amount of data transmitted can also be huge as has been shown by Macedonia et al. [4].

Distribution will also introduce new failure sources that are not present in a centralized solution [1], e.g. faults in the transmission medium used for synchronization. Therefore we must ensure that the system can continue operation during partial failure in the network and on individual hosts while retaining acceptable synchronization performance as experienced by all users involved.

Traditionally there are two major architectures used to host virtual worlds: client- server and peer to peer [2]. The centralized client-server architecture seen to the left in Figure 1.1 gives the service provider exclusive control over the simulation, it has therefore been, and still is, a very popular architecture to host multiplayer games on.

The distributed peer to peer architecture seen to the right in Figure 1.2 on the other hand has mostly been used in military simulations and foremost in controlled local area network (LAN) environments. It has for example been shown in simulations that the IEEE standard protocol for distributed interactive simulations (DIS) [3] would require 375 Mbit per second of network bandwidth to be allocated to each host to support 100 000 users [4], which is not acceptable with respect to today’s network hardware and performance attributes.

In 2004 a new middleware designed specifically for distributed simulations was presented [7]. This middleware is however not intended for interactive simulations, such as virtual environments, but for simulations of highly interactive systems like wireless and mobile networks. Still, it is relevant for this thesis as it has been shown to significantly reduce the communication and synchronization overhead between physical execution units on heterogeneous systems and communication services [7], something that middleware technologies in the application areas of virtual worlds must achieve.

1.3 Hypothesis

Choosing a peer-to-peer architecture instead of a centralized architecture for a distributed system can, according to Androtsellis-Theotokis et al. [13], be argued on the former’s ability to function, scale, and self-organize in the presence of highly transient node populations, all of which are very desirable abilities for a distributed virtual world.

To decide what type of peer-to-peer architecture is required a look at what it is that needs to be distributed and what is the expected result of the distribution will suffice.

Androtsellis-Theotokis and Spinellis [9] present five of categories in which peer-to-peer applications can be classified.

• Communication and collaboration

• Distributed computation

• Internet service support

3

(10)

Hypothesis

Figure 2.2: Illustration of the way in which various design features affect the main characteristics of peer-to-peer content distribution systems. [13]

• Database systems

• Content distribution

Out of these five categories, distributed computation and content distribution, are the most closely related candidates for classifying a peer-to-peer application for distributed virtual worlds involving massive multiplayer online games (MMOG). But on a closer look the computation in a virtual world is not sufficiently demanding or time consuming for there to be any need or anything to be gained from splitting some computation task and executing it on several machines, which is the definition of distributed computation [13]. The most demanding task at hand, regarding MMOG, is providing reliable access to the content of the world - the world state [9]. Therefore it is the content of the virtual world that will be distributed and the peer-to-peer architecture can be classified as a content distribution architecture [13].

From the characteristics in Figure 1.2 the most important functional requirements for a virtual environment emphasizing the application domain of MMOG would be scalability, performance, resource management and routing. Scalability and performance both depend on distributed object location and routing, content caching, replication and migration, while resource management and routing depend on semantic grouping. It would therefore be rational to assume that a peer-to-peer architecture for distributing world state information of an MMOG should focus on those design features. The hypothesis therefore stands as follows.

By distributing world state information in a structured content distribution peer-to- peer network and keeping the information synchronized using a content aware synchronization algorithm; it will be possible to reduce the required synchronization network traffic and thereby allowing more concurrently connected users while retaining the reliability of the system in terms of latency and responsiveness with the occurrence of network failures.

4

(11)

1.4 Approach and contribution

In this thesis, assertion of the hypothesis will be addressed in terms of three layers where each layer has a quantitative quality that can be calculated by means of an associated formula and measurements gathered in real time. On the top is the cognitive layer with responsiveness as the quality. In the middle is the information layer with capacity as the quality and at the bottom the physical layer with the availability quality. The reason for calculating qualities on more than one layer is to get a deeper understanding of how the system performs. If a quality measurement would be calculated at only one point in the system or for the system as a whole, it would be harder to determine what properties affected the quality and how different parts of the system performed.

The formulas for calculating the qualities are influenced by the work of Fox and Patterson [14]. In their work, Fox and Patterson argue that the software industry needs to focus more on minimizing recovery time from inevitable failures than on trying to prevent the failures from ever occurring. Their arguments have several valid points;

firstly it can be justified quantitatively as the formula for measuring availability (Equation 1.1) has a synergetic relationship between improvements to mean time to failure and mean time to recovery. A tenfold decrease to the mean time to recovery is as valuable as a tenfold increase to the mean time to failure. Secondly a short recovery time might serve as to completely cover up the failure, giving the user the impression of a very long MTTF. The other two qualities, capacity and responsiveness, can be calculated with very similar formulas and should be expected to represent similar qualitative properties regarding user experience of some system’s behavior as they are calculated from measurements that have the same relationships as those in the availability formula.

Starting from the bottom, the physical layer corresponds to the availability of the network architecture and is measured by the mean time to failure (MTTF) and mean time to recovery (MTTR).

MTTR MTTF

MTTF +

Equation 1.1:

Availability

The information layer deals with the capacity of the synchronization algorithm in real time by means of collecting data on the mean time to distribution (MTTD) and the mean time to synchronization (MTTS).

MTTS MTTD

MTTD +

Equation 1.2:

Capacity

Finally the cognitive layer deals with the responsiveness of the system, which aims to be an indication of how a user would perceive the system, by measuring the mean time to query (MTTQ) and the mean time to collect (MTTC), which is the mean time to when the requested information is collected.

MTTC MTTQ

MTTQ +

Equation 1.3:

Responsiveness

The three layers are interconnected and the quality sustained at one layer will affect the others. For example the availability of the network will affect how fast the information layer is able to synchronize which will affect the responsiveness of the cognitive layer.

The fact that the layers are interconnected and affects each other is also what will be exploited in this thesis. By combining data from the information layer with the routing in the physical layer a new synchronization technique can be attained. This technique or 5

(12)

Outline

algorithm is what will be called content aware in this thesis, meaning that it is able to act on the content of the synchronization data in real time.

To test the hypothesis a prototype system will be developed. This system will use peer to peer technologies to form a network where it will distribute a virtual environment and provide the possibility for user clients to access and update the state of the world. As there will exist replicated data in the network, synchronization will be a vital part of the program in order to hold the network in a consistent state. The system will be deployed in a laboratory network environment where it will be experimented upon to gather measurements on scalability, network latency and tolerance to network failure. Further information about the experiments and their outcome is presented in chapter five.

1.5 Outline

In chapter two, important properties of distributed worlds will be discussed and brought into perspective of the problem statement. Chapter tree will then continue to describe possible ways to address the problem of providing reliable access to distributed world state information. Chapter four is an overview of the prototype and the developed synchronization protocol whereas the following fifth chapter describes the experimentations as well as the results gathered from them. The sixth and last chapter contains the conclusion, the summary and a description of future and related work.

6

(13)

Chapter 2 Distributed virtual worlds

Distributing a virtual environment requires some thought at how the environment is to be represented, partitioned and communicated. The representation will typically consist of elements, objects that exist in the virtual environment but are more or less analogical to real world things such as terrain, vehicles, trees or actors. How the environment is represented has little value in itself. It will, however, affect how efficiently it can be partitioned and rendered. How the representation is partitioned can affect how the distribution performs under certain conditions and should therefore be well considered.

Assuming that we are able to partition the representation of the virtual environment and store it at different synchronized locations it has to be decided how to automatically identify such locations, how they should be connected and to what degree replication or redundancy should exist in the resulting network. The reason to use distribution in the first place was to increase scalability by spreading the host load of concurrent users over multiple and distributed host machines. But the more hosts that are involved in the distribution and the more redundancy that exists regarding the synchronized partitions the more problems will likely occur during networked synchronization which could result in perceivable inconsistencies to the users.

In this chapter follows a study of important aspects regarding the actual data or information in a distributed virtual environment. Section 2.1 gives an introduction to the concept of virtual worlds and studies the information that can be expected to exist in them and section 2.2 goes through how to represent that information. Section 2.3 discusses information partitioning, the importance there of and how it will affect the performance of the distribution. Then section 2.4 goes on to describe how the partitioned data should be spread out, the actual distribution and required synchronization.

2.1 Information

This section will study what kind of information constitutes a virtual world, such as a massive multiplayer online game (MMOG). Knowing this is important when we decide how to represent this information which will be covered in the next section. To properly analyze the information in virtual worlds we must first have a clear understanding of what a virtual world is and, therefore, this section will start with an introduction to the concept as such before proceeding to examine the information. It should be noted that every virtual world will of course not consist of the same information types; it is however plausible to assume that some types are common and it is those that will be discussed in this section.

According to Singhal and Zyda [8] virtual worlds can be defined by five features:

• Space - All participants have the illusion of being located in the same place, such as in the same room. That shared place represents the common location within which other interactions can take place. The place may be real or fictional. The shared space must present the same characteristics to all participants. For example, all participants should get the same sense of temperature and weather, as well as the acoustics. Though it needs not be presented graphically, the most effective

7

(14)

Information

networked virtual environments provide an immersive three-dimensional representation of the shared place.

• Presence - When entering the shared place, each participant takes on a virtual persona or agent, called an avatar, which includes a graphical representation. Upon entering the networked virtual environment, each participant can see the other avatars that are located in the shared place, and those users can see the new participant's avatar. Similarly, when a participant leaves the networked virtual environment, other participants should see the avatar depart.

• Time - Participants should be able to see each other's behavior as it occurs. In other words, the networked virtual environment should enable real-time interaction to occur.

• Interaction - Though visualization forms the basis for an effective networked virtual environment, most networked virtual environments also enable some communication among the participants; by gesture, by typed text, or by voice. This communication adds a necessary sense of realism to any simulated environment. It is a fundamental component of engineering or training systems.

• Sharing - The aforementioned elements effectively provide a high-quality video conferencing system. However, the true power of networked virtual environments derives from users' ability to interact realistically not only with each other but also with the virtual environment itself. Users should be able to pick up, move and manipulate items that exist in the environment, and they should be able to give items to other participants. Users might even be empowered to manipulate the environment by destroying existing parts of it or by building new ones.

Given this definition a conceptual model of a virtual world can be constructed. This model would contain the following.

• Agent - The representation of a user or a computer controlled avatar.

• Object - A general immobile and non-interactive element.

• Tool - An interactive element that can be used by the agents.

• Item - An element that can be picked up and maybe held by the agents.

• Terrain - The landscape or indoor environment that the other elements exist in.

Information about these elements is what makes up the virtual world from the users point of view. From this simplistic model it could be seen as if the virtual world consists only of visual elements, this would not create a very realistic experience. Sounds, lights and other effects also exist in the world and the element types described above should therefore be considered to include this information. The information needed to render these elements is in fact the same for all the element types, namely:

• Appearance - Information about how the client is to render the element to the user.

• Location - Information about where to place the element in the virtual world.

• Direction - Vector information about which way an element is facing or heading.

8

(15)

Element

Agent Interactive Object

Event Terrain

Item Tool

Element Collection

-Appearance -Location -Direction

Element

-Actors

Element Collection

-Duration -Actions

Event

Figure 2.2: The left figure shows an inheritance structure for a number of world elements. The right figure shows an example of how the property inheritance works for the event element type.

With those three properties it would be possible to render elements in a, for example three dimensional, virtual world. But without one more dimension, a notion of time, it would very difficult to facilitate for movement, communication and interaction. To be able to do this there needs to be one more type of information, something that presents what happened, what it affected and when it happened, simply put an event. The event information consists of:

• Actors - A collection of elements involved in the event.

• Action - Describes what happened.

• Location - Where the event fired.

• Duration - When the event fired and how long it lasted.

2.2 Representation

In the previous section it was studied what types of information can be expected to exist in a virtual environment, when this is known a representation of that information can be created which is what this section will cover. But how the information is represented will not only be affected by the type of the information. How the information is going to be created, accessed and manipulated also has a big influence. For the purpose of this thesis the information representation will therefore be optimized for distribution and synchronization and not rendering activities. The internal structures of the objects will therefore be structured to enable easy access to object parameters and the ability to update individual parameters. This ability is important when it comes to limiting network traffic; being forced to send the whole object when only one parameter is updated would of course be very inefficient.

As described in the previous section a virtual world consists of a number of elements and events which are all built up from the same basic information types. As many of the elements share the some properties but also contains type specific information and routines an inheritance structure as shown to the left in Figure 2.2 will

9

(16)

Partitioning

simplify representation by eliminating redundant information and retaining high object cohesion. A conceptual example is shown in to the right in Figure 2.2 where the event object inherits the properties of an element and an element collection and adds its own duration and actions properties.

The element collection class is used to instantiate collections of elements for various purposes. Which elements to include in the collection can be chosen arbitrarily based on element type, location or any other parameter. The event element-type inherits this ability and uses it to represent the actors of the event. The element collection type will also be used for specifying the partitions.

2.3 Partitioning

As the representation is going to be distributed it has to be partitioned so that different hosts in the distribution network can be responsible for different parts of the virtual environment. The partitioning might not be physical, all data could be present on all hosts when the distribution starts, but rather set by preconfigured responsibilities of the host. The partitions will be based upon the concept of element collections that was introduced in the previous section and each partition will consist of one or more of these collections. An element collection can contain elements of any type and which elements to include in the collection can be chosen arbitrarily based on element type, location or any other parameter. The parameter used will affect how the distribution behaves; care should therefore be taken when choosing the defining parameter.

Using the geographical areas inside the virtual world to define collections of elements, so that one collection contains all entities residing in a certain area, seems the most straight forward approach. The technique is called geometrical partitioning and is used in other application domains where geometrical proximity is highly relevant to the efficiency of the application [15]. This is very much the case in the application domain of massive multiplayer online games (MMOG) as entities that are in the same area of the virtual world are more likely to interact and require information from one another.

Having these entities in the same collection would reduce the amount of communication required between machines. Unfortunately the same problems that other applications using this technique face [15] would also apply to a virtual world.

As the entities move around in the virtual world they can also move to areas contained in other collections and partitions. The entity should then be transferred from the old collection to the one constituting the new geometric proximity. Obviously, problems might arise if many entities move to the same area; the collection containing this area and thereby machine responsible for this partition could potentially be overloaded. To counter this the system must be made resource aware; that is, it must be aware of the resources available at the machine and be able to adapt the partitioning based on that.

Another approach is to base the partitioning on the type of information in the representation, so that for example one partition contains all terrain information, one for all vehicle positions and so on. One benefit of this solution is that it presents a very clear definition of a partition and there would never be any need to transfer responsibility of objects between machines in the distribution. But of course there are considerations with this solution too. Some types of information are inevitably going to be more desired than others and, consequently, pose a load balancing problem for the overall system and its responsiveness. This approach also suffers from another flaw in that it is not adapted for interaction between entities and would therefore require excessive communication between machines responsible for different partitions.

Of the two options discussed here the geometrical proximity partitioning provides the greatest advantages. The problems with overloading will have to be countered by

10

(17)

anticipating worst case scenarios and providing adequate redundancy on high demand areas to ease the load on the responsible machines.

2.4 Distribution

With an application strategy for information partitioning at hand, it is time to distribute it. Firstly it should be established why we are trying to distribute the virtual world in the first place. For a large scale virtual world scalability will be a problem in a centralized solution. The general idea is therefore to increase scalability by distribution and spreading the load to several hosts. Burns and Wellings [1] give a few examples of potential advantages of distributed systems in general:

• Improved performance through exploitation of parallelism.

• Increased availability and reliability through the exploitation of redundancy.

• Dispersion of computation power to the locations in which it is used.

• The facility for incremental growth through the addition or enhancement of processors and communication links.

Point one, two and four are the most relevant for a distributed virtual world and this thesis will focus on the properties of point two, availability and reliability. But the advantages do not come without disadvantages. In many types of distributed systems coordination and synchronization is a challenge to the developers [1], and given the nature of distributed virtual worlds it is very much so in the case of massive multiplayer online games (MMOG). In a distributed virtual world every action and every change to information on one host may be of interest to any number of other hosts [12]. In a large- scale distributed virtual world with many simulated, and perhaps user controlled entities, the number of information updates will lead to a rise in communication overhead on the network in the order of n², where n is the number of hosts [12].

11

(18)

Distribution

12

(19)

Chapter 3 Reliable world state information access

While the previous chapter was about the information stored in the distribution network, this chapter will deal with how to provide reliable access to that information. To provide reliable access we must know what reliability means. Randell et al. defines the reliability of a system to be a measure of the success with which the system confirms to some authoritative specification of its behavior [6]. This specification should be complete, consistent, comprehensible and unambiguous [1] but the main measurement of a virtual environment’s performance is how the users perceive it. It must therefore be determined what it is that affects the users’ perceived quality of the world. For this purpose the three qualities presented in Section 1.4, responsiveness, capacity and availability will be used.

Responsiveness is a measurement of the systems ability to respond to user requests in a timely manner and greatly affects the users trying to interact with the world. The second criteria capacity, measures how often the system is able to send updated information to the users, e.g. the position of some vehicle. Low update frequency will result in the clients having to compensate when the update finally comes which in turn results in the user experiencing delays in an assumable real time phenomenon.

In an effort to improve responsiveness and capacity and thereby hopefully also improving the users’ perception of the world, this chapter will present a content aware synchronization algorithm and an area of interest multicast grouping mechanism. Both of these techniques are highly dependant on reliable communication between hosts in the network to perform well. Fortunately, well constructed peer-to-peer networks can provide for the ability of real time adaptation regarding routing around faulty links and nodes. Communication will therefore be carried out over such a peer-to-peer network.

Section 3.1 through 3.3 delves into synchronization and in particular how synchronization efficiency can be improved by making the algorithm content aware and able to prioritize. Section 3.4 through 3.6 then describes how reliability and performance can be further improved by exploiting peer to peer properties such as multicast grouping and dynamic routing. Section 3.7 further specifies the synchronization protocol.

3.1 Content aware synchronization

In a distribution with replicated information some kind of synchronization has to take place. The purpose of this synchronization is of course to keep the replicated information consistent on all locations in the distribution. But how important and time critical the synchronization is will depend on the nature of the distribution, the information itself, and how it is used. In the case of a distributed virtual environment the information tends to be time critical and very small latencies will affect how the users perceive the virtual environment they are immersed in. The strict time requirements coupled with other properties of virtual environments, such as the desire for scalability and “massive” amounts of users, presents some synchronization problems probably not seen in any other area.

In an effort to improve synchronization performance this section will present a synchronization algorithm that is able to distinguish between different types of synchronization data. This will create new opportunities for optimization and network

13

(20)

Information prioritization and categorization

Figure 3.1: A typical peer-to-peer network where a single information stream flows from one sender to the subscribing multicast group.

traffic reduction not available in algorithms that purely rely on the underlying network for routing. The version of the algorithm presented in this thesis will however not be a complete synchronization solution but a proof of concept prototype. Section 3.2 describes how the algorithm will use its content awareness to prioritize and categorize synchronization information and in what way this in itself could affect performance.

Section 3.2 goes on to describe how the information prioritizing can be used for the routing of information in the distribution network.

3.2 Information prioritization and categorization

This section will describe how the content aware synchronization algorithm will be able to prioritize and categorize information. This will be done to improve the efficiency in two areas of the algorithm, in the queuing of synchronization information which will be explained in this section and in the information routing which will be explained in the next section.

All information that makes up the virtual world is represented in the manner which was explained in the previous chapter. This representation makes it possible to distinguish between different types of information which in turn makes it possible for the synchronization algorithm to place the information in differently prioritized categories. How the information is categorized and how the categories are prioritized has the potential to affect both the performance of the algorithm and how the users perceive the virtual world. In the prototype the categorization is based upon statically defined degrees of importance of the information. Pieces of information deemed more crucial to the users’ experience will be higher prioritized than those with less impact on the same. For example, the position of agents could be highly prioritized while other non visible properties of the agents such as strength, agility or even their names might have lower priority. As stated, more sophisticated prioritization would be possible. This could be achieved by not only including the nature of the content of the information but also the context of the event that caused the update of the information, where it occurred and how many users will be affected by it or perceive it.

14

(21)

3.3 Information routing

By combining the knowledge of the prioritization and categorization described in the previous section with knowledge of the network topology and configuration of the distribution the content aware synchronization algorithm will try to improve routing efficiency. This section will describe how this is done.

When an information update is sent from a node, the node is not required to specify all other nodes to which the update is intended for. The sending node does in fact not need to specify at all who should receive the message. What it has to do is send the update to nodes that has registered as interested in the information category that the update is concerned with. The nodes that have registered as interested in the category might not them selves be interested in the information but have neighboring nodes that is. The message will continue to be routed through the network in this manner until it has reached all nodes that have registered their interest in the information category at the sending node or at any other node which has a possible route to the sending node. There might be cases when there is more than one possible route to the node which has registered interest in an information category. The update will in this case be sent on all possible routes to the destination node. If it reaches the destination or any other node on the way to the destination more than once the latter message will be discarded. A possible extension the algorithm which is not present in the current implementation is to note which route lead to a successful deliver and use this route for the next send operation. In cases where there are many routes to a destination node, such an extension would reduce network traffic at the cost of robustness and possible performance as with the current implementation the message will always reach the destination on the fastest route no matter which route that might be. This routing technique is tightly coupled with the multicast grouping that will be described in section 3.5 where the multicast groups correspond to information categories.

3.4 Exploiting peer to peer network properties

What network architecture to use as a base for communication in a distribution could greatly affect the performance and reliability of the system and is a key challenge when building a system that is to support maybe thousands of users or players [2]. In this thesis the peer-to-peer architecture [10] will be used for the inherent properties that will be discussed in this section. There are other architectures that might or might not be more suitable for this application, deciding this is however not the purpose of this thesis.

There are foremost two properties of peer-to-peer networks that will be examined in this section. First it’s the ability to form application level multicast groups which will be used for delivering information updates; this is studied in section 3.5. Second is the ability to adapt to changes in the network infrastructure and thereby coping with network failures which is covered in subsection 3.6

3.5 Multicast grouping

15 This section will describe how the synchronization algorithm will use application level multicast groups to spread the synchronization information. Multicast is a transmission method in which one source node communicates with one or more receivers with a single transmission [13]. This is in contrast to a broadcast transmission which has to be sent as many times as there are receivers. By only sending the transmission once network traffic is reduced but more responsibility is handed to the routing nodes on the way to the destination nodes. Multicast should be supported on the Internet Protocol (IP)

(22)

Network failure tolerance

Sim: A

Sim: B

Sim: A

Sim: A,B Sim: A,B

Sim: B User

User User User

User

User User

User User User User

User User User

Figure 3.2: This figure illustrates how a peer-to-peer network formed by the prototype system might look like from a conceptual point of view. The user nodes are tied together in multicast groups which have subscribed to updates from the simulation nodes.

level but its usability is limited by network hardware that does not effectively handle multicast packets. The default behavior of a layer two network switch is for example to forward all multicast traffic to all connected ports on the destination network. This defeats the purpose of the switch which is to limit the traffic to the ports that need to receive the data. Multicast is based on the concept of a group. An arbitrary group of receivers expresses an interest in receiving a particular data stream. This group does not have any physical or geographical boundaries—the hosts can be located anywhere on the Internet [17].

The prototype developed in this thesis does not use IP level multicast but has moved the routing and group management to the application itself, this is called application level multicast. The groups are formed based upon the interests and responsibilities of the nodes participating in the distribution. The simulation nodes, that is the nodes responsible for providing the content and running the computer controlled avatars, may have both interests and responsibilities, whereas the user nodes have only interests. A typical responsibility would be to control a specific type of creature in a specific area and a typical interest would be positions of specific creatures in a specific area. A network that has been formed employing this technique can be seen in Figure 3.2. Here groups of user nodes are connected to one or more simulation nodes which servers the content that the user nodes are interested in. This technique will be used in the prototype and is further explained in Section 4.2.

3.6 Network failure tolerance

16 Thanks to the peer-to-peer network technology used in the prototype the system will have a certain degree of network failure tolerance and the ability to recover from failed links. The tolerance to network failure comes from the possibility of redundant routes between nodes as shown if Figure 3.3. When redundant routes exist, messages will be sent on all possible routes while still avoiding loops. This means that when one route

(23)

A

B

D

C

E F

G

Figure 3.3: Routing in a partially operational network. Red connections are non-functional.

fails there is still the possibility of the message coming through on another. This method generates some extra network traffic, but it is fast and ensures that messages always get through in the fastest manner possible as long as there is one intact route to the destination node. A model of this can be seen in Figure 3.3. As this simple but robust routing technique is used the system will also be able to quickly recover when a lost link is reestablished. When the link is down the system will continue to try to send over it and therefore recover it as soon as it is reestablished.

3.7 Synchronization protocol specification

As a whole the synchronization protocol is built to maximize the throughput of messages between nodes in a reliable manner. As a result of this there are never any responses to messages that do not explicitly require it. Distributed objects are never in a locked state and rollbacks will only occur when an inconsistency is detected between nodes. Although a rather simplistic one, the protocol can therefore be view an optimistic synchronization protocol [19].

The synchronization protocol is based around two types of messages, information update messages and information request messages. The information update messages originate from the simulation nodes and are the primary messages used to keep the world state information on the simulation and user nodes synchronized. An information update message consists of three properties: a sequence number, an object id and a list of synchronization message parameters. A synchronization message parameter in turn is a container of two things: a name and a value. The name and value is used on the receiving node to dynamically invoke the set method of the correct parameter on the object that is to be updated. Which object to update is specified by the object id in the message. The sequence number is also an important part of the information update message. Without it, it would be impossible to know which update was the last one and therefore the correct one as messages can travel different routes and therefore may not arrive in the order in which they were sent. The node that receives an information update therefore compares the sequence number of the incoming message with the sequence number of the locally stored world object. If the incoming sequence number is larger or equal to that of the local object the local object and the local sequence of the object is 17

(24)

Synchronization protocol specification

updated. Otherwise a reply is sent back with the correct sequence and current state of the object to the original sender.

The information request messages are sent by user nodes to request certain information that is not directly pushed to them by the simulation nodes. An information request message consists of two properties: an object id and a list of the names of the properties of the object the user node requests. Upon receiving an information request message a simulation node will construct and respond with an information update message containing the information requested.

18

(25)

Chapter 4 Prototype implementation

To be able to assert the hypothesis and test the associated techniques described in the previous chapters a prototype had to be developed. This prototype should of course implement the mentioned techniques and have the ability to measure their performance.

Ideally the prototype would consist of a distributed interactive virtual environment using the content aware synchronization algorithm and the interest multicast grouping for communication. The problem is that developing a virtual environment is a great undertaking in and of itself and some features will therefore be considered out of scope for this prototype.

What the prototype does implement is a peer-to-peer communication layer able to form application level multicast groups based on the interests of the peers and a synchronization algorithm using the content aware prioritization techniques able to keep data synchronized over the network. To be able to determine the combined performance of the synchronization algorithm and the peer to peer network the prototype will also be able the gather relevant measurements.

Section 4.1 describes what the prototype is and what it is able to do while section 4.2 shows how it was designed and built. The last section then shows how the network architecture works.

4.1 Capabilities

The prototype will have capabilities that can be divided into two groups, distribution and measuring, both equally important for the purpose of this software. The distribution group includes capabilities that have to do with distribution of the virtual environment and the associated mechanisms such as synchronization of the distributed information.

The measuring group on the other hand includes capabilities only used for gathering measurements of the performance of the distribution capabilities. This section will provide descriptions of the capabilities from both groups in the corresponding subsections.

As it is stated in the introduction to this chapter this prototype is not in any way a complete distributed virtual environment. Capabilities expected of distributed virtual environments that are not found in this section should therefore not be thought to actually exist in the prototype.

Synchronization capabilities - The prototype is able to keep world state information synchronized on a number of hosts by implementing the synchronization protocol specified in the previous chapter. Each host in the distribution network is configured to be responsible for certain world entities, be connected to certain other hosts, and have certain interests by the use configuration files. The interests a host has will affect which updates it receives as specified in the protocol specification. This is something that should be determined dynamically in a real world distributed virtual world, but for the purpose of this prototype, which is to test the performance of the system in certain key scenarios, it is simpler and more effective to be able to specify the hosts’ interests, responsibilities and connections before execution.

19

(26)

Software design

Measuring capabilities - The actual prototype itself does in fact not perform any measuring but instead time stamps and logs all events such as sending and receiving of networks messages to a log file. To analyze this event log a log program has been developed that is able to extract the required measurements, do the calculations for various performance properties that will be described in section 5 and visualize the routing of network messages as seen in Figure 5.1.

4.2 Software design

The software design of the prototype will be explained in this section. This will be done with the help of UML diagrams, interaction diagrams and general explanatory text. The section will begin with a high-level description of how the prototype is built and how the interaction with other peers in the network will function, it will then work its way down to the lower levels and explain how specific parts of the synchronization algorithm and other mechanisms are implemented.

The prototype is built in three layers; visualization, logic and network. The visualization layer displays output from the system and handles the startup of the underlying layers. Contrary to what the name might imply, it does not actually visualize the virtual world as this is not needed for the purpose of the prototype. It is however in this layer that a future visualization solution such as a 3D engine would be implemented.

Below the visualization layer lies the logic layer. This layer holds the data that the virtual world consists of, it is responsible for keeping it synchronized with other nodes and also runs the computer controlled avatars if there is any. Three classes make up the core functionality of the logic layer; the WorldModel, the SynchronizationHandler and the Simulator. The WorldModel is used to access and modify virtual world data. It holds a number of world objects which represent individual elements of the virtual world and can be derived into specific types such as an item, a character or any other element of a virtual world. The “Simulator” is basically a loop which runs in its own thread and simulates WorldObjects that are computer controlled and within the responsibility of the node. The SynchronizationHandler, as the name implies, handles synchronization.

Whenever a WorldObject is updated the SynchronizationHandler will be notified and if the updated requires other nodes to be synchronized it will try to take care of this.

Incoming information updates from other nodes also ends up in the SynchronizationHandler which then uses the WorldModel to update the affected WorldObjects.

The lowest layer in the prototype is the network layer. This aptly named layer takes care of the network communication going to and from the node. Its tasks include construction of network messages, sending, receiving and routing of messages. In this layer the core functionality is made up of three classes, the Peer, the Router and the Transporter class. The Peer class functions as an interface from the logic layer and have public methods for sending messages to other nodes. It also fires off events when different types of messages are received that can be intercepted by other layers, like ResponseReceived and InformationUpdatedReceived. The Router class handles address lookup and routing of outgoing and passing-through messages. For this purpose it holds a list of the currently known addresses of all neighboring nodes. To send messages the Router uses the Transporter class. The Transporter transforms the internal message format to a binary network message format and sends the data to the specified address.

Incoming network messages are also taken care of by the Transporter which constructs an internal message from the binary network message and then notifies the Router.

20

(27)

21 Figure 4.1: Static structure diagram.

LogicPeer Simulator NetworkAbstraction

NetworkMessage

ConfigurationManager ConfigurationData Character

Item MobileObjectWorldModelWorldObject InformationRequestMessage

SynchronizationHander SynchronizationMessage

«delegate» SynchronizationDataReceivedEventHandler «delegate» InformationRequestReceivedEventHander «delegate» WorldObjectUpdatedEventHandler *

«uses»Logic Peer Router Transporter

Message Payload

InformationPayload InterestAccouncementPayload InterestUpdatePayload ResponsePayload

Network«delegate» ResponseReceivedEventHandler «delegate» InformationUpdateReceivedEventHandler «delegate» MessageReceivedEventHandler «delegate» ReceivingMessageEventHandler

Visualisation Main SynchronizationMessageParameter MessageType

(28)

Software design

Figure 4.2: Dynamic structure of the send operation.

22

(29)

Dynamic structure - Figure 4.2 shows the sequence of method calls that occur when an information update message needs to be sent from a node. To explain how the prototype works, each step will be explained below.

1. The simulator, running in its own thread, calls the WorldModel’s Simulate method on every simulation tic.

2. The WorldModel loops through all WorldObjects it contains checking against the ConfigurationManager if it is responsible for simulation the object. If it is, it calls the WorldObject’s Simulate method.

3. The WorldObject runs its simulation algorithm, which could be to move in a certain direction. When it has completed this, it checks if any properties, such as position, have changed. If that is the case it will fire off an event through the WorldObjectUpdatedEventHander. The event contains the identification number of the affected WorldObject and a list of SynchronizationParameters which describe what changes occurred to which properties of the object.

4. The SynchronizationHandler is listening on this event handler and will therefore be notified.

5. The SynchronizationHandler starts with creating a SynchronizationMessage containing the current sequence number of the object, its identification number and the list of SynchronizationParameters. It continues to determine what interests the update will affect. This is a crucial part of the process as this will affect which nodes the update is sent to. When the interests have been determined the NetworkAbstraction is called with the message and the interests as parameter.

6. The NetworkAbstraction will construct an InformationPayload which works as a container for the SynchronizationMessage and the interest list.

7. It will then call Deliver on the Peer in the lower network layer with the payload and the type of the message as parameters.

8. In the network layer the Peer consults its interest list to find which nodes are interested in the specified interest and constructs a recipient list of the ids of these nodes. A Message which contains the higher level SynchronizationMessage and the recipient list is then created and the Send method can be called. The Send method in turn calls the Route method on the Router with the message as parameter.

9. The Router uses a simple optimistic routing algorithm. It will look in a list of known nodes which it has constructed from incoming messages to find the internet protocol address and port of the nodes in the recipient list. For every known node the Router calls its own method Deliver with the message and address as parameters. The Deliver method just calls Send on the Transporter with the same parameters.

10. The Transporter serializes the Message to a binary format and sends the message to the specified recipient as a UDP datagram.

23

(30)

Software design

Figure 4.3: Dynamic structure of the receive operation.

24

(31)

Here follows the description of what happens on the receiving end, as seen in Figure 4.3.

1. The listening loop in the Transporter asynchronously begin to receive data and when completed call ReceiveFromCompleted. The message is here deserialized back into a Message and the ReceivingMessageEventHandler event is fired.

2. The event hander calls all who is listening.

3. The router which is listening to the event will be notified with the newly deserialized Message and sender internet protocol address as parameters. The address is stored in the list of known nodes along with the node identification number for later retrieval. The Router then checks if this node is in the list of recipients, if that is the case the MessageReceivedEventHandler event is fired with the message as parameter. If there are more recipients in the list after this the router will call its own Route method with the message as parameter to route the message on.

5. The Peer is listening to the event and receives a call to its MessageReceived method with the message as parameter. It begins by checking it the message has already been received from another node in which case it is ignored. It then proceeds to check what type of message it has received; in this case it is an information message. It could also have been a response message for example. As nothing special needs to be done at this stage to an information message the ExecutePayload method is called with the message’s payload and identification number as parameters. Depending on the type of payload the method will of course do different things; in this case it will just fire the InformationUpdateReceivedEventHandler event.

7. In the NetworkAbstraction the InformationReceived method is called with the payload and message identification number as parameters. This method investigates what type of payload it is and proceeds accordingly. In this case it was an information update, also known as a synchronization message so the SynchronozationDataReveivedEventHandler event will be fired.

9. The SynchronizationHandler was notified of the incoming synchronization message by a call to SynchronizationDataReceived with the message and message identification number as parameters. The identification number is only used if a response is required. This method is rather complicated as it must determine if the incoming synchronization data is in the correct sequence. This is done by checking the local sequence number of the object with the sequence of the incoming update.

If the sequence is the same or the incoming one is larger the update is applied to the object. If the incoming update’s sequence number is smaller that the local object’s an out-of-order response is sent to the sender with the local object’s state included. Assuming the incoming update is in the correct sequence the WorldObject’s Update method is called with the list of SynchronizationMessageParameters as parameter. The WorldObject applies the updates by dynamically invoking the properties specified in the list with the corresponding parameter values.

25 Network architecture - In this section the network architecture used for communication in the prototype will be explained. The architecture uses peer to peer technologies but whether it is truly a peer-to-peer architecture can be argued. The strictest definition of peer-to-peer refers only to totally distributed systems, where every node share the exact same set of functionality and tasks [13]. According to this definition the prototype is not a peer-to-peer system as the nodes have different tasks and even different functionalities when comparing user nodes to simulation nodes. But

RELIABLE ACCESS TO SYNCHRONIZED WORLD STATE INFORMATION IN PEER TO PEER NETWORKS