Delay Impact on Gaming Experience, For a First Person Shooter Game.

(1)

Delay Impact on Gaming Experience

For a First Person Shooter Game

Nagarajesh Garapati

Karlskrona, September 2009

Department of Telecommunication Systems School of Engineering

Blekinge Institute of Technology

371 79 Karlskrona, Sweden

(2)

(3)

Abstract

The growing market for online games and their significant share in the internet traffic has allured researchers to investigate the factors affecting the game playing experience of the end user. In this work an investigation has been done on how one-way delay affects the First Person Shooter game there by Quality of Experience of the player. The data that is analyzed was collected at Dream Hack Winter-2007 by Telecommunications research team of Blekinge Institute of Technology, Sweden. It consisted of 57 experiments conducted using Battle- Field2 game in a controlled LAN, with between 2 and 4 players. The network was monitored using Distributed Passive Measurement Infrastructure.

This thesis implemented tools to generate accurate delay plots for each relevant stream in the captured game traffic traces. The delay plots have been ensured by identifying what IP that was disturbed and when the disturbance occurred in the trace, as per the shaper settings. Then the investigation is mainly concentrated on the analysis of one-way delay and how it affects the end-user Quality of Experience. As a part of this work we also looked at the affect of packet delay variation on end user’s perception, delay distribution of the shaper and packet length distribution of the client - server and server - client packets.

Conclusions were made by stating that one-way delay and packet delay variation has considerable impact on end users perception. It is found that the delay of 400 ms is threshold delay value to get a good user response and packet delay variations of 100 ms and 200 ms with variable amounts of delay are detectable by the player.

Keywords

Quality of Experience, Delay analysis, Packet Delay Variation, FPS games, Battle- Field2, Synchronization, Clock drift.

(4)

(5)

Acknowledgements

First I would like to express my sincere appreciation to Dr. Patrik Arlos, for giving me the opportunity to be a part of this interesting research, and for his valuable support and guidance throughout the thesis.

I thank all of my fellow thesis works for their attitude towards knowledge sharing and group work. My sincere thanks to my family members for their love, unlimited support and encouragement.

Nagarajesh Garapati Karlskrona, September 2009

(6)

(7)

Abstract iii

Acknowledgements v

1 Introduction 1

1.1 Related work . . . 2

1.2 Contribution . . . 3

1.3 Outline . . . 3

2 Approach 5 2.1 Experimental Environment . . . 5

2.2 Collection of data . . . 9

2.3 Problem Identification And Solution . . . 9

2.3.1 Packet Identification . . . 10

2.3.2 Clock Synchronization . . . 10

2.3.3 Delay drift . . . 11

3 Results and Analysis 15 3.1 Shaping into the traffic from the Player, Exp: 1 . . . 16

3.2 Shaping into the traffic to the Player (Exp: 16) . . . 18

3.3 Shaping into the traffic to/from the player (Exp: 6) . . . 21

3.4 Experiments with abnormal behaviors . . . 23

3.4.1 Experiment: 2 . . . 23

3.4.2 Experiments: 3, 7, 10, 11, 12, 17, 19, 31 and 52 . . . 24

3.4.3 Experiments: 34 to 39 . . . 24

3.4.4 Experiment: 44 . . . 25

3.4.5 Experiment: 25 . . . 25

3.5 Quality of Experience of the user . . . 26

4 Conclusions and Future Work 37 4.1 Future work . . . 38

A Appendix 41 A.1 Developed Tools . . . 41

A.2 Synchronization and delay calculation . . . 41

A.3 Delay drift adjustment . . . 51

(8)

B Statistics 55

B.1 Delay statistics for unshaped players . . . 55

B.2 Delay statistics for shaped players . . . 62

B.2.1 Statistics of shaping to the player . . . 62

B.2.2 Statistics of shaping from the player . . . 66

C Results (Exp: 1 to 25) 71 C.1 Exp: 1 . . . 71

C.2 Exp: 2 . . . 73

C.3 Exp: 4 . . . 75

C.4 Exp: 5 . . . 77

C.5 Exp: 6 . . . 79

C.6 Exp: 7 . . . 81

C.7 Exp: 8 . . . 83

C.8 Exp: 9 . . . 85

C.9 Exp: 10 . . . 87

C.10 Exp: 11 . . . 89

C.11 Exp: 13 . . . 91

C.12 Exp: 14 . . . 93

C.13 Exp: 15 . . . 95

C.14 Exp: 16 . . . 97

C.15 Exp: 18 . . . 99

C.16 Exp: 19 . . . 101

C.17 Exp: 20 . . . 103

C.18 Exp: 21 . . . 105

C.19 Exp: 22 . . . 107

C.20 Exp: 23 . . . 109

C.21 Exp: 24 . . . 111

C.22 Exp: 25 . . . 113

D Results (Exp: 26 to 57) 115 D.1 Exp: 26 . . . 115

D.2 Exp: 27 . . . 117

D.3 Exp: 28 . . . 119

D.4 Exp: 29 . . . 121

D.5 Exp: 30 . . . 123

D.6 Exp: 31 . . . 125

D.7 Exp: 32 . . . 127

D.8 Exp: 33 . . . 129

D.9 Exp: 40 . . . 131

D.10 Exp: 41 . . . 133

D.11 Exp: 42 . . . 135

D.12 Exp: 43 . . . 137

D.13 Exp: 44 . . . 139

D.14 Exp: 45 . . . 141

D.15 Exp: 46 . . . 143

D.16 Exp: 47 . . . 145

D.17 Exp: 48 . . . 147

D.18 Exp: 49 . . . 149

D.19 Exp: 50 . . . 151

(9)

D.20 Exp: 51 . . . 153

D.21 Exp: 53 . . . 155

D.22 Exp: 54 . . . 157

D.23 Exp: 55 . . . 159

D.24 Exp: 56 . . . 161

D.25 Exp: 57 . . . 163

Bibliography 165

(10)

(11)

2.1 Block diagram of experimental setup . . . 6

2.2 DPMI trace formate . . . 9

2.3 Clock synchronization. . . 11

2.4 Delay traces with drift. . . 12

2.5 Adjusted delay traces. . . 13

3.1 Delay traces for unshaped players. . . 15

3.2 Shaping from client to server. . . 17

3.3 Delay distribution of shaper in Exp: 1. . . 17

3.4 Packet length distribution in Exp: 1 . . . 18

3.5 Shaping from server to client. . . 19

3.7 Packet length distributions in Exp: 16 . . . 20

3.8 Shaping from/to the client. . . 21

3.10 Packet length distributions in Exp:6 . . . 22

3.11 Packet length distributions in Exp: 2 . . . 23

3.12 Packet length trace for player input and server output (Exp: 2). . . 24

3.13 Delay plot of Exp: 36 (player 3 to server). . . 25

3.15 Delay trace with the silent period (Exp: 25). . . 26

3.16 Mean QoE vs. Delay . . . 28

3.17 Mean QoE vs. PDV (with variable amount of delay.) . . . 29

3.18 Mean QoE vs. PDV (with 200 ms of delay.) . . . 30

C.1 Delay plots for Exp: 1. . . 71

C.2 Delay distribution of shaper in Exp: 1. . . 72

C.3 Packet length distribution in Exp: 1. . . 72

(12)

(13)

D.1 Delay plots for Exp: 26. . . 115

D.2 Delay distribution of shaper in Exp: 26. . . 116

D.3 Packet length distribution in Exp: 26. . . 116

(14)

(15)

2.1 shaper settings for the experiments 1 to 25 . . . 7

2.2 shaper settings for the experiments 26 to 57 . . . 8

3.1 QoE vs. Delay . . . 28

3.2 QoE vs. PDV (with variable amount of delay.) . . . 29

3.3 QoE vs. PDV (with 200 ms delay.) . . . 30

3.4 delay statistics in the shaped region for the experiments with shaping to and from the player. . . 32

3.5 delay statistics in the shaped region for the experiments with shaping to the player. . . 33

3.6 delay statistics in the shaped region for the experiments with shaping to the player. . . 34

3.7 delay statistics in the shaped region for the experiments with shaping from the player. . . 35

3.8 delay statistics in the shaped region for the experiments with shaping from the player. . . 36

B.1 statistics of unshaped players in shaped region (Exp: 1-14) . . . . 56

B.2 statistics of unshaped players in shaped region (Exp: 15-32) . . . . 57

B.3 statistics of unshaped players in unshaped region (Exp: 1-14) . . . 58

B.4 statistics of unshaped players in unshaped region (Exp: 15-32) . . 59

B.5 statistics of unshaped players for entire region (Exp: 1-14) . . . 60

B.6 statistics of unshaped players for entire region (Exp: 15-32) . . . . 61

B.7 Delay statistics of shaped players from the experiments with shaping to the player (unshaped region)-1 . . . 62

B.8 Delay statistics of shaped players from the experiments with shaping to the player (unshaped region)-2 . . . 63

B.9 Delay statistics of shaped players from the experiments with shaping to the player (entire region)-1 . . . 64

B.10 Delay statistics of shaped players from the experiments with shaping to the player (entire region)-2 . . . 65

B.11 Delay statistics of shaped players for the experiments with shaping from the player (unshaped region)-1 . . . 67

B.12 Delay statistics of shaped players for the experiments with shaping from the player (unshaped region)-2 . . . 68

B.13 Delay statistics of shaped players for the experiments with shaping from the player (entire region)-1 . . . 69

(16)

B.14 Delay statistics of shaped players for the experiments with shaping

from the player (entire region)-2 . . . 70

C.1 Payer responses for Exp: 1. . . 72

C.8 Player response for Exp: 9. . . 86

D.1 Player response for Exp: 26. . . 115

(17)

(18)

(19)

Introduction

Today Massively Multiplayer Online games (MMO games) are popular and gen- erates significant share of Internet traffic [1]. MMO games are one of the largest growth areas in the field of interactive entertainment. There are millions of users playing MMO games across the globe [2]. The three main kinds of MMO games are Multiplayer Online Role Playing games (MMORPG), First Person Shooter (FPS) games and Real-Time Strategy games (RTS) [3]. Among these three games FPS games are very sensitive to the effects of network performance metrics such as delay, packet loss, jitter, bit rate and inter-packet time [4], [5], [6]. While playing online games players can easily experience these effects as disturbances in the game play.

As online game traffic is very active and have large number of users, Internet Service Providers (ISPs) can make better profits by offering the players with pre- mium network connectivity. ISPs that do not do this will see a loss of revenue as players migrate to other ISPs that fulfill their Quality of Service (QoS) require- ments on the network. QoS is a set of objective values of network performance metrics. A number of QoS parameters describe the speed and reliability of data transmission, e.g., throughput, transit delay, and error rate [7], [8]. A lot of research work has been done on the investigation of how network metrics affects the QoS which in turn affects Quality of Experience (QoE) of the player [5],[9],[8].

QoE is a measure of performance expectations of the end-user, it augments QoS by providing the quantitative link to user perception [3].

In this thesis an analysis has been carried out on how the players in a First Person Shooter (FPS) game are affected by different levels of network delay and jitter. Here delay refers to the time taken by a packet to travel from sender’s end to receiver’s end, this can also be called as one-way delay. The term jitter is nothing but the variation of delay from the mean delay value. Because of the vast usage of the term jitter differently in different scenarios the term Packet Delay Variation (PDV) has been used in this thesis instead of jitter [10]. This work can be useful for the ISPs to improve their existing traffic models thereby improve their QoS.

The data used in the analysis was collected from the LAN party Dream hack winter 2007 by the telecommunications research team of Blekinge Institute of Technology, Sweden. The game traffic traces were collected by using Distributed Passive Measurement Infrastructure (DPMI) [15]. This data consists of 57 games, conducted between 2 and 4 players.

(20)

1.1 Related work

This section discusses the previous work done on the game traffic analysis and the effects of different network metrics on the QoS.

[8] investigated the effects of packet loss and latency on user’s performance for a popular FPS game, Unreal Tournament 2003. They found that packet loss has no considerable impact on the user perception but latency does. Latencies as low as 100 ms were noticeable and latencies around 200 ms were annoying.

Authors suggested maintaining the delay below 150 ms and the packet loss below 3 percent.

In [5] authors investigated the impact of small amounts of delay and jitter on the FPS game, Unreal Tournament 2003. They conducted experiments by introducing the delays of 15 ms to 195 ms . They observed that the player experienced the disturbance in the game for the Round-Trip delay above 60 ms.

[6] explored the effects of background traffic on the Quake FPS game. For this investigation 3 sets of experiments were conducted: the first experiment is on isolated local network with clean traffic, the second one is over internet and the third one is using a simulator with varying amount of background traffic.

The simulator experiments shown that the background traffic can increase the delay.

[11] conducted a survey on how different network metrics affecting the subjective end user’s performance. The main factors which were analyzed in this study are player’s experience, delay and jitter. The survey was conducted with 309 participants; the results from this survey were related to the results of experiments conducted in a controlled environment. Authors evaluated three different kinds of games: two FPS games, a car racing simulation and a real time strategy game. While player experienced good quality of the game in the experiments with normal network conditions, delay and jitter have the large but game dependent impact on the game quality.

In [12] Authors started an interesting research on the QoS tolerances of game players with an assumption that the users are not sensitive to QoS. For this research they conducted some experiments by running the public game servers for FPS game Half-Life. They found that the degraded QoS in the form of network delay makes users to leave the server, but the players who have been playing the game from long time are unlikely to leave the server even with high delays.

[9] Investigated how sensitive are the end users with the network quality.

They analyzed the traces captured from a MMORP game Shen Zhou Online.

Authors made a statement that the game playing time is strongly affecting the network QoS and the player seems to be more sensitive for the packet loss than the latency.

In [13] investigators tried to establish performance bounds for two different games: Quake-3 and Halo. To do this they conducted experiments in a controlled environment with a user questionnaire for the subjective measure of user perceived quality and presented a list of existing assumptions those can be sup- ported by the findings of this investigation.

[3] conducted a study on how network delay affects the QoE of the user. For this purpose two experiments were conducted. First experiment was conducted in a network without introducing any delay with 12 players playing the game Battle field 2 (BF2). Second one is conducted in a controlled environment with 2

(21)

players in order to inquire the delay threshold. Author made a note that player felt the game is unplayable at 450 ms delay.

In [14] and [15] authors implemented simulation models for Quake-III and Half-Life respectively with packet size, packet inter - arrival times and data rate as main characteristics. To develop these simulation models traffic traces from the live experiments conducted over the LAN were analyzed with respect to the above mentioned characteristics. They found that the server - client packet sizes are affected by the number of players and map type in both games but the client - server packet sizes are not affected by the any parameters. They also stated that, while the sever - client inter - arrival times are not affected by the map or number of players except rat3 map in the Half-Life the client - server inter - arrival times are depends on the graphic cards of client machine, map in Quake-III and by graphic rendering software in Half-Life.

1.2 Contribution

This thesis began with the examination of traffic traces and the experimental setup. After that the real work started with the implementation of tools required to calculate delay and produce comprehensible delay plots from data traces captured during the experiments by synchronizing all client machines in the experiment and by adjusting delay drift in the plots. Then the content in the user questionnaires have been studied and an effective way to use available subjective user responses has been found out.

Afterwards investigation majorly concentrated on the analysis of impact of one-way delay and PDV on FPS game BF2 which affects the QoE of the game player. An investigation has also been done on the effect of packet loss, packet size distribution of server - client and client - server traffic and the delay distribution of traffic shaper. As there is very minute packet loss in the network it did not affect any aspects of the game unlike delay and PDV.

By relating the findings from the data analysis with the user questionnaires a scientific report has been produced which generalizes the relation between QoE and one-way delay, PDV.

1.3 Outline

The outline of upcoming paper is as follows: Chapter 2 describes about the experimental environment, problem identification and the method. The results from the experiments and the analysis are discussed in chapter 3 followed by conclusions and future scope in chapter 4.

(22)

(23)

Approach

In order to investigate the impact of one-way delay on FPS gaming experience the data collected from 57 experiments is available, out of them 42 were analyzed.

In remaining experiments some experiments have damaged traces because of uncertain data and some don’t have questionnaires from shaped players. The QoE investigation was done by relating the results of delay analysis with the user questionnaires. This entire work was carried out as follows:

1. Insight into experimental setup and examination of data traces collected from Dream Hack Winter 2007, see section 2.1 and 2.2.

2. Identification of problems and solutions, see Section 2.3.

3. Analysis of results, see section 3.

2.1 Experimental Environment

Figure 2.1 shows the network setup on which the experiments were conducted.

The device Dell OptiPlex 740 dual AMD was used as a game server. NetEm is used as a network emulator, it introduced specified amount of delay into the network at a specified time. P1, P2, P3 and P4 are the Dell XPS Quad P4 machines which were used by the players and configured with Nvidia GeForce 8600 GTS graphic cards, 3 GB RAM and 2.40 GHz processor speed. The game played in the experiments was BF2. MP1 and MP2 are two measuring points in the DPMI which are responsible for storing a copy of game traffic coming from Network Monitoring Cards. MP1 has DAG 3.6T cards and MP2 has DAG 3.5E cards [16]. DAG card uses a special memory buffer that allows game traffic can be copied with great speeds without dropping even a single packet by appending an additional header containing information about the location and time of capture.

Data from 57 experiments were collected for the investigation of one-way-delay behavior of FPS games, and to analyze the QoE of the user. While conducting the experiments games were started with the game start message received by every machine in the network and continued for around 16 minutes. Delays from 0 ms to 600 ms were introduced into the network with the variation of 0 ms to 400 ms to and from the client and server.

Shaper settings for all experiments are summarized in Table 2.1 and 2.2.

In the tables ‘Delay ± P DV ’ indicates the delay and PDV introduced into the

(24)

network. Delay was introduced into the network together with delay variation in the form of A ± B, where A is delay and B is delay variation. ‘Time’ is the time after which the network got disturbed (start time of the shaper). ‘Shaped player’

indicates the player who got shaped, ‘Server’ means that the server got disturbed in that specific game. ‘Direction’ indicates the data flow direction in which the disturbance occurred. To, From and To/From indicates that the shaping was introduced into the traffic to the player, from the player and to and from the player respectively. The last column ‘No.of players’ indicates the number of players participated in the experiments.

Figure 2.1: Block diagram of experimental setup

(25)

Table 2.1: shaper settings for the experiments 1 to 25 Game Delay ± P DV [ms] Time [s] Shaped

player Direction No.of players

1 400 ± 100 62 4 From 2

2 400 ± 100 76 2 To 1

3 400 ± 100 213 1 To/From 3

4 200 ± 50 221 3 To 2

5 200 ± 50 237 1 From 3

6 200 ± 50 287 2 To/From 4

7 200 ± 0 207 2 From 2

8 200 ± 0 268 1 To 4

9 200 ± 0 202 3 To/From 4

10 200 ± 200 190 1 From 4

11 400 ± 0 240 3 To 2

12 400 ± 0 241 3 From 4

13 400 ± 0 184 2 To/From 3

14 200 ± 100 209 1 To/From 4

15 200 ± 100 224 1 To 4

16 400 ± 200 213 2 To 2

17 400 ± 200 235 2 From 3

18 400 ± 200 150 2 To 2

19 200 ± 100 254 3 From 1

20 200 ± 200 189 2 To/From 1

21 200 ± 200 227 3 To 2

22 400 ± 400 261 3 To 1

23 600 ± 400 253 3 To 4

24 400 ± 400 292 1 To/From 4

25 600 ± 400 285 1 To/From 2

(26)

Table 2.2: shaper settings for the experiments 26 to 57 Game Delay ± P DV [ms] Time [s] Shaped

player Direction No.of players

26 600 ± 400 282 4 From 2

27 800 ± 0 294 3 To/From 4

28 800 ± 0 276 1 To 4

29 800 ± 0 181 3 From 2

30 400 ± 400 190 2 From 2

31 200 ± 50 245 3 To 1

32 100 ± 100 231 1 To 1

33 100 ± 100 262 3 From 1

34 0 ± 100 253 1 To 1

35 0 ± 100 206 3 From 4

36 0 ± 200 213 server To 4

37 1 ± 200 231 server from 4

38 1 ± 200 195 server From 4

39 1 ± 200 269 server To 4

40 100 ± 100 206 server To 4

41 100 ± 100 266 server From 4

42 1 ± 100 287 server To 4

43 1 ± 100 294 server From 4

44 1 ± 200 286 server To 4

45 1 ± 200 265 server From 4

46 200 ± 100 184 server To 4

47 200 ± 100 255 server From 4

48 200 ± 100 251 server To 4

49 200 ± 100 207 server From 4

50 200 ± 100 251 server To 4

51 200 ± 100 259 server From 4

52 200 ± 100 246 server To 4

53 200 ± 100 272 server From 4

54 200 ± 100 205 server To 4

55 200 ± 100 182 server From 4

56 1 ± 100 258 server To 4

57 1 ± 100 276 server From 4

(27)

2.2 Collection of data

The MP’s collected 96 bytes of data from the packets they saw on the network.

In these 96 bytes there are IP header, transport header and parts of payload.

Figure 2.2 shows the format of the captured data [16]. In the figure Capture Header (CH) is attached by the Capture Interface (CI) to all captured packets, MPID is an identifier of a measurement point in the DPMI, CI is the Capture Interface or dag card where the packet was caught, FL is the Frame Length on the link layer, Capture Length (CL) indicates how much data that was captured and stored in the packet payload and the timestamp indicates the time when the packet was captured [16]. This data is essential to analyze the games.

To analyze QoE of the players with respect to one-way delay behavior, participants were also asked to fill in a questionnaire after every session. Players gave grading to the following questions depending on the performance of the game and their experience during the game play:

1. I experienced that the other players moved normally in the game.

2. I felt that the game reacted promptly to my signals.

3. The game showed jerks’ or hacks’.

4. I was badly treated by the game.

5. My total impression of the game was good.

Figure 2.2: DPMI trace formate

2.3 Problem Identification And Solution

This work was started with the identification of Situations which needed to be solved during delay calculations to attain good accuracy. The identified problems are listed below.

1. Packet Identification.

2. Clock synchronization.

3. Delay drift.

(28)

2.3.1 Packet Identification

Packet identification is a procedure of identifying packets at both server and client ends with a unique identifier to measure one-way delay, packet loss, etc. Hashing is used to generate identifiers for each captured packet and they were produced through Secure Hash Algorithm-1(SHA-1). It accepts a message of maximum length 2⁶⁴ bits as input and gives 160 bit message digest as output [17].

The tool performs simple logical operations to the IP packet header fields to generate hash keys. Selection of header fields should be done based on two rules,“input values should be variable among the packets and input values should be static between network nodes” [18].

The first 40 bytes of header field has been used as an input for SHA-1 to reduce the number of hash over laps and it is found that there were no overlapped packets. This tool has been used to identify packets only to provide flexibility to evaluate even bigger traces with millions of packets, the sequence number of the packet can also be used as an identifier for the traces with less number of packets.

2.3.2 Clock Synchronization

One - way delay is the time taken by a packet to travel from sender’s end to receiver’s end. it can be calculated after the identification of each packet at both source and destination ends, by subtracting the time stamp in CH of the packet at sender’s side from the time stamp in CH of the same packet at the receiver’s side as shown in the equation 2.1.

di= ti,r− ti,s (2.1)

Where di is the delay of i^th packet, ti,r is the time when the i^th packet reached receiver and ti,s is the time when that packet was sent by the sender.

Here i indicates the same packet at both sender and receiver ends.

These delay calculations will give the required outcome only if all the clocks in the network show the same time at same instance. Let’s suppose that there is a sender’s clock and a receiver’s clock indicating different times at a same instance (at game start) in a network and shaper introducing no extra delay as shown in the Figure 2.3. Also assume that the minimum time required for a packet to travel from sender to receiver is 1 min and the game start times of sender and receiver are 3:00 hrs and 4:00 hrs respectively. In this scenario, a packet delivered by the sender at 3:05 hrs according to sender’s clock should reach receiver’s end at 4:06 hrs according to receiver’s clock. Now according to the equation 2.1 delay of the packet is 1 hr and 1 min. But it is clear that the obtained delay is wrong, it should be 1 min, to get this accurate delay synchronization was done by setting all clocks to zero at game start by subtracting game start time of receiver and sender from the time stamps of all packets captured at receiver’s side and sender’s side respectively. Equation 2.2 shows the formula for synchronization.

ti,j,Sync= ti,j− tgamestart,j (2.2)

where ti,j,Sync is the Synchronized time of i^thpacket at location j, ti,j is the time stamp of the i^th packet at the same location and tgamestart,j is the time corresponds to the game start message at location j.

(29)

Figure 2.3: Clock synchronization.

By substituting the above values in the equation 2.2 synchronized time of sender’s and receiver’s clocks are 5 min and 6 min respectively. Now the delay between the synchronized clocks become the actual delay of the packet, i.e. 1 min in the above assumption.

In the Dream Hack experiment setup there were six different clocks showing six different times at the game start. These clocks were synchronized by setting all clocks to zero at game start by implementation of the method explained above.

Delay calculations were done by assuming that all the clocks in the network were running with the same speed and then the graphs were produced separately for each relevant stream by considering the packets with the same source IP address and destination IP address as a stream.

2.3.3 Delay drift

While observing the delay plots it is found that the minimum delay keeps on changing. There was steep increase or decrease in the delay as shown in the Figure 2.4 called delay drift, this is the result of the clock drift. Because of the drift the clock does not run at the exact same speed compared to another clock i.e. after some time the clock drifts apart from the other clock. So it is difficult to analyze the graphs with considerable drift.

In the experiments receiver’s clock and sender’s clock were running with drift.

Sender’s clock was used as reference clock while calculating the delay. The following two situations can cause delay drift in the graphs:

1. Receiver’s clock lagging the sender’s clock.

2. Receiver’s clock leading the sender’s clock.

Other than theses two situations there is some minute error in the clocks even after the clock synchronization; Because of this error sender’s clock and receiver’s clock are apart from each other at the game start. There is a need to understand how this error behaves in the delay traces together with the clock

(30)

drift. Combining this error with the above mentioned situations there are four different cases to consider while correcting the delay drift.

case 1: Receiver’s clock is greater than the sender’s clock and receiver’s clock is leading: In this case receiver’s clock is greater than the sender’s clock at the game start and it runs faster than the sender’s clock because of the drift.

So throughout the game receiver’s clock is greater than the sender’s clock, the minimum delay is in positive axis and keeps on increasing as shown in the Figure 2.4 - A

case 2: Receiver’s clock is greater than the sender’s clock and receiver’s clock is lagging: In this case receiver’s clock is lagging the sender’s clock, so the distance between two clocks reduces with time. Due to this the delay starts in positive axis and decreases with time if the drift is really high there is a chance for the receiver’s clock to go less than the sender’s clock and the delay plot goes into negative axis. This kind of plot is presented in the Figure 2.4 - B

case 3: Receiver’s clock is less than the sender’s clock and the receiver’s clock is leading: Apart from first two cases, in case 3 receiver’s clock is less than the sender’s clock and it is running faster than the sender’s clock. So the delay starts in negative axis and difference between the receiver’s clock and sender’s clock is decreases with time. So the delay drifts towards positive axis as shown in the Figure 2.4 - C.

case 4: Receiver’s clock is less than the sender’s clock and the receiver’s clock is lagging: In this case receiver’s clock is already behind the sender’s clock and it is lagging, so the two clocks are going apart from each other with time. Since the graph starts in negative and axis and goes against the positive axis, Figure 2.4 - D shows this effect clearly.

0 200 400 600 800 1000

−5

−2 1 4 7 10 10

Delay [ms]

A − Case1

0 200 400 600 800 1000

−5

−2 1 4 7 10 10

B − Case2

0 200 400 600 800 1000

−5

−2 1 4 7 10 10

Time [s]

D − Case4

0 200 400 600 800 1000

−5

−2 1 4 7 10 10

Time [s]

Delay [ms]

C − Case3

Figure 2.4: Delay traces with drift.

A simple geometrical algorithm was used to adjust the delay drift in the output graphs, the steps of the algorithm are as follows.

(31)

1. Let t_i,jbe the arrival time of the packet i in location j, where j is the input location of the receiver.

2. the delay of i^th packet in the trace is di,j−k = ti,j − ti,k where k is the input location of the sender, given that i indicates the same packet at both locations. Simplified di,j−k is defined as (di, ti) where di and ti are delay and time corresponds to the same packet respectively.

3. r1 = min(di) where i < α and r1 6= 0, α is specified number of packets from the start of the trace.

4. r2 = min(di) where i > β and r2 6= 0, β is specified number of packets from the end of the trace.

5. Let the line joining (r₁, t₁) and (r₂, t₂) be the drift delay line where t₁ is the time corresponding to r₁ and t₂ is the time corresponding to the last packet in the trace.

6. Slope of the drift delay line k = _(r^(t²^−t¹⁾

2−r₁)

7. Let line joining the (r₁, t₁) and (r₁, t₂) be the actual delay line.

8. Distance between (di, ti) and (r1, ti) is Di =p((k × (t1− ti)) − r1)²where Di is the distnace between drifted delay and actual delay of i^thpacket.

9. ˆd_i =







d_i,j− Di, if k>0 di,j, if k = 0 d_i,j+ D_i, if k<0

10. di,Corrected=

( ˆd_i− (2 × r1), if r₁<0

dˆi, if r1> 0 where di,Correctedis the delay after drift adjustment.

0 200 400 600 800 1000

−5

−2 1 4 7 10 10

Delay [ms]

A − Case1

0 200 400 600 800 1000

−5

−2 1 4 7 10 10

B − Case2

0 200 400 600 800 1000

−5

−2 1 4 7 10 10

Time [s]

Delay [ms]

C − Case3

0 200 400 600 800 1000

−5

−2 1 4 7 10 10

Time [s]

D − Case4

Figure 2.5: Adjusted delay traces.

(32)

Delay drift was adjusted by implementation of the above algorithm in Perl and by selecting α and β in such a way that the delay values in the plot never goes below the drift delay line joining the points (r₁, t₁) and (r₂, t₂). In the Figures 2.5 - A, B, C and D you can also find the adjusted delay traces of all cases discussed above. Delay calculations with good precision were obtained after the drift adjustment. The graphs were produced between the packet sent time and one-way delay.

(33)

Results and Analysis

0 100 200 300 400 500 600 700 800 900 1000

0 1 2 3 4 5

Time [s]

Delay [ms]

B − Server to Player3

0 100 200 300 400 500 600 700 800 900 1000

0 1 2 3 4 5

Time [s]

Delay [ms]

A − Player3 to Server

Figure 3.1: Delay traces for unshaped players.

This section shows results from the experiments and analyzes the effects of delay on user performance with the help of user questionnaires.

From the analysis it is found that while the mean delay of shaped players is varying with respect to shaping, the mean minimum and maximum delays of unshaped players are 0.6 ms and 5 ms respectively. It is difficult to compare the delay plots of shaped and unshaped players with similar scales since there is a large variation in delay between them. So delay traces for the traffic to and from the unshaped client with small scales on Y-axis are presented in Figure 3.1. These traces are from the experiment 1 with 2 players and in this game the delay is varying from 0 to 4 ms. Players did not feel any disturbances in the game with this minute amount of delay. Only the traces from the shaped players are discussed in the rest of the paper. The remaining delay traces of shaped and unshaped players which are not discussed in the paper are shown in the appendix C and D. The delay traces of shaped players are clearly indicating the different shaper settings. These observations give support to our drift correction and delay evaluation tools.

(34)

For the detailed analysis, Dream hack experiments are divided into three groups depending on the shaper settings, shaping the traffic from the player, to the player and both. The delay statistics have been taken for these three different scenarios from shaped region, unshaped region and from entire length of the game trace. Shaped region is nothing but the region in which the shaper is active, unshaped region is the region where the shaper is inactive and the entire length of the game is a combination of both. Statistics from the unshaped region have been taken from the starting of the game trace to the point where the shaper became active. Statics of delay in the shaped region for all games separated with these three cases can be seen in the Table 3.4, 3.5, 3.6, 3.7 and 3.8. Statistics for unshaped and entire regions are shown in the appendix, section B.2 and the statistics of unshaped players from all regions for all scenarios are in the appendix, section B.1. The above mentioned scenarios are discussed in the following subsections by comparing with user responses.

3.1 Shaping into the traffic from the Player, Exp: 1

There are two players in the experiment 1. The shaper was set to introduce 400 ± 100 ms delay into the traffic between player to server after 62 s of game start until 300 s. Figure 3.2 clearly shows the variation of delay in shaped and unshaped regions corresponding to the shaper settings. The shaping started as per the shaper settings and continue until 300 ms. In the shaped region the delay is varying between 300 ms and 500 ms with the mean of 400 ms.

In this experiment with the shaping time of less than 5 min unshaped player gave the rating 5 on a scale of 1 to 5 while the shaped player gave the rating 3 for a question about the total impression on the game. But, while playing the game shaped player commented that there are hacks all the time in the game and he mentioned that he spends 42 hours per week on playing games. The player ratings can depend on many factors like animations in the game, experience of the player, curiosity of participating in the experiments and many more personal regions.

As a part of the analysis the delay distribution of the shaper has also been investigated. Figure 3.3 shows the Empirical Cumulative Distribution Function (Empirical CDF) and the histogram (with bin width of 25 ms) of delay for Exp: 1 from the shaped region. The Empirical CDF in Figure 3.3 - A shows the delay distribution is between 300 ms to 500 ms and the histogram in the Figure 3.3 - B shows that the shaper distributed the delay more or less uniformly throughout the shaped region.

Figure 3.4 shows the packet length distributions of the shaped player and the server. The entire length of the game has been considered to look at the packet length distributions. Figure 3.4 - B and D shows the packet size distribution of the packets coming from the client and the server respectively. From these figures it is clear that the client-server packets are smaller in size and have a narrow band of distribution. On the other hand server-client packets are bigger than client-server packets and the packet length distribution band is wider.

Generally client-server messages contain the information about single player’s game status such as button press, change orientation and movements [19]. There are few packets which are having greater size because of the non-game play messages and the massive triggered events of the player.

(35)

Packets coming from the server have the game status messages about all the players with in the client surroundings, so the packet size is usually bigger.

Server-client packet length might be dependent on the number of players involved in the game, type of map played, player events and the game type [20].

0 100 200 300 400 500 600 700 800 900 1000

Time [s]

Delay [ms]

Shaping from client to server

Exp: 1

Figure 3.2: Shaping from client to server.

0 100 200 300 400 500 600 700 800 900 1000

0 0.5 1

Delay [ms]

Probability

A − P4 to S (CDF) Exp:1

0 100 200 300 400 500 600 700 800 900 1000

0 0.1 0.2 0.3 0.4 0.5 0.5

B − P4 to S (Histogram)

Delay [ms]

Density

Figure 3.3: Delay distribution of shaper in Exp: 1.

(36)

Exp:1

0 500 1000 1500

0 0.5 1

Packet Size [bytes]

Probability

A − P4 input

0 500 1000 1500

0 0.5 1

Packet Size [bytes]

Probability

B − P4 output

0 500 1000 1500

0 0.5 1

Packet Size [bytes]

Probability

C − S output

0 500 1000 1500

0 0.5 1

Packet Size [bytes]

Probability

D − S input

Figure 3.4: Packet length distributions of client-server and server-client packets (Exp: 1).

3.2 Shaping into the traffic to the Player (Exp: 16)

In this experiment, the shaper introduced 400 ms of delay with the variation of 200 ms into the server to player traffic after 213 sec of normal play. Shaping was last until 899 sec. The shaped region and the delay variations are clearly shown in the Figure 3.5. Two players participated in this experiment.

Figure 3.6 and 3.7 shows the delay distribution and packet length distributions respectively. Delay and packet lengths are distributed similarly as explained in the above section.

Unshaped player rated his experience of the game with the rating 3 and the shaped player rated it as 2. Shaped player also made a comment in the questionnaires that there are jerks and hacks in the game with very long response time. He also stated that he spends 35 hours per week on games. He could have been imposing the bad impression on the game with the rating 2 because of the huge delay and PDV continued for more than 11 min in the game.

(37)

0 100 200 300 400 500 600 700 800 900 1000 0

100 200 300 400 500 600 700 800 900 1000

Time [s]

Delay [ms]

Shaping from server to client

Exp: 16

Figure 3.5: Shaping from server to client.

0 100 200 300 400 500 600 700 800 900 1000

0 0.5 1

Delay [ms]

Probability

A − S to P2 (CDF) Exp:16

0 100 200 300 400 500 600 700 800 900 1000

0 0.1 0.2 0.3 0.4 0.5 0.5

B − S to P2 (Histogram)

Delay [ms]

Density

(38)

Exp:16

0 500 1000 1500

0 0.5 1

Packet Size [bytes]

Probability

A − P2 input

0 500 1000 1500

0 0.5 1

Packet Size [bytes]

Probability

B − P2 output

0 500 1000 1500

0 0.5 1

Packet Size [bytes]

Probability

C − S output

0 500 1000 1500

0 0.5 1

Packet Size [bytes]

Probability

D − S input

(39)

3.3 Shaping into the traffic to/from the player (Exp: 6)

In experiment 6 shaping is introduced into the traffic, to/from the player. Shaping is started at 287 sec and ended after 899 sec. The amount of delay introduced is 200 ms with the variation of 50 ms. Figure 3.8 shows the delay plot of shaped player for this experiment.

From the Figure 3.8-A and B it is certain that the delay is not same as introduced, because the shaper divides the delay between the traffic to and from the player almost equally.

Figure 3.9-C and D shows that the delay distribution is not uniform in this case, there are majority of packets with delays between 75 and 125 ms. The packet length distributions in Figure 3.10 are similar to the experiments discussed above.

All the players participated in this experiment reacted very positively about the game even with the long shaping time and some of them gave the rating 4 and some gave 5. The shaped player gave the rating 4 and he made a comment that very few times he felt the jerks in the game and he said that he spends 15 hours per week on playing games.

0 200 400 600 800 1000

0 100 200 300 400 500 600 700 800 900 1000

Time [s]

Delay [ms]

Shaping from client to server

0 200 400 600 800 1000

0 100 200 300 400 500 600 700 800 900 1000

Time [s]

Delay [ms]

Shaping from server to client Exp: 6

Figure 3.8: Shaping from/to the client.

(40)

0 200 400 600 800 1000 0

0.5 1

Delay [ms]

Probability

A − P2 to S (CDF)

0 200 400 600 800 1000

0 0.5 1

Delay [ms]

Probability

B − S to P2 (CDF)

0 200 400 600 800 1000

0 0.1 0.2 0.3 0.4 0.5 0.5

C − P2 to S (Histogram)

Delay [ms]

Density

Exp:6

0 200 400 600 800 1000

0 0.1 0.2 0.3 0.4 0.5 0.5

D − S to P2 (Histogram)

Delay [ms]

Density

Exp:6

0 500 1000 1500

0 0.5 1

Packet Size [bytes]

Probability

A − P2 input

0 500 1000 1500

0 0.5 1

Packet Size [bytes]

Probability

B − P2 output

0 500 1000 1500

0 0.5 1

Packet Size [bytes]

Probability

C − S output

0 500 1000 1500

0 0.5 1

Packet Size [bytes]

Probability

D − S input

(41)

3.4 Experiments with abnormal behaviors

In most of the experiments delay, packet length and delay distribution plots exhibiting the same properties as explained in the above sections. But in some experiments there are some abnormal behaviors in the data flow between client and server, packet size distributions and the delay distribution of the shaper.

This section discusses the experiments with these behaviors.

3.4.1 Experiment: 2

In this experiment the delay and delay distribution plots are similar to the others experiments, but from the Figure 3.11 it is clear that the packet lengths of player input and server output are not similar (Figure 3.11-A and C)). This is due to the reason that there is a longer traffic trace from the measuring point at server side (MP1) than the measuring point at client side (MP2) as shown in the Figure 3.12. Because of this there are more packets captured at MP1 than MP2 with the same source and destination IP addresses, given that the packet size distribution is different. From the comparison of traces in Figure 3.12-A and 3.12-B between 0 and 700 s it is looking like there are some extra packets in the player input trace (Figure 3.12-B)which are not presented in server ouput trace. In reality those packets are same packets sent by the server but with some degradation of size due to the minute packet corruption in the Twisted pair cables used for the network connections.

There are some more experiments presented in the appendix which are affected by the similar behavior of the measuring points.

Exp:2

0 500 1000 1500

0 0.5 1

Packet Size [bytes]

Probability

A − P2 input

0 500 1000 1500

0 0.5 1

Packet Size [bytes]

Probability

B − P2 output

0 500 1000 1500

0 0.5 1

Packet Size [bytes]

Probability

C − S output

0 500 1000 1500

0 0.5 1

Packet Size [bytes]

Probability

D − S input

Figure 3.11: Packet length distributions of server-client and client-server packets (Exp: 2).

(42)

0 100 200 300 400 500 600 700 800 900 1000 0

200 400 600 800

Time [s]

Packet size [bytes]

Packet length trace for server output − A

0 100 200 300 400 500 600 700 800 900 1000

0 200 400 600 800

Time [s]

Packet size [bytes]

Packet length trace for player input − B

Figure 3.12: Packet length trace for player input and server output (Exp: 2).

3.4.2 Experiments: 3, 7, 10, 11, 12, 17, 19, 31 and 52

Experiment 3 is a failure because there is a big jump in the time stamps in MP1.

Due to this wrong measurement delay suddenly became very big (in the order of several thousand seconds) after 331 s.

In Experiment 12 there is no game start message delivered to the players for synchronization. So it became difficult to synchronize the players and calculate delay.

There are very short data traces from one measuring point in Experiments 17 and 52. These traces are ended in the middle of the shaped region, eventually there are very small delays at the starting of the trace (around 4 ms) and very high delays at the end (around 400 ms and 200 ms). As a result of this while adjusting delay drift there obtained a very huge positive slope which leads to incorrect drift adjustments.

There are no questionnaires from the shaped player for the Experiments 7, 10, 11 and 31, so we cannot measure the QoE of the user. In Experiment 19 shaped players did not give the rating for fifth question, even though delay traces for the players are presented in the appendix ( sections C.6, C.9, C.10 and D.6.

All these experiments have been skipped because of the reasons mentioned above.

3.4.3 Experiments: 34 to 39

In the Experiments 34 to 39 the shaper was configured to apply 0 ms delay with different amounts of variation. Shaper did not work properly for this setting as the shaper is not designed to handle this kind of inputs. An Example delay trace (Exp: 36, Player3 to Server) is shown in the Figure 3.13 to give an idea about how NetEm emulator acts for this kind of input. Figure 3.13 belongs to player 3 to server trace in experiment 36 with the introduced delay of 0 ± 200 ms.