Crowd-based Network Prediction : a Comparison of Data-exchange Policies

Institutionen för datavetenskap

Department of Computer and Information Science

Kandidatuppsats

Crowd-based Network Prediction:

a Comparison of Data-exchange Policies

av

Anton Forsberg och Jakob Danielsson

LIU-IDA/LITH-EX-G-15/057-SE

2015-06-24

Linköpings universitet Institutionen för datavetenskap

Kandidatuppsats

Crowd-based Network prediction:

a Comparison of Data-exchange Policies

av

Anton Forsberg och Jakob Danielsson

LIU-IDA/LITH-EX-G-15/057-SE

2015-06-24

Handledare: Niklas Carlsson

Examinator: Nahid Shahmehri

Students in the 5 year Information Technology program complete a semester-long software devel-opment project during their sixth semester (third year). The project is completed in mid-sized groups, and the students implement a mobile application intended to be used in a multi-actor setting, currently a search and rescue scenario. In parallel they study several topics relevant to the technical and ethical considerations in the project. The project culminates by demonstrating a working product and a written report documenting the results of the practical development process including requirements elicitation. During the final stage of the semester, students form small groups and specialise in one topic, resulting in a bachelor thesis. The current report rep-resents the results obtained during this specialization work. Hence, the thesis should be viewed as part of a larger body of work required to pass the semester, including the conditions and requirements for a bachelor thesis.

Abstract

Network performance maps can be used as a tool to predict network conditions at a given location, based on previous measurements at that location. By using measurement data from other users in similar locations, these predictions can be significantly improved. This thesis looks into the accuracy of predictions when using different approaches to distribute this data between users, we compare the accuracy of predictions achieved by using a central server containing all known measurements to the accuracy achieved when using a crowd-based approach with opportunistic exchanges between users. Using data-driven simulations, this thesis also compares and evaluates the impact of using different exchange policies. Based on these simulations we conclude which of the exchange policies provides the most accurate predictions.

Acknowledgements

First of all, we would like to express our gratitude to our fantastic supervisor Niklas Carlsson, for answering all of our question and for continuously providing us with useful input and advice. We would also like to sincerely thank Rickard Dahlstrand at .SE (The Internet Infrastructure Foundation) for providing us with the dataset that made this thesis possible. Furthermore, we would like to thank Tova Linder, Pontus Persson, Karl Andersson and Marcus Odlander for proof-reading our thesis and providing useful feedback during the whole process. Finally we would like to thank Marcus Bendtsen, Simin Nadjm-Tehrani, Eva T¨ornqvist, Nahid Shahmehri and all of our coursemates for all their support during this period.

CONTENTS CONTENTS

Contents

3.2 Crowd-based data sharing . . . 3

3.3 Exchange policies . . . 3 4 Methodology 4 4.1 The dataset . . . 4 4.2 Preparations . . . 4 4.3 Exchange policies . . . 5 4.4 Comparison . . . 6 4.5 The simulations . . . 6 5 Results 7 5.1 High-level characterization . . . 7

5.2 Baseline policy comparison . . . 9

5.3 Operator-aware policies . . . 11

5.4 Head-to-head comparison . . . 11

5.5 Comparison with entire dataset . . . 12

6 Discussion 14 6.1 Results . . . 14

6.2 Methodology . . . 14

6.3 Future work and possibilities . . . 15

6.4 Ethical considerations . . . 15

1 INTRODUCTION

1

Introduction

In a disaster scenario, managing resources such as energy and bandwidth is crucial in order to guarantee that important information can be transmitted and received when needed. Using network performance maps, a user can predict the available bandwidth at a certain location. With this information the user can adapt and optimize its network usage by downloading and uploading information based on the expected bandwidth conditions. This is important for reducing the mobile units energy usage. By using information and measurements from other users that have been at the same location, this optimization can be done more efficiently and with greater accuracy.

These crowd-based network maps can be distributed in different ways. One approach is using a central server from where the user can access all previous measurements. The problem with this centralized solution is that it needs to be available at all times. This means that this solution is sensitive to failures such as power or network outages which could be expected in a disaster scenario. The other approach is using an oppurtunistic exchange model, where users share known measurement information with each other using peer-to-peer technology.

Carlsson and Est´evez [1] show that using a network condition map together with a geo-smart scheduler can reduce download times and as an effect of this also reduce energy usage. This is because one can use the network condition map to decide where downloads should take place and make sure they occur at opportune times. Yao et al. [2] also use measurement data together with maps for network prediction. They show that the best indicator for accurately predicting future network conditions is to look at past experiences and measurements from that given location. This means that, when moving along a path, simply looking at your current bandwith does not say much about future conditions.

Our thesis investigates how these network performance maps can be expanded and improved by users sharing data and measurements with each other. We compare the accuracy of the predictions made from data shared between users, to the predictions made using data received from a central server. To do this we create and evaluate different exchange policies, which determine for example what data will be exchanged with whom and at what time. These policies are created and simulated using a dataset containing mobile bandwidth measurements, provided by Bredbandskollen.1 _{This dataset contains over 40 million measurements made by real persons}

using Bredbandskollen’s iOS and Android applications between November 2008 and February 2015. In this thesis, we focus on the measurements made since November 2014 in the central areas of Stockholm, Sweden.

1.1

Contributions

This thesis contributes with several things. First we compare the accuracy of throughput pre-dictions when using a central approach to the accuracy of a crowd-based approach in the sharing of networks performance maps. We also present a number of different data-exchange policies. These policies can be seen as rules or guidelines for data exchange between users. They control what data to exchange, who it should be exchanged with and when. We identified a policy that provides good throughput predictions and also keeps the number of transmissions low to reduce energy usage. This policy could then be used together with a geo-smart scheduler [1], to further reduce energy and bandwidth usage.

1.2

Thesis outline

In the next chapter we look at work that is related to the work we have done. In chapter 3 we present the theory behind the techniques we use. After that, in chapter 4, we present the dataset we have worked with. We also describe the method and the exchange policies used. In chapter 5 we present and briefly discuss our results. A deeper discussion of the results are presented in chapter 6 where we discuss the thesis in full. The last chapter of this thesis states our conclusions.

2 RELATED WORK

2

Related work

With the rise of smartphones and tablets, a number of studies have looked into network predic-tions for mobile networks. Gerber et al. [3] show a method of utilizing passively collected flow records to estimate achievable download speeds in mobile networks. The concept of mobile per-formance maps has been studied in a number of previous articles. A number of these studies has shown that it is possible to improve quality of service in multimedia applications by using perfor-mance maps based on previous measurement data [2, 4, 5]. Højgaard-Hansen et al. [6] show that the use of performance maps can reduce communication overhead and by this reduce battery and data usage. P¨olig and Wolf [7] show how a central approach can be used in the sharing of net-work performance maps between moving vehicles. Gregori et al. [8] show how a crowd-sourcing system called Portolan uses measurements from smartphones to populate a central server. This server then consolidates this data in the purpose of building a graph of the internet. Evensen et al. [9] present a working solution for mobile video streaming that can compensate for varying bandwidth using bandwidth prediction. In this solution they use a location-based QoS system, that uses crowd-sourced data containing GPS position and accompanying measurement data. Using this system in real-world experiments, they show that the perceived video quality could be significally improved

When writing a paper that makes an effort to do experiments on network prediction, like the ones mentioned above, it is important that one do these experiment on a large and well diversified dataset. Otherwise the results can be to dependent on the data. Yao et al. [2] describe their approach to gather this information. They built their own client-server measurement system. This system was later placed in a car and the car was driven along two certain routes in Sydney. These routes represent a daily commute. This information was gathered during a period of eight months and it resulted in 75 drives along the paths. We think this is a good approach to gather information since it gives a diversified and reliable dataset along the routes. However it only gathers information along the path, and is therefore not interesting for any experiments that does not use this particular path.

To collect data for network performance maps and network predictions, a crowd-based ap-proach can be used. Choffnes et al. [10] use one type of crowd sourcing to detect and monitor network events. In a similar way, Arlitt et al. [11] use what they refer to as passive crowd-based monitoring to observe quality of service (QoS) experiences of users across the world. Another approach is described by Sommers and Barford [12]. They have used 3 million user-initiated throughput tests done by iOS and Android phones, and the data collection is very similar to the one that were used to create the dataset we are using.

When a performance map has been created, it can be used to improve the performance of your mobile device. This is done by predicting where the throughput will be best and download files at that place. This is described by Murtaza et al. [13]. Tan et al. [14] provide another type of network performance map in which the bandwidth capacity of three mobile operators is captured. Using a crowd-based approach sounds really good, but it is not yet standard. There are always challenges when new technology is introduced to the market. Faggiani et al. [15] write about both the opportunities and challenges with a crowd-based approach.

3 THEORY

3

Theory

In this chapter we describe the three main techniques of this thesis. The techniques described in this chapter are network performance maps, crowd-based information sharing, and peer-exchange policies. These three techniques are related to each other by the fact that the exchange policies could be used together with the crowd-based information sharing service to populate a dataset. This dataset can then be represented as a network performance map.

3.1

Network performance maps

A network performance map is a map which displays the condition of a networks bandwith in a certain area. Though a network performance map always shows the condition of some network, it can be implemented in different ways and it could also be based on different types of data. The one that is the most intuitive and probably most common as well is the one that base the map on throughput or download speeds. Another type of data that could be useful in a performance map is latency. This type of map tells more about the servers capacity and the connection with the servers, and is therefore not the most interesting for this thesis. We have chosen to focus on download speed in this thesis.

A network performance map can be used for several purposes, like comparing coverage for different operators or for the operators to evaluate the performance of their networks. The most common however, is to display and predict how the network conditions will be at a certain location. These maps are sometimes extended to show more information, for example including timestamps in the base data. This way it becomes very simple to see how up-to-date the map is, and the user can determine how relevant the map is. It is also possible to take into account what mobile operator, network technology (3G, 4G, etc.) and mobile device that the data is derived from. This is highly relevant when one wants to predict the download speed, among other things, since the quality of these parameters will affect the average throughput.

3.2

Crowd-based data sharing

This thesis focuses on different ways of collecting and distributing measurement data. Both by crowd-based approaches and central approaches. It is very powerful to use a crowd-based approach since it gives the study a broad spectra of measurements. The measurements then make up a database that can be used in several different ways. Crowd-based measurements typically do not originate from a single source and a single area, but from different phone types, which has different capabilities [9, 12].

3.3

Exchange policies

An exchange policy is a set of rules that, in this thesis, decides what information will be shared. It also decides who will share information with whom. For example, to get the best possible prediction the only relevant measurements might be the ones performed by users with the same mobile operator or network technology.

4 METHODOLOGY

4

Methodology

4.1

The dataset

We have analyzed a large dataset provided by Bredbandskollen that contains over 40 million measurements done using Bredbandskollens application for Android and iOS. The measurements are performed both using smartphones and tablets. For each measurement the data contains several recorded parameters including: latitude, longitude, upload speed, download speed, and internet service provider (ISP). Latitude and longitude are used to indicate the location of each measurement and to map measurements to locations in a network performance map. The upload and download speed (i.e., throughput) is the primary variable of interest and will be used for both prediction and evaluation. In fact, past throughput measurements at a certain location has been shown to be the best predictor of the current throughput at that location [2]. As there are differences between mobile operators we also include the ISP. However, secondary variables that are not of any relevance for this thesis are not analyzed.

4.2

Preparations

In order to get as realistic results as possible we are only interested in measurements done by people using mobile networks such as 3G/4G and not WiFi. Another limitation we have made is that we decided to keep our area of interest to central Stockholm. This gives us an area of about 400 square kilometers, which is large enough to provide a reasonable amount of measurements while still being manageable in size. These limitations resulted in a file that contains 200 000 measurements in our chosen area over the period between November 2014 to February 2015.

Figure 1: Area of Stockholm that is used for the analysis

We wrote a simulation program that allowed us to divide our area into squares, 1 km x 1 km in size in order for us to group measurements done in the same area. The reason for using this specific size is that we have considered a commute scenario, in which a person travels along a path by car, subway or bus. This means that the squares needs to be a size that will be reasonable when traveling at somewhat high speeds. Using our simulation program we were able to populate each square with all measurements made inside that square. With the same tool we are also able to produce relevant statistics for all or single selected squares, such as number of measurements, maximum and average download speeds. This data is later used to compare and evaluate the different exchange policies we want to examine.

4.3 Exchange policies 4 METHODOLOGY

Since the selected squares covered a large area and many were in low-population areas or over water, we had to limit ourself in some way. This is where our mobility pattern comes in. We selected an imaginary route through Stockholm, from H¨asselby in the western suburbs to ¨ Oster-malm in the city centre, that was used for our analysis. Instead of looking at all measurements along the complete path, we selected eleven squares that make up selected parts of the route where we imagine that downloads could be performed. The reason for not including all squares is that some parts might be under ground where downloads are not possible or in areas where there is not enough measurement data. Figure 1 shows the original area of Stockholm containing the measurements that we created our mobility pattern from. Figure 2 shows our selected route and the squares along this path. Since we wanted the number of squares to be limited, but still be a fair representation of our path, these eleven squares were selected. They also provided a large number of measurements which makes them representative for the whole dataset.

Figure 2: The selected route shown along with the selected squares

4.3

Exchange policies

In the rest of the thesis we refer to each policy by the short names they have been given. First of all we decided to have some reference cases which we can compare our results with.

• Our first reference case we named ”all data”. This case has access to all data in the squares and can be seen as the central server in a central approach.

• The next reference case is ”no data”. This case is the opposite to ”all data” because this has no access to the data at all. It makes guesses without any information.

• Our last reference case, ”random percent” has access to a certain percent of random data from the dataset. We use 50%, 25% and 10% random measurements in our results. Here is the list of the exchange policies that we evaluate and discuss later in this thesis. • Our first policy we have named ”full sharing”. With this policy users always share all

available data with all other users that are in range. This open and unrestricted data sharing will lead to the most peer-to-peer transmissions and will require the most overhead. • The second policy is called ”same technology sharing”. Users with this policy shares information with persons that uses the same network technology. This will make sure that users get data from the same internet capacity range as the users who did the measurement. • The next policy is ”same operator sharing”. With this policy users will share information with other users that has the same mobile operator. The advantage of this is that users only get information from the same mobile operator, which reduces the number of incorrect predictions caused by the operators differences.

4.4 Comparison 4 METHODOLOGY

• The last policy is called ”restricted sharing”. Users who apply this policy will only share information with users that has the same mobile operator and uses the same network technology (3G/4G). This is the most restricted policy and it will result in least amount of peer-to-peer transmissions and overhead data.

We have also implemented something we chose to call a simulation-persons ”expert areas”. We have assumed that a ”simulation-person” lives in square 1 and works in square 11. Therefore we have assumed that this is where the person spends most time during the day. As a consequence of this our policies will result in 20% of the measurements from square 1 and 11, but only 5% from the other squares. We believe that this makes the prediction better in the expert areas and a little worse in the path between these.

All of these policies aim to make good predictions, but with some differences with how to achieve this. Full sharing will result in an very broad and open dataset which will make it hard to make good predictions in general. Restricted sharing will make good prediction since it rules out the most significant differences. It will however be more difficult to populate a dataset with this policy since it is so restricted.

4.4

Comparison

When we have prepared all the data and created the exchange policies it is time to use these to simulate our crowd based information sharing service. In order to do this we compare the different exchange policies with the reference cases, but also with each other. The evaluation will be based on the dataset we have retrieved. Based on these simulations and comparisons we expect to be able to present a policy that produces predictions with good accuracy and at the same time keeping the number of exchanges low to keep energy usage to a minimum.

The dataset include some outliers that are hard to explain. For this reason we will also compare the policies after different degrees of filtering for such anomalies. Our filters selects a subset of mobile operators and remove some extreme values.

4.5

The simulations

In order to perform our simulations, we have written a simulation program in Java. This program reads our data file, creates ”location buckets” and places each measurement in the correct bucket. After this, the program picks out the eleven buckets representing the squares along our path that we are using for our scenario. At this point, each of these eleven buckets contain all measurements for the corresponding square from our data file.

Our program then lets us choose what policy to use in our simulated data exchange. The program then picks out data from the entire dataset, based on the filtering specified in the chosen policy. This results in a new dataset on which we make our predictions. The predictor simply calculates the average download speed in the exchanged dataset for each square and ranks them according to the averages seen in the sample data. The result of the prediction is a list that specifies in what order we rank the squares, from the square we predict to have the highest download speed and ending with the one with the lowest predicted download speed.

To estimate the speed that our person actually receives at these locations, our program runs a simulation. This simulation works on the original data in each square and for each one picks a random measurement from this data. After this, we use the previously mentioned list of predic-tions to determine in which squares our user would perform downloads. Since we are interested in results from downloading in more than one location, the simulation program calculates the average speed achieved when downloading at different numbers of squares, ranging from one to all of the squares. So for one location the result is the speed achieved from downloading at only the highest ranked square from our prediction. In the same way, the result for two locations is the average of the speeds achieved from downloading at the two highest ranked squares and so on.

This process is repeated a number of times and our final result is the average download speed from all iterations.

5 RESULTS

5

Results

5.1

High-level characterization

The dataset appears to include some potential anomalies. To illustrate these anomalies and to better understand the impact on the overall dataset we first show the Cumulative Distrubu-tive Function (CDF) and the Complimentary CumulaDistrubu-tive DistribuDistrubu-tive Function (CCDF) of the download speeds for different limitations that we have applied.

Figure 3 shows the CDF for all ISP’s in our dataset. This includes ISP’s that seemingly does not offer any mobile network services, but only wired connections. Each line in the graph shows a different network technology limitation. We can see from the graph that measurements made over 4G are generally faster than 3G measurements, as is expected. We can also see that there are a fairly large number of measurements that seem to be on the high end. Especially on 3G measurements, we see that over 10% of measurements has a download speed over 50 Mbps and that there are a number of measurements that surpasses even 150 Mbps, which is all too high for 3G.

To illustrate the amount of unreasonably high measurements, Figure 4 shows the CCDF for the same dataset as in Figure 3. Here we can see clearly that 10% of 3G measurements are over 50 Mbps. We can also see that 1% of 3G measurements are far over 100 Mbps confirming our suspicions that the dataset includes anomalies.

Figure 3: The download speed of data from all ISP’s in the chosen squares.

5.1 High-level characterization 5 RESULTS

In Figure 5 we have filtered out only measurements made from the four big mobile operators in Sweden and show the CDF for these. These four operators represent each line in the graph, and they are TeliaSonera, Tele2, Telenor and Hi3G. In this graph, we see that the download speeds in general are a bit lower. Another thing we observe is that there are much fewer measurements over 100 Mbps, and practically no results over 150 Mbps. This seems to suggest that the dataset containing all ISP’s might include some results that were not conducted over a mobile connection, but were erroneously labeled as such.

Looking at the CCDF for the same dataset in Figure 6 we can see the difference compared to Figure 4. The speeds are generally lower, and the amount of very high results is significantly lower.

Figure 5: The download speed of data from all mobile operators in the chosen squares.

Figure 6: CCDF for the same dataset as above.

Figure 7 shows the CDF for each of the four mobile operators compared to the entire dataset. We observe that the mobile operators show quite similar results with the exception of Telia, that has noticeably higher results. One possible reason for this might be that Telia is one of the major ISP’s for wired broadband in Sweden, and some of the Telia measurements might actually have been performed over WiFi and a wired connection.

Figure 8 shows this in greater detail. We clearly see that Tele2, Telenor and Hi3G all have fairly similar results, and have very few measurements with speeds over 100 Mbps. The higher results of Telia is also clearly visible here, deviating greatly from the other three operators. The number of high measurements from Telia compared to the others seem to support our suspicion that Telia’s measurements might include a small number of results produced over WiFi. However, these results are more representative of what we expected to see, in terms of realistic speeds, compared to the dataset containing all ISP’s.

5.2 Baseline policy comparison 5 RESULTS

Figure 7: The download speed of data for all network technologies, grouped by operator.

Figure 8: CCDF for the same dataset as above.

From these graphs we can conclude that the dataset made up of all ISP’s contain a consid-erable number of unreasonably high measurements. We see that all of the four mobile operators seem to have more realistic numbers. Based on this we have made the decision to filter our dataset and only use the data produced using the four mobile operators. This hopefully elimi-nates most of the anomalies in the dataset and give us results that are closer to reality.

5.2

Baseline policy comparison

With our data filtered we can begin running simulations and comparisons of our exchange policies. We started by running simulations using our reference cases in order to get a baseline we can compare our policies with. The results are visualized in Figures 9(a) to 9(d). These graphs show a comparison between the predictions made based on the dataset resulting from our exchange simulation, as explained in Section 4.5, and a theoretical ”oracle”. This ”oracle” will always make the perfect prediction, meaning it will always choose the download point with the highest download speed. The horizontal axis represents the number of download points used. So ”3/11” means that three download points are used, with the resulting average speed shown on the vertical axis being the average of these three points as described in section 4.5. On the line showing the oracle in these graphs, we also show a 95% confidence interval. This means that if we pick a random value out of our 50 simulations, there is a 95% probability that the value is within the shown interval.

5.2 Baseline policy comparison 5 RESULTS

(a) All data (b) 10% random data

(c) 5% random data (d) 1% random data

Figure 9: Differences between predictions using different amount of data and an ”oracle”.

From these graphs we can see that ”all data”, ”10% data” and ”5% data” results in fairly similar speeds when using only one download point, and ”1% data” resulting in the lowest speed. When using more than one download point, we see that all of the cases presents results that are fairly similar. To give a better overview of these four graphs we present a graph combining these results in Figure 10.

Figure 10: Shows differences from perfect prediction for our reference cases.

Figure 10 shows a combination of the four oracle graphs above but in a different way than before. Here we can see how far away the different predictions with our reference cases are from the perfect oracle. We can see that for prediction in one square ”all data”, ”10% data” and ”5% data” is the same. ”1% data” results in worse results, which we think is natural. We can also see that for downloading in two or more squares, ”all data” presents somewhat better results than the others, but the difference is not big. The other three reference cases present results that are fairly similar. To create a picture of how large the dataset are, the average square holds 78 measurement when 10% of the data is used.

5.3 Operator-aware policies 5 RESULTS

5.3

Operator-aware policies

Figure 11 illustrates the similarities and differences between the four mobile operators we have focused on in this thesis. This experiment was based on all data in the dataset. The first obser-vation we made was that all of the predictions made by filtering out different mobile operators were better than the previous example with a certain percentege of the data. We also observed that Telenor performed worse than the three others and Telia has the most stable prediction curve with a constantly decreasing distance to the perfect prediction for each download point used.

These observations shows that it is better to use a more restrictive policy, which provide more relevant data. The anomaly with Telenor we believe could have a statistical explanation. We have noticed that out of these four, this is the operator that has the least number of measurements. Because of this, the dataset derived using Telenor as carrier might be too small to make good predictions.

Figure 11: A comparison between predictions made with ”same operator” policy for different carriers.

5.4

Head-to-head comparison

Figure 12 shows a comparison of the different exchange policies we have defined. The graph shows how far off from a perfect prediction each of the policies are, measured in percent. The simulations were carried out with a ”simulation-person” who uses Telia as operator and 3G as network technology. This means that our simulated speeds for the experiments comes from a dataset limited to Telia and 3G, to allow our simulations to be more realistic.

5.5 Comparison with entire dataset 5 RESULTS

From the figure we can see that our three defined policies perform quite similarly, while both ”full sharing” and ”no sharing” results in significally worse predictions. We can see that ”restricted sharing” has a slightly better prediction than the other two policies when using only one download point. This means that ”restricted sharing”, on average, makes a slightly better prediction as to which download point will give the best download speed.

We can also see that ”full sharing” gives significantly worse predictions compared to our policies. From this observation we can conclude that the relevance of the data is a lot more important than the amount of data when making this kind of prediction.

Figure 13: Difference between predictions made using the defined policies for 4G and a perfect prediction.

Figure 13 shows the same comparison as in Figure 12, but our ”simulation-person” uses 4G instead of 3G. Telia is still used as the operator. The results are similar to the results with 3G, all three exchange policies performs much better than ”full sharing” and ”no sharing”, with ”restricted sharing” resulting in the best prediction.

One big difference from the 3G comparison is that the ”same operator sharing” policy makes a noticably worse prediction when using one download point compared to the other two policies. The reason for this might be that the dataset on which the prediction is based on contains a majority of 3G measurements, since the ”same operator sharing” does not filter on network technology. If the location with the highest average speed over 3G differs from the one with the highest 4G speed, which is not unlikely, this could very well lead to ”same operator sharing” making the wrong prediction for the best location.

To compare the results of our exchange policies to a central based approach we can compare the two previous figures to the results for our reference cases in Figure 10. The reference case called ”All data” could be seen as using a central server with knowledge of all measurements. We can see that the distance from a perfect prediction for ”All data” is over 100% when using two or fewer download points. The same difference for our exchange policies in Figures 12 and 13 is considerably smaller with the exception ”same operator sharing” which for one download point with the network technology 4G, is worse. This would imply that our exchange policies result in better predictions than a central server approach.

5.5

Comparison with entire dataset

To verify that the results we have produced are reasonable and applicable for other scenarios than the path we selected for this thesis, we have ran a number of simulations to see the deviation of the predictions. The results from these simulations were compiled into boxplots. These plots were produced by reproducing the previous simulations for 20 different scenarios, where each scenario consists of 11 squares picked at random from the entire original area of Stockholm. Predictions were made using each of the presented policies and the results are shown in Figures 14 to 16. The first graph shows results when picking one square to perform the download in. Figure 15 shows the result when using four download points and Figure 16 shows the result when using eight download points.

5.5 Comparison with entire dataset 5 RESULTS

Figure 14: Boxplot showing deviation from perfect prediction for 1 of 11 squares.

Figure 15: Boxplot showing deviation from perfect prediction for 4 of 11 squares.

Figure 16: Boxplot showing deviation from perfect prediction for 8 of 11 squares.

Looking at the boxplots we can see that the results from our thesis is recognizable here too. Our three policies produces significantly better results than both ”full sharing” and ”no sharing”, which is consistent with our previous results. These plots shows that our results are representative for not only our originally chosen squares but also for the whole dataset.

6 DISCUSSION

6

Discussion

This chapter mainly discuss two parts of the thesis. The result and the method which has been used to get them. However, we also discuss and present limitations that has been made through-out this thesis.

We made the biggest limitations of this thesis very early in the writing process. This limita-tions was that we decided to only use measurements from a certain area in Stockholm. On top of that we threw away all measurements that were made with WiFi as network technology. We also threw away all measurements that was done before 2014. This was beacuse we only wanted to use measurements with the latest technology, and since the technology evolves very quickly we chose this rather restrictive limit. This left us with a little over 100 000 measurements in an area as big as 400 square kilometres. This area is shown in Figure 1. Although the area is big we have in this thesis focused on a path of squares that is shown in Figure 2.

In this thesis we have only investigated how different exchange policies could affect network predictions in a crowd-based system. How the data is shared and stored is not something we have taken into consideration for this thesis. If the system were to be implemented in real life, this is naturally something that has to be solved for the system to work.

6.1

Results

Our thesis shows, through numerous simulations and experiments, that a prediction made with a lot of unfiltered data is going to be worse than a prediction done with a smaller amount of more relevant data. This becomes very clear when looking at Figures 12 and 13, where the three more restrictive exchange policies outperform the non-restrictive ”full sharing” policy. This is also observed when comparing Figure 10, where predictions are made with different amounts of unfiltered data, with Figure 11, where predictions are made with data filtered by its mobile operators.

In Figure 10 we show differences between predictions made with different amount of unfiltered data. In this graph we can see how big impact the amount of data has on the prediction. We can see that when downloads are made in two or more squares ”all data” is better than the others. The others performs similar results, with ”1% data” being a bit worse when downloading in one square. This means that if one are to download in two or more squares there is no need to exchange large amounts of data since it takes up unnecessary storage space and transmission overhead.

Even though we can see significant differences between predictions made with the exchange policies and the predictions made by unfiltered data, there is still work to do on the predictions. In Figure 13 the differences between the exchange policies are displayed. We can observe that ”restricted sharing” is the best of the policies, with ”same technology sharing” and ”same operator sharing” close behind, when downloading in more than one download point. This means that ”restricted sharing” should be used when it is possible, in order to keep the number of exchanges to a minimum.

6.2

Methodology

We have used a measurement and data driven method. The decision to base the experiments on the big dataset from Bredbandskollen was crucial for this thesis to happen. If we were to collect all the data for ourselves it would have taken too much time. This also makes sure that our experiments are using realistic data that originates from real-world measurements. Something that makes the experiments more reliable is that they are based on a big dataset. This makes the averages realistic and credible. Another part of the method that further improves the results of this thesis is our software we use for simulations. This software allows us to do a number of simulations in a short amount of time. This means that we can repeat our experiments several times and in that way confirm our results. This also increases this thesis reliability, because if someone were to do the same experiments as we have done in the making of this thesis, they could expect to get very similar results as we do.

6.3 Future work and possibilities 6 DISCUSSION

We think it was a good choice to limit the area of interest to a smaller area than the entirety of Sweden, in this case we limited it to central Stockholm. One could argue that we throw away a lot of measurements, which is true. However, the results of the experiments are difficult to interpret if not enough data is used as basis. For this reason it is wise to use a big city as basis if similar experiments is to be made.

Despite being able to run the simulations several times and the fact that the simulations operates on a big dataset, our exchange policies are still based on a number of assumptions. These assumptions might make the results differ from reality. This might affect the validity of this report negatively since we cannot be sure that the exchange policies we have simulated in theory, actually works in practice.

6.3

Future work and possibilities

This thesis has presented a number of different exchange policies and shown that they provide a significant improvement compared to using random data. These results and the data presented could be used in different ways and further improved.

In our work, we have limited ourselves to a single path. Similar simulations could be per-formed on a number of locations and paths, preferably in areas with different coverage and a mix of urban and more rural areas.

Our location squares were 1km x 1km in size, which could lead to some misleading results if applied in reality. A square of this size might contain areas with different degrees of coverage. It would certainly be possible to look at smaller squares, allowing for more localized and more accurate predictions.

The policies presented have only been evaulated and compared using simulations. While this gives some representation of reality, it would be very interesting to recreate the comparison using real-life experiments. Performing these exchanges in real-life situations would result in a dataset much more representative of reality to base predictions on. The results from a real-life implementation could then be compared to our results to evaluate the accuracy of our simulations. Another interesting possibility would be to use a more refined network prediction method, for example using more advanced statistical models, and see how the results differ from the results we produced with our very simple prediction method.

6.4

Ethical considerations

Since this thesis is based on a dataset comprised of real world measurements, privacy and anonymity must be considered. It is very important for us that the confidentiality of the data remains intact. The dataset provided by Bredbandskollen was anonymised by removing all IP-addresses, to prevent any possibility of identifying specific persons. While working on this thesis, no data has been used by the authors to identify the specific user behind any measurement. Privacy and anonymity has also been considered in the presentation of data and our results in this thesis. All results are presented as aggregated data to eliminate the possibility of identifying specific measurements.

Also if this type of crowd-sourced system was to be implemented in reality, the privacy of the users would have to be considered. Data sent between users might have to be anonymized to prevent anyone from identifying a single user and for example finding out specific mobility patterns or common locations for this user.

7 CONCLUSIONS

7

Conclusions

This thesis presents a comparison between different exchange policies. By looking at the results of this thesis we can draw a number of conclusion. We can see that both ”full sharing” and ”no sharing” results in worse predictions than the policies we have defined. From this we make the conclusion that you need a more restricted policy than these to make any decent predictions. We can also see that there is a very little difference between the policies ”same technology sharing”, ”same operator sharing” and ”restricted sharing”. We conclude that, in a big city, it is not so important what operator you have, it is the internet technology that matters the most. We recommend to use ”restricted sharing” if the possible data exchanges in the area allows it, since this policy limits the number of transmissions compared to ”same technology sharing”, which in turn limits energy usage. We also recommend that if you are in areas where less possible exchanges is expected you should use ”same technology sharing” or ”same operator sharing” rather than ”restricted sharing” in order to get as many data exchanges as possible.

REFERENCES REFERENCES

References

[1] A. G. Est´evez and N. Carlsson. “Geo-location-aware Emulations for Performance Evalu-ation of Mobile ApplicEvalu-ations”. In: Proceedings of the IEEE/IFIP Conference on Wireless On-demand Network Systems and Services (WONS). Obergurgl, Austria, Apr. 2014. [2] J. Yao, S. S. Kanhere, and M. Hassan. “Improving QoS in High-Speed Mobility Using

Bandwidth Maps”. In: IEEE Transactions on Mobile Computing 11.4 (Apr. 2012), pp. 603– 617.

[3] A. Gerber, J. Pang, O. Spatschek, and S. Venkataraman. “Speed testing without speed tests: estimating achievable download speed from passive measurements”. In: Proceedings of the ACM Internet Measurement Conference (IMC). Melbourne, Australia, Nov. 2010. [4] I. D. Curcio, V. K. M. Vadakital, and M. M. Hannuksela. “Geo-predictive Real-time

Me-dia Delivery in Mobile Environment”. In: Proceedings of the Workshop on Mobile Video Delivery (MoViD). Firenze, Italy, Oct. 2010.

[5] J. Yao, S. S. Kunhere, and M. Hassan. “Geo-intelligent Traffic Scheduling for Multi-Homed On-Board Networks”. In: Proceedings of the Workshop on Mobility in the Evolving Internet Architecture (MobiArch). Krakow, Poland, June 2009.

[6] K. Højgaard-Hansen, T. K. Madsen, and H.-P. Schwefel. “Reducing Communication Over-head by Scheduling TCP Transfers on Mobile Devices using Wireless Network Performance Maps”. In: Proceedings of European Wireless. Poznan, Poland, Apr. 2012.

[7] T. P¨ogel and L. Wolf. “Prediction of 3G Network Characteristics for Adaptive Vehicular Connectivity Maps”. In: Proceedings of the IEEE Vehicular Networking Conference (VNC). Seoul, South Korea, Nov. 2012.

[8] E. Gregori, L. Lenzini, V. Luconi, and A. Vecchio. “Sensing the Internet through crowd-sourcing”. In: Proceedings of the IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops). San Diego, CA, Mar. 2013. [9] K. Evensen, A. Petlund, H. Riiser, P. Vigmostad, D. Kaspar, C. Griwodz, and P. Halvorsen.

“Mobile Video Streaming Using Location-Based Network Prediction and Transparent Han-dover”. In: Proceedings of the ACM international workshop on Network and operating sys-tems support for digital audio and video (NOSSDAV). Vancouver, BC, Canada, June 2011. [10] D. R. Choffnes, F. E. Bustamante, and Z. Ge. “Crowdsourcing Service-Level Network Event

Monitoring”. In: Proceedings of ACM SIGCOMM. New Delhi, India, Sept. 2010.

[11] M. Arlitt, N. Carlsson, C. Williamson, and J. Rolia. “Passive Crowd-Based Monitoring of World Wide Web Infrastructure and its Performance”. In: Proceedings of the IEEE International Conference on Communications (ICC). Ontario, ON, Canada, June 2012. [12] J. Sommers and P. Barford. “Cell vs. WiFi: on the performance of metro area mobile

con-nections”. In: Proceedings of the ACM Internet Measurement Conference (IMC). Boston, MA, Nov. 2012.

[13] G. Murtaza, A. Reinhardt, M. Hassan, and S. S. Kanhere. “Creating personal bandwidth maps using opportunistic throughput measurements”. In: Proceedings of the IEEE Inter-national Conference on Communications (ICC). Sydney, Australia, June 2014.

[14] W. L. Tan, F. Lam, and W. C. Lau. “An Empirical Study on 3G Network Capacity and Performance”. In: Proceedings of the IEEE International Conference on Computer Com-munications (INFOCOM). Anchorage, AK, May 2007.

[15] A. Faggiani, E. Gregori, L. Lenzini, V. Luconi, and A. Vecchio. “Smartphone-based crowd-sourcing for network monitoring: Opportunities, challenges, and a case study”. In: IEEE Communications Magazine 52.1 (Jan. 2014), pp. 106–113.

P˚

a svenska

Detta dokument h˚alls tillg¨angligt p˚a Internet – eller dess framtida ers¨attare – under en l¨angre tid fr˚an publiceringsdatum under f¨oruts¨attning att inga extra-ordin¨ara omst¨andigheter uppst˚ar. Tillg˚ang till dokumentet inneb¨ar tillst˚and f¨or var och en att l¨asa, ladda ner, skriva ut en-staka kopior f¨or enskilt bruk och att anv¨anda det of¨or¨andrat f¨or ickekommersiell forskning och f¨or undervisning. ¨Overf¨oring av upphovs-r¨atten vid en senare tidpunkt kan inte upph¨ava detta tillst˚and. All annan anv¨andning av dokumentet kr¨aver upphovsmannens medgivande. F¨or att garantera ¨aktheten, s¨akerheten och tillg¨angligheten finns det l¨osningar av teknisk och adminis-trativ art.

Upphovsmannens ideella r¨att innefattar r¨att att bli n¨amnd som upphovsman i den omfattning som god sed kr¨aver vid anv¨andning av dokumentet p˚a ovan beskrivna s¨att samt skydd mot att dokumentet ¨andras eller presenteras i s˚adan form eller i s˚adant sammanhang som ¨ar kr¨ankande f¨or upphovsmannens litter¨ara eller konstn¨arliga anseende eller egenart.

F¨or ytterligare information om Link¨oping University Electronic Press se f¨orlagets hemsida http://www.ep.liu.se/

In English

The publishers will keep this document online on the Internet – or its possible replacement – for a considerable time from the date of publication barring exceptional circumstances.

The online availability of the document implies a permanent permission for anyone to read, to download, to print out single copies for your own use and to use it unchanged for any non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional on the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility.

According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement.

For additional information about the Link¨oping University Electronic Press and its proce-dures for publication and for assurance of document integrity, please refer to its WWW home page: http://www.ep.liu.se/

c