Software engineering Thesis no: BSE-2019-0? 2019
Comparison of Video file transmission
over Dat protocol and Hypertext transfer protocol
Jonathan Artback
Dept. Computer Science & Engineering Blekinge Institute of Technology
This thesis is submitted to the Department of Computer Science & Engineering at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Bachelor of Science in Software engineering. The thesis is equivalent to 20 weeks of half-time studies.
Contact Information: Author(s):
Jonathan Artback
E-mail: jodm15@student.bth.se
University advisor:
Dr. Ahmad Nauman Ghazi
Department of Software Engineering
Dept. Computer Science & Engineering Internet : www.bth.se Blekinge Institute of Technology Phone : +46 455 38 50 00 SE–371 79 Karlskrona, Sweden Fax : +46 455 38 50 57
Abstract
Context. Scalability and bittrate is an large concern when build-ing infrastructure and application for video content delivery. Many research and companies have for a long time looked at peer to peer technology as a way of solving the issue by utilizing the clients capac-ity.
Objectives. The first objectives is to see if Dat protocol would be a good fit for video file delivery. Comparing with the widely used HTTP too see the differences in scalability and bitrate. The challenges with implementing such a solution and tactic for the future if choosing to. Methods. In order to achieve the objectives of our research a compar-ing network experiment in similar manners and with the same client base was conducted.
Conclusions. The Dat protocol show good scalability and perfor-mance especially on larger number of clients compared to HTTP. Some future works on Dat protocol such as peer-selecting and load balancing are needed before Dat can be used as protocol for video delivery on commercial sites.
Contents
Abstract i
1 Introduction 1
1.1 Content delivery network . . . 2
1.2 Peer to peer . . . 2
1.3 Research Aim . . . 2
1.4 Objectives . . . 3
1.5 Problem domain . . . 4
1.6 Research questions and motivation . . . 5
2 Background 7 2.1 Types of P2P network . . . 8 2.2 Topology of P2P network . . . 8 2.3 Dat protocol . . . 8 2.3.1 Dat procedure . . . 9 2.3.2 HTTP . . . 10
2.4 Benefits of Peer to peer technology . . . 11
2.5 Flaws of Peer to peer technology . . . 12
2.6 Security in peer to peer . . . 12 2.7 HTTP vs Dat . . . 12 3 Related Work 14 4 Method 17 4.1 Research Methodology . . . 17 4.2 Prerequisites . . . 17 4.3 Variables . . . 18 4.3.1 Dependent Variables . . . 18 4.3.2 Independent Variables . . . 19 4.3.3 Extraneous Variables . . . 20 4.4 Procedure . . . 20 4.4.1 HTTP . . . 20
4.4.2 Dat peer to peer . . . 21
5 Results 24 6 Analysis and discussion 27 6.1 Analysis of Research Q1 . . . 27 6.2 Analysis of Research Q2 . . . 28 6.3 Analysis of Research Q3 . . . 28 6.4 Discussion of Research Q1 . . . 29 6.5 Discussion of Research Q2 . . . 29 6.6 Discussion of Research Q3 . . . 30 6.7 Threats to validity . . . 31 iii
7 Conclusions and Future Work 32
7.1 Conclusion . . . 32
7.2 Future work . . . 33
References 34
Appendices 38
A Result of network expriement 39
Chapter 1
Introduction
Video content delivery on sites such as YouTube, Vimeo, and Netflix is a huge part of the internet today. The volume of video content traffic on the internet just keeps growing for every year[1]. With higher video resolutions and more advanced image technologies, it is a development that won’t cease in the nearest future[2]. To keep the clients happy when doing "video content providing" consistent quality of service is very important[3]. In a regular client-server type of network, this puts huge pressure on the server to handle the massive loads. Often a form of content delivery network is used to help the server. hypertext transfer protocol(HTTP) is a very common application layer protocol and also often used for video content. When used for video content delivery it is often combined with MPEG-dash and HLS which are protocols for chunking video streams and adopting the video bandwidth to the network. A technique not covered in this thesis.
Chapter 1. Introduction 2
1.1
Content delivery network
A content delivery network is a subset of servers located on multiple data centers. The servers fetch files from each other and cache them locally. This speeds up the experience for the end user as they can fetch from a server closer to them[4]. A CDN could also be partly supported by a network where end users can download from each other so-called peer to peer[5].
1.2
Peer to peer
Peer to peer(P2P) networks have for long times attracted large attention both in industry and Academics for various reasons. Peer to peer technology solves the scalability issue of content delivery in theory by levering the resources of the participating peers[6]. That has been proved in projects such as P2Plive where a 10 Mbps server bandwidth served 400 Kbps video streams to 1.5million users[7].
1.3
Research Aim
The aim of this report was to tell if Dat protocol can be a good replacement protocol for centralized protocols used for video streaming today most commonly
Chapter 1. Introduction 3 HTTP. The issue with scalability has been a problem that has been in focus of research for a long time. It motivates me to see if with open decentralized technology can serve more clients and faster connections without added costs. Bypassing political censorship on the web is also much easier with p2p technology as there is not one but many servers.
1.4
Objectives
The aim of this experiment and research is to learn more about DAT(P2P), HTTP(Centralized) and how their scalability work. Given below are the objec-tives which will help us achieve the required outcome.
• To identify the differences in bandwidth when scaling the different network protocols in similar manners.
• To analyze the reason for differences between the two technologies in re-spect to variables such as number of clients, network upload and download bandwidth.
• To identify the possibility of using Dat as an alternative for solving scala-bility issues in content and video delivery applications compared to existing HTTP solutions.
Chapter 1. Introduction 4
1.5
Problem domain
Centralized client-server are often used to serve video content on the web today. A big concern with client-server is the scalability. The server needs to grow its capacity with more users. Often a CDN is used. A CDN could provide a better quality to end-users and scale pass the limits of the source server when the workload is within the provisioning limits. A centralized based CDN is expensive to deploy and maintain[8]. This type of CDN is often owned by a large cloud corporation such as AWS CloudFront. The cost of using AWS Cloudfront is $850 for 10000 users streaming 1GB(as of 2019-04-06). P2P could achieve high scalability with low server requirements. It could lower the server cost and lower the need for advertising to support the hosting of websites. It could also be a profit for the climate as servers use a lot of electricity.
Chapter 1. Introduction 5
1.6
Research questions and motivation
• Q1 How or will Dat get a higher maximum bitrate compared to HTTP client-server infrastructure for video content providing(Mbit/s)?
Motivation: Hypertext Transfer Protocol is an Application protocol for distributed content and is part of the standardized World Wide Web[9]. Although no reliable study can show which internet protocol is the most used on internet, HTTP is a very common protocol and used for most websites today. This is the protocol used on sites such as Google, YouTube and Vimeo. A http site can be distinguished by the http:// or https:// in front of the Uniform Resource Locator(URL). https is the secure/encrypted version of http.
Video streaming traffic is the biggest part of internet traffic and puts great importance on high bandwidth to delivery the videos that keep growing in quality and size.
This question will give an understanding of what difference the Dat protocol will do compared to HTTP client-server system. It can also help content providers decide if a rebuild for using DAT protocol will be worth it in terms of scalability. Without considering that none of the widely used web browsers yet has implemented Dat protocol.
• Q2 Can the same number of clients watch higher quality content which they can when the bitrate gets higher using Dat protocol compared to HTTP client-server infrastructure for on-demand video content providing?
Motivation: This research question helps us identify the scalability differ-ences in HTTP as a centralized content providing technology and Dat as a P2P technology. It is costly to scale to large user-bases on purely centralized systems.
With this question, we can identify if a Dat implementation will benefit the end users in higher video qualities on the same server hardware and network.
The answer to this question will show if there is any idea in continue consid-ering Dat an alternative in the video contents case. Or if researchers should look at other technologies or continue developing and using HTTP.
Chapter 1. Introduction 6 • Q3 How will the maximum video content quality/ maximum bitrate that can be sent rise with the number of seeders per video? In other words "how do scalabilty work for Dat protocol and HTTP and will get better performance with with more users?"
Motivation: The motivation between this question is to see what and if end users will benefit from using a decentralized technology vs a centralized technology and how it correlates to what bandwidth the network is deliver-ing(upload and download) and also the number of users/seeders. A seeder is a user or someone who already downloaded parts of the file and are now resharing it to other users.
Chapter 2
Background
According to a study made by Cisco, video streaming will stand for 82% of all the internet traffic 2022[10]. The most common architecture model for content delivery on the internet is the client-server model. The client is requesting data from a powerful server that respond with the requested data[11]. With this model more clients give a higher demand and server load and the server resources need to scale up.
Peer to peer networks is another architecture model. In peer to peer networks, clients can request and download from other clients who have the content. A great advantage with peer to peer networks is that when scaled with more clients its capacity grows.This is because of the networks ability to use clients/peers as servers. By doing so the combined uploading bandwidth in the network also grows with more users. It’s also more resilient in case of failures as without single server that can be a single point of failure. The data can be downloaded from alternative peers if one peer goes down[11].
Using peer to peer technology for video is very common in illegal file sharing. Often with the BitTorrent protocol, examples are Thepiratebay and 1337x and commonly used tools are UTorrent and Transmission. It has also been discussed to use peer to peer technology for legal sites such as Netflix[12].
Chapter 2. Background 8
2.1
Types of P2P network
• Pure P2P network means a network which has now central service of any kind[13]. That means that the discovery phase, file sharing, and commu-nication is done completely without servers. The benefits of this being a very robust network as long as peers are connected. Building a pure P2P network is quite uncommon as the peers need to send a lot of control mes-sages to keep track of other peers. Example of pure P2P are Freenet[14] and Gnutella[15].
• Hybrid P2P network on the other hand depends partially on a central server but the peers download direct between each other[13]. Often time the server maintains a registry of the peers and the files in the network. That gives the creator a larger control of the data and less overhead data meaning better performance. But the control server is also a single point of failure. As without it the peers can’t find each other[16].
2.2
Topology of P2P network
The topology in the P2P network is about how the clients are connecting to each other. A tree-based topology has parent nodes and child node. A parent is one that forwards data and a receiving node are called a child node. A tree-based topology can have many levels meaning that a node can be both parent and child[17].
A mesh network is described as an unstructured network with no specific data delivery order. Mesh topology does not have the problem with constant repairs as it does not have to retain a specific order between nodes. A node in a mesh network maintains a connection with multiple neighbor peers[18].
2.3
Dat protocol
Dat protocol is a relatively new peer to peer decentralized web protocol found in 2013. It was made to share data between computers and was made initially made for researchers and activists but also for application creators[19]. Dat also exist as more of a utility for the end user based on many different libraries and technologies. It exist as a command line interface that can be used to share files between computers and there is also a browser called Beaker browser that can run Dat websites.
Chapter 2. Background 9 Dat can use TCP or μTP as a transport protocol. μTP(Micro Transport Protocol) is a UDP based protocol(experimental) that has better congestion control than TCP.TCP is The default is TCP. [20].
2.3.1
Dat procedure
The procedural steps for a file share with Dat are[21]:
Discovery
HyperDiscovery
Discovery keys are used for finding other peers who are interested in the same Dat feed as you[21]. The discovery keys are hashed based on the BLAKE2b function[22]. In a local area, network peers send a broadcast for the data they are interested in. This technique is called multicast DNS and works like regular DNS expect using local broadcasts.
In a wide area network, DNS is used and for clients to stay connected the need to re-announce themselves every 60 seconds. It is a centralized technique and Dat has a dedicated server running on discovery1.datprotocol.com and a fallback on discovery2.datprotocol.com.
The strength of using DNS is that it is quick to search and find other peers. The weakness lays in the centralization of the DNS table. It’s a central point of failure and the DNS can see everyone’s metadata.[?] Meaning it can be a target point for hacker attacks. This means when in Local network Dat act more as a Pure P2P and on WAN it acts as a hybrid P2P. Source discovery is an ongoing process everyone having one chunk or more equals a source.
Exchanging data
Hypercore
HyperCore is a library in the Dat protocol used to replicate feeds of binary data. HyperCore is the low-level library that Hyperdrive is built on top of[23]. HyperCore is public key based. Meaning that everyone with the public key can download the content and redistribute the content but only the one with the
Chapter 2. Background 10 private key can edit the content[21]. In Dat this public key is the Uniform Resource Locator or link that is used to locate the Dat content.
How it works
1. Breaks file content into pieces called chunks. Fixed size chunks, default is 16KB
2. Hashes every piece
3. Create a Merkle tree. A tree structure where every leaf is a data block hash and every non-leaf is labeled with the cryptographic hash. The top node hash is the Dat link. This is also the technique used to verify transactions in bitcoin[24].
Files and folders
HyperDrive
HyperDrive is a library for replicating files and folders structures. It uses two coupled feeds to represent files and folders. One is the metadata feed it contains names, sizes and other metadata for each file. Then it has a content feed which contains the actual files contents. The metadata feed points to where in the content feed each file is located. This is so that it only needs to fetch the contents of the files that are interesting[21].
2.3.2
HTTP
HTTP is a application-level protocol for hypermedia information and has been used by the world wide web since 1990[9]. HTTP is a client-server protocol and it is the client who establish the connection.
The procedure for HTTP file share, also called session is[25]:
Establishing a connection
In client-server protocols the clients establish the connection. This means first off all initiating a connection in the lower transport layer.Usually this is TCP. The Uniform Resource Locator for the domain name, and the port number. The port number can be omitted if using the standard port 80[25].
Chapter 2. Background 11 Sending request
When connection on transport level is established the user can send a request. This is done by sending a request containing 3 blocks. First the requested path to the document, Then a header containing options such as language, filetype and other data altering behaviors. The final block is the optional data that can be included in the request. Mostly used for POST request when a user want to upload data to the server.
There is many different methods for the http protocol such as: GET which is used for retrieving data from the server.
POST which is used to post content to the server. DELETE which is used to delete data on the server.
Server response
The server receives the request process it and respond. The response contains of a standardized status message telling the user-agent(browser) how the request went.The response also contains of headers and an optional data block.
2.4
Benefits of Peer to peer technology
Distributing a video on centralized technology puts a lot of pressure on the server and could lead to overload and scaling issues. With peer to peer, scaling is solved by using the user’s computers as peers and have them share the content they have previously downloaded.
It could speed up the download time and lower the network load as the stream of the file can be taken from others in your local network or from the surroundings rather than a server far away. The viewing patterns of users in regions is often similar as humans tend to mimic the surrounding. This is something that is concluded by Huan Yan... they state that the views of the top 10% gets 80% of the views in a city[26].
The peer to peer network could have the features of a caching system without the need of the ISP or content provider having to pay for expensive servers. The users need for fast network applications will grow. So will the power and network
Chapter 2. Background 12 speed of the end users. So why not lend some of their capacity!
2.5
Flaws of Peer to peer technology
One security concern in a P2P network is called poisoning. That a peer introduces malicious data or inserts false records in the index table leading other unaware users to the wrong result[27].
Another problem as it is decentralized that without servers that are controlled by the service provider, peer to peer technology is trusting in the users(peers) to provide and scale the network for other users. Meaning the owner has none to little control of the state of the network and that it actually provides the needed quality to keep users happy. In "Understanding hybrid CDN-P2P", peer to peer is described as a self scaling but it is apposed to more reliable and maintainable advantages of CDN and client-server.[28]
2.6
Security in peer to peer
One way of protecting network traffic and make it less vulnerable to unwanted changes or man in the middle attacks[29] is encryption[30]. Encryption is one of the features of Hypercore[31].
Hypercore is also build on binary append-only streams. Which means that data can’t be changed or removed from a Dat stream only added by the owner[32]. That could possibly make poisoning attacks impossible.
2.7
HTTP vs Dat
Comparison in similarity and differences.
HTTP has a centralized architecture where request are sent to the server. Either a domain name or IP address has to be provided in the request so that the protocol knows where to send the request[25].
Dat is a decentralized architecture where requests are announced in the network for other peers to respond to.
Chapter 2. Background 13 Once a peer is discovered Dat will open a connection to the other peer to enable the file share. This is similar to how HTTP send a request to a server[?].
Both protocols use TCP are most commonly using TCP on the transport layer but this is not a requirement. Dat can also use micro datagram protocol, a User Datagram Protocol(UDP based protocol.[?] HTTP only requirement is a stable connection that don’t lose messages. Google have been experimenting with UDP transport protocol called QUIC combined with HTTP which gives a more efficient experience and less round-trips[25].
Both protocols allow for fetching separate resources from the same URL by spec-ifying the resource/file name with the hostname/Dat key.
Dat has built in versioning making it easy to handle data history something that pure HTTP does not[33]. Dat can also handle partly downloading a resource[?]. HTTP can solve this with query parameters[25].
HTTP is part of the World Wide Web and is supported by all browsers and web servers. Dat is only supported by there own libraries and experimental browsers such as Beaker browser[21].
Chapter 3
Related Work
This chapter presents recent related work. Search words include: Peer to peer, Video on demand, Hybrid peer to peer, P2P, CDN, and content delivery network P2Cast discusses the problem with unicast client services and how to design an overlay peer to peer network for glitch-free VoD(video on demand) streaming [34]. The paper describes the problem with server bottlenecking in client-server VoD services. They discuss the problem with scalability in client-server and that the server is the only "contributor". All clients connect and consume a part of the bandwidth leading eventually to slow connections and request rejections. They develop an algorithm called Best fit(BF) algorithm that is used to construct a streaming delivery tree of multicast. P2Cast then uses peer to peer technology to serve the missing parts to an arriving client, this is called a patch. The com-bination of multicast and peer to peer technology shows much better scalability than client-server and lowers the server-load.
The problem with scalability in video on demand and peer to peer alternatives have been discussed in recent works. "Streaming Live Media over Peers" discusses the limitation of clients regarding the high bandwidth need for video streaming, They evaluate a tree-based overlay network called PeerCast[35]. They do a similar study to this although the type of peer to peer network they are building is different. They are comparing on more metrics that could be interesting such as packet loss and join/leave policies. They focus on tree-based peer to peer network topology.
A performance comparison was written on the differences in mesh and tree-based topology[17]. The performance of the tree-based topology is drastically reduced by a high churn rate. Meaning the rate at which nodes leave the network. This is because it is a structured network. So that after a node left it needs to
Chapter 3. Related Work 15 ture to get back the order. The variables used are more focusing on the delay between an initiated download and that the users can start to watch. Based on an independent variable of video bitrate, a number of peers and source bandwidth. They conclude that overall the video end-to-end delay is smaller with the tree-based topology. They also state that the tree-tree-based model has fewer frame drops that the mesh based. Tree-based topology seems to be best fitted for when the peer churn is static. When conducted with dynamic peer churn on larger node networks the mesh network outperforms the tree network[17].
A study by Yang Guo proposes a directory-based multicast video service peer to peer protocol called DirectStream[36]. They address much of the same problems as this thesis such as scalability issues in Video-on-demand streaming. Their technology caches the video at the client. They have a more advanced caching system which only caches the most recently viewed content at the client. To restrict this is necessary because users will not be willing to serve other clients for an unforeseen future after watching the content. It will also fill their cache space. A directory service means a system that maps resources to network addresses and keeps tracks of the peers [37]. The Directory service which is proposed in Directstream is called AMDirectory and is an application-level multicast without the single point of failure of a centralized service. It employs multicast groups that users can join. They also implement proximity aware features.
In a paper by Venkata N. Padmanabhan, the problem domain of distributing streaming content is discussed. They consider the problems of overwhelmed cen-tralized servers and propose a cooperative client peer to peer video streaming service called CoopNet. CoopNet clients cache video content they have down-loaded recently much like DirectStream, these clients then redistribute to other clients. CoopNet is built for users that are not willing to redistribute content for more than a few minutes. The paper discusses problems such as quality on peer failures[38].
Other similar research projects are PPstream [39] and Joost both software for internet TV distribution[40]. Peer5 being a modern alternative to a Peer to peer hybrid CDN solution. It uses traditional CDN technology in combination with peer-assisting technology[5].
In "Understanding hybrid-P2P" the potential gains using hybrid CDN-P2P for the leading CDN companies Akamai and Limelight are discussed. The paper dis-cusses P2P as a popular alternative to CDN in recent years. They are mentioning recent works such as PPLive. Their approach is to combine CDN and P2P to get "the best of both worlds". They found that this hybrid approach reduces the cost of pure Content delivery networks. Their study is a large scale comparison of pure CDN and CDN-P2P with 1.25 million unique clients. They show that
Chapter 3. Related Work 16 hybrid-CDN could cut the server load by more than 2/3 even when the clients are only uploading a video while the play it. If cached videos are uploaded the benefit could be even higher. Their conclusion is that they believe the future will be dominated by hybrid-CDN because of the showed benefit[28].
Summary
Recent works on peer to peer and CDN show that scaling applications are a true problem in video content delivery. The literature study also showed that content delivering applications could benefit from peer to peer technology. The problem with delivering a good Quality of service is solved by using different peer-selecting algorithms. Combining P2P with multicast saves the server from having to serve multiple connections i.e. lowering the server load. The problematic with peers leaving the network and the best solution. The literature showed that the best fit topology depending on the churn rate (nodes leaving). Tree-based if churn rate is static or mesh network if the churn rate is large and dynamic.
Chapter 4
Method
4.1
Research Methodology
To answer the research question the best fit would be to make an empirical experiment comparing HTTP as a centralized technology and Dat protocol as a P2P protocol. This chapter will state the important variables for the experiment and decide on the data points as well as the procedure.
4.2
Prerequisites
All experiments were done using Linux operative system. The distribution of Linux and hardware are independent of the result.
The toolbox of FFmpeg will come to use with this experiment. It is a set of tools for converting, stream and read metadata of video files.
Wget is a tool for Linux for downloading content from the web. Used to download the video file from the HTTP server. The reason for not using curl is that the output was easier with Wget and besides that they are similar[41]. Wondershaper is a tool that can limit an adapter’s bandwidth. Used for the HTTP server. Create a virtual interface and run the server on[42].
wondershaper l o : 0 −d 1024 −u 1024 −d 1024
This command will limit the up- and download speed of virtual interface lo:0 to 1 Mbit/s. The example IP address will be 123.123.22.22 in the experiment.
Chapter 4. Method 18 Trickle is a tool for Linux for limiting bandwidth on a process. It simulates clients with a constraint on the upload and download speed. Used for HTTP(Wget)[43]. Netbrake is a tool for Linux for limiting bandwidth on a process. It simulates clients with a constraint on the upload and download speed. Used this when doing the P2P experiment[44].
The reason for using two different tools is that trickle didn’t work with node processes and Netbrake doesn’t work with Wget[45].
Use FFmpeg to find the bit rate of the video file which you are looking to stream. Write down the value that says bitrate.
FFmpeg −i <video−filename >
The video bitrate value is the minimum data value which the network will have to provide for streaming of that video file to not lag.
4.3
Variables
4.3.1
Dependent Variables
• Maximum bitrateThis value is the same as the maximum bitrate that can be transmitted by the network. This is the value which by comparison will answer the research questions Q1, Q2, and Q3.
Q1: The outcome of this variable on the experiments will be able to answer which network protocol can give the highest maximum bitrate and at what state that is true.
Q2: The outcome of this variable on the two technologies of the experiments at the same number of clients show which is performing better.
Q3: The outcome of this variable compared on how many clients are con-nected can show if more seeders give a different result.
Chapter 4. Method 19
4.3.2
Independent Variables
• Clients
The variables control how many simultaneous downloads there is in the network. It help answer Q2 and Q3.
• Seeders
The variables control how many seeders are in the network. P2P tech-nology needs to have some seeders that help the server redistribute the video file otherwise P2P losses its advantages. A seeder is a peer/com-puter who distribute content for others in a P2P network. This is a P2P only variable. Helps answer Q2 and Q3.
• Network bandwidth
Control the networks upload and download speed to see how the bitrate correlates with the speed of the network. The bandwidth is the same for upload and download.
Chapter 4. Method 20
4.3.3
Extraneous Variables
• bitrate of video file
The quality and size of the video file being viewed. It is an important value when streaming since a larger bitrate will need a larger band-width to be playable.
• System
What operative system and hardware is used • Software such as network manager, drivers
Software and versions for various task that are involved in the experi-ment
4.4
Procedure
For a peer to peer network to provide benefits in scalability it needs to have seeders. Therefore start half of the clients. Then have a sleep period of 10 seconds and start the next half of the client group. By doing that the first half of the client group can now be seeders for the other clients. In some of the experiments, the 10 seconds will be enough time for the first half of the clients to already finished downloading the video before the waiting clients start to download.
4.4.1
HTTP
The HTTP parts of the experiment are conducted with a server and one or many clients. We run the experiments on the virtual loopback device(localhost) of the computer and limit the network speed accordingly.
1. What: Create a node server. We use HTTP module from node
h t t p . c r e a t e S e r v e r ( f u n c t i o n ( r e q u e s t , r e s p o n s e ) { . . . } Why: The server will receive the request and respond with the given file
2. What: Create a read Stream for the video file
s o u r c e = f s . createReadStream ( ’ . . / v i d e o . mkv ’ ) ; Why: The file readstream is what we will send as response to the client
Chapter 4. Method 21 3. What: Pipe the video file stream and send it as a response
s o u r c e . p i p e ( r e s p o n s e ) ;
4. What: Set the server to listen on the limited loopback interface . l i s t e n ( 5 0 0 0 , ’ 1 2 3 . 1 2 3 . 2 2 . 2 2 ’ ) ;
5. What: start the node server. node s e r v e r . j s
6. What: Use Wget to spawn up the clients and output the result of each download.
wget 1 2 3 . 1 2 3 . 2 2 . 2 2 : 5 0 0 0 / −o <r e s u l t −f i l e > −O /dev/ n u l l
Why: We output the downloaded video file to /dev/null since for the sake of the experiment the only necessary is to see network bandwidth. The results file gives the average download speed in bytes for the video file this is the maximum bitrate at that network speed and number of clients. To make this step easier a script is recommended that spawn X amount of clients and collect the result.
4.4.2
Dat peer to peer
The peer to peer experiment has two types of processes running one is the server or Master peer the other is the peers. The server is the process of creating the Dat stream and reading in the video. Server / Master peer
1. What: Create a hypercore feed.
f e e d=h y p e r c o r e ( ram , { o v e r w r i t e : t r u e } ) ; 2. What: Add a callback that is ran when the feed is ready.
f e e d . on ( " ready " , f u n c t i o n ( ) { . . . }
3. What: Create a swarm with hyperdiscovery and connect it to the feed when feed is on.
swarm=h y p e r d i s c o v e r y ( f e e d ) ;
Chapter 4. Method 22 1. What: In the peer/client we call the HyperCore function but with the key
to the feed created by the server.
f e e d=h y p e r c o r e ( ram , <key >,{ o v e r w r i t e : t r u e } ) ; 2. What: Create a read stream from the feed
stream=f e e d . createReadStream ( { s t a r t : 0 , l i v e : t r u e } ) ; 3. What: Create a function which is called when a new chunk is downloaded
stream . on ( " data " , f u n c t i o n ( data ) { . . . } ) ;
Why: This is the part where the content is downloaded. We could save it to file or watch the content as it is downloaded. Here we could also record the download speed by recording the size and download speed.
Chapter 4. Method 23 The experiment will be run on 4 different network setups. The values are:
• 1000mbit/s server and 100mbit/s per client • 100mbit/s server and 100mbit/s per client • 100mbit/s server and 22.5mbit/s per client • 22.5mbit/s server and 22.5mbit/s per client
The values were chosen based on the common network speeds provided by in-ternet service providers on fiber networks. 22.5Mbit/s being used because of it being average connection speed for Sweden provided in a report from 2017 by Akamai[46].
All these network setups will be run on four different numbers of clients. • 1 client
• 2 clients • 10 clients • 20 clients
The reason for these values is that they correlate well with the network setup. If using a larger value, for example, many millions the need to raise the network bandwidth as otherwise, the HTTP experiment would take to a long time. At least in theory with client-server(HTTP) where all clients connect to one source/server. The maximum bitrate will be the bandwidth of the client if the client’s combined bandwidth is lower than the bandwidth of the server or else it will follow:
s e r v e r bandwidth / number o f c l i e n t s
So with millions of clients, we would also need millions of Megabit in server. Start half of the clients at time 0. Wait for 5seconds and start the other half of the clients. That is done to make it possible for P2P to have a few seeders oth-erwise P2P technology can’t give a benefit. Both HTTP and P2P are conducted similarly.
Chapter 5
Results
This chapter presents the result obtained from the experiment. The results in this chapter are taken from the figures presented in the Appendix.
Out of the four network setups we tested on we saw a rise in maximum bitrate for the client on three. The rise of benefits of using P2P was at highest when many concurrent clients were connected. At most, we had 20 clients. The expectation to this was the experiment with a 1000Mbit/s server bandwidth and 100Mbit/s clients in this HTTP outperformed Dat protocol as can be seen in figure 5.2. The result showed where there where more than one client connected and that the client’s download bandwidth was higher than the combined server bandwidth. In those cases, the result is a higher bitrate with Dat protocol than with HTTP.
Chapter 5. Results 25 1 2 10 20 0 10 20 30 23.7 13.1 9.5 4.9 20.4 11.2 2.1 1 Clients Mbit/second(Higher is b etter) P2P HTTP
Figure 5.1: 22.5Mbit/s Server and 22.5Mbit/s Clients bandwidth
With larger client groups where the server bandwidth is insufficient of providing each client their maximum bandwidth. The result in the HTTP test approxi-mately follows:
Chapter 5. Results 26 1 2 10 20 0 20 40 60 80 100 120 95.2 73.3 58.1 30.6 98.4 98.4 97.6 75.36 Clients Mbit/second(Higher is b etter) P2P HTTP
Figure 5.2: 1000Mbit/s Server and 100Mbit/s Clients bandwidth
This is not the case when the server have sufficient bandwidth to server all the clients. An example of this can be seen in the result of the 1000Mbit/s server and 100Mbit/s clients. Then the maximum bitrate will be a value close to the bandwidth of the client.
When the server can provide the maximum 100Mbit/s bandwidth to all clients, The P2P technology gives no benefit and its overhead gives a lower maximum bitrate.
Chapter 6
Analysis and discussion
In this first part of this chapter, we will analyze the answer and trying to answer the research question. After that, we will discuss the result in correlation with related works.
6.1
Analysis of Research Q1
"How or will we get a higher maximum bitrate using Dat compared to HTTP client-server infrastructure for on-demand video content providing (Mbit/s)" The data shows that when the server has sufficient upload bandwidth to serve the X number of clients with maximum bandwidth the bitrate for HTTP is higher. This is proven in Figure 5.2.
The opposite is true when the network is set up so that the server doesn’t have sufficient bandwidth to serve all the clients. e.g. 22.5Mbit/s server bandwidth for 10 contemporary clients with 22.5 Mbit/s bandwidth. That will theoretically give ∼ 2.5Mbit/s bitrate for the HTTP protocol where all clients are connected to the same server which is also proven by figure 5.1. In the same figure, we can see that for 10 clients P2P did have an average bitrate of 9.5Mbit/s. As the network condition is the same we can see that the scalability is better for the Dat protocol.
Starting 20 clients directly would cut the maximum bitrate in half but the 10 first clients are started 10 seconds before the other. So most of the data for these clients are downloaded before starting the next 10. As so we don’t have 20 simulations downloads and that gives us a better maximum bitrate.
Chapter 6. Analysis and discussion 28
6.2
Analysis of Research Q2
"Can the same number of clients watch higher quality content will using Dat pro-tocol compared to HTTP client-server infrastructure for on-demand video content providing?"
After analyzing the data it shows that in 3 of 4 tested network setups the higher bitrate was using Dat at least for the 10 and 20 clients experiment. The difference is between 1-5 times higher bitrate with Dat protocol in these cases. E.g. in figure A.2 with 20 clients the Dat protocol is performing five times as high bitrate. The result is showing that when the server has enough bandwidth to serve the X number of clients with there maximum bandwidth the maximum bitrate was higher for HTTP. If this is not the case the Dat protocol performed better.
6.3
Analysis of Research Q3
"How will the maximum video content quality rise with the number of seeders per video?"
The data did not show a rise in bitrate adding more seeders to a video. it showed the opposite. E.g. in the 22.5Mbit/s and 22.5Mbit/s in figure 5.1 we see that with only one peer downloading got a bitrate of 23.7 Mbit/s but with 2 peers we have only half. With 10 and 20 peers half of which are seeders to the other, the data showed even lower bitrate.
Chapter 6. Analysis and discussion 29
6.4
Discussion of Research Q1
"How or will we get a higher maximum bitrate using Dat compared to HTTP client-server infrastructure for on-demand video content providing (Mbit/s)" Though many research projects are done on P2P technology and comparing it with centralized alternatives no previous researches have been done on Dat. In [34] the same analysis is made about the client-server(HTTP) model. The paper confirms the same view of centralized Video streaming. "Each client sets up its own connection with the server over a unicast channel. As the video popularity increases, the server needs to serve a large number of clients and soon becomes the bottleneck. How to optimally provide VoD(Video on demand) service in a scalable manner remains an open question to be solved." their comparison includes the client-server model, IP-multicast and their proposed P2P protocol P2Cast.
6.5
Discussion of Research Q2
"Can the same number of clients watch higher quality content will using Dat pro-tocol compared to HTTP client-server infrastructure for on-demand video content providing?"
To give a true meaning to the values obtained from the experiment a correlation with real video quality values is needed.
Netflix requires a connection speed of 0.5 Mbit/s for the lowest quality. For Full HD quality, 5.0 Megabits per second is needed[47]. According to Vimeo an SD video with good quality is between 2-5Mbit/s and a 1080p video between 5-10 Mbit/s bitrate.[48]. That means by using Dat protocol on a 22.5Mbit/s server and 10 clients could with 10 and 20 connections watch HD content but with HTTP have to settle for SD content or lower.
Chapter 6. Analysis and discussion 30
6.6
Discussion of Research Q3
"How will the maximum video content quality rise with the number of seeders per video?"
The result I was expecting was more or less an exponential rise in the bitrate when adding more seeders or at least not a significant drop.
After seeing the result of the experiment E.g in Figure A.1. I saw that the maxi-mum bitrate value was getting lower with more clients. I suggest the explanation could be that the file is only downloaded from one peer at a time. If this wasn’t true and Dat did download separate chunks from multiple peers the files would download faster with more clients.
More efficient protocols would load balance the traffic between the clients/peers. An example of a peer to peer service that does this is DirectStream[36]. They call it a Quality of service peer selection. This is based on distance and bandwidth and the connecting peer selects the peer with the smallest distance-bandwidth. Something I haven’t found notes on Dat doing. This could be why the result shows lower and lower bitrate with more clients on the experiments. If the protocol is not controlling which node a peer connects to they could end up downloading from the same peer which then gets overloaded with requests.
PeerCast is more focused on live content. Content that is still updating something
that could also have been done with Dat as HyperCore support appending to feeds live. Peercasts base is a multicast tree that they call "base tree". When a client joins the network they join this multicast and all the coming content are downloaded here. The first section of the video from the start of the session to the client joining is obtained as a "patch" from other peers. They call the algorithm best-fit(BF) that is tree based and has a peer selecting alike the one inDirectStream.
Chapter 6. Analysis and discussion 31
6.7
Threats to validity
This section will present possible threats and limitations that where found during experimentation.
The experiment was only conducted on a simulated LAN/loopback device. Due to the design of the Dat protocol with a DNS server when used on WAN and Multicast on LAN only the multicast part was tested. Maybe the performance would be drastically different when used on WAN. That could prove the research invalid as most likely a video network at large scale would run on WAN. However, the reason for choosing to run on a local network was so that other traffic wouldn’t affect the result.
I could have written the HTTP client with Node.js HTTP module and collected the data in a similar way as the Dat protocol experiment. The data collection function could have been the same and the same network limiter could have been used. Would give a higher similarity to the experiments and higher credibility to the experiment.
Preferably each experiment could have been presented multiple times to verify similarity in the result. Due to fiddling with the research method too long the time for multiple runs didn’t exist. The experiment method was however tested multiple times to see that it performed consistently.
Chapter 7
Conclusions and Future Work
7.1
Conclusion
We could see that Dat showed better scalability using the same server and client bandwidth. Which answered RQ1 that we will get higher bitrate using Dat and also that the maximum video quality will be higher. The reason for this was because as a peer to peer protocol Dat utilize the clients so that other clients can download content directly from them.
Even though it performed well in the experiment could Dat be used as an alter-native to HTTP? I do not think so as the inconsistency of the outcome is too big. It lacks the control of from which peer the file is downloaded. By that, it can randomly download from the slowest peer or the fastest. This gives an inconsistent experience that users don’t want.
Features such as downloading chunks from multiple peers simultaneous would also be a needed feature for faster bandwidth and to be able to use Dat as a standard web protocol. Using HTTP as most video content delivery does today is giving the service provider a larger control of performance. The scaling can then be done reliably with a CDN even though the price is higher.
A service owner wants to be sure that the clients are getting the quality of service they expect as otherwise, they will get that from someone else. Right now that is not possible with Dat. As Dat protocol is open source and seems to be highly active it could probably be implemented as an extension.
I conclude that for now, Dat protocol is not a good fit for video content delivery. The Merkle tree design of HyperCore is good and it makes it easy to validate
Chapter 7. Conclusions and Future Work 33 files. Dat’s open source implementation could make it a good fit in the future after more research and extensions.
7.2
Future work
If you still want to use Dat for decentralized video. It would be a recommenda-tion doing research in extending the peer selecting algorithm this is part of the discovery phase. This is needed so that the content can be downloaded from the most efficient peer based on location and bandwidth. Some form of load balanc-ing features to know that the peers are not gettbalanc-ing overflooded with requests. Users won’t like that all there network capacity is consumed by serving others. More extension such as a system that rewards users that contribute by seeding. Something alike CoopNet where users who reshare/seed are rewarded with higher bandwidth [38] could be interesting.
Before further work is done on Dat I would recommend checking out and compar-ing with BitTorrent protocol as it has been used for file sharcompar-ing and streamcompar-ing for a long time. Google has a new centralized protocol in the making called QUIC or HTTP/3 that is interesting to consider as the future web protocol for video content and should be compared when doing further work.
It would also be nice to be able to run it in the commonly used browsers such as Firefox and Chrome. The implementation of Dat that was tested could only be run in the experimental Beaker Browser. This will probably need a WebRTC implementation to experiment with. WebRTC is the standard used for P2P in the browser today[49].
After that, a larger scale experiment would be interesting. Implemented and testing on a real-world video application as a type of CDN. A few serverless CDN networks for video are implemented today such as Peer5 [5]. They use WebRTC but are proprietary software and the implementation can’t be reviewed.
References
[1] “Cisco visual networking index: Forecast and methodology, 2015-2020,” 2016. [2] A. Technologies, “Users demand high quality video.” Youtube Video, apr
2016.
[3] S.-H. Lin, R. Pal, B.-C. Wang, and L. Golubchik, “On market-driven hybrid-p2p video streaming,” IEEE Transactions on Multimedia, vol. 19, no. 5, pp. 984–998, 2017.
[4] R. Oskoui, “How content delivery networks work,” Jan
2019. https://www.cdnetworks.com/web-performance/
how-content-delivery-networks-work/. [5] Peer5, “Faq,” 2019.
[6] J. Li, “On peer-to-peer (p2p) content delivery,” Peer-to-Peer Networking and
Applications, vol. 1, no. 1, pp. 45–63, 2008.
[7] G. Huang, “Pplive: A practical p2p live system with huge amount of users,” in Proceedings of the ACM SIGCOMM Workshop on Peer-to-Peer Streaming
and IPTV Workshop, pp. 22–28, 2007.
[8] T. T. T. Ha, J. Kim, and J. Nam, “Design and deployment of low-delay hybrid cdn–p2p architecture for live video streaming over the web,” Wireless
Personal Communications, vol. 94, no. 3, pp. 513–525, 2017.
[9] P. J. Leach, T. Berners-Lee, J. C. Mogul, L. Masinter, R. T. Fielding, and J. Gettys, “Hypertext transfer protocol–http/1.1,” 1999.
[10] Cisco, “Complete visual networking index (vni) forecast,” 2019.
[11] B. Mitchell, “Introduction to client server networks,” Retrieved 3rd January, 2012.
References 35
[12] J. Roettgers, “What if netflix switched to p2p for video
streaming?,” mar 2014. https://gigaom.com/2014/03/21/
what-if-netflix-switched-to-p2p-for-video-streaming/ Accessed:
2010-05-28.
[13] D. Moltchanov, “Client/server and peer-to-peer models: basic concepts,”
Utopia.duth.gr, vol. 1, 2013.
[14] Freenet, “Freenet project,” 2019. https://freenetproject.org Accessed: 2010-05-28.
[15] Gnutella, “Webarchive, gnutella,” 2006. https://web.archive.org/web/ 20060716053753/http://www.gnutella.com/ Accessed: 2010-05-28. [16] S. Lui and S. Ho Kwok, “Interoperability of peer-to-peer file sharing
proto-cols,” SIGecom Exchanges, vol. 3, pp. 25–33, 06 2002.
[17] C. Y. Goh, H. S. Yeo, H. Lim, P. K. Hoong, J. W. Lim, and I. K. Tan, “A comparative study of tree-based and mesh-based overlay p2p media stream-ing,” International Journal of Multimedia and Ubiquitous Engineering, vol. 8, no. 4, pp. 97–105, 2013.
[18] Y. Liu, Y. Guo, and C. Liang, “A survey on peer-to-peer video streaming systems,” Peer-to-Peer Networking and Applications, vol. 1, pp. 18–28, 03 2008.
[19] D. Project, “A bit more about dat,” mar 2019. Accessed: 2010-05-28. [20] D. Project, “Docs concepts,” mar 2019. https://docs.datproject.org/
concepts Accessed: 2010-05-28.
[21] D. Project, “Dat protocol,” jan 2019. https://www.datprotocol.com/ Ac-cessed: 2010-05-28.
[22] Blake, “Blake2,” mar 2019. https://blake2.net/ Accessed: 2010-05-28.
[23] M. Buus, “Hypercore,” aug 2018. https://devhub.io/repos/
mafintosh-hypercore Accessed: 2010-05-28.
[24] Bitcoin, “Merkle tree.” https://bitcoin.org/en/glossary/merkle-tree Accessed: 2010-05-28.
[25] jswisher, “An overview of http,” 2019. https://developer.mozilla.org/ en-US/docs/Web/HTTP/Overview Accessed: 2010-05-28.
[26] H. Yan, J. Liu, Y. Li, D. Jin, and S. Chen, “Spatial popularity and similarity of watching videos in large-scale urban environment,” IEEE Transactions on
References 36
[27] “Content poisoning in peer to peer network,” jun 2014. http://www.
researchpublish.com/download.php?file=Content%20Poisoning%
20in%20Peer%20to%20Peer%20Network-251.pdf&act=book Accessed:
2010-05-28.
[28] C. Huang, A. Wang, J. Li, and K. W. Ross, “Understanding hybrid cdn-p2p: why limelight needs its own red swoosh,” in Proceedings of the 18th
Inter-national Workshop on Network and Operating Systems Support for Digital Audio and Video, pp. 75–80, ACM, 2008.
[29] incapsula, “Man in the middle (mitm) attack,” 2019. https://www.
incapsula.com/web-application-security/man-in-the-middle-mitm. html Accessed: 2010-05-28.
[30] J. Li, “A survey of peer-to-peer network security issues,” Retrieved November, vol. 29, p. 2010, 2007.
[31] D. Project, “how-dat-works, encryption,” jan 2019. https://datprotocol. github.io/how-dat-works/#encryption Accessed: 2010-05-28.
[32] M. B. Paul Frazee, “Dep-0002: Hypercore,” feb 2018. https://www.
datprotocol.com/deps/0002-hypercore/ Accessed: 2010-05-28.
[33] D. Project, “Dat intro,” may 2019. https://docs.datproject.org/docs/ intro Accessed: 2010-05-28.
[34] Y. Guo, K. Suh, J. Kurose, and D. Towsley, “P2cast: peer-to-peer patching scheme for vod service,” in Proceedings of the 12th international conference
on World Wide Web, pp. 301–309, ACM, 2003.
[35] H. Deshpande, M. Bawa, and H. Garcia-Molina, “Streaming live media over peers,” tech. rep., Stanford InfoLab, 2002.
[36] Y. Guo, K. Suh, J. Kurose, and D. Towsley, “Directstream: A directory-based peer-to-peer video streaming service,” Computer Communications, vol. 31, no. 3, pp. 520–536, 2008.
[37] Technopedia, “Directory services.” https://www.techopedia.com/
definition/18887/directory-services Accessed: 2010-05-28.
[38] V. N. Padmanabhan, H. J. Wang, P. A. Chou, and K. Sripanidkulchai, “Dis-tributing streaming media content using cooperative networking,” in
Proceed-ings of the 12th international workshop on Network and operating systems support for digital audio and video, pp. 177–186, ACM, 2002.
References 37 [39] X. Su and L. Chang, “A measurement study of ppstream,” in 2008 Third
International Conference on Communications and Networking in China,
pp. 1162–1166, Aug 2008.
[40] J. Lei, L. Shi, and X. Fu, “An experimental analysis of joost peer-to-peer vod service,” Peer-to-Peer Networking and Applications, vol. 3, pp. 351–362, Dec 2010.
[41] Wget, “wget.” https://www.gnu.org/software/wget/ Accessed: 2010-05-28.
[42] S. Krishna, “How to limit network bandwidth on
linux,” 2016. http://www.tutorialspoint.com/articles/
how-to-limit-network-bandwidth-on-linux Accessed: 2010-05-28. [43] marius a. eriksen, “Trickle,” 2018. https://github.com/mariusae/trickle
Accessed: 2010-05-28.
[44] S. Krishna, “Netbrake,” 2007. https://github.com/pld-linux/netbrake Accessed: 2010-05-28.
[45] J. Yu, “How to limit node.js bandwidth,” jul 2013. https://stackoverflow.
com/questions/14508348/how-to-limit-node-js-bandwidth Accessed:
2010-05-28.
[46] D. belson, “Akamai’s state of the internet q1 2017,” tech. rep., 2017.
[47] Netflix, “Internet connection speed recommendations.” https://help.
netflix.com/en/node/306 Accessed: 2010-05-28.
[48] vimeo, “Compression.” https://vimeo.com/help/compression Accessed: 2010-05-28.
List of Figures
5.1 22.5Mbit/s Server and 22.5Mbit/s Clients bandwidth . . . 25
5.2 1000Mbit/s Server and 100Mbit/s Clients bandwidth . . . 26
A.1 22.5Mbit/s Server and 22.5Mbit/s Clients bandwidth . . . 40
A.2 100Mbit/s Server and 22.5Mbit/s Clients bandwidth . . . 40
A.3 100Mbit/s Server and 100Mbit/s Clients bandwidth . . . 41
A.4 1000Mbit/s Server and 100Mbit/s Clients bandwidth . . . 41
Appendix A
Result of network expriement
These appendix show the result of the network experiments that was conducted. The result is presented in bar diagrams.Each bandwidth test is split into own diagram. A higher bar is better. Each diagram is then split into four separate bars for each number of clients 1,2,10 and 20 making it easy to see which technology was performing best at each state. The provided result are average values. One iteration was conducted.
Appendix A. Result of network expriement 40 1 2 10 20 0 10 20 30 23.7 13.1 9.5 4.9 20.4 11.2 2.1 1 Clients Mbit/second(Higher is b etter) P2P HTTP
Figure A.1: 22.5Mbit/s Server and 22.5Mbit/s Clients bandwidth
1 2 10 20 0 10 20 30 23.7 22.3 17.2 10.6 22.3 22.3 9.8 4.8 Clients Mbit/second(Higher is b etter) P2P HTTP
Appendix A. Result of network expriement 41 1 2 10 20 0 20 40 60 80 100 120 95.3 95.5 33.3 19.8 90.4 22.3 9.8 4.8 Clients Mbit/second(Higher is b etter) P2P HTTP
Figure A.3: 100Mbit/s Server and 100Mbit/s Clients bandwidth
1 2 10 20 0 20 40 60 80 100 120 95.2 73.3 58.1 30.6 98.4 98.4 97.6 75.36 Clients Mbit/second(Higher is b etter) P2P HTTP
