• No results found

A Comparison of RTMP and HTTP Protocols with respect to Packet Loss and Delay Variation based on QoE

N/A
N/A
Protected

Academic year: 2022

Share "A Comparison of RTMP and HTTP Protocols with respect to Packet Loss and Delay Variation based on QoE"

Copied!
68
0
0

Loading.... (view fulltext now)

Full text

(1)

Master Thesis

Electrical Engineering December 2012

A Comparision of RTMP and HTTP Protocols with respect to Packet Loss and Delay Variation

based on QoE

Ramesh Goud Guniganti and Srikanth Ankam

School of Computing

Blekinge Institute of Technology 37179 Karlskrona

Sweden

(2)

This thesis is submitted to the School of Computing at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Master of Science in Electrical Engineering. The thesis is equivalent to 20 weeks of full time studies.

This Master Thesis is typeset using LATEX

Contact Information Author 1:

Ramesh Goud Guniganti Address: Karlskrona, Sweden E-mail: ramesh493@gmail.com Author 2:

Srikanth Ankam

Address: Karlskrona, Sweden

E-mail: srikanthankam@hotmail.com University advisor:

Dr. Adrian Popescu, Prof.

COM/BTH

School of Computing

Blekinge Institute of Technology 371 79 KARLSKRONA, SWEDEN

Internet: www.bth.se/com Phone: +46 455 385000 SWEDEN

(3)

Abstract

In recent year’s multimedia services like Video-on-Demand (VoD) and mo- bile video streaming, videos for e-learning, video conferencing are growing predominantly, and the user’s expectations towards the quality video are in- creasing as the technology is developing. There are different video streaming protocols are used for streaming videos from servers to the client. Recently, Adobe Systems developed Real Time Messaging Protocol (RTMP) (propri- etary) for streaming audio, video and data over the Internet between a Flash player and a Media Server. On the other hand, Hypertext Transfer Protocol (HTTP) is a well-known and efficient protocol; it has achieved the popu- larity in multimedia services like VoD. Hence, a qualitative research is to be performed on comparing the two Transmission Control Protocol (TCP) based protocols, under sustainable network conditions for tracing the QoE results from acquired User Ratings (UR).

This thesis investigates the quality assets on network parameters over VoD streaming. The study addresses the subjective assessment of RTMP and HTTP streaming protocols, by varying network parameters (like packet loss and delay variation) in a controlled and repeatable environment. The packet loss and delay variation are altered by the network emulator NetEm [1, 2] in between the server and client. The video collected at the client end are eval- uated by using subjective assessment, MOS (Mean Opinion Score), following the International Telecommunication Union (ITU) Recommendations [3].

Based on our results it was found that HTTP is having better ratings, when there are more packet losses compared to RTMP. RTMP accomplished better at minimum loss of packets. However, in the case of delay perfor- mance of HTTP is better than RTMP.

Keywords: HTTP, Mean Opinion Score, Quality of Experience, RTMP, Subjective Assessment, Network Emulator.

i

(4)

Acknowledgements

First and foremost, we offer our sincerest gratitude to my supervisor, Prof.

Adrian Popescu, who has supported throughout our Thesis with his pa- tience and knowledge. We attribute the level of our Masters degree to his encouragement and effort and without him this Thesis would not have been completed or written.

We also owe our deepest gratitude to Dr. Patrik Arlos for providing us the experimental test bed and support throughout the Thesis work.

We would like to thank all the video quality assessment survey partici- pants, who have contributed towards survey part of this work. Finally, we would also like to thank our loved parents for supporting us both morally and financially. Without their encouragement and motivation we could not able to complete this project. We would also like to thank our friends who helped us with their valuable suggestions and support.

Ramesh Goud Guniganti Srikanth Ankam

ii

(5)

Contents

Abstract i

Acknowledgements ii

Contents iii

List of Figures vi

List of Tables vii

Acronyms viii

Introduction 1

1 Introduction 2

1.1 Aims and Objectives . . . 3

1.2 Research Questions . . . 4

1.3 Expected outcomes . . . 4

1.4 Research Methodology . . . 5

1.5 Outline of Thesis . . . 5

Background 6 2 Background and Related Work 7 2.1 Flash Player . . . 9

2.2 H.264 Video Codec . . . 9

2.3 FFmpeg . . . 9

2.4 Quality of Experience . . . 10

2.5 Subjective Quality Assessment . . . 10

2.6 SSCQE (Single Stimulus Continuity Quality Evaluation) . . 11 iii

(6)

Implementation 12

3 Experimental Setup 13

3.1 Method . . . 13

3.2 Experimental Setup . . . 13

3.3 Setup Design . . . 14

3.3.1 Measurement Point . . . 14

3.3.2 MArC . . . 15

3.3.3 Consumer . . . 15

3.3.4 Wowza Media Server . . . 15

3.3.5 Wowza Client . . . 15

3.3.6 NetEm . . . 16

3.3.7 Packet Loss . . . 16

3.3.8 Packet Delay Variation . . . 16

3.3.9 RTMP at Client . . . 17

3.3.10 HTTP at Client . . . 17

3.3.11 FFmpeg Encoding . . . 17

3.3.12 FFmpeg Splitting . . . 17

3.3.13 FFmpeg Grabbing . . . 17

3.3.14 Scale . . . 18

3.4 Experimental setup and procedure . . . 19

3.5 Assessment of Videos . . . 19

Results 22 4 Results and Discussions 23 4.1 Mean scores calculations . . . 23

4.2 Confidence Interval Calculations . . . 24

4.3 RTMP and HTTP Packet loss . . . 24

4.4 RTMP and HTTP Delay Variation . . . 29

4.5 Validity Threats . . . 32

Conclusions and Future Work 33 5 Conclusions and Future Work 34 5.1 Conclusion . . . 34

5.2 Future Work . . . 35

iv

(7)

Bibliography 36

Appendix 42

A Apendix A 43

A.1 Network Configuration . . . 43

A.2 Speed and Duplex settings . . . 43

A.3 Emulator Commands . . . 44

B Apendix B 45 B.0.1 BTH HTTP Packet loss ratings . . . 46

B.0.2 BTH RTMP Packet loss ratings . . . 47

B.0.3 Cable car HTTP Packet loss ratings . . . 48

B.0.4 Cable car RTMP Packet loss ratings . . . 49

B.0.5 Big Buck Bunny HTTP Packet loss ratings . . . 50

B.0.6 Big Buck Bunny RTMP Packet loss ratings . . . 51

B.0.7 BTH HTTP Delay variation ratings . . . 52

B.0.8 BTH RTMP Delay variation ratings . . . 53

B.0.9 Cable car HTTP Delay Variation ratings . . . 54

B.0.10 Cable car RTMP Delay Variation ratings . . . 55

B.0.11 Big Buck Bunny Delay Variation HTTP ratings . . . . 56

B.0.12 Big Buck Bunny Delay Variation RTMP ratings . . . . 57

v

(8)

List of Figures

3.3.1 Experimental Setup . . . 14 3.5.1 Login Screen . . . 20 3.5.2 Assessment tool for video survey . . . 20 4.3.1 MOS with 95% CI vs. Packet loss for HTTP BTH and RTMP

BTH . . . 25 4.3.2 MOS with 95% CI vs. Packet loss for HTTP cablecar and

RTMP cablecar . . . 26 4.3.3 MOS with 95% CI vs. Packet loss for HTTP Big Buck Bunny

and RTMP Big Buck Bunny . . . 26 4.3.4 Standard deviation for HTTP and RTMP packet loss . . . 27 4.4.1 MOS vs. Delay Variation for HTTP BTH and RTMP BTH . 29 4.4.2 MOS vs. Delay Variation for HTTP Cable car and RTMP

Cable car . . . 30 4.4.3 MOS vs. Delay Variation for HTTP Big Buck Bunny and

RTMP Big Buck Bunny . . . 31 4.4.4 Standard deviation for HTTP and RTMP Delay variation . . 31

vi

(9)

List of Tables

3.3.1 Description of Videos . . . 18

3.3.2 Characteristics of Videos . . . 18

3.3.3 ITU-T SCALE OF MEDIA QUALITY IMPAIRMENT . . . 18

4.3.1 MOS Ratings for Packet Loss . . . 28

4.4.1 MOS Ratings for Packet Delay Variation . . . 32

B.0.1Packet loss user ratings on BTH HTTP Video . . . 46

B.0.2Packet loss user ratings on BTH RTMP Video . . . 47

B.0.3Packet loss user ratings on Cable car HTTP Video . . . 48

B.0.4Packet loss user ratings on Cable car RTMP Video . . . 49

B.0.5Packet loss user ratings on Big Buck Bunny HTTP Video . . 50

B.0.6Packet loss user ratings on Big Buck Bunny RTMP Video . . 51

B.0.7Delay variation user ratings on BTH HTTP Video . . . 52

B.0.8Delay Variation user ratings on BTH RTMP Video . . . 53

B.0.9Delay Variation user ratings on Cable car HTTP Video . . . . 54

B.0.10Delay Variation user ratings on Cable car RTMP Video . . . 55 B.0.11Delay Variation user ratings on Big Buck Bunny HTTP Video 56 B.0.12Delay Variation user ratings on Big Buck Bunny RTMP Video 57

vii

(10)

Acronyms

3GPP 3rd Generation Partnership Project AVC Advance Video Codec

BTH Blekinge Tekniska Högskola CGA Colour Graphics Adapter CI Confidence Interval

DAG Data Acquisition and Generation

DSCQS Double Stimulus Continuous Quality Scale DPMI Distributed Passive Measurement Infrastructure EGA Enhanced Graphic Adapter

FPS Frames Per Second

GPS Global Positioning System HAS HTTP Adaptive Streaming HTTP Hypertext Transfer Protocol IP Internet Protocol

IPTV Internet Protocol TeleVision

ISO International Organization for Standard IEC International Electro technical Commission

ITU-R International Telecommunication Union, Radio Communication Sec- tor

ITU-T International Telecommunication Union, Telecommunication Stan- dardization Sector

JVT Joint Video Team

viii

(11)

LTE Long Term Evolution

MArC Measurement Area Controller MOS Mean Opinion Score

MP Measurement Point

MPEG Moving Pictures Expert Group PL Packet Loss

PDV Packet Delay Variation

PEVQ Perceptual Evaluation Video of Quality PQoS Perceived Quality of Service

PSNR Peak Signal-to-Noise Ratio QoE Quality of Experience

QoS Quality of Service QoP Quality of Presentation QoD Quality of Delivery

RTMP Real Time Messaging Protocol RTP Real-Time Transport Protocol RTSP Real Time Streaming Protocol SP Service Providers

SS Single Stimulus

SSCQE Single Stimulus Continuous Quality Evaluation TC Traffic Control

TCP Transport Control Protocol TS Traffic Shaper

UR User Rating

VGA Video Graphics Array VoD Video-on-Demand

VCEG Video Coding Expers Group VQEG Video Quality Expert Group WMS Wowza Media Server

ix

(12)

Introduction

1

(13)

Chapter 1

Introduction

In recent years, multimedia technology has been widely developed. In the video streaming there is enormous development and research in the field of streaming protocols like RTSP (Real Time Streaming Protocol), RTP (Real- Time Transport Protocol), HTTP and RTMP . However, these protocols performance may vary in streaming due to traffic influencing parameters such as packet loss and delays in the network. Because of this, choosing an appropriate video streaming protocol becomes challenging task. Using an efficient protocol in video streaming would improve the Quality of Service (QoS) when there is large data transmission over the network. Recently, RTMP protocol was designed by Adobe for video streaming. HTTP is well known protocol for video streaming now-a-days [4].

To stream a video over a network, protocols are required. In a case, when a client wants to view the video client sends a request by using a protocol, when the servers receives request from the client it starts sending the packets. To stream videos over any network, many protocols are available but an appropriate protocol is to be chosen.

RTMP refers to the proprietary protocol developed by Adobe Systems for streaming audio, video, and data over the Internet between a Flash player and a Flash Media Server. Like RTSP, RTMP is an example of a traditional streaming protocol, though it is only one of many versions of streaming protocols for the web. RTMP is defined as a stateful protocol, meaning that from the first time a client connects until the time it disconnects, the streaming server keeps track of the clients actions. The client communicates its actions, or “states”, to the server by issuing commands such as PLAY or PAUSE. When a command between the client and the server is established, the server begins sending the media as a steady stream of small informa- tion packets. This behavior continues and repeats until the server or player client closes the session [4, 5]. RTMP was designed for high-performance transmission such as audio, video, and data between Adobe Flash Platform technologies, including Adobe Flash Player [6].

2

(14)

CHAPTER 1. INTRODUCTION 3

HTTP refers to the protocol used to deliver webpages images and videos cross the Internet worldwide. HTTP is an adopted, open standard the most ubiquitous mode of online delivery. It is a “stateless” protocol think of it as an airline ticket to anywhere. It can be delivered by a variety of web servers, both commercial and open source [4].

When the network factors such as packet delay variation (PDV) and packet loss (PL) are in the network, the quality of the video is affected by the video artifacts such as blockiness, blur jerkiness, freezes affects the streaming video. When these artifacts deploy on the video quality degrades due to the network behaviour. In user point of view, the user will not show interest in knowing about the video codec or video bit rate while experiencing the streaming video in VoD or Live. The end-user considers the quality of video instead of Quality of Service.

This work focuses on Quality of video presenting to the end-user while streaming over protocols like HTTP and RTMP. To continue our experi- mentation, VoD streaming has been done on RTMP and HTTP by varying packet loss and delay variation with network emulator NetEm [1, 2] in be- tween the server and client. Wowza server was used on the server side, and a flash player is embedded in a web page at the client side. The videos are streamed over RTMP and HTTP protocols from server to client. FFmpeg is used at client side to record the video sequences, that are being streamed from the server.

In our thesis, we are investigating the impact of network parameters such as packet loss and packet delay variation for both RTMP and HTTP proto- cols by streaming three different video sequences over a controlled network.

In the quality of video assessment by human subjects, the subjective ratings were obtained for the resulting test sequences using SSCQE (Single Stim- ulus Continuous Quality Evaluation) defined by ITU-R Recommendation BT.500-11 [3]. Similarly, Mean Opinion Score (MOS) quality assessment over videos is conducted with No-reference metric design [7].

1.1 Aims and Objectives

The aim of this research is to investigate the performance of the RTMP and HTTP protocols for video streaming, and also to provide recommendations on suitable protocol.

The objectives are as follows:

• Literature review is performed to identify the drawbacks and performance of RTMP and HTTP protocols for video streaming servers.

(15)

CHAPTER 1. INTRODUCTION 4

• Streaming videos over RTMP and HTTP video streaming server via a controlled traffic network.

• Relating the network parameters quality of streamed videos from server to client with QoE by MOS.

• Analyze the threshold level of delay in both RTMP and HTTP streaming servers based on MOS.

• Giving recommendations on the performance of protocols based on the analysis of results.

1.2 Research Questions

1 What is the impact of packet loss on RTMP and HTTP video streaming with regard to video quality?

2 Which one of the protocols RTMP and HTTP achieves better video quality when delays are introduced in the network?

3 What recommendations can be given for RTMP and HTTP pro- tocols based on the network performance in video streaming?

1.3 Expected outcomes

The following results are reported based on the work done:

1 The impact of packet loss and delay variation on RTMP and HTTP protocols on video quality is investigated by measuring MOS for the two protocols under varying network disturbances.

2 By performing a subjective video quality assessment of video streamed over RTMP and HTTP protocols, conclusions are drawn regarding their performance.

3 Final results also contain QoS parameters that are taken into consideration while relating QoE.

4 By analyzing the obtained results for RTMP and HTTP protocols for various network disturbances like delay variation, packet loss, recommendations will be given based on the results.

(16)

CHAPTER 1. INTRODUCTION 5

1.4 Research Methodology

1 In the early stage of our research, we do literature review related to the video streaming protocols RTMP, HTTP and QoE, stan- dards. Finally, QoS parameters like delay and packet loss.

2 Study of different video streaming servers such as Adobe flash media, Red5, Wowza, VLC. Selecting the appropriate one that suits our research in video streaming.

3 In the next stage we choose the videos recommended by standard groups that are encoded into H.264 using FFmpeg encoder for conducting experiments using RTMP and HTTP protocols.

4 Study about the constraints that affect the QoE.

5 Once the literature survey is done, experimentations are done with appropriate video streaming servers for RTMP and HTTP, traffic shaper and standard videos

6 During the experimentations packet loss and delay variation are introduced by using appropriate traffic shaper in the network.

7 Streaming selected videos over RTMP and HTTP protocols. We stream from the server to client, in a controlled environment.

8 Videos streamed at the client are observed and MOS are collected using subjective analysis method.

9 The results obtained from the user related survey of different videos of both these protocols are quantitatively analysed.

10 Based on the final results and analysis, we provide conclusions and recommendations between these two protocols.

1.5 Outline of Thesis

This report is organized as follows. The Chapter 2 describes back- ground work and research related to this thesis work. The Chapter 3 describes the experimental setup. The Chapter 4 contains results and discussion and final chapter 5 describes conclusion and future work.

(17)

Background and Related Work

6

(18)

Chapter 2

Background and Related Work

This section provides the background related to the thesis work. Sev- eral factors like packet loss and delay variation in network may vary the video quality, and because of some unreliable protocols are influ- encing the end user video applications. However, there may be smooth play out interruptions due to bandwidth fluctuations or long retrans- missions, which affect the QoE for the video applications [8].

In [9] the author has explained about the overview of HTTP Adap- tive Streaming (HAS) concepts, and also an end-to-end QoE evaluation study on HAS conducted over 3GPP LTE networks. In paper [8], the authors proposed no reference QoE monitoring module for adaptive HTTP streaming based on Random Neural Networks, models the im- pact of both factors. In [10] authors has proposed a design for the academic institutions to distribution, uploading videos and scheduled broadcast by using Wowza media server, Stream Class Scheduler Mod- ule, Flash media encoder, FFmpeg, VLC and JWplayer tools are used in this work. In paper [11], to investigate the relation between QoS and QoE, the author has conducted a set of experiments on different videos, by using different shapers like NetEm, NISTnet and KauNet by varying packet loss, delay variation and bit rate, and also to chosen the best shaper for the future. Finally, authors concluded on NetEm showing the accurate results based on one-way delay evaluation. In paper [12] the author focuses on revealing challenges and offering con- cepts associated with the incorporation of the Quality of Experience (QoE) into the design of mobile video systems. In [13] the author ex- plained concept of QoE in engineering is which also known as Perceived Quality of Service (PQoS) and the term QoS which is finally perceived by the end-user. The quality perception of H.264/AVC coded video containing packet loss is analyzed based on the results of a combined subjective video quality and eye tracking experiment. In the field of

7

(19)

CHAPTER 2. BACKGROUND AND RELATED WORK 8

networking, multidimensional QoS analysis has not yet received much attention with regards to quality services [14].

The QoE of users degrades due to defective encoding, bottlenecks, channel change time too long, excessive compression, order failure for VoD, Transmission Unavailable, audio-video out of sync [15]. In pa- per [16] the authors explained the technical and non-technical param- eters aspects of QoE, in technical, application- and network-level QoS and in non-technical user perception, experience and expectations. In paper [17] the author discussed about video streaming effects by using QoS parameters like delay, loss and bandwidth. To characterize the effects of the parameters such as delay and loss are accessed by QoE where the MOS assigns each video sample by human subjects. In pa- per [18] the authors had done series of experiments on 3G networks, to observe the effect of PDV on the end-user QoE. These results are adequate to predict UR (User Ratings) values over PDV, related to QoE of streaming video users.

In paper [19], the author describes about applications and access technologies that best suit the user needs, thereby combining these features of access technologies to provide an enhanced user perceived experience. In [20], work was done for small set of experiments, con- sidering the web based server for RTMP video streaming. The authors mentioned that they had Mean Opinion Score for only 10 users. The server communicates with the flash media player by using the Real Time Messaging Protocol. RTMP was designed for high speed audio, video and data with maintaining persistent connection [21].

In paper [5], RTMP is used as real-time internet class room for web- based collaborative work between teachers and students to interact over exchanging messages such as flash data, audio, video. RTMP is basically a TCP/IP based protocol which delivers flash content. In [22,23], with the RTMP specification, developers and companies will be able to provide users with optimized audio, video and data streaming, no matter what kind of device the user is on, or where the content is coming from.

The end user satisfaction is very much important, in evaluating a product or a service. There is a correlation between QoS and QoE to evaluate the end user satisfaction, of a product on the internet [18].

In paper [9] HAS (HTTP Adaptive Streaming) provides the ability at client to fully control over the streaming session, i.e., HTTP can intelligently manage the on-time request and smooth play out of video frames, potentially adjusting bitrates.

The evolution of new multimedia solutions requires new ways to optimize future wireless networks for video services towards delivery to

(20)

CHAPTER 2. BACKGROUND AND RELATED WORK 9

the end user. In the case of HTTP Adaptive Streaming, the content delivered at client side is in multiple bit rates that are higher or lower bit rates. HTTP Dynamic Streaming supports both live and on-demand media content that adjusts to viewer connection speed and processing power using standard HTTP protocol Infrastructures [24].

In this [25] authors considered an Eclipse IDE application to stream HTTP and RTMP, for the streaming they used a web server (Apache Tomcat and Red5) and flash client. They fetched that flash client browser is used to stream in both protocols. In scalable 3G live stream- ing over HTTP and RTMP protocols will stream different encoded videos over various computers and mobile devices. Here, videos are encoded with On2 VP6 and H.264 baseline, it showed better encoding and decoding performance against High profile [26].

2.1 Flash Player

Here, a web based flash player is used at client side to receive en- coded video sequences. Flash Player 10.1 has more features and it will increase the options available to deliver high quality media including H.264 over many different protocols such as RTMP [27].

2.2 H.264 Video Codec

H.264 is a video compression technology, or codec, that was jointly de- veloped by the ITU as H.264 and International Organization for Stan- dardization/International Electro technical Commission (ISO/IEC) Mov- ing Picture Experts Group (as MPEG-4 Part 10, Advanced Video Cod- ing, or AVC). Thus, the terms H.264 and AVC mean the same thing and are interchangeable. A video codec in H.264 integrated into mul- timedia container format, and is frequently produced in the MPEG-4 container format, which uses the .MP4 extension [28]. In [29] ITU-T H.264/MPEG-4 (Part 10) Advanced Video Coding (commonly referred as H.264/AVC) is the newest entry in the series of international video coding standards. H.264/AVC is currently the most powerful and state-of-the-art standard. It was developed by a Joint Video Team (JVT) consisting of experts from ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Moving Picture Experts Group (MPEG).

2.3 FFmpeg

FFmpeg is an open source tool that can be used as both encoder and decoder. It has a powerful multimedia processing capability [30]. FFm-

(21)

CHAPTER 2. BACKGROUND AND RELATED WORK 10

peg provides multimedia visual adaptation functionalities such as res- olution reduction, frame rate reduction, cropping etc. [31]. It is a very fast video converter that can grab live video source [32].

2.4 Quality of Experience

It is the perceived quality of service by the end-user. In [33] the authors portrayed the purview of QoE definition, QoE is the overall perfor- mance of a system from the user point of view, i.e., what a user really perceives in terms of usability, accessibility, retain-ability and integrity.

The author discusses about QoE parameters, has more important than the QoS parameter [11]. To know about the better service of the net- work the QoE determines in the form of results from user perspective.

In [11, p. 26- 28] the QoE depends on services like Quality of Presenta- tion (QoP), Quality of Delivery (QoD) and QoS to get user experience over network services. Research has been done on user satisfaction has been mainly forward motion in multimedia technology such as in video streaming [8–12, 14]. In paper [34] for the service providers (SP) is to provide PQoS services, assurance based on multimedia content to the end-user. Subjective analysis test should be measured via users. The quality of perceived video is rated in the subjective analysis by the user.

The ITU-T definitions for QoE are:

Quality of Experience includes the complete end-to-end system ef- fects (client, Terminal, network, services infrastructure, etc)

• The overall acceptability may be influenced by user expectations and con- text. [35].

• The overall acceptability of an application or service, as perceived subjectively by the end-user by ITU-T P.10/G.100.

2.5 Subjective Quality Assessment

Subjective assessment methods are thus often considered as a ground truth for quality prediction by user [13, p. 7]. The feeling of the in- dividual taking part in the analysis process determines the outcome.

In QoE the subjective results vary from user-to-user [36]. There are two types of standards to measure subjective analysis defined by In- ternational Telecommunication Union, single and double stimulus. In SSCQE (Single Stimulus Continuous Quality Evaluation) method the quality of the distorted video is rated without any reference to the original stimulus.

(22)

CHAPTER 2. BACKGROUND AND RELATED WORK 11

In DSCQS (Double Stimulus Continuous Quality Scale) method the reference video is given to measure the distorted video quality. In this project SSCQE method is used in subjective analysis. In this scenario single SSCQE allows end-users to dynamically rate the quality of an arbitrarily long video sequence using quality scale, be more useful for evaluating real-time quality monitoring systems [37]. This subjective test was performed in a room conforming to ITU-R BT.500 [3].

2.6 SSCQE (Single Stimulus Continuity Quality Eval- uation)

In the SSCQE continuous video sequences are once presented to user, to rate the video quality. The presented video sequences may contain impairments. Human subjects will evaluate the instantaneous quality in real time using a slider with a continuous scale. This SSCQE is recommended by ITU-R BT. 500-7. This method is preferred in QoE subjective analysis.

similarly, another method for objective analysis is No-reference method.

It is also known as reference free method [11]. In this case, the QoE is not measured by comparing an original video. This method tries to detect artifacts such as blockiness, blur, jerkiness directly in the video [13] for quality prediction. This approach is based on the idea that customers don’t know the original content. In this method the users will rate the video with experienced quality and distortions.

(23)

Implementation

12

(24)

Chapter 3

Experimental Setup

3.1 Method

This section describes the methodology and methods to answer our research questions. There are two steps in our proposed method. They are experimentation and user related survey i.e. subjective assessment.

The first step of our methodology is discussed in the next section 3.2, and the second phase involves video quality assessment. The video quality assessment can be done in two ways namely objective assess- ment and subjective assessment.

In objective assessments involves mathematical algorithms and mod- els that can estimate the Human perceptual behavior. The models like peak signal-to-noise ratio (PSNR), Mean Square error are some ex- amples of objective assessment. These models are may vary for the different codecs and others parameters. They may or may correlate with the subjective assessment values.

The other method for video assessments is subjective assessment where a user will participate in the survey to provide ratings for the videos. In our research we have chosen subjective assessment to rate the videos, because the objective assessment ratings may or may not correlate the perceptual ratings of the human behavior.

3.2 Experimental Setup

This chapter elucidates the experimental setup, to observe the charac- teristic of packet loss and packet delay variation affects in video quality at end user and a set of experiments conducted in different scenarios.

13

(25)

CHAPTER 3. EXPERIMENTAL SETUP 14

3.3 Setup Design

In this experiment, streaming is done form server to client over HTTP, RTMP. Here, server and client machines are on Linux operating sys- tem. In between server and client the required equipments are placed to trace out the results for packet loss and packet delay variations. In this setup two hosts are equipped with the traffic shaper in between. The traffic shaper is used to control the traffic in network going from host A (server) to host B (client). The experimental setup is configured to stream video from host A (video originator) to host B (video receiver) at the other end. The connections are made with Ethernet cables with required static IP address. In this scenario host A sends packets to host B via emulator. In the emulator losses and delays are introduced to find out the video quality at host B client side while streaming video over RTMP, HTTP. The setup is mount to Measurement Point (MP) to capture the traffic flowing form sender to receiver. The experiment setup is shown in Figure 3.3.1.

Figure 3.3.1: Experimental Setup 3.3.1 Measurement Point

MP is used for measuring overall timestamps of packets. MP obtains the sender time as well as receiver time of packet. With the help of wiretaps MP collects the data at sending and receiving side. The role of wiretaps is to recognize the traffic flowing in network. It will duplicate the traffic and passes to MP. The MP consists of two Endace

(26)

CHAPTER 3. EXPERIMENTAL SETUP 15

DAG 3.5E cards [38]. These DAG cards are used near sender and near receiver. These cards are synchronized with respect to time and frequency respectively; by utilizing Global positioning System (GPS) antenna for obtaining timestamps accuracy of 60ns in network DAG cards [39].

3.3.2 MArC

MArC (Measurement Area Controller) is a central subsystem in Mea- surement Area manages the MP’s. According to applied rules of MArC the MP will tap the traffic and filters. It recognizes the users request information and send it to MP’s [39].

3.3.3 Consumer

Consumer is system controlled by the user. It is a Linux environment based system. The replicated packets captured by Data Acquisition and Generation (DAG) cards are stored in the consumer. It contains libcap-utils-7.0.8 is used to convert the binary traces to human readable text format. Here, the network traffic are saved in text files. These stored files are used for further analysis of protocols [38].

3.3.4 Wowza Media Server

Wowza Media Systems has server software that delivers streaming video and audio content as streaming content on a variety of platforms [40].

Wowza Media Server (WMS) can deliver multi-bitrate live and on- demand media [41]. The paper [10] discusses about both VoD and live streaming by using Wowza meadia server. It is fully interactive server for streaming multimedia content with full support for H.264, they have used FFmpeg tool for video conversion and video length.

In [42, 43] WMS software produces the broadest any-screen coverage over Flash and Silverlight-capable computers, tablets, phones, set-top boxes, media players, and game consoles. In this project Wowza media server 3 is used on Linux platform, which is and the DELL Laptop with Intel i5 M370 2.40GHz is used.

3.3.5 Wowza Client

Wowza client is a flash player, which is on a web browser. It is used to receive the flash content delivered by WMS. Here, the flash content are encoded video sequences. The Wowza client side HP Pavilion g6 laptop with an Intel i3 M370 2.40GHz processor architecture running Linux Ubuntu 10.04 LTS (Lucid Lynx) is used.

(27)

CHAPTER 3. EXPERIMENTAL SETUP 16

3.3.6 NetEm

Traffic shaping and emulation of network properties is useful in cases like delay, loss, duplication and re-ordering [1]. NetEm or TC shaper belongs to the Traffic Control (TC) bandwidth provisioning package of Linux [2]. In this project NetEm is placed in between server and client to disturb the ongoing traffic. To capture the video traffic before and behind the shaper, the Distributed Passive Measurement Infrastruc- ture (DPMI) is used [39]. It acts as an interface. All the packets are travelling through the interface will be affected with properties like loss and packet delay variation. The paper [38] explains about the NetEm shaper having more accuracy emulation property, compared to NIST- net and KauNet. It is installed on Intel Celeron(tm) 710 MHz. RAM:

392192KB with Ubuntu 10.04 LTS (Lucid Lynx), and Kernel version 2.6.32-38-generic-pae.

3.3.7 Packet Loss

The packet loss determines the number of packets that have been lost at the destination side compared to the source. It illustrates as ratio of packets lost (PL) at destination to packets sent (PS) by source on an Ethernet packet switched network.

P acketloss = PL PS

In our experiments we have used 0%, 2.5%, 5%, 10%, 15% and 20%

packet loss.

3.3.8 Packet Delay Variation

Packet Delay Variation (PDV) requires end-to-end measurements be- tween source and destination. In this era Packets can arrive at its destination node with a random time distortion. That clearly states, the time between packets at the destination is different from that at the source [44]. ITU-T recommendations have been applied for delay variations in audio and video transmissions. The delay and delay vari- ation values are expressed as D ±4D, where D is the fixed delay and 4D is variable delay. [45] ITU-T Rec. G.114 suggests that 0 to 150 ms limits can be used for one-way transmission time. 150 to 400 ms:

acceptable when provided that administrations are aware of the trans- mission time impact on the transmission quality of user applications.

The constant delay (D) value is considered as 150 ms. The delay (D) and variable delay (4D) settings used for these experiments are D±

4D=150 ms±{0 ms, 10 ms, 15 ms, 25 ms, 50 ms, 100 ms, 150 ms}.

(28)

CHAPTER 3. EXPERIMENTAL SETUP 17

3.3.9 RTMP at Client

RTMP is a sophisticated TCP-based real time networking protocol. It supports an efficient exchange of multimedia messages like video [5].

3.3.10 HTTP at Client

In [8] HTTP streaming using TCP is the popular choice of many web based applications. HTTP Dynamic Streaming supports both live and on-demand media content that adjusts to viewer connection speed, it enables high-quality H.264, network efficient HTTP streaming for mul- timedia delivery that is tightly integrated with flash platform for video content [4].

3.3.11 FFmpeg Encoding

It is a cross-platform tool. In this project the role of FFmpeg is to interact with videos in digitizing different dimensions such as encoding, splitting and capturing. It acted as an interface between Wowza client video and user, here QoP (Quality of Presentation) of FFmpeg matters to present the videos to users by an Application. The raw videos Big- Buck-Bunny, Cable car and BTH are encoded to one of the most widely used codecs are H.264/AVC (Advanced Video Coding) [46]. We had used FFmpeg with libx264 for encoding purpose. The high profile of H.264 was used. The encoder used was x264 [29]. The each video sequence has a frame rate of 25 fps and a bit rate of 800 kbps.

3.3.12 FFmpeg Splitting

The videos are taken from different sources, are split to 50 secs due to the long duration for the convenience of subjective assessment.

3.3.13 FFmpeg Grabbing

In this two offsets are used i.e., X offset and Y offset, mainly these terms are defined as XOFF and YOFF of application. The Geometry specifications are considered to grab the videoed using X, Y OFF. Off- sets must be given as pairs. The layout of window can be placed in four edges of display i.e., +0+0 (upper left edge of window), -0+0 (up- per right edge of window), -0-0 (lower right edge of window) and +0-0 (lower left edge of window) [47]. In this case +0+0 to Grab the res- olution of 640x480 (EGA) video with required parameters at client side.

(29)

CHAPTER 3. EXPERIMENTAL SETUP 18

Video name Description

Big-Buck-Bunny A computer animation video. Big-Buck-Bunny from the Netherlands computer graphics teacher Sacha Goedegebure, a comedy about a fat rabbit and three ir- ritating rodents [48].

Cable car This is video took from Wowza. These video visuals are scene at mountain area, in a same motion.

BTH This is real time advertisement video of Blekinge In- stitute of Technology.

Table 3.3.1: Description of Videos

Video Sequences Big-Buck-Bunny, Cable car, BTH.

Codec H.264/AVC

Resolution EGA (640X350) Frame−rate 25fps

Extension name .MP4

Encoder Libx264

Table 3.3.2: Characteristics of Videos

The video resolution 640x350 is used in this project. This resolu- tion is preferred in computer screens. In [49] IBM display standard is 640x350 (EGA) introduced in 1984. It is a subset of VGA. EGA (Enhanced Graphic Adapter) is in between CGA and VGA.

3.3.14 Scale

MOS scale which is recommended by ITU-T. The QoE subjective ratings for videos is generally given on a scale from 1 to 5. To measure the video quality by the user a five point scale Mean Opinion Score (MOS) is used. The User Rating (UR) for videos sequences using the scale Excellent (5), Good (4), Fair (3), Poor (2) and Bad (1). This scale is recommended by ITU-T P.910 [50] shown in table 3.3.3.

Scale Quality Impairment

5 Excellent Imperceptible

4 Good Perceptible, but not annoying

3 Fair Slightly, annoying

2 poor Annoying

1 Bad Very Annoying

Table 3.3.3: ITU-T SCALE OF MEDIA QUALITY IMPAIRMENT

(30)

CHAPTER 3. EXPERIMENTAL SETUP 19

3.4 Experimental setup and procedure

The experimental setup consists of a video streaming server, video player at client, shaper and MP. The Streaming server namely Wowza Media Server is used to send the encoded video sequences to the Wowza client. The video originator will send packets to client using HTTP and RTMP protocols. The NetEM traffic shaper is used to control the loss and variable delay shaping of traffic in the network going from server to client. To capture the video traffic at sender and receiver side of the shaper. Distributed Passive Measurement Infrastructure (DPMI) [39]

is used, with Endace DAG cards equipped in MP [51] to verify the packet loss and delay variation shaping. Streamer and player were in- stalled on Linux Ubuntu 10.04 platform the shaper also run on same Linux platform. These connections are installed with Ethernet cables.

Here, a full duplex link bandwidth of 10Mbps is concerned. The IP addresses are assigned in a network. Here, static IP address for the server is: 192.168.0.2 and for the client: 192.168.0.3 with net mask:

255.255.255.0. Here, two port numbers are concerned. The port num- ber for RTMP is: 1935 and port number for HTTP is: 8080. These are default port numbers for RTMP as well HTTP. Intially wire shark is used to check the traffic that is flowing from one device to another device [52] and finding out the extra traffic.

The different videos are collected by varying packet loss and delay variation at client by using FFmpeg grabbing, as the client side video is played by flash player. Here, three video sequences preferred namely BTH, Big Buck Bunny and Cable Car.

3.5 Assessment of Videos

In this work, subjective analyses were conducted among 35 viewers.

Among 35 viewers, 30 of them were males and 5 females are partici- pated. We had not considered 5 user ratings as they were not given genuine ratings. According to the recommendation by the ITU-R [3]

the number of human viewers participating in a subjective quality ex- periments should not be lower than 15. The standard group VQEG (Video Quality Expert Group) suggest at least 24 human observers are needed for quality assessment [53]. The observers belong to BTH university students. Nine students had been participated before in the video quality assessment. Participants with an average age of 23. For the survey of videos we had design a tool shown in the Figures 3.5.1 and 3.5.2 With local database in the backend. When user click on the opinion score and submit, the corresponding value will store in the

(31)

CHAPTER 3. EXPERIMENTAL SETUP 20

database. The values collected from the database are mathematically analyzed.

Figure 3.5.1: Login Screen

Figure 3.5.2: Assessment tool for video survey

Subjective testing for visual quality assessment has been formalized in ITU-R Rec. BT.500 and ITU-T Rec. P.910. Our survey was ac- cording to the ITU-R recommendations [3], using the Single Stimulus

(32)

CHAPTER 3. EXPERIMENTAL SETUP 21

(SS) method. Total 78 videos are rated by the end users, according to 5 point scale as shown in the Table 3.3.3. When each user rates the video, the score will be stored in the local database. All the values from the database are taken and analyzed using statistical methods discussed in the next chapter.

(33)

Results

22

(34)

Chapter 4

Results and Discussions

This chapter explains the detailed description of the obtained results.

These results are based on the experiments performed in the previ- ous chapter. These results gives the Subjective assessment and quality of the two video streaming protocols RTMP and HTTP based on the packet loss and delay variation. However, MOS rating gives best opin- ion on video quality.

To discuss and achieve these results we had used statistical methods such as Mean and standard deviation acquired from the set of data of the subjective assessment of video quality survey. Later these results are analyzed.

Based on the Recommendation BT. 500 subjective assessment in the quality of television pictures, of the International Telecommunications Union Radio communications Sector (ITU-R) [3], we had calculated mean scores of the MOS and Confidence Interval (CI) for the set of videos like BTH, Big Bunky video and Cable car for both the protocols HTTP and RTMP, based on the packet loss and delay variation.

4.1 Mean scores calculations

We have to calculate, the mean score for each and every single presen- tation, and the mean is defined as,

jk = 1 N

N

X

i=1

ijk

The standard deviation defined as,

23

(35)

CHAPTER 4. RESULTS AND DISCUSSIONS 24

δjk = v u u t

N

X

i=1

( ¯Xjk − ¯Xijk)2 N − 1 ,

Where N = Number of observers Xijk = Score of the ith observer for test condition j and video sequence k .

4.2 Confidence Interval Calculations

Once all the results of mean scores are calculated, and as the mean scores are always associated with CI, 95 % Confidence Intervals for all the mean scores are calculated.

With the 95% Confidence Interval, the exact value of difference between experimental mean score and the true mean score, will be ob- tained.

The 95% of CI is given by:[ ¯Xjk − γjk, ¯Xjk+ γjk] Where marginal error γjk,

γjk = 1.96 ∗ δjk

√N γjk = Standard deviation

N = Number of observers

4.3 RTMP and HTTP Packet loss

The below Figures 4.3.1, 4.3.2, 4.3.3 gives graphical representation of BTH advertisement, Cable Car and Big Buck Bunny streamed videos with MOS ratings on Y-axis and packet loss % on X-axis respectively for both RTMP and HTTP.

(36)

CHAPTER 4. RESULTS AND DISCUSSIONS 25

Figure 4.3.1: MOS with 95% CI vs. Packet loss for HTTP BTH and RTMP BTH

In Figure 4.3.1 the BTH RTMP and BTH HTTP videos have “GOOD”

ratings at 0% packet loss, above “FAIR” ratings from 2.5%- 10% packet loss. Once the packet loss is increasing from 10% - 15%, 20% the ratings of the users are degrading to “POOR” and “BAD” for BTH advertise- ment video for the HTTP and RTMP protocols. The both protocols have similar performance as the user ratings are matching for BTH advertisement video.

The Figure 4.3.2 it illustrates that the Cable car RTMP and Cable car HTTP videos are having slightly “GOOD” ratings at 0% packet loss, “FAIR” ratings from 2.5% - 10% packet loss. Once the packet loss is increasing from 10% to 20% the ratings of the users are degrading to

“POOR” and “BAD” for RTMP protocol. For HTTP video the ratings are slightly less when compared to RTMP up to 10% packet loss. As packet loss is increasing 10%-20% HTTP is having “POOR” and “BAD”

ratings, and slightly better than RTMP.

(37)

CHAPTER 4. RESULTS AND DISCUSSIONS 26

Figure 4.3.2: MOS with 95% CI vs. Packet loss for HTTP cablecar and RTMP cablecar

Figure 4.3.3: MOS with 95% CI vs. Packet loss for HTTP Big Buck Bunny and RTMP Big Buck Bunny

Figure 4.3.3 it illustrates that Big Buck Bunny HTTP video is hav- ing “GOOD” ratings at 0% packet loss, “FAIR” ratings from 2.5% - 15% packet loss, and when the packet loss is increasing from 15% to 20% the ratings of the users are degrading to “POOR”. For RTMP pro-

(38)

CHAPTER 4. RESULTS AND DISCUSSIONS 27

tocol it has similar ratings from 0%-5%, compared to HTTP, “FAIR”

ratings from 5% - 10%, and “POOR” ratings from 10% - 20%. As the packet loss is increasing from 5% RTMP protocol is having less ratings compared to HTTP protocol.

Figure 4.3.4 illustrates the standard deviation of all videos for both RTMP and HTTP on Y-axis and Packet loss % on X-axis. The values are ranging from 0.49 to 1.19. Comparing all the standard deviations, there is minimum divergence for all the videos up to 5% packet loss and more divergence from 10% - 20% packet loss.

Figure 4.3.4: Standard deviation for HTTP and RTMP packet loss 95% Confidence intervals for all the three video sequences for both RTMP and HTTP are shown with MOS respectively in the Figures 4.3.1, 4.3.2, 4.3.3 BTH, Cable Car, Big Buck Bunny videos. CI is mainly depends on total number of people participated during the as- sessment. X-axis shows the packet loss % and Y-axis shows the MOS of UR. From the graphs we can observe that the UR are decreasing as packet loss is increasing. But when we compare the ratings of both RTMP and HTTP, it is clear that user ratings are better for RTMP when there is minimum loss of packets and as the loss is increasing HTTP is having better ratings.

(39)

CHAPTER 4. RESULTS AND DISCUSSIONS 28

PacketLoss % BTH HTTP

BTH RTMP

Cable Car HTTP

Cable Car RTMP

Big Buck Bunny HTTP

Big Buck Bunny RTMP

0% 4.433 4.6 4 4.0666 4.56 4.6

2.5% 3.7666 3.9 3.1 3.3 3.7 3.9

5% 3.7666 3.7 3 3.166 3.7 3.666

10% 3.633 3.533 2.8 3.133 3.633 3.23

15% 2.7666 2.633 2.466 2.033 3.066 2.26 20% 1.533 1.666 1.566 1.633 1.866 1.933

Table 4.3.1: MOS Ratings for Packet Loss

From the both cases we can observe that users are giving “GOOD”

ratings at 0% packet loss, “FAIR” ratings from 2.5% to 10%, and as packet loss is increasing MOS ratings are “POOR” by the users. RTMP is having better performance at low packet loss compared to HTTP, but when observed from the graphs the behavior of RTMP streamed videos quality is degrading compared to HTTP streamed videos.

(40)

CHAPTER 4. RESULTS AND DISCUSSIONS 29

4.4 RTMP and HTTP Delay Variation

The below Figures 4.4.1, 4.4.2, 4.4.3, gives the graphical representa- tion of BTH advertisement, Cable Car and Big Buck Bunny streamed videos with MOS ratings on Y-axis and Delay variation in ms on X- axis respectively for both RTMP and HTTP. In Figure 4.4.1 the BTH HTTP streamed video has “FAIR” ratings until 0ms, as the delay vari- ation is increasing from 10ms - 50ms it has “POOR” ratings by users.

From 50ms - 150ms user ratings are tending towards “BAD”. For the BTH RTMP video at 0ms - 15ms user ratings are above “FAIR”. As the delay variation is increasing from 15ms - 150ms user ratings are tend- ing towards “BAD” from “POOR”. As the delay variation is increasing BTH HTTP is having better performance compared to BTH RTMP.

Figure 4.4.1: MOS vs. Delay Variation for HTTP BTH and RTMP BTH From the Figure 4.4.2 it illustrates that Cable Car HTTP video is having “FAIR” ratings at 0ms. As delay variation is increasing from 10ms - 50ms it has above “FAIR” ratings, and from 100ms to 150ms the ratings are tending towards “POOR” to “BAD”. In case of Cable Car RTMP video, it has “BAD” ratings at 0ms and as delay variation is increasing from 10ms to 150ms, the ratings are changing from “POOR”

to “BAD”.

From the Figure 4.4.3 it illustrates that Big Buck Bunny HTTP video is having “GOOD” ratings by the user at 0ms and while delay variation is varying from 0ms - 15ms it has “FAIR” ratings. From 15ms to 150ms the HTTP video has “POOR” ratings. For Big Buck

(41)

CHAPTER 4. RESULTS AND DISCUSSIONS 30

Figure 4.4.2: MOS vs. Delay Variation for HTTP Cable car and RTMP Cable car

Bunny RTMP video, it has “FAIR” ratings from at 0ms - 25ms and very “POOR” ratings from 25ms - 150ms.

The Figure 4.4.4 illustrates the standard deviation of all videos for both RTMP and HTTP on Y-axis and delay variation in ms on X- axis. The values are ranging from 0.46 to 1.014. As comparing all the standard deviations, there is minimum divergence for RTMP compared to HTTP videos until 25ms and more divergence of ratings for RTMP videos compared to HTTP videos. This intends that users are giving

“GOOD” ratings for HTTP rather than RTMP when there are more delays.

(42)

CHAPTER 4. RESULTS AND DISCUSSIONS 31

Figure 4.4.3: MOS vs. Delay Variation for HTTP Big Buck Bunny and RTMP Big Buck Bunny

Figure 4.4.4: Standard deviation for HTTP and RTMP Delay variation The 95% Confidence intervals for all the three video sequences for both RTMP and HTTP are shown in the Figures 4.4.1, 4.4.2, 4.4.3 refers to CI of Big Buck Bunny, Cable Car and BTH videos respectively.

CI mainly depends on total number of people participated during the assessment. X-axis shows the packet loss % and Y-axis shows the MOS

(43)

CHAPTER 4. RESULTS AND DISCUSSIONS 32

of user ratings. From the graphs we can observe that user ratings are decreasing as delay variation is increasing. But when we compare the ratings of both RTMP and HTTP, it is clear that user ratings are

“GOOD” for HTTP when compared to RTMP.

Delay Varia- tion (ms)

BTH HTTP

BTH RTMP

Cable Car HTTP

Cable Car RTMP

Big Buck Bunny HTTP

Big Buck Bunny RTMP

150±0 3.166 3.7 3.4 3.3233 4.166 3.866

150±10 2.366 3.4 2.866 2.833 3.266 3.633

150±15 2.466 3.3 2.5666 2.166 3.166 3.266

150±25 2.266 1.933 2.4666 1.766 2.8 2.5

150±50 2.2 1.6 2.4666 1.7333 2.433 1.866

150±100 1.7 1.433 1.9 1.6 2.233 1.566

150±150 1.5 1.366 1.7 1.5 2.233 1.666

Table 4.4.1: MOS Ratings for Packet Delay Variation

From both cases we can observe that users are giving “FAIR” ratings at 0ms delay variation, as we observe from 10 ms to 50 ms and as delay variation is increasing MOS ratings are given “POOR” by the users. By observing the graphs the behaviour of RTMP streamed videos quality is degrading Compared to HTTP streamed videos.

4.5 Validity Threats

In our thesis we are lacking real time physical network, in order to overcome we had proposed a small network in a lab environment.

We had recorded the videos by using ffmpeg tool, for which the recorded video quality depends on tool. The videos are recorded for both RTMP and HTTP to compare at different packet loss and delay variation. We have chosen ffmpeg to record the videos at client as there are no other tools to record the RTMP videos.

(44)

Conclusions and Future Work

33

(45)

Chapter 5

Conclusions and Future Work

5.1 Conclusion

Experimental results of our thesis shows the video quality of RTMP and HTTP video streaming protocols based on subjective video quality assessment. As a part of our experiment, we had streamed the videos from the server to client over traffic shaper, by varying the factors like packet loss and delay variation, for different video sequences such as BTH, Cable car and Big Buck Bunny. All the videos at the client side are recorded by changing the network factors. By using QoE subjective assessment methodology, we had collected feedback from different user perceptions. The collected feedback is later analyzed by statistical methods like mean and Confidence intervals.

Our first research question deals with impact of packet loss on RTMP and HTTP protocols on video quality. From the subjective assessment analysis, we can observe the quality between both RTMP and HTTP protocols for different video sequences. As the packet loss is increasing the MOS ratings are changing from “GOOD“ to “POOR”

by the users, as there is a degrade in the video quality. RTMP pro- tocol is having better ratings than HTTP from 0%-5% packet loss, as packet loss metric is increasing, HTTP is slightly better as compared to RTMP.

Our second research question deals with impact of delay variation on RTMP and HTTP protocols on video quality. From the subjective assessment analysis, we can observe the quality between both RTMP and HTTP protocols for different video sequences. As delay variation is increasing MOS ratings are pointed less for the video sequences.

When the delay variation is very low, users are giving better ratings for RTMP streamed videos when compared to HTTP video sequences.

When high delay variations are applied, HTTP performs better than

34

(46)

CHAPTER 5. CONCLUSIONS AND FUTURE WORK 35

RTMP. Form the analysis of results HTTP is having good performance than RTMP.

Our third research question deals with what recommendations can be given for both protocols when network disturbances like packet loss and delays places. Form the analysis of the results, we can say that RTMP protocol is having less performance with respect to video qual- ity, and more performance examined by HTTP when there is more packet loss and delay variation. We recommend, using HTTP protocol is preferred for streaming the videos in wide networks like web, public network domains. As RTMP is a very secure protocol, and it having better quality when there are fewer disturbances in the network, we suggest using RTMP in small set of networks for video conferences and e-learning video lectures in the universities, offices.

5.2 Future Work

Future research has to be done on 3G and 4G networks with large set of experiments should be conducted for both RTMP and HTTP.

Comparing the two protocols videos by using PEVQ software with subjective and objective assessments.

(47)

Bibliography

[1] (2012) NetEm. Linuxfoundation.org. [Online]. Avail- able: http://www.linuxfoundation.org/collaborate/workgroups/

networking/netem

[2] S. Hemminger, “Network emulation with netem,” in Linux Conf Au. Citeseer, 2005, pp. 18–23.

[3] ITU-R Recommendation BT.500-11, “Methodology for the subjec- tive assessment of the quality of television pictures,” International Telecommunication Union, Geneva, Switzerland, Research Report, 2002.

[4] C. group, “HTTP versus RTMP: Which Way to Go and Why?,.”

[5] W. Premchaiswadi, A. Tungkasthan, and N. Jongsawat, “En- hancing learning systems by using virtual interactive classrooms and web-based collaborative work,” in Education Engineering (EDUCON), 2010 IEEE, pp. 1531–1537.

[6] (2012) Dynamic streaming in Flash Media Server.

Adobe.com. [Online]. Available: http://www.adobe.com/devnet/

flashmediaserver/articles/dynstream_advancedpt1.html

[7] ITU-T Telecommunication G.1070, “Opinion model for video- telephony applications,” International Telecommunication Union, Research Report, 2007.

[8] K. Singh, Y. Hadjadj-Aoul, and G. Rubino, “Quality of experi- ence estimation for adaptive HTTP/TCP video streaming using h. 264/AVC,” in Consumer Communications and Networking Con- ference (CCNC), 2012 IEEE, pp. 127–131.

[9] O. Oyman and S. Singh, “Quality of experience for HTTP adaptive streaming services,” IEEE Communications Magazine, vol. 50, no. 4, pp. 20 – 7, 2012/04/. [Online]. Available:

http://dx.doi.org/10.1109/MCOM.2012.6178830 36

(48)

BIBLIOGRAPHY 37

[10] D. Cymbalák, F. Jakab, and M. Michalko, “Next generation iptv solution for educational purposes,” in Emerging eLearning Tech- nologies and Applications (ICETA), 2011 9th International Con- ference on. IEEE, 2011, pp. 41–46.

[11] T. Minhas, “Network Impact on Quality of Experience of Mobile Video,” Ph.D. dissertation, Blekinge Institute of Technology, 2012.

[12] M. Fiedler, H. Zepernick, L. Lundberg, P. Arlos, and M. Petters- son, “Qoe-based cross-layer design of mobile video systems: Chal- lenges and concepts,” in Computing and Communication Technolo- gies, 2009. RIVF’09. International Conference on. IEEE, 2009, pp. 1–4.

[13] U. Engelke, “Modelling perceptual quality and visual saliency for image and video communications,” Ph.D. dissertation, Licentiate, Blekinge Institute of Technology, 2012.

[14] K. D. Vogeleer, “On the Quality of Delivery for Variable Bit Rate Video,” Ph.D. dissertation, Licentiate, Blekinge Institute of Tech- nology, 2012.

[15] (2012) Gain control over your audiences quality of experience.

ip-label.co.uk. [Online]. Available: http://www.ip-label.co.uk/

end-user-experience-monitoring-solutions/image-tv-video/

[16] T. Hosfeld, M. Fiedler, and T. Zinner, “The QoE provisioning- delivery-hysteresis and its importance for service provisioning in the Future Internet,” in Next Generation Internet (NGI), 2011 7th EURO-NGI Conference on, june 2011, pp. 1 –8.

[17] M. Venkataraman, M. Chatterjee, and S. Chattopadhyay, “Eval- uating Quality of Experience for Streaming Video in Real Time,”

in Global Telecommunications Conference, GLOBECOM 2009.

IEEE, pp. 1 –6.

[18] S. Ickin, K. De Vogeleer, M. Fiedler, and D. Erman, “The effects of Packet Delay Variation on the perceptual quality of video,” in Local Computer Networks (LCN), 2010 IEEE 35th Conference on, oct. 2010, pp. 663 –668.

[19] J. Kellokoski and T. Hamalainen, “User-centric approach to always-best-connected networks,” in Ultra Modern Telecommuni- cations and Control Systems and Workshops (ICUMT), 3rd Inter- national Congress on, oct. 2011, pp. 1 –6.

(49)

BIBLIOGRAPHY 38

[20] H. French, J. Lin, T. Phan, and A. Dalal, “Real time video QoE analysis of RTMP streams,” in Performance Computing and Communications Conference (IPCCC), IEEE 30th International.

IEEE, 2011, pp. 1–2.

[21] W. Wei, “Flash Media Server and Its Application in the Simula- tor of VHF Marine Radio Station,” in Information Engineering and Computer Science. ICIECS 2009. International Conference on, pp. 1 –4.

[22] (2012) Adobe developers connection. Adobe.com. [Online].

Available: http://www.adobe.com/devnet/rtmp.html

[23] (2012) Adobe to Open Flash Platform Mes- saging Protocol. Adobe.com. [Online]. Avail- able: http://www.adobe.com/aboutadobe/pressroom/

pressreleases/pdfs/200901/012009RTMP.pdf

[24] (2012) HTTP Dynamic Streaming. Adobe.com. [Online]. Avail- able: http://www.adobe.com/products/hds-dynamic-streaming.

html

[25] X. Gong and D. Zhang, “Research on web server based on red5, tomcat and apache,” Advanced Materials Research, vol. 282, pp.

721–725, 2011.

[26] Q. Chang, Z. Yang, and Y. Song, “A scalable mobile live video streaming system based on RTMP and HTTP transmissions,” Ad- vanced Research on Computer Science and Information Engineer- ing, pp. 113–118, 2011.

[27] (2012) RTMP, RTMFP and HTTP - The multi-protocol experience with Flash Media Delivery on Flash Player 10.1.

Adobe.com. [Online]. Available: http://blogs.adobe.com/ktowes/

2009/10/rtmp_rtmfp_and_http_the_multi.html

[28] O. Jan. (2011) what is H.264? . Streamingmedia.com. [Online].

Available: http://www.streamingmedia.com/Articles/Editorial/

What-Is-.../What-is-H.264-74735.aspx

[29] Joint Video Team, “Draft ITU-T Recommendation and Final Draft International Standard of Joint Video Specification (ITU- T Rec. H.264 | ISO/IEC 14496-10 AVC),” 8th Meeting: Geneva, Switzerland, 23-27 May, Tech. Rep., 2003.

(50)

BIBLIOGRAPHY 39

[30] B. Zhang, S. Zeng, R. Xu, D. Guo, J. Yan, and W. Wang, “Design and implementation of a scalable system architecture for embed- ded multimedia terminal,” in Electrical and Control Engineering (ICECE), 2011 International Conference on. IEEE, 2011, pp.

618–621.

[31] F. Caron, S. Coulombe, and T. Wu, “A Transcoding Server for the Home Domain,” in Portable Information Devices, 2007.

PORTABLE07. IEEE International Conference on, may 2007, pp.

1 –5.

[32] (2012) FFmpeg Document. ffmpeg.org. [Online]. Available:

http://ffmpeg.org/ffmpeg.html

[33] J. Schumacher, M. Dobler, E. Dillon, G. Power, M. Fiedler, D. Er- man, K. De Vogeleer, M. Ramos, and J. Argente, “Providing an user centric always best connection,” in Evolving Internet (IN- TERNET), 2010 Second International Conference on. IEEE, 2010, pp. 80–85.

[34] M. Sidibe, H. Koumaras, I. Kofler, A. Mehaoua, A. Kourtis, and C. Timmerer, “A novel monitoring architecture for media services adaptation based on network QoS to perceived QoS mapping,”

Signal, Image and Video Processing, pp. 307 – 20, 2008. [Online].

Available: http://dx.doi.org/10.1007/s11760-008-0083-2

[35] ITU-T FG IPTV, “Focus group on IPTV standardization:

Definition of quality of experience (QoE),” International Telecom- munication Union Telecommunication Sector Publications, Research Report, 2007. http://ties.itu.int/ftp/public/itu-t/

fgiptv/readonly/PreviousMeetings/20070122MountainView/il/

T05-FG.IPTV-IL-0050-E.html.

[36] S. Baraković, J. Baraković, and H. Bajrić, “QoE dimensions and QoE measurement of NGN services,” in Proceedings of the 18th Telecommunications Forum, TELFOR, 2010.

[37] M. Pinson and S. Wolf, “Comparing subjective video quality test- ing methodologies,” in SPIE Video Communications and Image Processing Conference, vol. 5150, 2003, pp. 573–582.

[38] P. Arlos, “On the Quality of Computer Network Measurements,”

Ph.D. dissertation, Blekinge Institute of Technology, 2005.

[39] P. Arlos, M. Fiedler, and A. A. Nilsson, “A Distributed Passive Measurement Infrastructure,” in PAM2005, 2005, pp. 215–227.

(51)

BIBLIOGRAPHY 40

[40] M. McClure, Managing Costs in a Multipaltform World, Vol. 35 Issue 1 ed., Jan/Feb2012, pp. 20-24.

[41] (2012) Wowza Media Systems, LLC. Wowza.com. [Online]. Avail- able: www.wowza.com/resources/WowzaMediaServerUsersGuide.

pdf

[42] (2012, July) Wowza Media Systems at IBC. wowza.com. [Online].

Available: http://www.wowza.com/news/ibc2012-preview

[43] (2012) Wowza Media Server 3.1. wowza.com. [On- line]. Available: http://broadcastengineering.com/automation/

wowza-server-streaming-media-east

[44] J. Welch and J. Clark, “A Proposed Media Delivery Index (MDI),”

2006.

[45] ITU-T G.114, “One-way transmission time,” International telecommunications union telecommunication sector, Research Re- port, 2006.

[46] ITU-T REC-H.264, “H.264: Advanced video coding for generic audiovisual services,,” International telecommunications union telecommunication sector publications, Tech. Rep., 2010.

[47] (2012) How to Specify the location of windows. mit.edu. [Online].

Available: http://stuff.mit.edu/afs/athena/project/consultdev/

answers/xwindows/xwindows_geometry.html

[48] (2012) Bigbuckbunny Homepage. bigbuckbunny.org. [Online].

Available: http://www.bigbuckbunny.org/

[49] IBM, IBM Enhanced Graphics Adapter. minuszerodegrees.net.

[Online]. Available: http://www.minuszerodegrees.net/oa/OA%

20-%20IBM%20Enhanced%20Graphics%20Adapter.pdf

[50] ITU-T Rec. P.910, “Subjective video quality assessment meth- ods for multimedia applications,” International telecommunica- tions union telecommunication sector, Research Report, 1999.

[51] (2012) Endace time distribution server. Endace.com. [Online].

Available: http://www.endace.com/time-distribution-servers.

html

[52] (2012) Wireshark. Wireshark.com. [Online]. Available: http:

//www.wireshark.org/docs/

(52)

BIBLIOGRAPHY 41

[53] Video Quality Experts Group, “Final report from the video quality experts group on the validation of objective models of multimedia quality assessment, phase I,” VQEG, Sept, 2008.

(53)

Appendix

42

(54)

Appendix A

Apendix A

A.1 Network Configuration

To get network in a proper condition the permanent settings are changed at network interface configuration used /etc/network/interfaces at ter- minal. Where ifup and ifdown settings are generated. These conditions will change the interface identity.

# ifconfig ethX ipaddress netmask 255.255.255.0 up

The above command is used to change the settings at Client and Server. This feature will change physical interface to logical interface.

To take place the above settings restart is required to networking service under Linux OS.

# sudo /etc/init.d/networking restart

A.2 Speed and Duplex settings

A full duplex bandwidth of 10Mpbs is prefered, to change the speed and duplex settings the following command is used

# ethtool -s ethX speed 10 duplex full advertise 0x002 autoneg on To get speed and other information of ethX

# ethtool ethX

43

(55)

APPENDIX A. APENDIX A 44

A.3 Emulator Commands Packet loss

# tc qdisc add dev eth2 root netem delay X %

# tc qdisc change dev eth2 root netem delay X %

Delay Variation

# tc qdisc add dev eth2 root netem delay X.ms

# tc qdisc change dev eth2 root netem delay X ms Y ms

In the above stated command the term X stands for Delay, the term Y stand for delay variation.

The FFmpeg commands used in this project are:

FFmpeg Grabbing

f f mpeg −f x11grab −sameq −r25 −s 640x350 −i : 0.0+X−of f set, Y − of f set −vcodec libx264 −vpre losslessultraf ast −threads 0 video.M P 4

FFmpeg Splitting

f f mpeg −i f ile.mp4 −sameq −ss hh : mm : ss −t hh : mm : ss outf ile.mp4

(56)

Appendix B

Apendix B

This section gives the detail MOS ratings over videos took for subjective analysis. In upcoming figures clearly explains the ratings on three videos in packets loss and delay variation.

45

(57)

APPENDIX B. APENDIX B 46

B.0.1 BTH HTTP Packet loss ratings

Table B.0.1: Packet loss user ratings on BTH HTTP Video

(58)

APPENDIX B. APENDIX B 47

B.0.2 BTH RTMP Packet loss ratings

Table B.0.2: Packet loss user ratings on BTH RTMP Video

(59)

APPENDIX B. APENDIX B 48

B.0.3 Cable car HTTP Packet loss ratings

Table B.0.3: Packet loss user ratings on Cable car HTTP Video

(60)

APPENDIX B. APENDIX B 49

B.0.4 Cable car RTMP Packet loss ratings

Table B.0.4: Packet loss user ratings on Cable car RTMP Video

(61)

APPENDIX B. APENDIX B 50

B.0.5 Big Buck Bunny HTTP Packet loss ratings

Table B.0.5: Packet loss user ratings on Big Buck Bunny HTTP Video

(62)

APPENDIX B. APENDIX B 51

B.0.6 Big Buck Bunny RTMP Packet loss ratings

Table B.0.6: Packet loss user ratings on Big Buck Bunny RTMP Video

References

Related documents

46 Konkreta exempel skulle kunna vara främjandeinsatser för affärsänglar/affärsängelnätverk, skapa arenor där aktörer från utbuds- och efterfrågesidan kan mötas eller

Det finns många områden som kan studeras inom ämnet, men vi har valt att avgränsa oss till att undersöka om minskad arbetstid är det som önskas av anställda eller om det är andra

What is interesting, however, is what surfaced during one of the interviews with an originator who argued that one of the primary goals in the sales process is to sell of as much

Qualitative Analysis of Video Packet Loss Concealment with Gaussian Mixtures.. Daniel Persson, Thomas Eriksson and

Experiment 1 Mean Delay [ms].. Experiment 2 Mean

The main aim of each industry is to provide better products with higher quality, but what makes them successful, is the process of preparing a product. These processes must consume

In order to analyze the impact of delay and delay variation on user‟s perceived quality in video streaming, we designed an experiment in which users were asked

Swedenergy would like to underline the need of technology neutral methods for calculating the amount of renewable energy used for cooling and district cooling and to achieve an