How to evaluate usage of media flow systems: and how to use statistical metrics for rule based control.

(1)

and how to use statistical metrics for rule based control.

HELEN ELEMIDA

Master’s degree project report at CSC Supervisor: Olov Engwall

Examiner: Olle Bälter

(2)

(3)

Evaluation of complex systems is as necessary now as ever.

To conduct a successful evaluation, in other words an evaluation that shows something interesting and perhaps unknown about the system, one should be structured and plan well. This degree project used the goal question metric approach to conduct an evaluation of how the system MAMA, a media flow system used for television broadcasting ser- vices, was used during one year. By using the goal question metric approach it was possible to specify goals, questions and metrics that provided data interesting for system operators. The model facilitated the planning phase of the evaluation of MAMA and clearly showed how one keeps a good structure throughout an evaluation. This degree project report may serve as a guide of how to conduct a fruitful evaluation of any system. Automation and dynamic flow control are in high demand when human workers are ex- pensive. A natural step in modernising flow systems is to make more flows automated and dynamic. A suggestion of how to set the time of deadline of an order was produced.

By constructing a fuzzy logic controller it was found the- oretically possible to obtain reasonable results. The fuzzy logic controller that was suggested, aims to mimic the decisions that otherwise would be made by human operators.

The suggested controller was believed to be able to provide a reasonable deadline, with the restrictions in the system.

(4)

Att utvärdera användande av mediaflödessystem

och hur man använder statistikmått for regelstyrning.

Utvärdering av komplexa system är lika nödvändigt nu som tidigare. För att genomföra en lyckad utvärdering, med andra ord en utvärdering som visar något intressant och kanske ej tidigare känt om systemet, bör man vara struk- turerad och planera väl. Detta examensarbete använde goal question metric approach för att genomföra en utvärde- ring av hur systemet MAMA, ett mediaflödessystem som används för tjänster vid sändning av television, användes under ett år. Genom att använda goal question metric approach var det möjligt att specificera mål, frågor och mått som försåg operatörer med intressant data. Modellen under- lättade arbetet med att ta fram en plan för utvärderingen av MAMA och visade på ett tydligt sätt hur man behåller struktur i planeringen. Detta exjobb kan användas som en guide för hur man kan genomföra en utvärdering av god- tyckligt system. Ett naturligt steg i modernisering av flö- dessystem är att göra fler delar av flöden automatiserade och dynamiska. Ett förslag på hur deadline av en order kan sättas togs fram. Genom att konstruera en regulator som använder oskarp logik fanns det vara teoretiskt möjligt att uppnå rimliga resultat. Den föreslagna regulatorn har som mål att härma de beslut som annars skulle tas av mänsk- liga operatörer. Regulatorn tros kunna ta fram en rimlig deadline, med de villkor som gäller i systemet.

(5)

This degree project report has been conducted at the school of computer science and communication,CSC, KTH. Sunstone Systems AB has been kind to supply the idea for the degree project as well as help, support, resources and good company all which I would like to thank them for. I would also like to thank my supervisor Prof. Olov Engwall, for steady supervision and help with the degree project report and Marcus Carrigan, to whom I have turned to when the need to discuss problems have arisen as well as the need for some nonsense befitting two five year olds. Last but not least I would like to thank Erik Jerilgård and my mother Agneta Edin, for love and support along the way.

Introduction

This degree project intend to use information attainable in a metadata- and media flow system to gain knowledge of the usage of a workflow system and to suggest a more dynamic way to control some of the flows of the system. This degree project was carried out at Sunstone Systems AB, a Swedish company specialised in ad- vanced information handling.

When using large and complex systems it can be difficult to know how a system is used. During development of such systems it is common to test the usage, performance and correctness by developing test cases. But when a system already is in use it becomes increasingly difficult to extract the information necessary to gain knowledge of the usage of the system.

1.1 Background

Sunstone Systems AB has developed a system integration and workflow platform, MAMA, which handles metadata and directs media flow between multiple IT- systems of customers within the television channel industry. Further development of the platform has been conducted continuously since launch. From delivery of a file from its producer, MAMA handles all treatment of the file needed for it to become ready for airing.

MAMA is used to make media ready for broadcast. One company is supplying the service of operating MAMA, which several channels pay for. This one company responsible for the daily operations wishes to gain a deeper knowledge of how each channel utilises MAMA.

Examples of operations performed on a media file is reformatting and addition of subtitles and speaker tracks. In order to detect problems in time, such as delayed deliveries or hardware failures, the system aims to have the media files ready for airing a certain number of days before they are scheduled for broadcast. This

(8)

number of days was at the time of writing this degree project report equal to two.

The deadline for delivery is today set automatically to ten days before broadcast.

1.2 Problem description and objectives

Which statistical metrics are desirable and possible to measure from the subtitle and speaker script flow of MAMA in purpose to gain knowledge of how the system is used, and how may these metrics be used to determine a suitable deadline?

The first part of this degree project aims to decide which metrics are suitable to elucidate the usage of a specific system, MAMA.

Some of the flows of the system in question are at this time managed by human operators with expert knowledge of the aforementioned system. The interest of collecting appropriate statistical metrics comes from a desire to visualise the usage and change in usage over time, as well as a desire of being able to control some of the system flows in a more dynamic manner. The second part of this degree project aims to suggest a method or technique suitable for controlling such flows. This suggestion will not be implemented, and in extension not tested. The suggestion should be viewed as purely theoretical.

The results of this degree project are:

• A list of metrics to be collected that may be used to elucidate the usage of the subtitle flow within MAMA.

• Charts displaying the collected metrics mentioned above.

• A suggestion of how more dynamic control of setting the deadline of an order within the subtitle and speaker flow may be achieved.

1.2.1 Delimitations

For the first part of this degree project, finding metrics, only one of the many flows within MAMA was considered, namely the flow controlling subtitles and speaker scripts from order to delivery. A review of this flow is illustrated and described in section 3.3.2.

This degree project report will give an example of how one may simplify the daily use of systems which today need human operators to make experienced decisions.

The example will only consider the specific decision of when to set the deadline of

(9)

an order to a translation company. This will contribute to the knowledge development of companies similar to Sunstone Systems AB, companies which employ and develop workflow systems without dynamic flow control.

(10)

(11)

Theory, workflow evaluation

In this chapter relevant theory regarding workflow evaluations will be presented along with a recollection of earlier work.

To gain knowledge of how a system is used, which the first part of this degree project is all about, it is helpful to have a framework or a method to use or follow.

The goal question metric approach, described below in section 2.1, is a method or approach to conduct a workflow evaluation. The approach was used in this degree project to obtain structure before and during the first part of this degree project, i.e., finding metrics.

A workflow performance evaluation is usually undertaken for a specific purpose.

This purpose might be to minimise production errors or to locate process bottle- necks or just to get a grip on what is happening. Historically only computer hardware performance was evaluated [1], but today software must be evaluated as well as hardware. To monitor a system one collects data of the performance of an ex- isting system and later analyses this data. How the data is collected depends on the system subjected to the evaluation. Data can for example be collected by using system logs or by using code built into the system. According to Basil, Calidera and Rombach [2] as well as Nick, Althoff and Tautz [3] measurement must be focused and based on goals and models. This is the base premise of the goal question metric approach.

2.1 The goal question metric approach

Basil and Weiss [4] described a data collection method for evaluating software development methodologies using goal-directed data collection. In the article the authors states that the main steps of a software evaluation should be:

• Establish the goal of the data collection

(12)

• Develop a list of questions of interest

• Establish data categories

• Design and test data collection forms (when data is collected by people filling out forms)

• Collect and validate data

• Analyse data

An approach where one first decides what to measure and then how to use the collected data will not work since there are too many observable characteristics in software [2]. The goal question metric approach, or GQM as hereafter mentioned, is an approach to measure organisations and software where one first needs to define goals of the measurement in question. The approach was first defined in order to evaluate defects in certain projects at NASA Goddard Space Flight Center environment [5] as described in [2].

The advantages of the goal question metric approach are [6]:

• Ensure adequacy, consistency and completeness of a measurement plan and in extension of data collection.

• Manage the complexity of the measurement plan. Increased complexity occurs when there are to many attributes to measure.

• Stimulate a structured discussion and promote consensus about measurement and improvement goals.

Disadvantages of the goal question metric approach are [7]:

• It can only be used by experienced users of the system who can define and answer questions relevant to the purpose of the evaluation.

• Does not allow for comparison of software systems.

• Does not provide a means of combining different attributes to a single value for the sake of comparison.

The result of use of the GQM is a specification of a measurement system, with regard to a specific set of issues, and a set of rules of how to interpret the measurement data. This resulting specification has three levels:

1. Conceptual level (GOAL): A goal is defined for an object, for a variety of reasons, with respect to various models of quality, from various points of view and relative to a certain environment. Objects of measurement may be products, processes and resources.

(13)

• Products: Artifacts, documents and deliverables that are produced dur- ing the system life cycle, for example specifications, programs, test suites.

• Processes: Software related activities often associated with time, for ex- ample specifying, designing, testing, interviewing.

• Resources: Data used by processes in order to produce their output, for example personnel, hardware, software, office space.

2. Operational level (QUESTION): Questions are used to define the way the assessment/achievement of a specific goal is to be performed based on a char- acterising model. The questions should try to characterise the object of measurement with respect to a selected quality issue.

3. Qualitative level (METRIC): A set of metrics are associated with each ques- tion in order to answer it in a quantitative way. The term metric is here loosely defined; it can mean a base metric, a derived metric or a composite metric. The collected data may be objective or subjective.

• Objective: If the data only depend on the object that is being measured and not on the viewpoint from which they are taken. Examples: number of documents, size of a file.

• Subjective: If the data depend on both the object that is being measured and the viewpoint from which they are taken. Examples: readability of a text, level of user satisfaction.

2.1.1 Setting goals

The process of setting goals in the conceptual level is a critical part of a GQM approach. It is the first step towards finding which metrics to collect. Below follows a description of how to form these goals.

A goal consists of three parts. An Issue, an Object, and a Viewpoint along with a Purpose. See Figure 2.1

To form these parts of a goal, information gathering is needed. The first source of information is policies and strategies of the organisation conducting the evaluation. By analysing corporate policy statements, strategic plans and interviewing relevant subjects in the organisation, one derives the issue and the purpose of the goal. The second source of information is the description of processes and products of the organisation that are relevant to the measurement to be performed. From this one can derive the object of the goal. The third source of information is the model of the organisation. This gives the viewpoint of the goal.

From the specification of a goal one can derive meaningful questions that characterise said goal in a quantifiable way [2, 3]..

(14)

In [8] it was pointed out that one of the strong points of GQM is that it is multi- faceted and supports multiple perspectives.

Figure 2.1: Information gathering for goal definition. Based on information in [2]

2.1.2 Alternatives to GQM Balanced scorecard

Kaplan sand Norton defined the balanced scorecard (BSc) method as a multidimen- sional framework for describing, implementing and managing strategy at all levels of an enterprise. This is done by linking objectives, initiatives and measures to an organisations strategy. The approach starts with the vision and strategy of the organisation and then look at this from four different perspectives of the organisation:

• financial perspective

• customer perspective

• business process perspective

• learning and growth perspective

The factors to the overall vision and strategy of the organisation from each of these perspectives are investigated to form indicators. From the indicators one find the

(15)

metrics to be collected. "In summary, a scorecard is to be used to facilitate the translation of strategy into action." [9]

Application of measurement in industry method

The application of measurement in industry method (AIM) is the result of a ESPRIT funded project derived by nine organisations from France, Germany, Italy and the UK. It is a handbook of a method based on multiple evaluation methods, GQM is one of them. AIM consists of four main stages, Assess, Analyse, Measure and Improve. The objectives are first to understand where development is starting from, and then control and improve the status of the software. By identifying development shortcomings it is possible to establish primary goals. The metrics are derived in the same way as in GQM through questions and related metrics. The method continues by explaining how these metrics should be collected and validated. The AIM was derived to aid in development of software. [10]

2.2 Previous work, GQM

In this section follows a recollection of some earlier uses of the goal question approach.

Extended version of the GQM

Berander and Jönsson [11] presented an extended version of GQM. The authors found that the ordinary GQM approach had three issues, as follows:

1. The amount of metrics might be quite large and may be a problem for an industrial application of a measurement framework.

2. The resulting measurement framework may not cover all relevant perspectives of the process it measures.

3. Only the perspectives of those defining the goals of the GQM are taken into account.

The extension of the ordinary GQM consisted of a prioritisation of goals and questions in order to deal with list items number one and three and a categorisation of questions to deal with number two.

There exists multiple extensions of the original GQM with different modifications.

This shows that there is a consensus that the GQM is a good base to work with that one can modify to fit one’s needs.

(16)

Successful use of the GQM approach, 1

Schlumberger RPS, the Retail Petroleum Systems division of Schlumberger, success- fully used the GQM approach to assess the effectiveness of the RPS division’s measurement practices [12]. Schlumberger RPS had collected data from most projects from the late 1990s to 1993, when an evaluation was conducted. The evaluation revealed several weaknesses. Data points were often incomplete, inconsistent and lacked context and explicit purpose. Schlumberger RPS found that they could not draw conclusions about the effectiveness of their practices. The implemented GQM-based measurement approach helped avoid many of the earlier problems in the organisation’s former measurement activities.

This is a great example of what can be accomplished by using a structured plan for an evaluation. When Schlumberger did not use the GQM they simply tried to collect all they could think of, without structure or a plan.

Successful use of the GQM approach, 2

Gencel, Petersen and Imran proposed in [13] a framework to support software organisations making informed decisions in metrics selection when planning measurement programs, based on the GQM approach. The case study showed that by using their GQM-based framework the cost of an evaluation became transparent to the decision makers. The framework included an iterative goal-based metrics selection process with prioritisation of goals.

(17)

How to realise a GQM based evaluation and what was used to generate charts of the result

3.1 Scientific method

This degree project used rational discussion as a means of gaining knowledge from experts. Controlled observation was used to gather the metrics decided upon in the evaluation. This since there was no possibility to affect variables of the system, because it only exists one version of the system and it is in sharp use every hour of the day.

3.2 GQM based measurement

This degree project report utilised the GQM approach when deciding which metrics to collect from MAMA. This decision was made since there was a need for structure and consistency during the planning of the evaluation. One of the benefits of using the GQM is that it stimulates discussion, a needed element during the beginning of this evaluation. The disadvantages of the GQM did not pose a problem during this evaluation. As a result a GQM based specification was produced without regard to whether the metrics were feasible to obtain. Next another GQM based specification was produced where the feasibility and expected effort of collection were taken into account.

3.2.1 Establishing goals

As described, in Chapter 2, the GQM method in [4] starts with a step where the goals of the data collection are to be established. This step corresponds to the conceptual level described in section 2.1.

(18)

Issues and purposes were obtained by discussion of the organisational policies and strategies of the organisation utilising MAMA. Objects were obtained by discussion of the products and processes of MAMA. Viewpoints were obtained by discussion of the organisational model of the organisation utilising MAMA. These combined formed the goals of the evaluation. All discussion were conducted with resources from Sunstone Systems AB, since they are experts on MAMA, both concerning system specifics and operation.

Policies and strategies of the organisation

The organisation in question is the company responsible for the service of making programmes, commercials, films and other ready for broadcast utilising MAMA.

This company supplies this service to multiple channels. Lets hereafter call this company the broadcasting company. The organisational policies of the broadcasting company were not relevant in this evaluation. This since organisational policies mostly deals with how the staff should handle different situations, such as fires.

The strategies of the broadcasting company on the other hand were found relevant.

The broadcasting company’s business strategy consists of upholding a contract written together with a customer. This contract may specify an amount of late deliveries that are permitted, what portion of bandwidth a channel is guarantied when full capacity of the system is used and more. Each contract is different from the other.

What all the contracts have in common is some notion of time and quality aspects throughout the flow.

Products and processes

As described earlier in this degree project report, the products of the subtitle and speaker flow of MAMA is either a subtitle file or a speaker file. The processes are described in section 3.3.2. The objects of the GQM were found to be the timeliness and quality of deliveries.

Organisational model

There are only two stakeholders relevant to the subtitle and speaker script flow.

One is the broadcasting company and the other is the translation companies. This means that the viewpoint of this evaluation is either the one of the broadcasting company or the one of the translation companies.

3.2.2 Finding questions of interest

The next step of the method described in [4] is to develop a list of questions related to each goal. These questions were produced as a means to more fully specify the goals. They were produced by discussion with system experts.

(19)

3.3 Data and compilation

Data from the subtitle and speaker flow have been collected from the year 2011 and forward. It was decided that the data to be used in this degree project were to be the data from the year 2011. The data was in a format not suitable for this degree project, but some of the metrics decided upon were possible to extract from this data. The extracted data was stored in a MongoDB instance [14], as decided by Sunstone Systems AB.

3.3.1 MongoDB -The big difference from relational database management systems

MongoDB is a NoSQL open-source document database. The use of a MongoDB database in this degree project was predetermined by Sunstone Systems AB.

In contrast to relational database management systems (RDMS) where the database schemas are static, they are dynamic in NoSQL databases. This means NoSQL databases are more easily changed when the needs for an application changes over time. A better name for the collection of NoSQL databases would be Non relational databases. A document is a complex data structure that can consist of many different key-value pairs, or key-array pairs, or nested documents. The documents in MongoDb have the structure of JSON, JavaScript Object Notation. An example document could look like:

{

name: "sue", age: 26, sex: "F",

groups: ["sports", "news"]

} Structure of collected data

When deciding the data model to use in a MongoDb database instance one should consider how the data will be used. There are two main ways to represent related data. One way is to use what is called embedded data. The second way to represent related data is to use references.

Embedded data is a data model that allows an application to store related data in the same document. The use of embedded data provides better read performance, along with an ability to access and retrieve related data in a single operation [14].

Embedding of documents may lead to document growth after creation of a document. Document growth in a collection of documents may have negative impact on write performance and may lead to data fragmentation.

(20)

References store a relationship between related documents. This is similar to nor- malised data models.

This degree project used the embedded data model since there was a natural way to structure the data in documents with embedded subdocuments and since good read performance was wished for. The risk of document growth was concluded to be small since nearly no updates or writes to documents would be performed after creation.

3.3.2 A review of the subtitle and speaker flow within MAMA

A review of the subtitle and speaker flow is presented below to better understand the collected data. The resulting products of the subtitle flow of MAMA is either a subtitle-file or a speaker-file. Figure 3.1 shows the possible events of the workflow from submission of an order to the point where the subtitle or speaker script is ready for airing (RFA).

(21)

Figure 3.1: Normal flow for a subtitle or speaker script order

Below follows a step by step description of the events of the workflow.

(22)

1. Order.

An order is submitted along with a deadline. A specific translation company is asked to perform the task requested. The task may be to create subtitles in one or more languages to a media-file or to create a speaker script. The translation company either accept the order or reject it.

2. Update.

An update of the order may be submitted at any time from the broadcasting company. To exemplify this an update has been inserted in figure 3.1 between an acceptance of an order and an transfer of attachment. The lines are dotted to represent the non-certainty of occurrence. An update may be a new deadline or that a new media-file or attachment-file will be used. The deadline may be postponed or brought forward. After this the translation company either decide to to accept it or to reject it, If the update is rejected the broadcasting company confirms they have seen the rejection.

3. Update Confirmed. Sometimes the translation company and the broadcaster come to an understanding and the update may be considered to be confirmed from the beginning. When this takes place there are only this event and no update as described above. The lines in figure 3.1 are dotted to represent the non-certainty of occurrence.

4. Transfer Attachment. At this point the broadcaster tries to send an attach- ment to the translation company. Typically this is an original script. If there are no Transfer Attachment Failed the transfer of an attachment was success- ful.

a) Transfer Attachment Failed. The transfer of an attachment was unsuccessful. A human operator need to press a button to re-send the attachment-file.

5. Transfer Media. At this point the broadcaster tries to send a media-file to the translation company. Typically this is a video file. If there are no emph- Transfer Media Failed the transfer of media was successful.

a) Transfer Media Failed. The transfer of media was unsuccessful. A hu- man operator needs to press a button to re-send the media file.

(23)

6. Delivery of product. The translation company delivers what was ordered, ei- ther subtitles or a speaker script.

7. Automatic Check. The delivered subtitles or speaker script are checked for syntax errors. This is an automatic process.

a) Check OK. The delivered subtitles or speaker script passed the automatic syntax control.

b) Check Failed. The delivered subtitles or speaker script did not pass the automatic syntax control. The translation company is notified.

8. Manual Check. The delivered subtitles or speaker script are checked for tex- tual errors. This is a manual task performed by a human operator.

a) Check OK. The delivered subtitles or speaker script passed the manual text control.

b) Check Failed The delivered subtitles or speaker script did not pass the manual text control.

9. Ready For Airing (RFA). Now the delivered subtitles or speaker script are ready for broadcasting.

3.3.3 Chart tool

As a means to compile the collected data a web application was developed. The application pulled data from a MongoDB database instance and used libraries from Google Charts [15] to generate charts of the collected data. Google Charts is a JavaScript library made by Google which provides classes and methods for creating many different types of charts.

(24)

(25)

Theory, of flow control

In this section necessary theory relating to the second part of the degree project, flow control, will be presented. The goal of this part of the degree project was to en- able a more automatic and dynamic way to set a deadline of a translation delivery.

At the time of writing this degree project report the process of setting a deadline was automatic but not dynamic. The deadline was set by taking the date of when the media file was to be broadcast and subtract ten days. The number of days to be subtracted was decided by system experts, and have been changed multiple times throughout the life of the system.

By encapsulating the knowledge of these system experts in rules the setting of a deadline can be made more dynamic and automated. Below in section 4.1 follows a general description of what a rule based system is.

4.1 Rule based systems

The goal of rule based systems is to encapsulate the knowledge of human experts in rules.

A rule based system controller consists of a rule base, a database and an inference engine. The rule base consist of rules. Most rules have the form:

IF condition THEN action

The database is where all information of the system is stored. It is this data that constitutes the condition part of the rule. Binding them together is the inference engine. The inference engine extracts the data corresponding to the state of the system and matches it to a rule in the rule base. The output is an action.

When constructing such a rule-based system controller a large portion of time will consist of interviews with system experts, this in order to acquire information needed

(26)

to write rules.

A fuzzy logic controller is a controller much in the same way as the rule based system controller described above in section 4.1, with two major differences. The first difference is that a fuzzy controller uses fuzzy input sets, described in section 4.2. The second difference is that the inference engine of a fuzzy controller uses fuzzy logic, also described in section 4.2, instead of classical logic. Several applications has shown the usefulness of fuzzy logic when there are imprecisions such as large time delays between the application of a control signal and its effect, non-linearities in the system dynamics, or when the sensor information may be untrustworthy [16].

In the case of the subtitle and speaker flow of MAMA there may certainly be non- linearities in the system dynamics, as time dependencies that depend on human beings tend to not be linear.

An introduction to fuzzy logic follows and further on in section 4.3 a fuzzy controller will be described in more detail.

4.2 Fuzzy logic

To understand how a fuzzy controller works one must first have some knowledge of fuzzy logic and fuzzy set operators. This section gives an introduction to fuzzy logic and section 4.2.1 describes the fuzzy operators. Fuzzy logic is an extension to classical logic that takes into account the un-precise nature of real systems.

In 1965 Lotfi A. Zadeh proposed the fuzzy set theory [17] on which fuzzy logic is built.

In classical logic, the sets are crisp and the law of excluded middle is one of three laws of thought. The three laws of thought are three fundamental axiomatic rules generally attributed to Aristotle. The law of excluded middle, also called the law of the excluded third, states that any statement is either true or its negation is. Fuzzy logic is a multi-valued logic where the law of excluded middle does not pertain.

Variables may have values in between true and false, such as nearly true or mostly true. The range of all possible values in a set is called a universe of discourse [17].

One could for example use fuzzy logic in order to describe outdoor temperatures.

At 10^◦C the outdoor temperature could be said to be both cold and warm, and at 25^◦C both warm and hot. Figure 4.1 shows how the membership functions of this example would look.

(27)

Figure 4.1: Fuzzy sets describing temperature regions

A fuzzy set is defined through specification of a membership function µ [17] that assigns each element a degree of membership, between zero and one, to the set.

The overlap of the membership functions reflects the imprecise nature of the underlying concept. These membership functions can take different forms, such as trapezoidal, triangular, bell shaped or some other arbitrary form. The most used form of membership function is the triangular shape, which has been shown by experience to be a very good choice in most cases [18].

4.2.1 Fuzzy set operators

At some point when using a fuzzy controller, described in the next section, it is likely that some logic operators will be needed. Take the outdoor temperature described above, where 10^◦C could be considered both cold and warm. This example shows a union between two fuzzy membership functions. Below follows a description of the fuzzy logic operators that may be needed when building and using a fuzzy logic controller.

Union, intersection and complement are standard operators within classical logic and are defined within fuzzy logic as well. These operators are defined below.

The union of two membership functions µ_A and µ_B is defined as the maximum of the two membership functions. This corresponds to the logical OR operator [17].

See Figure 4.2a.

µ_A∪B= max(µ_A, µ_B)

This way to define the union is called the maximum criterion.

(28)

The intersection of two membership functions µ_A and µ_B is defined as the minimum of the membership functions. This operation corresponds to the logical AND operator. See Figure 4.2b.

µA∩B= min(µ_A, µB)

This way to define the intersection is called the minimum criterion

The complement of a fuzzy set A with membership function µ_A_¯ is defined as the negation of the membership function. This corresponds to the logical NOT operator [17]. See Figure 4.2c

µA¯= 1 − µ_A

(29)

Figure 4.2: Fuzzy operators

(a) Fuzzy union

(b) Fuzzy intersection

(c) Fuzzy complement

(30)

4.3 A fuzzy logic controller

A fuzzy logic controller [19] consists of:

• A fuzzification interface that involves the functions:

– mapping of the range of possible input values into fuzzy membership functions/sets

– giving these fuzzy sets linguistic values that can be seen as labels of the sets.

– measuring the values of input variables

– performing fuzzification. This is done by the inference engine. This means converting measured input values into fuzzy values using the membership functions

• A rule base consisting of:

– a database of definitions necessary for the linguistic control rules – a fuzzy control rule base

• An inference engine which infers fuzzy control actions by using fuzzy implica- tion and the rules of inference in fuzzy logic.

• A defuzzification interface which:

– maps the range of output values into corresponding universes of discourse – performs defuzzification, which gives a non fuzzy control action from an

inferred fuzzy control action

Below follows further explanation of the parts needed for a functioning fuzzy controller.

4.4 Fuzzy rule base

An example of a rule using classical logic could be:

IF temperature IS 100 AND rpm IS 1200 THEN risk IS 86%

describing the risk of overheating the engine of a motorcycle. By using fuzzy logic this rule may be written as:

IF temperature IS hot AND rpm IS V eryHigh THEN risk IS high [20]

(31)

Which is more readable and more general. The first parts concerning temperature and rpm is called antecedents and the latter concerning the risk is called the consequent [20]. These rules should be derived from information gathered by interviewing system experts. In the example above these experts would be either motorcycle mechanics or the engineers who designed the engine in question. By using this linguistic way of expressing rules one can design a controller with a much smaller set of rules than if one were to use conventional rule based control [21].

4.5 Inference engine

The inference engine of a fuzzy controller is the part that calculates an output from input and rule base. There are two major types of inference engines. One is called Mamdani type inference engine, described in section 4.5.1 below, and the other is called Sugeno type inference engine. The main difference between them is the characteristics of their output. A Mamdani type inference engine, described in section 4.5.1 below, outputs a fuzzy set and a Sugeno type outputs a linear function or a constant value.

According to MathWorks Documentation Center [22], the Mamdani inference engine is more intuitive than the Sugeno type and has gained widespread acceptance. Since the task was to find an example of how a deadline may be decided more dynamic and automatic, intuitive was suitable. Because of this a mamdani inference engine was used. There will not be any description of the Sugeno type inference engine.

4.5.1 Mamdani type inference engine

The most used inference engine was created by Mamdani and Assilian in 1975 [23].

It was first used to control a steam engine and boiler. The Mamdani method is widely accepted for capturing expert knowledge [24] and was therefore used in this degree project.

The first thing an inference engine does is to evaluate the antecedent for each rule.

This first step is called input fuzzification and is the process of obtaining membership values from crisp inputs. If there are more than one part of an antecedent of a rule, a fuzzy operator will be applied to obtain a single membership value. This degree project report will continue with the example with the motorcycle engine mentioned earlier.

Figure 4.3a shows an evaluation of the antecedents of rule 1;

"IF temperature IS hot AND rpm is high then risk IS high"

The two input values are: a temperature of 78 degrees Celsius and an rpm of 12800/min. The Mamdani inference engine first fuzzify the inputs (the two left-

(32)

most graphs of 4.3a) to obtain the membership values of respective antecedent.

Then the two parts of the antecedents are joined by a fuzzy operator, in this case fuzzy conjunction (AND), to obtain a single antecedent value. This antecedent value is then used to get an output set from the consequent membership function.

The inference engine will evaluate every rule that applies to the input values. In this case there was one more rule applicable to the input values. Figure 4.3b shows the evaluation of rule 2;

"IF temperature IS warm AND rpm is normal then risk IS normal".

As a last step the inference engine uses an aggregation operator on the output sets to produce a final combined output set, see figure 4.3c. The most used aggregation operator is called the max aggregation [25]. This will be the aggregation operator used in this degree project. The max aggregation operator calculates the final output set by taking the maximum from the individual outputs where there is an overlap of one or more output sets [25].

(33)

Figure 4.3: Evaluation of the antecedents of the rules..

(a) Evaluating antecedent of rule 1

(b) Evaluating antecedent of rule 2

(c) Combining outputs from rule 1 and rule 2 into a single fuzzy set

(34)

4.6 Defuzzification

Defuzzification is the process of calculating a crisp result from the output of a Mamdani inference engine. Basically the result of the defuzzification is a weighted average. Different defuzzification methods use different weighting. Defuzzification can be done in a number of ways but only the one most used will be described below.

4.6.1 Center of gravity (COG)

This method is referred to in literature as either center of gravity (COG) or centroid of area (COA). In this degree project report the metod will hereafter be referred to as COG. Using the COG defuzzification method [26, 27] will make a crisp output, Z, change continuously when the input does, which is a desirable property in most modelling and control applications [28]. The COG method finds the point where if a vertical line were placed it would divide the aggregated output set into two sets of equal mass.

Z_COG= Pmax

i=minx_i· µ(x_i) Pmax

i=minµ(xi) (4.1)

where max is the number of rules used. The value on the horizontal axis of the center of gravity of the i^thoutput set is x_i and µ(x_i) is the value of the membership function of the fuzzy set in question at x_i.

4.7 Previous work, fuzzy logic

The fundamental idea of using formal rules in order to make decisions has existed a long time. Aristotle (384-322 BC) was said to be the first to formulate a formal system of syllogisms for proper reasoning given initial premises [29].

4.7.1 Early progress with rule based expert systems

DENDRAL was the first knowledge-intensive system, it utilised a large amount of special-purpose rules [30, 29]. DENDRAL had as input an elementary formula of a molecule together with a mass spectrum provided by a mass spectrometer. The program then generated all possible molecular structures consistent with the formula, calculated what the mass spectrum would be for each structure and compared these to the mass spectrum given [31].

MYCIN was developed in 1975 to diagnose blood infections and was able to per- form as well as some experts. What MYCIN brought to the evolution of rule based systems was an incorporation of a calculus of uncertainty [32].

The first successful commercial system based on rules was R1 [33] that helped

(35)

configure orders for new computer systems. This system is said to have saved the Digital Equipment Corporation an estimated 40 million dollars a year by 1986 [29].

4.7.2 Automatic fuzzy control of a subway train

One of the most famous applications of fuzzy logic is the control of the Nanboku line of the Sendai Subway system in Sendai, Japan [34]. The control developed by Hitachi used a fuzzy controller to run the train and proved itself to be capable of performing operation comparable to skilled human train operators.

In showing that fuzzy logic can be used in such an every day thing as a subway train the market for fuzzy control was opened up. The application of fuzzy logic in a subway train is a prime example of the strengths of a fuzzy logic controller.

4.7.3 Software cost estimation using fuzzy logic

Mittal, Parkash and Mittal presented a new model using fuzzy logic to estimate effort required in software development [35]. Triangular fuzzy membership functions were implemented and the defuzzification method used was a weighted average of an optimistic cost estimate, most likely cost estimate and an pessimistic cost estimate. The authors found, through a comparison study, that their proposed model was best on the basis of mean absolute relative error.

This software cost estimation shows how many possible areas fuzzy logic may be used in.

4.7.4 Maximum power point tracking using fuzzy logic

In [36] a control method for maximum power point tracking of a photovoltaic system was presented. The proposed method uses a fuzzy logic controller applied to a DC-DC converter device (a converter with an output voltage greater than its input voltage). The results show that the method increase the efficiency of energy production.

4.7.5 Energy management of an electric vehicle using fuzzy logic The goal of the authors of [37] was to find a solution to the problem of battery over-discharge and associated damage in an plug-in series hybrid electric vehicle resulting from inaccurate estimates of the state of charge. A new quantity called the battery working state (BWS) was defined using fuzzy logic. The quantity was based on battery terminal voltage and state of charge of a battery. The BWS was used by a fuzzy logic energy-management system to make a decision on the power split between battery and engine. Three gaussian curve membership functions was used to identify acceptable values and two trapezoidal membership functions to identify very high or very low values. Simulations were performed and showed

(36)

that the energy-management system was effective in preventing the battery from over-discharging.

(37)

Finding suggestions for flow control

The suggestion of method or technique to be used for flow control is based on the literature study conducted in the early stages of this degree project. There was no implementation of this. Since there was no implementation there was no evaluation of the suggested method or technique.

In order to in theory design the fuzzy controller there were a list of things to be done.

1. Decide upon the fuzzy sets.

2. Decide upon membership functions for the fuzzy sets.

3. Determining a set of fuzzy rules.

4. Decide upon an inference engine.

5. Decide upon a defuzzification method.

The inputs to the fuzzy controller were decided upon during discussion with system experts. They were found by discussing the question; If the time of deadline would not have been set automatically to 10 days before scheduled broadcast, what would weigh in to the decision?

The fuzzy sets, membership functions and rule base were decided upon together with system experts. This to make sure that the inputs would be handled like it would by human operators. The shape of the membership functions were chosen to be trapezoidal or triangular, considering the nature of the fuzzy set and the underlying concept. This since the trapezoidal and triangular shape are both the easiest and are in most cases a good choice [18]. The inference engine to be used was decided to be of Mamdani type, described in section 4.5.1, since it provides the most suitable output. As a defuzzification method the COG was used since this is the most widely used method today [28].

(38)

(39)

Results

In this chapter the resulting metrics will be presented. Relevant visualisations of the collected metrics, generated with the visualisation tool will be supplied. What is needed for the flow control will be presented here.

6.1 Results from GQM

The results from the conducted GQM evaluation was divided into two segments.

One segment where all goals, questions and metrics were included and another segment where only those which metrics were possible to collect within the time frame of the degree project were included.

6.1.1 GQM specification-wanted

Tables showing the results from the GQM evaluation follows, with no respect taken to if the resulting metrics were possible to collect at the time of this degree project.

(40)

Table 6.1 illustrates a goal to gain knowledge of how deliveries from the translation companies to the broadcasting company match requested quality and timeliness.

The questions more fully specify which information that was requested from this evaluation. During discussions it came to be known that system experts knew there were deliveries made after deadline, but not how many and not how much after.

At todays date the time between deadline and scheduled broadcast is set to a pre decided number of days for all deliveries (currently 10 days).

Table 6.1: GQM specification. Deliveries to the broadcaster Goal 1 Purpose Gain knowledge of

Issue the quality and timeliness of

Object deliveries from the translation company to the broadcasting company

Viewpoint in the broadcasting company’s perspective

Question 1.Q1 How long time before or after deadline are translations¹delivered²?

Metric Time in milliseconds/days and hours

Question 1.Q2 How long time after subtitles are delivered is text-check made²?

Question 1.Q3 How many delivered subtitles are rejected in text-check²? Metric 1.M Number of rejections

Question 1.Q4 How many³delivered speaker scripts are rejected in syntax- check²?

Metric 1.M Number of rejections

Question 1.Q5 How large is the time span between deadline for delivery and scheduled broadcast²?

Metric 1.M Time in milliseconds/days and hours

Question 1.Q6 How long before scheduled broadcast is required material available to the broadcaster for sending to translation company?

1Translations not approved in text-check are not regarded.

2To be possible to analyse with regard to variations during specific months, weeks, weekdays and hours. Variations over time is also to be possible to be analysed

3The sought answer is wanted in both absolute numbers and percentage.

(41)

Table 6.2 illustrates a goal to gain knowledge of how deliveries from the broadcasting company to the translation companies match requested quality and timeliness. In this goal, quality represents the need for continuity. The broadcasting company can not be said to provide good quality if the orders are often changed for example.

Table 6.2: GQM specification. Deliveries to the translation company Goal 2 Purpose Gain knowledge of

Object orders and deliveries from the broadcaster to the translation company

Viewpoint in the translation company’s perspective

Question 2.Q2 How long time before deadline of translation delivery are orders submitted²?

Question 2.Q3 How long before deadline of translation delivery has required material been sent from broadcaster?

Question 2.Q4 How many orders are changed after submission, by whom and why?

Metric Amount

Metric Percentage of total

2To be analysed with regard to variations during specific months, weeks, weekdays and hours.

Variations over time is also to be analysed

(42)

Table 6.3 illustrates a goal to gain knowledge of how the system bandwidth is used.

Both how the different customers (channels) of the broadcasting company utilises bandwidth and how much of the usage consists of re-transfers. A re-transfer could be needed, for example when a transfer did not succeed at first, but it could also be unnecessary. A transfer may take more time than the operator think it should, and when said operator try to transfer the file one more time, two transfers will be registered by the system.

Table 6.3: GQM specification. Resource utilisation Goal 3 Purpose Gain knowledge of

Issue utilisation of

Object bandwidth for transfer of media and attachments from broadcaster to translation company

Viewpoint in the broadcaster’s perspective

Question 3.Q1 How is the transfer bandwidth utilised^{2 4}? Metric Amount bytes transferred

Amount of attachments and media transferred Percentage utilised bandwidth

Question 3.Q2 How much of the utilisation consists of re-transfers?

Metric Amount re-transfers in percentages of total

4All answers to questions is to be possible to group by translation company, channel, sort of event and more.

(43)

Table 6.4 illustrates a goal to gain knowledge of errors in the system. More specific if there are some flows that are more error prone than others and what kinds of errors there are.

Table 6.4: GQM specification. Error frequency Goal 4 Purpose Gain knowledge of

Issue (technical) error frequencies

Object in subtitle and speaker script workflows Viewpoint in the broadcaster’s perspective

Question 4.Q1 What is the (technical) error frequency per flow²?

Metric Amount of errors in flow Amount of non errors+amount of errors

Question 4.Q2 How many of the (technical) errors are due to each kind of error²?

Metric Amount

Table 6.5 illustrates a goal to gain knowledge of how much time is spent waiting for a resource to become available.

Table 6.5: GQM specification. Error and disruption rate, media flow Goal 5 Purpose Gain knowledge of

Issue holdups

Object in video workflows

Viewpoint in the broadcaster’s perspective

Question 5.Q1 How much media is in line for a resource?

Metric Amount in absolute number

Metric Amount in kB

Question 5.Q2 How long time does media spend wanting in line for usage of a resource?

Metric Amount in absolute number

(44)

6.1.2 GQM Specification-Possible to collect

Not all of the GQM goals specified above were in the end evaluated, this due to their complexity when it comes to collection and lack of time. A specification from which metrics were actually collected follows. The resulting specification consists of tables 6.6 and 6.7 described in section 6.1.1.

Table 6.6: GQM specification. Deliveries to the broadcaster Goal 1 Purpose Gain knowledge of

Object deliveries from the translation company to the broadcasting company

Viewpoint in the broadcasting company’s perspective

Question 1.Q1 How long time before or after deadline are translations¹delivered²?

Question 1.Q2 How long time after subtitles are delivered (all) is manual check made²?

Question 1.Q3 How many delivered subtitles are rejected in manual check²? Metric 1.M Number of rejections

Question 1.Q4 How many³ delivered speaker scripts are rejected in automatic check²?

Metric 1.M Number of rejections

Question 1.Q5 How large is the time span between deadline for delivery and scheduled broadcast²?

Metric 1.M Time in milliseconds/days and hours

1Translations not approved in text-check are not regarded.

(45)

Table 6.7: GQM specification. Deliveries to the translation company Goal 2 Purpose Gain knowledge of

Object orders from the broadcaster to the translation company Viewpoint in the translation company’s perspective

Question 2.Q2 How long time before deadline of translation delivery are orders submitted²?

Question 2.Q3 How long before deadline of translation delivery has required material been sent from broadcaster?

Question 2.Q4 How many orders are changed after submission, by whom and why?

Metric Amount

6.2 Charts-Display of collected metrics

In this section charts of the collected metrics will be displayed. The charts were rendered with a visualisation tool, described briefly in section 3.3.3. The order in which the results are presented follows the order of the questions in the resulting GQM specification above in section 6.1.2.

(46)

Figure 6.1 shows the result from the collected time differences between delivery and deadline. As can be seen there were many deliveries delivered after deadline. One may also see that during the second half of 2011 those deliveries delivered after deadline were delivered a longer time after deadline than those during the first half of the year. The reason for this is unknown, but could probably be explained by the right person working at the broadcasting company. Perhaps the general workload was larger the second half of the year or the broadcasting company was working under new directions. The author of this degree project report may only speculate and think it is best to leave this sort of speculation to the ones with knowledge of historical events. The average value, when only considering positive values, was calculated to be 3.1 days if rounded to nearest first decimal. The amount of negative values were 2864 and the amount of positive values were 9029, which means that 31.7% of the deliveries made in 2011 were delivered after deadline.

Figure 6.1: Timespan between delivery and deadline by date of delivery

(47)

A ratio of 31.7% late deliveries is high. An explanation to why so many deliveries were late may be that the translation companies are aware of that the deadline is set to 10 days before scheduled broadcast. The translation companies sometimes use this and deliver the translation late. Another possibility is that the translation companies and the broadcasting company came to an agreement that it was acceptable to push the deadline but the broadcasting company did not change the deadline in the system. This means that the system contains erroneous information.

To get correct data and in turn correct charts, the broadcasting company needs to be strict in entering correct information into the system and to enter all changes.

The mean value of the timespan of the delayed deliveries was 4 days.

(48)

Figure 6.2 shows the result from the collected time differences between delivery and text-check. An increase in the time differences started around June and in December it started to decrease. The highest value was between 1836 and 1890 hours. The lowest value was between -162 and -216 hours. Negative values indicate text-checks made before the time of delivery. These values were believed to be values entered incorrectly into the initial datasource, from where the data in this degree project were extracted. The big bulk of values are very close to the zero line, indicating that the text check was made more or less directly after delivery. This is how the broadcasting company wants it. The increase of the timespans in the second half of the year could relate to the increase of the timespans of late deliveries in figure 6.1.

If a delivery does not arrive when planned it might end up in a queue of deliverables to be text-checked when possible. Deliveries made late on Fridays also contribute to the larger timespans since no or at least very few text-checks are made during the weekend. The average value, when only considering positive values, was calculated to be 8.6 hours if rounded to nearest first decimal.

Figure 6.2: Timespan between delivery and text-check by date of delivery

(49)

Figure 6.3a shows the result from the collected amounts of rejections in text check grouped by date. In absolute numbers the most often occurring value was one rejection and the highest value was 13 rejections. The highest percentage of rejections occurred in April, when both of the delivered deliverables were rejected. The percentage of rejections were mostly under 10 % during the year of 2011.

Figure 6.3b shows the result from the collected amounts of rejections in text check grouped by weekday. In absolute numbers the highest value of 46 rejections was on Mondays. There were no rejections in text-check on Saturdays or Sundays, which is reasonable since text-check is a manual check and the regular staff does not work on weekends. The percentage of rejections was high at the beginning of the week and decreased throughout the week. The amount of deliverables to be text-checked pile up at the end of the week which means there are more to text-check, and in extension more possible rejections, in the beginning of the upcoming week.

Figure 6.3: Rejections in text-check. Total amount of submitted orders in 2011 was 12194.

(a) Rejections in text-check by date of rejection.

(b) Rejections in text-check by weekday of rejection. Sunday -1, Monday-2, Tuesday- 3, Wednesday-4 , Thursday-5, Friday-5 and Saturday-7.

Charts displaying rejections in text check grouped by week and month may be found in Appendix A.

How to evaluate usage of media flow systems: and how to use statistical metrics for rule based control.

Att utvärdera användande av mediaflödessystem

Contents

Introduction

1.1 Background

1.2 Problem description and objectives

Theory, workflow evaluation

2.1 The goal question metric approach

2.2 Previous work, GQM

How to realise a GQM based evaluation and what was used to generate charts of the result

3.1 Scientific method

3.2 GQM based measurement

3.3 Data and compilation

Theory, of flow control

4.1 Rule based systems

4.2 Fuzzy logic

4.3 A fuzzy logic controller

4.4 Fuzzy rule base

4.5 Inference engine

4.6 Defuzzification

4.7 Previous work, fuzzy logic

Finding suggestions for flow control

Results

6.1 Results from GQM

6.2 Charts-Display of collected metrics