Server Design to Ensure Quality and Fairness in Mobile Crowdsourcing

(1)

IN

DEGREE PROJECT INFORMATION AND COMMUNICATION TECHNOLOGY,

SECOND CYCLE, 30 CREDITS ,

STOCKHOLM SWEDEN 2019

Server Design to Ensure Quality and Fairness in Mobile

Crowdsourcing

VILLE GRANFORS

KTH ROYAL INSTITUTE OF TECHNOLOGY

SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE

(2)

(3)

Abstract

Mobile crowdsourcing solves complex problems by utilizing the untapped power of a crowd, connected through the fantastic mobile devices we use in our daily life. These gadgets are equipped with a versatile set of sensor that could be used for gathering data about a specific location in combination with questions to the human carrier. Common problems in crowdsourcing systems is how to ensure that the contributed data from the crowd is of a high quality and how to do task allocation fairly. A large population of users is often needed to ensure a high quality of data and coverage, every participant is important and the system have to do be designed with the population in mind.

In this thesis we discuss the responsibilities of the server in a crowdsourcing system and presents a system model which tries to tackle the problems of quality and fairness. A prototype of the system model was developed (CrowdS) to determine its potential, for both Android and iOS devices.

A long running test was performed to evaluate the performance of CrowdS with the main focus on determine how well the system performed in terms of coverage of the search area and fairness of earnings and prices. The test was executed on both platforms for a couple of weeks. The vast majority of all completed tasks were finished within 10 minutes of being created, with a median time of 3 minutes and 32 seconds seconds. Jain’s fairness index measured an overall high fairness for both the price of tasks at 0.944 and the earnings made by participants at 0.941. The radius of the search area had to be extended to maximum of 800 meters for roughly on third of the tasks to find the required number of participants.

Keywords:

Mobile Crowdsourcing, Quality, Fairness, Server, Sensor, Real Life Data

(4)

Sammandrag

Mobil crowdsourcing löser komplexa problem genom att utnyttja kapaciteten av en grupp, sammankopplad med de mobila enheter vi använder dagligen. Dessa maskiner är utrustade med en m˚angsidig uppsättning av sensorer som kan användas för att samla in data om en specifik plats i kombination med fr˚agor till den mänskliga bäraren. Vanliga problem med gruppbaserad problemlösning är hur man försäkrar att den insamlade informationen är av en hög kvalité och hur man utför uppdragsallokering rättvist. En stor användargrupp behövs för att uppn˚a en hög kvalité av data och täckning, varenda deltagare är viktig och systemet m˚aste utformas med denna användargrupp i ˚atanke.

I denna avhandling diskuterar vi ansvaret som en server har i mobil gruppbaserad problemlösning och presenterar en systemmodel som försöker hantera problemen med kvalité och rättvisa. För att utvärdera systemmodellen utvecklades en prototyp (CrowdS), för b˚ade Android och iOS-enheter.

Ett l˚angvarande test utfördes för att utvärdera Crowds prestanda med huvudfokus p˚a att bestämma hur bra systemet presterade inom täckning för sökomr˚adet och rättvishet för vinster och priser. Testet utfördes p˚a b˚ada plattformarna under ett par veckor. Den stora majoriteten av alla utdelade uppgifter var avklarade inom 10 minuter efter att de hade skapas, med en median tid p˚a 3 minuter och 32 sekunder.

Jain’s rättvise index användes för att f˚a en uppfattning om hur rättvist vinster och priser var för uppgifter, det räknades till 0,944 respektive 0,941. Radien för sökomr˚adet behövde utsträckas till dess maximum av 800 meter för cirka en tredjedel av uppgifterna för att hitta det tillräckligt m˚anga deltagare.

(5)

Acknowledgements

This project would not have possible without all the help and support from friends and family. Special thanks and recognition to my fellow workmate Johan Waller and the number one tester Mihhail Matskin for giving me the chance.

(6)

CONTENTS CONTENTS

1 Introduction

Mobile devices are great tools for retrieving all kinds of information. This diverse piece of equipment that we carry around in our pockets is used everyday by a growing part of the Swedish population [1]. With multiple kinds of sensors it can read its surroundings and quickly relay the gathered data over an internet connection. The potential of these gadgets is unrealized and could be used much more frequently when solving complex problems by asking a crowd. Crowdsourcing is a problem solving model which taps into the potential of this crowd by asking these volunteers to together solve a problem which might not been possible otherwise. Wikipedia [2] is a commonly used crowdsourcing application, in which anyone knowledgeable about a subject can provide their wisdom to an online repository for all to read.

1.1 Background

Mobile crowdsourcing is a subfield of the crowdsourcing research which incorporates mobile devices as the main tool for connecting participants in a crowd and retrieving information. It is a growing research area and has been for the last decade, the work performed in this thesis has already been used as a base for another paper that was presented and published on the CSCE’18 congress [3]. The mobile devices supplied by the crowd is an important resource in a crowdsourcing system, the data can be automatically gathered and sent back to solve simpler tasks. If the task is to complex to be solved by the use of sensors alone, the possibility to forward the task to device owner is always available. Having an incentive to participate is an important property of a mobile crowdsourcing system since it costs resources such as battery, time and bandwidth when replying to a task. Without a functional reward module in the system the size of the set of possible long term participants is limited to strictly volunteers, retaining a healthy population of participants is a problem of itself.

1.1.1 Problem formulation

The first focus of this thesis is to define the responsibilities of a mobile crowdsourcing server in terms of fairness and quality. Fairness in regard to the task allocation, how the set of service providers chosen for a task should be selected. And what can be done by the server to ensure a high quality of data. The second focus is to develop and implement a system model that incorporates the findings from the first focus and then evaluate its performance.

1.1.2 Problem statement

What responsibilities does the server have to ensure fairness and a high quality data in a mobile crowdsourcing system?

1.2 Purpose

The purpose of the degree project is to research and implement a mobile crowdsourcing system, that can be used to solve complex tasks using the crowd. The resulting framework can be used as a platform for future mobile crowdsourcing projects, at KTH.

1.3 Goal

The main goal is to develop and implement a mobile crowdsourcing system. Which includes modules for managing; task allocation, reward management, reputation and quality management. A secondary goal is to evaluate the system by collecting data from users participating in an experiment and analyzing the results.

(9)

1.4 Method 1 INTRODUCTION

1.4 Method

The research performed in this thesis applies a quantitative research method with a deductive approach.

Data were collected from experiments and statistically analyzed.

The established information in the research area decides what approach, inductive or deductive, is best suited for the experimental design. The inductive approach bases a hypothesis after observing the outcome of experiments, searching for a pattern that can be replicated and confirmed with more experiments. This approach is best suited for instances when the established information is insufficient, making it next to impossible to base an hypothesis without first performing experiments. If the research area already have established information such as known facts, physical laws and theory, it would be possible to apply the deductive approach. This approach utilizes the established information to formulate a hypothesis and then confirms or denies it depending on the collected data from experiments. The problem in 1.1.2 is fundamentally deductive, multiple mobile crowdsourcing systems exists with knowledge about the research area. Based on the established knowledge the responsibilities for the server can be defined and implemented.

Experiments are performed to determine how well the implementation performs.

A mobile crowdsourcing system was developed for this study, including a complete set of components necessary for the distribution and creation of tasks. To determine system performance a 3 week long test was performed with the collaboration of course students in KTH Kista. During the duration of the experiment data was gathered and then analyzed to make conclusions about the system.

1.5 Delimitations

This project is done in close collaboration with Johan Waller, and we both worked on the completion of CrowdS. However, we have distributed the work so that my main focus is the back end of the project, the server and the persistent storage while Waller is responsible for the front end, the mobile devices.

1.6 Outline

The remainder of this thesis is organized as follows. In the second chapter, the research area and related works are briefly explained. In the third chapter, the system model is described. In the fourth chapter the implementation is explained. In the fifth chapter experiments and the results are discussed. Last but not least, our contributions are summarized in the sixth chapter together with possible future directions of our work.

(10)

2 RELATED WORK

2 Related Work

In this chapter some background of mobile crowdsourcing systems will be presented along with a previous works in the area.

2.1 Background

A mobile crowdsourcing system is a system that uses the crowd and their mobile devices to solve problems.

These problems can be divided into two different categories, human intelligence and sensor tasks. Human intelligence tasks, are tasks that require the aid of the device owner to be completed, answering a question or taking a photo for instance. Sensor tasks on the other hand are fully automatic and takes full advantage of the diverse mobile devices supplied by the crowd to collect information about the environment. All users are not equal, and should not be treated as such if the goal is to achieve a high quality of data. For instance, there is a wide range of different sensor of different quality in our mobile devices, and some humans are more knowledgeable than others. The history of past interactions with both man and machine has to be accounted for when deciding what users to distribute the task to. The location of the users is also a deciding factor when allocating tasks in a mobile crowdsourcing system. Completing tasks consumes resources like bandwidth, battery and time, and most people do not voluntarily participate. An incentive of some kind is required to get the users to complete their tasks. Taking the mentioned problems in this section into consideration, the architecture of a mobile crowdsourcing system needs to have the following components to handle those problems; task allocation, reputation and quality management and reward management.

2.2 Previous Work

In [4], Mrazovic et al. presents a conceptual architecture for context-mobile crowdsourcing, in which devices are treated as intelligent problem solvers for both sensing and human intelligence tasks. The work done in this paper is a continuation of their work, applying their proposed system architecture as a guideline for the mobile crowdsourcing system. The natural step would have been to continue and contribute to the development of MobiCS but that work is unfortunately not available anymore. Therefore, the implementation of CrowdS had to be done from scratch.

In [5], ¨Ozyagci defines a reverse auction incentive mechanism for mobile crowdsourcing systems and integrates Vickrey-Clarke-Groves (VCG) to determine if truthful bidding would become the dominant strategy. According to the authors simulations the fairness of prices and earnings was observed to be generally better when the mechanism was applied. To confirm their findings and to accomplish truthful bidding in our system we implemented the rVCG mechanism, as described in the paper [5], as the main task allocation scheme.

In [6], Zheng et al. provide a detailed survey on 17 existing algorithms for truth inference which is used in quality control. The algorithms are summarized in terms of task types, task modeling, worker modeling and inference techniques, and are extensively compared and analyzed on five real data sets in real crowdsourcing platforms. In order to decide what quality control algorithm to apply in our work, this survey was used. As found in [6], the Majority Voting strategy is the fastest strategy when working with relatively small data set of answers to sufficiently infer the truth.

In [7], Cheng et al. investigates how workers should be selected so that the task in hand can be completed successfully and economically by solving the Jury Selection Problem (JSP). JPS is defined in the paper as follows: “Given a monetary budget, and a set of decision-making tasks (e.g., ‘Is Bill Gates still the CEO of Microsoft now?’), return the set of workers (called jury), such that their answers yield the highest ‘Jury Quality’ (or JQ)”. The authors solves the problem by applying an successful heuristic of Bayesian voting which is inherently NP hard. We decided against implementing Bayesian voting as the algorithm for the task allocation as a result of it requiring the quality of all workers which we did not have at the time.

(11)

2.2 Previous Work 2 RELATED WORK

In [8], Luck et al. addresses the problem of trust between agents in dynamic open systems by developing TRAVOS (Trust and Reputation model for Agent-based Virtual OrganisationS). Probability theory is applied using past interactions between agents when possible to calculate a trust value, otherwise the model requires reputation information from third parties (e.g. other agents). We implemented the basic notation from TRAVOS, that the trust is based upon if the contract is fulfilled or not, if a participant completes an assigned task. Further implementation of the trust module was outside the scope of the project.

In [9], Juong-Sik Lee and Baik Hoh proposes a new incentive mechanism for participatory sensing applications. The Reverse Auction based Dynamic Price (RADP) incentive mechanism addresses the problem of adapting to user’s dynamic true valuations (the value each user consider the service they provide worth). They observed that fixed price solutions that cannot adapt lead to users dropping out from using the application. The reversed auction performed by RADP is a solution to that problem by letting users bid to sell their sensing data. The service provider then selects a set amount of the lowest bids, and reward them with their bid price. But there was still a problem with this solution, that it did not retain users with higher true valuations. These users would get discourage from losing each round of the auction and finally stop participating. Their solution was to introduce a virtual participant credit (VPC), given to participants who lost a round of the auction. This credit was rewarded for each lost round until the user won an auction, encouraging continued participation. Another mechanism added to combat the loss of participants was to reveal the highest selling price to only dropped users, in an attempt to make them rejoin incase their true valuation was lower. The main problem that I have with that attempt is that it would be fairly easy to make a second user to circumvent the restriction of only letting the dropped out users know the max price. The idea of VPC is an interesting idea for keeping users participating in the application but not fair since it allows users to win an auction they should had lost otherwise.

In [10], Jaimes, Vergara-Laurens and Labrador explores another possibility for an incentive mechanism in participatory systems. They demonstrate that by incorporating the location of each user into the selection process, improves the coverage of the area of interest. Selecting two users that are located close to each other, is inefficient and renders pretty much the same data. They propose a Greedy Incentive Algorithm (GIA), a recursive auction-based incentive mechanism that selects users according to a fixed budget and their respective locations. The location of users is not static and changes over time, they respected that fact by adding a displacement vector to their model that gives a user a probability to move to a new location in each round of the auction. Every sensor in GIA is modeled as a disk, with a set radius from their center that they cover. The problem is then to find a subset of all sensors that cover the area of interest, also known as the NP-Complete Set Cover Problem. I believe it is necessary to consider the location of users to solve some problems in participatory sensing systems, and users in close proximity to the area of interest should be prioritized when selecting users. I find it troublesome to exclude users (that would been chosen otherwise) simply because they are in located very close to each other. The generalization that each user would provide “pretty much the same information” is to excessive. The age, model, and location (pocket or in your hand) would affect the rendered information.

(12)

3 SYSTEM MODEL

3 System Model

In this section the system model will be presented and motivated in regard to the responsibilities of the server.

Figure 1: Mobile crowdsourcing system overview

3.1 Components

The system model visualized in figure ?? consists of three major components; provider, requester and a server. The requester and provider are roles for mobile devices that communicates anomalously with each other through the server but never directly. If a mobile device created a task, it is considered as the requester for that task. If a mobile was assigned a task it is defined as one of the providers. Mobile devices in the system can take both of these roles simultaneously but never for the same task, it will never give an answer for its own task. The focus of this thesis is on the server and this paper will not go into more details about the mobile devices. If the reader wants more information I recommend reading this paper [11], that is written on the same project.

3.2 Auction

To give an incentive to use the system, providers will be paid for each time they complete a task. This payment will come from the requester and is based upon a bid the provider made when applying for the task. To decide whom to pick for the task an auction will take place and the winners will be assigned the task.

3.3 Fairness

The server is responsible for the task allocation and should be fair when deciding which providers to assign a task to. The task allocation is based on an auction in which the requester pays a price that is split up among the winning providers as earnings based on their bid. This thesis will measure how fair the task allocation is by calculating Jain’s fairness index [12] for prices and earnings on completed tasks.

(13)

3.4 Quality 3 SYSTEM MODEL

3.4 Quality

To complete any task in a mobile crowdsourcing system, data has to be collected by the providers. This data is then used to compile an answer to send back to the requester. Quality control is needed to ensure that the answer given is as good as possible. Depending on the data collected, different algorithms can be applied for quality control [6]. For instance, when handling numeric data outlier detection algorithms are applied that excludes observations that are not following the overall trend. In other cases voting strategies can be used, letting the majority decide what the correct data should be. To measure the quality of the data provided by a user we decided to keep different records depending on where the data comes from. For example, one bad light sensor does not necessarily mean that all of the sensors on the mobile device gives bad readings. This fine grain quality division should give a better overall picture of providers.

3.5 Reputation

Before considering what kind of quality a user can provide, the server should filter out users that has a bad reputation in the system. This history of bad reputation might come from erroneous contributions or not completing tasks given. The server will take this history into consideration and give priority to providers that have been excellent contributors in the past.

(14)

4 IMPLEMENTATION

4 Implementation

In this section the implementation of the server side of the system model (CrowdS) will be explained, including the modular components, software and hardware used.

4.1 Hardware and Software

CrowdS was hosted on a Linux computer running a PHP server with a MySQL database for persistent data.

Firebase was used for cloud messaging, to send messages from the server and deliver them cross-platform to the mobile devices. With limited resources we had little choice in what hardware we could use and had to make due with what was available to execute the code. The server language SQL and a MySQL database was fitting since I had prior experience working with it in another project [13].

4.2 Modularity

An important property that had to be implemented for CrowdS was modularity. It had to be modular so that parts could be replaced without having to redo the code and to let future iterations of CrowdS to take any path they desire. This version is only the foundation for future developers to keep working upon. All of the components mentioned for the server in section 3 should be modular. The code can be divided into two parts; the core, and modules. The core is the part of the code that provides the service of the system and cannot be replaced, while the modules delivers the implementation.

Just being modular is not enough, the modular parts should be easy to exchange as well. That is why a website was created to be able to access and configure the server. Simply add the component file into the appropriate location and then choose the new component as the current one using this website and that code will be run instead. To make this possible all created component files have to implement a corresponding interface with pre defined functions that the core expects to be available. For instance the current Task Allocation component implements the function getUsers, as long as the component returns a set of users, the core has does not enforce anything else on the component as long as the set is returned.

4.3 Task Allocation

CrowdS task allocation implements a sealed reversed auction with multiple winners which is common among many MCS systems according to our literature survey. The implementation is based on Vickrey- Clarke-Groves (VCG) mechanism [14–16] and is beneficial for truthful bidders, truthful bidding has the same or better utility than underbidding to win an auction [5].

CrowdS handles task allocation as follows:

• A task requiring n number of providers is requested and sent to the core system.

• The core system finds all available and online providers.

• Filters out unsuitable providers, for instance:

– Task giver

– Missing necessary sensor

– To far away from the requested location (see 4.3.1)

• If the remaining number of providers is less than the requested amount n, a failure notification is sent back to the task requester.

• Applies reversed VCG using the Task Allocation module (see 4.2)

(15)

4.4 Quality Control 4 IMPLEMENTATION

– If n equals the amount given to TA, simply return all of them. Their payment equals to their bid.

– Otherwise

∗ Sort providers by lowest bid

∗ If two providers has the same bid, prioritize providers that has been online the longest.

∗ Return the first n providers from the sorted list. Their payment equals to the bid of provider n + 1.

4.3.1 Location

To find providers close to the target location, the distance from available providers is calculated using their last given location in longitude and latitude. Using this distance all providers within the base search area of 25 meters is counted. If the number of providers found is as least as many as the amount needed, all of the providers are returned. Otherwise the search area is doubled, and in range providers counted again. This goes on until either enough providers are found and returned or the maximal search area of 800 meters is reached. If the later occurs, the search cannot be completed.

4.4 Quality Control

Depending on the type of data supplied from the providers for a task, different algorithms are needed to calculate the quality of the data. Even so, all algorithms have one thing in common, to infer the truth. Given the truth their job is to calculate how far away from the truth the given data were. Currently two types of data is considered by CrowdS, text and numeric. For textual data, the current implementation uses majority voting and a distance formula for numeric values. Each registered provider has one value representing its overall quality, which is the average of the quality calculated from the last 20 or so tasks. During the bootstrap time, until a provider has completed 20 tasks it keeps a quality base value of 0.5.

4.4.1 Majority Voting

The majority decides the truth, if an option is chosen by the majority of the providers, that option is considered the truth. There is only two types of tasks that currently returns textual data, single and multiple choice tasks. Multiple choice tasks are split up into a decision task for each answer option. And each one of those tasks has their own inferred truth if it was chosen by majority or not.

Example of multiple choice task:

Where do people want to live? Alice Bob Eve

Apartment x x x

Bungalow x

Cottage x x

Drugstore x x

Split up into decision tasks:

Do people want to live in an apartment? Yes No

Alice x

Bob x

Eve x

Do people want to live in an cottage? Yes No

Alice x

Bob x

Eve x

Do people want to live in an bungalow? Yes No

Alice x

Bob x

Eve x

Do people want to live in an drugstore? Yes No

Alice x

Bob x

Eve x

(16)

4 IMPLEMENTATION 4.5 Reputation Management

The majority has chosen the following as the truth:

• People want to live in: apartments, cottages and drugstores

• People do not want to live in: Bungalows.

The quality is calculated by dividing the number of correct answers with the number of questions. In the example above all three providers ended up with the same quality value of 0.75, three out of four correct answers.

Example of single choice task:

Where do people want to live the most? Alice Bob Eve

Apartment x x

Bungalow x

Cottage

In the example above, apartment was voted as the place where people want to live the most. Since its a single choice question, the resulting quality value is either 1 or 0 if the majority agrees with the providers pick. If two or more choices ends up sharing the most amount of votes, all of them are considered as if they had the majority when calculating the quality.

4.4.2 Distance

Numeric values have to be handled differently than textual, since its next impossible to vote for a majority with unlimited number of possible choices. CrowdS uses the mean value of all the received values as the inferred truth instead, and calculates each providers quality by their distance from that value.

4.5 Reputation Management

The reputation CrowdS considers and keeps track of is the rate of a providers task completion. If the system assigned a task to provider, it should be completed as well in an ideal world. But that is not always the case, and some providers might be worse than others to complete their given tasks. That is why the system keeps track of the last 20 tasks distributed to each provider and calculated their reputation by dividing the number of completed tasks by the number of given tasks. Until the provider has been given 20 tasks, it keeps a base value of 0.5 for its reputation.

4.6 Reward Management

Its important to give incentive to the providers for completing their tasks otherwise they have no reasons the spend their resources to begin with. As explained in section 4.3, the reversed Vincent-Clarke-Groves (VCG) auction rewards the providers with their bid as payment, or the bid of the n+1 as payment if there are more than n providers to chosen from.

(17)

5 RESULTS AND DISCUSSION

5 Results and Discussion

Please note that this chapter has been written in collaboration with Johan Waller, his paper [11] share the same results and may have textual similarities.

5.1 Experiment

The first larger test for CrowdS took place during the end of the year of 2017. Its main purpose was to gather information about how the system performed regarding fairness and coverage. How fairly were providers chosen for a task? How large of a search area is required to find a provider? These were some of the questions the test aimed to address. To increase the number of participants, students taking the ID2209 course could earn some extra credit by participating. Besides developers, friends and family, the students were the biggest contributing part. Every participant was asked to create 50 human intelligence tasks (HIT ) and 50 sensor tasks (ST ), and wait for their completion. Tasks that expired or terminated without an answer was not considered to be completed and had to be redone.

5.1.1 Bids

The bids used in the reversed auction was static and set when creating an account for the test. Each user was asked to answer some questions by selecting among 3 different predetermined options which had different hidden values (points). The sum of all the points was then used as the bid. The following list contains the questions asked when creating an account and the hidden values of their options.

• How old is your phone?

– 1 point: Less than 1 year

– 2 points: Between 1 and 2 years – 3 points: More than 2 years

• How motivated are you to participate?

– 1 point: Very – 2 points: Moderate – 3 points: Little

• How long does your battery last with normal usage?

– 1 point: More than 24h

– 2 points: Between 12h and 24h – 3 points: Less than 12h

(18)

5 RESULTS AND DISCUSSION 5.2 Results

5.1.2 Test Variables

Another list with variables and their respective values that are necessary to replicate the test:

• Radius of the search area – Initial value: 25m – Maximal value: 800m – Growth: double

• Time to complete – HIT: 3h

– ST: ((n − 1) ∗ 9 + 15) min, n = number of readings

• Providers

– Exactly one provider needs to be found – Shown on map

5.2 Results

This section contains the results of the experiment 5.1. The number of people that participated in the test was 25, and only 5 of them fully completed the test requirements. The most used operating system was Android with 18 users while the remaining 7 selected iOS.

5.2.1 Bids

3 4 5 6 7

0 2 4 6

3

6

7

5

4

bid

users

Figure 2: Distribution of user bids

Figure 2 shows the resulting bids from the questions at account creation 5.1.1. The bid was the sum of all points from 3 questions, with each answer being worth from 1 to 3 points. A lower bid increases the likelihood of winning an auction and the answers are using this fact to prioritize motivated users with newer devices, by giving a lower amount of points. Even though 8 and 9 bids were possible, no bids exceeded the 7 mark. The main reason behind this was that no user answered that they had a phone with a battery that lasted less than 12 hours with normal usage. Phone batteries are getting larger and larger and it seems we reached the point were it is common place for it to last the whole workday without a recharge, an observation that could not been seen just a decade ago.

(19)

5.2 Results 5 RESULTS AND DISCUSSION

5.2.2 Task

Figure 3: Task distributions

ST 49.9%

HIT 50.1%

MCT 8.5%

SCT 28.9%

NT 62.7%

PST

49.3% LST

50.7%

ST : Sensor Task NT: Numeric Task PST : Pressure Sensor Task HIT : Human Intelligence Task SCT: Single Choice Task LST : Light Sensor Task

MCT: Multiple Choice Task

Figure 3 presents 3 different pie charts with information about completed tasks in the test. The first pie chart shows all tasks and how they are divided into the two main categories, either Sensor Tasks (ST ) or Human Intelligence Tasks (HIT ). Both categories has about the same number of completed tasks, which is expected since the participants were asked to create 50 of each (5.1). The second pie chart contains information about the distribution of the HIT category of tasks, which includes; Numeric Tasks (NT ), Single Choice Tasks (SCT ) and Multiple Choice Tasks (MCT ).More than half of all the tasks completed in this category were NT , and the most likely explanation behind this observation is that they were the fastest HIT to create. NT only required the tester to provide a question to be answered while SCT and MCT demands different answers to be provided as well. This fact in conjunction with students often taking the path of least resistance when possible is most likely the reason why NT were chosen most of the time. The final pie chart shows completed tasks in the second category (ST ) divided into Pressure Sensor Tasks (PST ) and Light Sensor Tasks (LST ). Both tasks require the same information at creation and should therefor not affect the outcome when choosing between the types. That decision is instead based on user preferences and slightly more seems to prefer LST over PST .

5.2.3 Response Time

Completion time for Human Intelligence Tasks (HIT ) depends heavily on the response time from the humans that have been assigned the task. The average response time for HIT is an interesting metric to observe for information about how eager providers are to reply to a given task. A user with a low response time could be given priority when selecting providers for a task, if that is something that the requester asks for. The vast majority of tasks created during the test were completed within 10 minutes with a median time of 3 minutes and 32 seconds. Tasks without a response within 3 hours are considered expired and stated in 5.1.2.

(20)

5 RESULTS AND DISCUSSION 5.2 Results

5.2.4 Coverage

109 25m

84 50m

135 100m

242 200m

249 400m

404 800m

0 100 200 300 400

number of tasks

Figure 4: Radius of search area for task creation

The diagram shows how big the search radius was when the required number of providers was found to create a task. The ideal case for this diagram would have been to have as small radius as possible while always finding enough providers to distribute the task. This was not the case and most of the time the search area had to be extended to the maximum of 800 meters before finding enough providers. The information about how many times the location of the search area had to be moved due to not finding enough providers was not recorded and is therefore missing from the diagram. The location of all nearby providers was displayed on a map on the devices user interface with markers when selecting a location for a task. The reason behind why we decided to show everyone nearby on the map was to make it easier to create tasks.

With so few providers online during the test it would have been a difficult to find anyone otherwise. There is two main reasons that we believe contributed to why most of the tasks were created with a search area of 800 meters. The first one is that the participants did not prioritize precision while choosing a location for a task, as long as the task was created the location mattered little. The second reason was that the map would not let markers be placed on top of each other which lead participants to place the markers right next to the targets instead.

5.2.5 Earnings

For completing a task each provider was given an amount of virtual currency 3.2 as an incentive for providing the information. To collect the information some resources are spent by the provider and the value of those resources are unknown to us. The information that we had available was the total cost of the auction and the value of each winning bid by the providers. If we assume that each provider were motivated to make a profit then that means that their bid had to be greater than the value of the resources spent. The earnings e for a provider is the virtual currency v minus the cost of the resources r spent on that task.

e= v − r (1)

The problem with this equation is that r is still unknown, the only solution is to tackle it from a different angle. And it starts with a normalization of the equation by diving it with the providers bid b.

e=v− r

b (2)

(21)

5.2 Results 5 RESULTS AND DISCUSSION

There is another way to look at the equation now, v− r

b can be interpret as the expected earnings in relation to the providers bid. If we make the weighty assumption that all providers expect to earn about the same amount in relation to the resource spent by them, then it would be possible to trivialize r to a constant and ignore it for this final form of the equation.

e= v

b (3)

If no other providers are selected in an auction the earnings for that provider is 1 and greater than 1 otherwise. Jain’s fairness index [12] can be used as a measurement for the fairness of the earnings made by all participants in the test and it was 0.941, with 1 being the highest possible value. A surprisingly high number at first but not so much if you consider that only 61 out of 802 tasks found only 1 possible candidate for the task. And the fact that the difference between the highest and lowest bid was little (see 5.2.1) results in a low variance in earnings.

5.2.6 Prices

A task requester pays for each created task equal to the sum of all the selected providers bids. In this test only 1 user was chosen for each task, if two or more users were available the bid of second lowest was chosen as the price. In 741 out of 802 tasks only 1 user was in the area which resulted in that user’s bid ending up as the final price for the task. As shown in 5.2.1 all of the bids were about the same, this in combination with only picking one user for every task lead to a Jain’s fairness index of 0.944.

(22)

6 CONCLUSIONS AND FUTURE WORK

6 Conclusions and Future Work

This paper defines the responsibilities of a server in a mobile crowdsourcing system in terms of fairness and quality. Fairness in regard to its task allocation, how the set of service providers for a task should be selected.

And what the should be done server side to ensure a high quality of data. A system model is presented and motivated in regard to these responsibilities. Based on this system model a prototype crowdsourcing system CrowdS was developed and experimented upon to evaluate its performance. CrowdS allows the creation and distribution of both Human intelligence Tasks (HIT ) and Sensor Tasks (ST ) on mobile devices, with support for both iOS and Android. Based on the work done while developing CrowdS for this thesis work another paper was written, presented and published on the CSCE’18 congress [3]. The system was evaluated by executing a long-running experiment on live subjects during the end of 2017 and collecting data about interactions between participants. The main purpose of the experiment was to determine how well the system performed in terms of coverage and fairness of earnings and prices. Jain’s fairness index shows an overall high fairness for both the price of tasks at 0.944 and the earnings made by participants at 0.941.The coverage was underwhelming since roughly one third of all tasks had the search area extended to its maximum of 800 meters which is the worst case scenario. There is a number of improvements that could be done in this area mentioned in the results. Future work on the server side involves replacing or improving existing modules for quality, reputation, task allocation and reward management. This is likely done on the existing prototype CrowdS due to it having modular components designed with this scenario in mind. More extensive tests of its performance is needed in the future to get a better evaluation of the system.

(23)

REFERENCES REFERENCES

References

[1] Statistiska Centralbyr˚an, “Privatpersoners anv¨andning av datorer och internet 2016,” 2016. [Online].

Available: https://www.scb.se/Statistik/ Publikationer/LE0108 2016A01 BR 00 IT01BR1601.pdf [2] Wikimedia Foundation. (2018) Wikipedia. Accessed: 2018-05-02. [Online]. Available: https:

//www.wikipedia.org/

[3] V. Granfors, J. Waller, P. Mrazovic and M. Matskin, “Crowds: Crowdsourcing with smart devices,”

in Proceedings of the 2018 International Conference on Internet Computing & Internet of Things (ICOMP’18), Las Vegas, NV USA, 2018, pp. 10–16.

[4] P. Mrazovic and M. Matskin, “Mobics: Mobile platform for combining crowdsourcing and participatory sensing,” Taichung, Taiwan, pp. 553–562, July 2015. [Online]. Available:

http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7273666

[5] O. Z. Ozyagci and M. Matskin, “Truthful incentive mechanism for mobile crowdsensing,” Atlanta, GA, USA, pp. 282–287, June 2016. [Online]. Available: https://pdfs.semanticscholar.org/3230/

ddab5d0e09c6b09d7514ef853388764e4cca.pdf

[6] Y. Zheng, G. Li, Y. Li, C. Shan, and R. Cheng, “Truth inference in crowdsourcing: Is the problem solved?” Proc. VLDB Endow., vol. 10, no. 5, pp. 541–552, Jan. 2017. doi:

10.14778/3055540.3055547. [Online]. Available: https://doi.org/10.14778/3055540.3055547

[7] Y. Zheng, R. Cheng, S. Maniu, and L. Mo, “On optimality of jury selection in crowdsourcing,” 2015.

doi: 10.5441/002/edbt.2015.18. [Online]. Available: http://hdl.handle.net/10722/208740

[8] W. T. L. Teacy, J. Patel, N. R. Jennings, and M. Luck, “Travos: Trust and reputation in the context of inaccurate information sources,” Autonomous Agents and Multi-Agent Systems, vol. 12, no. 2, pp. 183–198, Mar 2006. doi: 10.1007/s10458-006-5952-x. [Online]. Available:

https://doi.org/10.1007/s10458-006-5952-x

[9] J.-S. Lee and B. Hoh, “Sell your experiences: A market mechanism based incentive for participatory sensing,” in Proc. 8th Annu. IEEE Int. Conf. on Pervasive Computing and Commun. (PerCom ’10), Mannheim, Germany, 2010, pp. 60–68.

[10] L. G. Jaimes, I. Vergara-Laurens, and M. A. Labrador, “A location-based incentive mechanism for participatory sensing systems with budget constraints,” in 2012 IEEE International Conference on Pervasive Computing and Communications, March 2012, pp. 103–108.

[11] J. Waller, “Client design for a mobile crowdsourcing platform,” 2019, Master Thesis Report, KTH.

[12] R. Jain, D.-M. Chiu, and W. Hawe, “A quantitative measure of fairness and discrimination for resource allocation in shared computer systems,” Sep. 1984, DEC-TR-301.

[13] V. Granfors and J. Waller, “Skapande av ett medlemskortsverktyg med hj¨alp av uml,” 2015. [Online].

Available: http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-177502

[14] W. Vickre, “Counterspeculation, auctions, and competitive sealed tenders,” vol. 16, pp. 8–37, Mar.

1961.

[15] E. H. Clarke, “Multipart pricing of public goods,” vol. 11, pp. 17–33, Sep. 1971.

[16] T. Groves, “Incentives in teams,” vol. 41, pp. 617–631, Jul. 1973.

[17] T. Luo, H. Tan, and L. Xia, “Profit-maximizing incentive for participatory sensing,” in IEEE INFOCOM 2014 - IEEE Conference on Computer Communications, April 2014. ISSN 0743-166X pp. 127–135.

(24)

REFERENCES REFERENCES

[18] The PHP Group. (2018) php. Accessed: 2018-05-22. [Online]. Available: http://php.net/downloads.

php

[19] Oracle Corporation. (2018) MySQL. Accessed: 2018-05-22. [Online]. Available: https:

//www.mysql.com/downloads/

[20] Oracle Corporation. (2018) MySQL Workbench. Accessed: 2018-05-22. [Online]. Available:

https://dev.mysql.com/downloads/workbench/

[21] Android Open Source Project. (2018) Download Android Studio and SDK Tools. Accessed:

2018-05-22. [Online]. Available: https://developer.android.com/studio/#downloads

[22] Google Developers. (2018) Add Firebase to Your Android Project. Accessed: 2018-05-22. [Online].

Available: https://firebase.google.com/docs/android/setup?authuser=0

[23] Google Developers. (2018) Maps SDK for Android. Accessed: 2018-05-22. [Online]. Available:

https://developers.google.com/maps/documentation/android-sdk/signup

[24] Apple Inc. (2018) Apple Developer Program. Accessed: 2018-05-22. [Online]. Available:

https://developer.apple.com/programs/

(25)

A SERVER DOCUMENTATION

A Server Documentation

This chapter contains documentation for CrowdS server.

A.1 Run Instructions

By following the instructions in this section you should (hopefully) be able to get the current version of CrowdS Server up and running.

A.2 Operating Systems

These instructions have been tried and confirmed working on Windows (v10). They do not however work on macOS (High Sierra v10.13.6), there was a problem with the mysqli connection that PHP does to the MySQL database, further configuration probably needs to be done on the installed PHP Server. CrowdS was previously hosted on a Linux (Ubuntu) system, and should work, but not been tested with this guide.

There could be other things that needs to be done to get the system up and running, that I did but forgot to mention in this guide. Its been a while since the time when I first set up the system.

A.2.1 CrowdS Repository

Before you start you have to download CrowdS repository from https://bitbucket.org/jowalle/

crowds/downloads/. If you can’t access this page please contact Mihhail Matskin and ask to be granted permissions.

A.2.2 Software

The software used for this guide is the following:

• PHP

• MySQL

• MySQL Workbench A.2.3 Install PHP

• Install the latest version of PHP from http://php.net/downloads.php.

• Extract the php folder to a place of your choosing.

• Add the path to that folder to your environment variables if necessary.

A.2.4 Test PHP

To confirm if you successfully managed to install the PHP server, we are going to launch CrowdS Admin website by using PHP built in web server.

• Open up a terminal or equivalent for your operating system.

• Change the directory to the html folder inside CrowdS repository. For instance:

– cd PATH/TO/CrowdS/CrowdS Server/PHP/www/html

• Host the website by running the command:

– php -S localhost:8000

(26)

A SERVER DOCUMENTATION A.2 Operating Systems

• If everything worked out you should be able to access http://localhost:8000 on your browser and see the login screen.

– Please note that you can not log in before the MySQL system is installed and configured.

A.2.5 Install MySQL and MySQL Workbench

• Install the latest version of both MySQL and MySQL Workbench

– Use the documentation on https://www.mysql.com/downloads/ as reference.

– Depending on MySQL version you might be asked to provide a password for the root user, do so and make a mental note of it.

A.2.6 Configure MySQL

• Modify MySQL Server configuration file

– The name of the file is my.ini or perhaps my.cnf (different extensions depending on your operating system).

– The location might vary, default path for windows is:

∗ C:/ProgramData/MySQL/MySQL Server 8.0/

• Add or change the following lines:

– below [client]:

∗ default-character-set = utf8 – below [mysql]:

∗ default-character-set = utf8 – below [mysqld]:

∗ collation-server = utf8 unicode ci

∗ character-set-server = utf8

(27)

A.2 Operating Systems A SERVER DOCUMENTATION

A.2.7 Configure Database

• Start MySQL Workbench

• Open a new connection to the MySQL Server you made while installing.

• Select Data Import/Restore from the Navigator (see image below).

• Import CrowdS SQL tables from datafile (see image below).

1. Select Import from Self-Contained File

2. Browse and select the tables only SQL file from the downloaded CrowdS repository (CrowdS/CrowdS Server/MySQL/tables only.sql)

3. Create a new schema and name it cs and use it as Target Schema (the name of the new schema will the name of your database).

4. Click Start Import

• Switch from the Administration tab to Schemas in the Navigator.

• Check if you can find the newly created schema

– If not press the refresh button next to SCHEMAS.

– You should now be able to see all 5 tables (compare with image).

(28)

A SERVER DOCUMENTATION A.2 Operating Systems

• Finally we have to modify the root user for the database.

– Currently, php mysqli extension do not support the new caching sha2 authentication feature.

– Make the following query:

∗ ALTER USER ’RootUser’@’localhost’ IDENTIFIED WITH mysql native password BY ’RootPassword’;

∗ Replace RootUser and RootPassword with the username and password you picked before.

A.2.8 Initial Configuration of Website

If you followed the guide and got this far, well done! Just one final step, if you remember the website we hosted back when we wanted to test our PHP server, its time to configure it as well.

• Open the file config.php

– Its inside the html folder (CrowdS/CrowdS Server/PHP/www/html/config.php) – Change the value of PHP URL

– The value should be the IP address and Port you are hosting the PHP server on.

– If you want people to reach your server outside the local network, make sure the port forwarding on the router is set up correctly.

• Open the file mysql.php

– Its inside the database folder)

– Change the value of username and password

(29)

A.3 CrowdS Server API A SERVER DOCUMENTATION

A.3 CrowdS Server API

The following section contains information about all the PHP files that can communicated with by sending a HTTP Post request. The section is divided into 4 different subsections depending on what kind of field each file represents. The purpose of each file is explained, what kind of data it expects as input in the HTTP headers, and finally what kind of encoded message is sent back to the mobile device.

Purpose: The main purpose of the file.

Input: The variables that the file expect to be passed via the HTTP POST method.

Reply: The PHP server will reply a JSON encoded message with a status key which value represents the outcome of calling the file.

A.3.1 Account Files

Files to communicate with when working with user accounts.

register user.php

Purpose Store a new user’s info in database

Input

email Email of the new user username Username of the new user password Password of the new user

device os Operating system of the user’s mobile device device model Model of the user’s mobile device

firebase Firebase registration token

bid Static bid used in the reversed auction device sensors Binary representation for sensor availability Reply ERROR User already exist

OK User is now registered

delete user.php

Purpose Remove a user’s info from the database Input email Email of the user that needs to be deleted Reply ERROR User could not be deleted

OK User is now removed

update user.php

Purpose Update user with new user data Input

email Email of the user that needs to be updated password Password of the user (could be new) username Username of the user (could be new) Reply ERROR Could not update user

OK User updated with given data

(30)

A SERVER DOCUMENTATION A.3 CrowdS Server API

A.3.2 Session Files

Files to communicate with for user accounts. There is an additional key in the JSON encoded reply message for ERROR messages, check the reason key for additional information about why the ERROR occurred.

login.php

Purpose Allow a registered user to access CrowdS

Input

email Email of the user password Password of the user

firebase Firebase token from the user lat Current latitude position lng Current longitude position

os Operating system of the user’s mobile device version What version of CrowdS the user is running

Reply ERROR

Server is currently under maintenance The user is not registered

Unknown operating system (has to be android or iOS) The user is running an outdated version of CrowdS OK User has been granted access

logout.php

Purpose Marks a user as offline

Input email The email of the user going offline Reply OK User is now considered offline

positions.php

Purpose Get the positions of all other active users Input email Email of the user requesting positions

Reply OK Position of all other active users, check the coords key

(31)

A.3 CrowdS Server API A SERVER DOCUMENTATION

A.3.3 Task Files

Files to communicated with when working with tasks. There is an additional key in the JSON encoded reply message for ERROR messages, check the reason key for additional information about why the ERROR occurred.

create task.php

Purpose Create a task with given data and distribute it to other users

Input

email Email of the user that gave the task description A description of what kind of task it is lat Latitude position of task location lng Longitude position of task location type What type of task (hit or sensor)

if type is hit Input

hit type What kind of HIT

question What is the question that needs to be answered file What file has the handles the updates for the task

if type is sensor

Input

duration The duration the sensor is read from

readings The number of readings that should be done sensor The sensor used

file What file has the handles the updates for the task Reply ERROR No devices within the maximum search area

No device with the required sensor OK Task has been distributed to providers

update task.php

Purpose Store new data for a task in progress

Input

email Email of the user that has been given a task

id Id for task and provider number for that task (e.g. 568:0) data The new data

type Task type

file The file that handles updates for the task in question Reply OK Task was updated

ongoing task.php

Purpose Get all tasks that are in progress for the user Input email Email of the user that needs all ongoing tasks Reply OK An array with all ongoing tasks

task history.php

Purpose Get all completed tasks that has been given by the user Input email Email of the user that needs all completed tasks

Reply OK An sorted array with all completed tasks after completed time

(32)

A SERVER DOCUMENTATION A.3 CrowdS Server API

A.3.4 Reward Files

Files to communicate with for data about a users collected rewards for participating.

reward.php

Purpose Get the collected rewards for a users participation Input email Email of the user that needs the reward data Reply OK All rewards collected for participating

(33)

A.4 Task Management A SERVER DOCUMENTATION

A.4 Task Management

I have divided this section into 5 parts, all concerning different kinds of task management. Each part will list all files on the server that it concern and a brief description about their purpose. For more information I recommend studying the source files.

A.4.1 Task Allocation Files

All files are located in the task allocation folder.

• random: Selects users by randomly picking users that are above a quality and reputation threshold, if none are found it lowers the threshold and tries again.

• vcg: Implements a reversed auction using the VCG mechanism to benefit truthful bidding.

A.4.2 Task Creation Files

All files are located in the task creation folder and handles the creation of the current available types of tasks for CrowdS. Included in the file are functions to notify the user about the newly created task through firebase.

• multiple choice: Creates and stores a multiple choice task.

• numeric: Creates and stores a numeric task.

• sensor: Creates and stores any sensor task.

• single choice: Creates and stores a single choice task.

A.4.3 Task History Files

All files are located in the task history folder. These files are used by the task history file.

• hit: Returns all completed hit tasks for a user.

• sensor: Returns all completed sensor tasks for a user.

A.4.4 Task Ongoing Files

All files are located in the task ongoing folder. These files are used by the ongoing task file.

• hit: Returns all ongoing hit tasks for a user.

• sensor: Returns all ongoing sensor tasks for a user.

A.4.5 Task Update Files

Files are located in the task update folder and the html folder. These files are used by the update task file. Included are functions to send a notification through firebase if the task was completed.

• multiple choice: Appends new multiple choice task data to database.

• numeric: Appends new numeric task data to database.

• sensor: Appends sensor task data to database.

• single choice: Appends single choice task data to database.

• finish task: If a task did not completed within its time frame, this file will terminate it.

(34)

A SERVER DOCUMENTATION A.5 Heartbeat

A.5 Heartbeat

Heartbeats are scheduled using at commands to check if the user are still connected to CrowdS. This is done to remove providers that failed to logout or got disconnected from the pool of active users, to reduce the chance that they are assigned a task. The following files are used to schedule and ping users to see if they still are connected:

• heartbeat: Sends an heartbeat through firebase, and schedules a heartbeat check in the near future.

• heartbeat check: Checks if the users replied to the heartbeat, and schedules a new heartbeat.

• heartbeat update: Called by the user to notify the server that it received the heartbeat.

A.6 Quality Control

These files can be found in the quality control folder and are the current available algorithms for the quality control performed by CrowdS.

• distance: Calculates quality by a distance formula.

• majority voting: Calculates quality by majority voting.

A.7 Reputation Management

This file can be found in the reputation folder and implements CrowdS current simple reputation management.

• completion ratio: Simply calculates user reputation by completion ratio of given tasks.

A.8 Reward Management

This file can be found in the reward folder and implements CrowdS current simple reward management.

• easy points: Calculates and updates a users collected rewards.

A.9 Web Page

This section is divided into 5 parts, all connected to the administrative website for CrowdS. Each part has a list of the files related and a brief description about their purpose. As usual, check the source files for more information.

A.9.1 Page Files

Each file in this section is a HTML page, but since they execute some PHP code as well the type had to be of the PHP extension. Every file represents a single web page on the administrator site.

• index: The login screen, can only be accessed by administrators. Current way of granting access is by making a query in MySQL Workbench.

• main: The web page that is displayed after passing the login screen.

• account: A page that has some account management functions, as updating or deleting users.

• active users: Displays all current online users of CrowdS through Google Maps.

• server settings: A settings page which allows an administrator to change settings for the server. For instance what quality algorithm that should be applied. Currently under development and saving the settings is disabled.

(35)

A.9 Web Page A SERVER DOCUMENTATION

A.9.2 Session Files

A session is created and managed while an administrator logs in the website. The following files contains the source code for that process.

• admin login: Creates an session at successful admin login.

• admin logout: Destroys session and changes the header location to login page.

• forced logout: Used to forcefully log out all online users. Usefully if the server has to go down for maintenance.

A.9.3 Settings Files

These files are related to or has important settings that the rest of the system includes.

• config: A ton of definitions the server use, a lot of them are paths for the different modules. The idea is that you should be able to change settings dynamically for the different modules, without interfering with the rest of the system. Other noteworthy settings are for number of devices, flag for activating logs and maintenance.

• session: Simple short file used for login sessions.

• settings: A JSON file containing the settings that can be made on the settings webpage with additional fields for what the latest version is for android and ios.

• update server settings: Used to update server settings.

A.9.4 Image Files

Images used by the account page for a better looking web interface.

• checkmark-icon: Check mark image.

• cogwheel-icon: Cogwheel image.

• delete-icon: Red cross image.

A.9.5 Other Files

A number of files that don’t quite fit in under another topic.

• find user: Finds the set of all online users.

• system utils: A helper file with utility functions for other files.

• info: Displays PHP info.

(36)

A SERVER DOCUMENTATION A.10 Interface

A.10 Interface

The following file has all the interfaces that CrowdS Server API requires the rest of the modules to implement. If you would like to design a new module I would recommend implementing the interface for that module as well.

• interfaces: This file has interfaces for:

– Database – Task Creation – Task Update – Ongoing Tasks – Task Allocation – Quality Control

– Reputation Management – Reward Management

A.11 Firebase

Firebase is used for cross-platform communication with the mobile devices. The mobile devices sends a registration token while creating an user account. To be used for later communication in the future.

• firebase: Main file for firebase related things.

• push: Used for push notifications.

• notification: Used for normal notifications.

A.12 Database

A MySQL database is used for persistent storage. The files PHP files mentioned in this section handles all the queries done by the server.

• database: A class that the rest of the system uses for all database related queries. It uses the current select database that implements the database interface.

• mysql: This file implements the database interface and are use for all queries.

(37)

B CSCE 2018 REPORT

B CSCE 2018 Report

The following section contains the report from CSCE2018 [3] along with referee evaluation points and comments.

B.1 COMPILED REFEREES’ evaluation points

Originality (1 Poor - 20 Very Good): 18 Technical Quality (1 Poor - 20 Very Good): 15

Significance of Contribution (1 Poor - 20 Very Good): 17 Readability and Organization (1 Poor - 20 Very Good): 15 Relevance (1 Not relevant - 20 Very relevant): 20

Overall Recommendation (1 Reject - 20 Strong Accept): 17

Degree of Expertise of Reviewer in the topic (1 Poor - 20 Good): 16

B.2 Comments

Specific comments to the authors.

B.2.1 Referee A

The authors in this paper present a design and implementation of a crowdsensing system, named CrowdS, which allows the creation of both sensing and human intelligence tasks and allocate them to smart devices in a community network. This is an excellent piece of work and so my recommendation is a definite accept.

The English write-up of the paper is ok but can still be enhanced. The web sites mentioned in the reference list should each have the date of retrieval (since the content of the web sites can change by their owners).

Good job.

B.2.2 Referee B

I very much liked this paper. It is truly IoT related. The proposed CrowdS is impact-full and has the potential to be highly cited. I would have hoped to see a little more discussion about the reported results.

My recommendation is to accept (with high level of confidence).

Server Design to Ensure Quality and Fairness in Mobile Crowdsourcing

Server Design to Ensure Quality and Fairness in Mobile

Crowdsourcing

VILLE GRANFORS

Contents

1 Introduction

1.1 Background

1.2 Purpose

1.3 Goal

1.4 Method

1.5 Delimitations

1.6 Outline

2 Related Work

2.1 Background

2.2 Previous Work

3 System Model

3.1 Components

3.2 Auction

3.3 Fairness

3.4 Quality

3.5 Reputation

4 Implementation

4.1 Hardware and Software

4.2 Modularity

4.3 Task Allocation

4.4 Quality Control

4.5 Reputation Management

4.6 Reward Management

5 Results and Discussion

5.1 Experiment

5.2 Results

6 Conclusions and Future Work

References

A Server Documentation

A.1 Run Instructions

A.2 Operating Systems

A.3 CrowdS Server API

A.4 Task Management

A.5 Heartbeat

A.6 Quality Control

A.7 Reputation Management

A.8 Reward Management

A.9 Web Page

A.10 Interface

A.11 Firebase

A.12 Database

B CSCE 2018 Report

B.1 COMPILED REFEREES’ evaluation points

B.2 Comments