EVALUATION CRITERIA FOR SELECTION OF API PRODUCTS: Practitioners' Perspective

Master of Science in Software Engineering February 2017

Faculty of Computing

Blekinge Institute of Technology

Evaluation criteria for selection of API Products

Practitioners’ perspective

SAI SANDEEP CHIKKALA

This thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of MSc in Software Engineering. The thesis is equivalent to 20 weeks of full time studies.

Contact Information:

Author(s):

Sai sandeep chikkala

E-mail: saca14@student.bth.se

University advisor:

Samuel Fricker DIPT

Faculty of Computing

Blekinge Institute of Technology SE-371 79 Karlskrona, Sweden

Internet : www.bth.se

Phone : +46 455 38 50 00

Fax : +46 455 38 50 57

i

A ^BSTRACT

Context. The approach of developing software systems with the use of third party components i.e. COTS or OSS has increased globally. In this study API product refers to either a software component or a software service or both packaged together, that can be accessed through an API. Developers are faced with plethora of alternative choices to select an API product. With this increase in components adoption, API product providers are faced with challenge of designing their product to be more attractive than others. This needs the providers to be educated about the developer behavior when they choose an API product.

Understanding the selection practices of developers can help providers to improve the packaging of API products, making them more suitable for selection.

Objectives. The objectives of this study is to investigate the criteria that developers use when reasoning about acceptability of a software component.

Methods. A background study is performed to identify the evaluation criteria proposed in the literature. An empirical study using Qualitative content analysis is performed. In the study the 480 reviews of different API products are analyzed to understand the criteria from practitioners’ perspective.

Results. 9 relevant criteria that developer use to reason about accepting or rejecting an API Product are identified. 30 sub criteria related to the 9 criteria are described in the study.

Conclusions. This study concludes that the identified 9 criteria play an important role in developer assessment of the API product. It is also found that the criteria have significant impact on the ratings of API product. These criteria could guide API product providers to make better choices when developing the product.

Keywords: API product, Evaluation criteria, content

analysis

A CKNOWLEDGEMENTS

I would like to thank my supervisor Dr. Samuel Fricker for his support, patience and especially the valuable inputs to craft the thesis process. It has been a great learning experience under him. His impeccable command on software engineering research and ability to offer a fresh perspective about any topic in hand have helped to overcome hurdles faced in this journey. Timely meetings coupled with quick skype replies have eased the communication and pushed to stay motivated throughout the thesis.

I am more than thankful to my parents and brother who supported me in all the ways

possible to help me finish the thesis. I would also like to thank my friends for giving that

much needed nudge during crucial times. All of you have made this journey memorable and

enjoyable.

C ^ONTENTS

ABSTRACT ...I CONTENTS ...II

1 INTRODUCTION ...6

2 RELATED WORK ...7

2.1 WHY DEVELOPERS USEAPIPRODUCTS?... 7

2.2 WHY SELECTION PROCESS OFAPIPRODUCTS ISIMPORTANT? ... 7

2.3 SELECTIONPROCESS... 7

2.4 SELECTIONCRITERIA:... 9

2.5 RESEARCHGAP:... 11

3 METHODOLOGY ... 12

3.1 AIM:... 12

3.2 OBJECTIVES: ... 12

3.3 RESEARCHQUESTION:... 12

3.4 RESEARCH APPROACH OVERVIEW... 12

3.5 WHY THIS RESEARCH METHOD?... 13

3.6 RESEARCHIMPLEMENTATION:... 13

3.7 DATAEXTRACTION: ... 14

3.8 DATAANALYSIS:... 15

3.8.1 Why content analysis? ... 15

3.8.2 Analysis Procedure ... 15

4 RESULTS ... 17

4.1 SUMMARY OF SAMPLE: ... 17

4.2 FINDINGS FROM CONTENT ANALYSIS... 19

5 ANALYSIS ... 24

5.1 ANSWERING THERESEARCHQUESTION:... 24

5.2 DETAILED ANALYSIS OF EACH CRITERIA... 27

5.2.1 Criteria: Documentation... 28

5.2.2 Criteria: Learnability... 30

5.2.3 Criteria: Community Support... 33

5.2.4 Criteria: Usability ... 35

5.2.5 Criteria: Functionality... 38

5.2.6 Criteria: Flexibility ... 41

5.2.7 Criteria: Interaction with other components ... 43

5.2.8 Criteria: Performance... 45

5.2.9 Criteria: Pricing... 47

6 DISCUSSIONS AND LIMITATIONS... 49

6.1 CONTRIBUTION: ... 49

6.2 COMPARISON WITHLITERATURE:... 49

6.3 IMPLICATIONS TO PRACTITIONERS: ... 52

6.4 THREATS TOVALIDITY... 52

7 CONCLUSION AND FUTURE WORK ... 54

REFERENCES ... 55

APPENDICES: ... 57

List of Tables

Table 1 - List of criteria identified in literature ...10

Table 2 - Distribution of reviews across various domains...17

Table 3 - Domain and the API products from which reviews are extracted...18

Table 4 - The journey of raw data emerging into category ...21

Table 5 - codes, sub categories and categories ...23

Table 6 - Main findings of the study ...26

Table 7 Comparison between current study and Literature...52

List of Figures

Figure 1- Analysis process followed in this research ...16

Figure 2 Screenshot of a typical review on G2crowd...18

Figure 3 Snapshot of Word document at initial analysis stage...19

Figure 4- Snapshot of analysis process in excel at step 4...20

Figure 5 – No. of times Documentation mentioned by reviewers across 1-5 ratings...28

Figure 6 – Percentage (%) of reviewers who liked and disliked Documentation across 1-5 ratings ...28

Figure 7 – Occurrence percentage of each sub criteria in documentation...29

Figure 8 - No.of times Learnability mentioned by reviewers across 1-5 ratings...31

Figure 9 - % of reviewers who liked and disliked Learnability across 1-5 ratings...31

Figure 10 - Occurrence percentage of each sub criteria in learnability ...31

Figure 11 - Occurrence percentage of each sub criteria in Community support ...33

Figure 12 - No. of times "Community support" mentioned by reviewers across 1-5 ratings.33 Figure 13 - % of reviewers who liked and disliked "Community support" across 1-5 ratings ...34

Figure 14 - No. of times "usability" mentioned by reviewers across 1-5 ratings ...35

Figure 15 - % of reviewers liked and disliked "Usability" across ratings 1-5...36

Figure 16 - Occurrence of each sub criteria in Usability...36

Figure 17 - No. of times "Functionality" mentioned by reviewers across 1-5 ratings...38

Figure 18 - % of reviewers who liked and disliked "Functionality" across 1-5 ratings ...39

Figure 19 - occurrence percentage of sub criteria in functionality ...39

Figure 20 - No. of times "Flexibility" mentioned by reviewers across 1-5 ratings ...41

Figure 21 - % of likes and dislikes for "Flexibility" across 1-5 ratings...41

Figure 22 - - occurrence percentage of sub criteria in flexibility ...42

Figure 23 - No. of times "Interaction with other components" mentioned, across 1-5 ratings ...43

Figure 24 - % of likes and dislikes for "Interaction with other components" across 1-5 ratings ...43

Figure 25 - - occurrence percentage of sub criteria in Interaction with other components ....44

Figure 26 - No. of times "performance" is mentioned, across 1-5 ratings...45

Figure 27 - % of likes and dislikes for "performance” across 1-5 ratings...45

Figure 28 - - occurrence percentage of sub criteria in Performance...46

Figure 29 - No. of times "Pricing and License" was mentioned, across 1-5 ...47

Figure 30 - - occurrence percentage of sub criteria in Pricing...47

Figure 31 - % of likes and dislikes for "Pricing and License", across 1-5...48

1 I NTRODUCTION

With the emergence of software ecosystems, an increasing number of components and services are offered to practitioners for integration into software development. Avoiding the

“need to re-invent the wheel” has critically motivated developers to adopt software components [1]. The benefits of component adoption also include quick to market, less development effort and improved innovation [2][1]. The competitive edge provided with these benefits have driven more industries to practice component based software development. Software components generally used by developers come under COTS (Commercial off the shelf) products or Open source software (OSS). COTS are commercial products provided by a vendor, often with no user access to source code. OSS are free software developed by fellow developers or a community and users can have access to source code [2]. The term API product in this paper refers to either a software component or a software service or both packaged together, that can be accessed through an API. Example of software component are COTS and OSS. Example for software service are Amazon Web services or Microsoft Azure. In this study the terms developer and practitioners mean the same and used interchangeably.

Due to the growing popularity for API products, there are wide range of API products across all domains. Internet is the marketplace for choosing API products. Hosting domains like sourceforge [3] , Eclipse marketplace [4], and google code [5] offer diverse API products, catering to needs of any developer. With this growth, developers are regularly confronted with numerous API products that satisfy their need. This demands product managers of API products to make their API product more appealing to practitioners than their competitors.

To make an attractive API product, the providers need to understand the behavior of developers, when developers perform selection. The criteria followed by developers to evaluate the API product can serve as valuable inputs to design attractive API product. To understand the selection practices few researchers studied the developer behavior when selecting API products [6] [7]. These studies concluded that developers follow informal or ad-hoc procedures to make API product selection. Although they highlighted few selection criteria, they are not exhaustive and could not be generalized. Majority of remaining literature on selection of API products have not studied the developer behavior. The suggested selection criteria and selection algorithms are based on their prior experience. This presses the need for more research on selection process from developers’ perspective.

In pursuing developers’ opinion on selection criteria, reviews of diverse API product reviews were analyzed in this study. The quantity and quality of API product reviews available online has motivated the author to choose this approach. The opportunity to extract the selection criteria and to map it with the reasoning of developers could provide interesting insights for API product providers. This research also helps new developers to understand the various criteria they need to consider when selecting an API product. Finally, the results of this study could help researchers to improve their selection models to make them more applicable to practical use.

The remainder of this document are organized as follows: chapter 2: Related work, chapter

3: Research Methodology, chapter 4: Results of the study, chapter 5: Analysis of the results,

chapter 6: Discussions and limitations of the research and chapter 7: Conclusions and future

work.

2 R ^ELATED W ^ORK

Th is sec t ion de ta i ls the backg round know ledge requ ired to unde rs tand the con tex t o f the research . The con ten ts of th is sec t ion desc r ibe the impor tance of API produc ts in sof twa re deve lopmen t , impor tance o f API se lec t ion p rocess , Ex is t ing L i te ra ture abou t API se lec t ion and how cu rren t resea rch cou ld make a re levan t con tr ibu t ion to f ie ld of s tudy .

2 .1 Why deve lopers use API products?

An API produc t is a p iece of th ird par ty sof twa re , in teg ra ted by the deve lope rs to o f ten deve lop la rger sof tware sys tems . The use o f API produc ts in indus try has surged in recen t years . A s tudy c la ims tha t ou t o f 769compan ies 33% prov ide OSS based p roduc ts and 46 .8% of 569 compan ies in teg ra teOSS in sof tware sys tems[8][1]. The increas ing adop t ion cou ld be exp la ined by the benef i ts ga ined from API produc ts . The var ious advan tages offe red by in teg ra t ing API produc ts a re : Fas te r t ime to marke t , cos t e ffec t ive so lu t ion , encou rages innova t ion and inc reased sof tware qua l i ty and p roduc t iv i ty[9 ][6 ][1 ]. These advan tages g ive indus tr ies a huge compe t i t ive edge tha t v irtua l ly make i t imposs ib le to ignore the usage o f API produc ts[6 ].

2 .2 Why se lect ion process of API products is Important?

Desp i te these advan tages , i t cou ld prove to be cos t ly if the API produc t adop t ion is no t prope r ly orches tra ted . The adop t ion of API produc ts in to so f twa re deve lopmen t ma jo r ly inc ludes se lec t ion , in teg ra t ion and ma in tenance of the componen ts[1]. The deve lopmen t us ing so f twa re componen ts is of ten re fe rred as componen t based so f tware deve lopmen t (CBSD) . I t is s trong ly c la imed in l i te ra ture tha t success of CBSD is heav i ly re l ian t on se lec t ion o f the mos t appropr ia tecomponen ts[1][10]. M ismanagemen t of API se lec t ion process cou ld dr ive sof twa re pro jec ts towards more r iskyand uncer ta in env ironmen ts[6].

The se lec t ion process and the ex is t ing work upon i t d iscussed be low .

2 .3 Se lect ion Process

The se lec t ion process o f API produc ts is genera l ly carr ied ou t in three s teps[11][7][1 ][12]:

 Loca t ing the API produc t .

Th is s tep invo lves search ing for the API produc t . How deve lope rs end up f ind ing the re levan t API produc ts tha t are easy to eva lua te and compare.

 Eva lua t ing API produc t w i th prede te rm ined user requ iremen ts .

The ob jec t ive o f th is s tep is to eva lua te the API produc t compa t ib i l i ty w i th respec t to the ex is t ing user requ iremen ts .

 Choos ing API produc t .

Th is s tep refers to compar ison of the po ten t ia l lyma tch ing API produc ts to f ind the bes t f i t for the g iven s i tua t ion .

Locat ing the API product:

The f irs t s tep in se lec t ion process is loca t ing the API produc t . In terne t has been the ma jo r

hos t of the API produc ts . The marke tp lace prov ided fo r in te rac t ion of API produc t des igne rs

and use rs[6 ][1 ].Trad i t iona l ly var ious mechan isms and too ls a re p roposed to loca te API produc ts . Search eng ines w i th var ious techno log ies were imp lemen ted to sk im through componen t repos i tor ies and f ind the re levan t p roduc ts upon h i t t ing a sea rch s tr ing . Few Examp les of these too ls a re Goog lecodeSearch , IPScom–in te l l igen t por ta l fo r search i ng componen ts[13]. However i t is found ou t tha t these me thods a re rare ly used in p rac t ice . The prac t ica l me thods used to loca te the API produc tare unsys tema t ic and ad- hoc[6 ][14 ][15 ].

Few prac t ices men t ioned in l i tera tu re for iden t ifying API produc t are :

 Iden t if ied componen t us ing prev ious expe r ience[1 ][14][9 ]:

I f any of the team member has prev iousexper ience us ing a spec if ic componen t then i t is se lec ted fo r the pro jec t w i th no fur the r sea rch ing . Th is is the mos t fo l lowed approach in indus tr ies . Here fam i l iar i ty o f the API produc t has encou raged the se lec t ion

 Perfo rm ing informa l in te rne t search[1 ][14][7 ]:

When deve lope rs have no pr ior expe r ience us ing componen ts , then they tu rn to in te rne t sea rches on goog le . Mos t ly th ey s tumb le upon API marke tp lace l ike Source fo rge to search fo r the app rop r ia te componen t . They a lso fo l low the recommenda t ions and sugges t ion of pee rs on in te rne t who have dea l t w i th s im i la r requ iremen ts .

 Fo l low ing API marke tp lace[14]:

Th is approach is fo l lowed to p roac t ive ly iden t ify the componen ts . Deve lopers usua l ly mon i tor the API marke tp lace to s tay upda ted on la tes t trends and too ls . They a lso fo l low var ious fo rums and subsc r ibe to ma i l ing l is ts and news le t ters . S tay ing upda ted prov ides them a head s tar t when they need API produc t .

Informa l in te rne t sea rch is men t ioned mos t as common ly fo l lowed approach toiden t ify the API produc ts[1 ][14][7][9 ].

Eva luat ing API product w ith pre-de term ined user requ irements:

For eva lua t ion of API produc t , var ious me thods are proposed w i th use o f many modern techno log ies . The p roposed me thods have used dec is ion suppo r t sys tems , s imu la t ion , me thod eng ineer ing , s tra teg ic con trac t ing and procu remen t and many o thers to he lp deve lopers make an info rmed dec is ion[6 ]. These me thods are f irs t in troduced to he lp use rs se lec t COTS produc ts . A L i te ra ture rev iew of the ex is t ing me thods fo r eva lua t ion of COTS p roduc ts has beens tud ied by A .Mohamed e t .a l[16]. These me thods fo l low a common pa t tern in i mp lemen ta t ion . In i t ia l ly the requ iremen ts shou ld be iden t if ied , then a se t of we l l- def ined eva lua t ion c r i te r ia are evo lved w i th these requ iremen t , then ass ign we igh tage to the eva lua t ion cr i te r ia , and compare the Po ten t ia l API produc ts us ing the techno log ies spec if ied above[9].

The above me thods a re in i t ia l ly deve loped for COTS se lec t ion un t i l the popu la r i ty and demand for OSS increased . W i th g row th in OSS adop t ion , the need to propose OSS eva lua t ion me thods inc reased . Bu t the eva lua t ion me thods used fo r COTS a re no t su i tab le to OSS as i t posed some cha l lenges . The emergence of in terne t as a v i ta l hos t fo r w ide var ie ty of OSS sof twa re has made i t d iff icu l t to f ind a l lthe po ten t ia l API produc ts[14]. A lso , un l ike Commerc ia l so f tware OSS are no t pushed upon peop lew i th any marke t ing s tra tegy[14].

Th is makes i t hard to loca te and eva lua te the OSS p roduc ts . Address ing these cha l lenges many me thods , frameworks spec if ic to OSS are proposed . These inc lude Qua l iPSo mode l of trus twor th iness[17], Qua lOSS mode l framework[18]and open sou rce ma tur i ty model[14].

A l though these eva lua t ion me thods fo r COTS and OSS are success fu l in the ir

imp lemen ta t ion , the ev idence from l i tera tu re s trong ly c la ims tha t these me thods are se ldom

used in p rac t ice[14][19][20][1 ]. In prac t ice deve lopers a re mos t ly unaware o f these fo rma l

me thods and they usua l ly fo l low in forma l approach for eva lua t ion of API produc ts . The common ly fo l lowed approaches for API produc t eva lua t ion a re :

 Prev ious wo rk ing exper ience[14 ][6]

The pos i t ive exper ience o f work ing w i th an API produc t is the bes t ind ica to r to use i t the nex t t ime . I f any member o f the team is fam i l ia r w i th the componen t then they of ten choose i t w i th no fu r ther eva lua t ion

 Rev iews of othe r deve lopers on in te rne t[14]

The feedback prov ided by o ther deve lopers who have used i t fo r s im i la r requ iremen ts is cons ide red to eva lua te the p roduc t . These deve lope r rev iews are of ten found on techno logy b logs , commun i ty forums and o the r deve lope r or ien ted webs i tes .

 API produc ts tha t have bu i l t repu ta t ionand has proven track reco rd[14]

The repu ta t ion ofAPI produc t a lso p lays a key ro le for se lec t ion . I f the p roduc t is used success fu l ly in o ther p ro jec ts then i t is p referred by compan ies .

 Deve lop ing a pro to type[14 ][6 ]

Deve lopers down load and use the API produc t to bu i lda pro to type in the ir pre fe rred env ironmen t . Successfu l imp lemen ta t ion of p ro to type is ano ther eva lua t ion process .

 Reading the produc t in forma t ion[6 ]

Sof twa re a t tr ibu tes a re of ten men t ioned in the on l ine marke tplace . Th is enab les deve lope rs to qu ick ly go through the specs o f so f twa re and check if i t’s f i t fo r need in hand .

Choos ing API produc ts:

Choos ing of an API produc t is gene ra l ly made a pa r t of eva lua t ion o f API produc ts . The forma l eva lua t ion me thods d iscussed above have encouraged to compare the po ten t ia l API produc ts and then make a dec is ion to choose the bes t f i t so f twa re . Bu t s ince i t is repea ted ly men t ioned in l i tera tu re tha t these me thods are obso le te fo r prac t ica l app l ica t ion in indus try , choos ing the bes t f i t approach is deba ted . The s tudy done by Oyv ind hauge e t .a l[14]

conc ludes tha t a f irs t f i t approach is fo l lowed ra ther than the bes t f i t . Deve lopers when search ing for an API produc t , of ten tend to exam ine the f irs t po ten t ia l componen t they encoun te r . Exam ining is usua l ly tes t ing the produc t or deve lop ing a pro to type . I f the componen t so lves the need then i t is se lec ted , e lse the search proceeds for nex t po ten t ia l componen t . As deve lopers sea rch th rough API produc ts they ga in know ledge abou t the produc ts andthe p rob lem more . Th is know ledge is used fo r the nex t search to improve the chances o f iden t ify ing the componen t tha t so lves the p rob lem[14].

F ina l ly the dec is ion to adop t an API produc t a f ter eva lua t ion is of ten made by the pro jec t team . Some t imes , c l ien ts had thef ina l say in dec id ing wh ich API produc t to choose[6 ].

Mos t of the t imes pro jec t team or the boss i .e . p roduc t manager has the au thor i ty to make f ina l dec is ion[6 ]. I n ano ther s tudy[9]i t is men t ioned t ha t predef ined ru les of the company gu ide and l im i t the se lec t ion process o f API produc t . Bu t i t is no t men t ioned wh ich fac tors of the company ru les w i l l gu ide the eva lua t ion process .

2 .4 Se lect ion Cr iter ia:

Se lec t ion cr i ter ia are the c r i te r ia app l ied by the prac t i t ione rs of API produc t to eva lua te and

adop t the API produc t . These c r i ter ia has been ex tens ive ly d iscussed in the l i tera ture . The

forma l COTS and OSS se lec t ion me thods men t ioned in above sec t ion have proposed a lo t o f

cr i te r ia tha t deve lopers shou ld use . The prob lem w i th these se lec t ion cr i ter ia is tha t , they are

no t ga the red from the deve lope rs’ perspec t ive i .e . they are no t used by indus try prac t i t ione rs

when se lec t ing an API produc t . There is overwhe lm ing amoun t of l i tera tu re tha t c la ims tha t

t he ex is t ing forma l se lec t ion me thods and the ir se lec t ion cr i te r ia are no t fo l lowed bythe prac t i t ioners[7 ][6][14][15][19 ][20 ].Th is reve la t ion has led researchers to a l ign the ir research towards mak ing emp ir ica l ana lys is abou t se lec t ion o f API produc ts . Resea rche rs s ta r ted to inves t iga te the API se lec t ion prac t ices fromthe pe rspec t ive o f deve lopers . These s tud ies[6][14 ][7 ][21 ]sugges t tha t ra ther ad hoc procedure is fo l lowed in prac t ice by prac t i t ioners to se lec t API produc ts . These in forma l se lec t ion p rac t ices a re men t ioned above . However , on ly few s tud ies have tr ied to e l ic i t the se lec t ion c r i ter ia . These se lec t ion cr i te r ia iden t if ied by these s tud ies a re dep ic ted in be low tab le :

Tab le1-L is t of cr i ter ia iden t if ied in l i te ra ture

Husey in Dagdever in e t .a l[22]havemade observa t ions from COTS /OSSmarke tp lace and iden t if ied5 fac torstha t cou ld be dec is ivetoaffec t the marke t ing ofCOTS /OSS .They have des igned a ques t ionna ire and eva lua ted 264 componen ts w i th the ques t ionna ire . They conc lude tha t fo l low ing the 5 fac to rs can enhance the marke t ing of COTS /OSS.

Se lec t ion Cr iter ia Research paper

 Func t iona l i ty

 Pr ice

 Suppor t and Ma in tenance

 Env ironmen t or p la tfo rm

 Qua l i ty Assurance

An Exp loratory S tudy for Effec t ive COTS and OSS Product

Marke t ing [22 ]

 Cos t

 Cus tom iza t ion requ iremen ts

 Componen t charac ter is t ics

 L icens ing

 Ma in tenance and suppo r t

Open Source Reuse in Commerc ia l F irms[7]

 Func t iona l comp l iance

 Evo lu t ion of p roduc t

 Proven success record of componen t

 Suppor t ava i lab i l i ty

 Fam i l iar i ty w i th componen t

 Ease o f in teg ra t ion

 Perfo rmance and sca lab i l i ty

 L icens ing te rms

 Pr ice

 Documen ta t ion Qua l i ty

 Source code ava i lab i l i ty

Se lec t ion o f th ird par ty so ftware in Off-The-She lf-based so ftware deve lopmen t—An interv iew study

w ith industr ia l pract it ioners[6]

 Func t iona l i ty

 Re l iab i l i ty

 Cos t

 Ease o f use

 Vendor repu ta t ion

 Ease o f cus tom iza t ion

 Ease o f Imp lemen ta t ion

Beyond COTS: The dr ivers o f

COTS app l ica t ion va lue[23]

Madanmohan et.al in [7] performed structured interviews to understand the selection practices of OSS products. As part of their results they have proposed the above mentioned five criteria as selection criteria for selecting OSS.

In [6], Claudia Ayala et.al investigated about industrial practices about component selection.

Their results have declared that informal selection practices are followed in industry. They have also identified above mentioned criteria for selection of software components.

Mark Kell et.al in [23] investigated the factors of COTS products that are perceived as most valuable by the developers. They have identified the factors from previous studies and performed a survey with COTS selectors to rank the factors according to their importance.

In [14], oyvind hauge et.al conducted interviews with developers from 16 Norwegian companies to understand the selection process of OSS. Their research concludes that the selection criteria is situational in nature and there cannot be a generic criteria that fits in every context. They also highlighted that the situation depends on various factors like company policies, project constraints, use of technologies and infrastructure.

2.5 Research Gap:

The majority of the literature related to selection practices of API product is mainly based on researcher experience and didn’t account for the industry practices. The selection methods and selection criteria proposed by this literature are seldom followed in industry. So, it is essential to investigate selection practices of API product from developers’ perception. Few researchers have aligned their research and performed empirical studies. Their results claim that developers follow informal or ad-hoc practices for selection.

Even though these studies have extracted the selection criteria, the empirical evidence is quite limited. Madanmohan et.al in [7] have extracted the criteria that are specific for OSS and Mark kell et.al in [23] identified criteria that are specific to COTS. Also, chen W et.al in [15] performed research only about Chinese companies, whereas API product adoption is a global phenomenon. Claudia Ayala et.al in [6] although has outlined the criteria, a clear description of what developers liked and disliked about each criteria are not provided.

The criteria identified in these studies are followed during selection. But the evaluation criteria i.e. which criteria developers consider when choosing one API product over other are not identified. The selection criteria and evaluation criteria may overlap sometimes. So, there is a need to improve the existing little empirical evidence on evaluation criteria for selection of API products. Also the evaluation criteria has to be exhaustive, including the developers reasoning for considering a specific criteria. This is the aim of current research. With the increase in OSS adoption, API marketplace has got significance in selection of API products.

There is so much data available in feedbacks that developers provide to API product vendors on these marketplaces. Analyzing these feedback channels could provide insights into developers’ decision making activities.

To achieve this objective, developer reviews of API products are analyzed in this study.

Developer reviews for an API product can include reports about bugs, request for feature

additions and other crucial information about different criteria that could have affected their

liking. A list of evaluation criteria that are grounded in developers’ experience and the

reasoning used for liking and disliking of the criteria are the major outcomes expected from

this study.

3 M ^ETHODOLOGY

3.1 Aim:

To understand the criteria that influence the selection of API products, from the perspective of practitioners.

3.2 Objectives:

Following these objectives would help in attaining the research aim:

1) Enquire the state-of-art that discusses the criteria and selection process of an API product.

2) Empirically validate the criteria that are gathered from literature. Validation includes confirming, complementing, removing and changing of factors based on the empirical evidence.

Collecting the selection criteria from literature through background study will help reach objective 1. Objective 2 is attained through studying the developer reviews of API products.

This empirical study helps to validate, complement the criteria gathered from literature.

3.3 Research Question:

RQ: What are the criteria that developers use in reasoning about the acceptability of an API product?

This research question helps to bring out all the criteria that influence selection of a particular API product over other alternatives. In this question, developers are users of API products.

3.4 Research approach overview

Background study is performed as initial part of the study. To get familiar with the research context and so align the future research steps, the existing research on API products selection is studied. All research papers that discuss about the selection of third party components (COTS) or open source products (OSS) or service oriented software are searched in engineering village and Scopus. After studying the literature, relevant data from these papers is extracted. Analysis and synthesis of the background study in the context of this research is presented in section 2.

The second part of the study deals with the empirical research. In this study, the online

developer reviews for various API products are extracted and analyzed to answer the

research question. Qualitative content analysis is the research method employed to extract

insights from the online developer reviews. The guidelines mentioned by Hseih and Shannon

[24] to implement qualitative content analysis are followed for this research.

3 .5 Why th is research method?

Accord ing to research ob jec t ive , the op in ions of deve lopers need to be co l lec ted to f ind the cr i ter iatha t a ffec t the deve lope rs’ l ik ing of an API produc t . A su rvey or a case s tudy cou ld be performed to e l ic i t the fac tors in th is con tex t . The ob jec t ive of the resea rch is to f ind fac tors tha t cou ld be genera l ized o r common ly app l icab le to w ider popu la t ion of API produc ts . S ince case s tudy canno t prov ide resu l ts tha t cou ld be app l icab le to w ider popu la t ion , i t is no t cons idered . Survey is very much app l icab le in th is scenar io bu t qua l i ta t ive con ten t ana lys is of on l ine rev iews is prefe rred over su rvey .There is so much da ta ava i lab le on l ine abou t the rev iews on w ide range of API produc ts . Th is enab les access to more informa t ion abou t deve lopers ’ op in ion and g ives more reach to qua l i ta t ive da ta tha t survey cou ld no t prov ide .A lso , few prev ious s tud ies[6][15 ][7][14]have used survey for s im i lar k ind of research . There fore a l te r ing the resea rch me thod cou ld prov ide oppor tun i ty to make new con tr ibu t ions o r make the ex is t ing c la ims s tronger .

3 .6 Research Imp lementat ion:

The f irs t s tep in imp lemen t ing the research me thod is da ta co l lec t ion i .e . co l lec t ing on l ine rev iews . The da ta co l lec t ion process is ou t l ined in fo llow ing s teps : choos ing a webs i te o r a marke tp lace tha t hos t API produc ts , se lec t ion o f doma in and API produc ts from wh ich rev iew da ta is ex trac ted .

Choos ing the on l ine marketp lace o f API produc ts: To choose a webs i te tha t hos t API produc ts , I have approached few deve lopers who have exper ience work ing w i th the API produc ts . The pr imary recommenda t ions a re : apache .org [25], ec l ipse .o rg [4], sourcefo rge .com[3], b inp ress .com[26]. Ou t of these apache .org d id no t have any user rev iews to the produc ts hos ted . A lso i t d idn ’ t have any produc t re la ted me tr ics l ike user ra t ings or number of down loads . Ec l ipse .org had p roduc t me tr ics and use r rev iews to some produc ts ,bu t they are no t comprehens ive of ten con ta in ing a s ing le sen tence descr ib ing e i ther user l iked or d is l iked the p roduc t . Same was the case w i th sou rce forge and b inpress . A t th is s tage a l is t o f c r i ter ia has emerged to choose the re levan t webs i te for the resea rch .

 Webs i tes tha t cons is ts w ide range of API produc t w i th user rev iews

 Webs i tes tha t prov ide any k ind of p roduc t me tr ics l ike , user ra t ings o r user down loads

 Webs i tes tha t have comprehens ive user rev iews tha t a re au then t ica ted .

The ra t iona le to inc ludethe second cr i ter ia i .e . access to produc t me tr ics is tha t , i t he lps to f ind co rre la t ion be tween the fac tors ex trac ted from rev iews and the user ra t ings or user down loads . Fo r examp le , rev iews of produc t w i th h igh ra t ing o r more down loads can be ana lyzedto f ind wha t fac tors lead to more ra t ings . In pursu i t of webs i tes w i th above se t cr i te r ia in m ind , au tho r found thesetwo webs i tes : G2c rowd .com[27], trus trad ius .com[28].

A l though these webs i tes don’ t hos t any API produc ts they co l lec t user rev iews for vary ing

range of API produc ts . Ou tof these g2crowd is chosen as i t had re la t ive ly more API

produc ts sp read across vary ing sof tware doma ins . They have a lso p rov ided some key me tr ics

l ike user ra t ings fo r each rev iew . Mos t o f the rev iews are comprehens ive , de ta i l ing wha t

aspec ts users l ikedand d is l iked abou t the API produc t . The rev iew mechan ism in G2crowd

asks user to l ink the ir L inkedIn accoun t to ensure the au then t ic i ty of the rev iews . The

webs i te se lec t ion p rocess was ca rr ied throughou t the research to loca te any be t ter webs i tes .

Few o the r s im i lar webs i tes were s tumb led in the process , bu t they were ignored as they d idn’ t have the amoun t of da ta tha t g2crowd p rov ided .

Se lec t ion o f doma in and API products: The nex t s tep is to choose API produc ts and rev iews . API produc ts have vary ing a ttr ibu tes l ike Doma in , Pa id or free , produc t ra t ings (h igh , med ium or low) . Accord ing to resea rch ob jec t ive , i t is impor tan t to f ind the fac tors tha t a ffec t user l ik ing towards the API produc ts . The l ik ing of use r con tr ibu tes to the p roduc t ra t ing . I t is obv ious tha t a produc t w i th h igh ra t ings , w i l l have rev iews tha t exp la in wh ich fac tors led to use rs l ik ing of the p roduc t . S im i la r ly s tudy ing rev iews of low ra ted produc t w i l l en l igh ten us abou t the fac tors tha t led to use r d is l ik ing of the p roduc t . So , theproduc ts were d iv ided in to th ree sub g roups based on the ir ra t ings . Th is app roach is s tra t if ied random samp l ing .

S tra t if ied random samp l ing is fo l lowed when the popu la t ion is he terogeneous and when researcher wan ts to know abou t a spec if icsub g roup[29]. Th is me thod enab les the popu la t ion to be d iv ided in to non- over lapp ing sub g roups or s tra ta and then perfo rm ing s imp le random samp l ing among each s tra ta[29 ]. In the con tex t o f th is resea rch ,the s tra ta are formed based on p roduc t ra t ings . Th is cou ld he lp us to know abou t fac tors tha t con tr ibu te to h igh low ra t ings of the API produc t . The nex t s tep is to d iv ide the API produc ts based on the ra t ings in to th ree s tra ta i .e . Low ra ted , Med ium ra tedand h igh ra ted . Bu t th is cou ld no t be ach ieved due to skewed ra t io of h igh and low ra ted API produc ts on the webs i te g2crowd .com . Accord ing to au thor obse rva t ion a t th is s tage of research , 88% API produc ts w i th rev iews be longed to H igh ra t ings , 8% be longed to Med ium ra t ings and 4% be longed to low ra t ings . Mos t of the produc ts w i th low ra t ings d id no t have comprehens ive rev iews . Th is asymme try in API produc t d is tr ibu t ion has caused for ad jus tmen t in samp l ing process . The compos i t ion o f s tra ta a re a l te red .Ins tead of choos ing API produc ts , API produc t rev iews are chosen in each s tra ta . To exp la in in de ta i l : Each rev iew has a use r ra t ing tag to i t . The ra t ing ranges from 1-5 . Ra t ings 1 , 1 .5 , 2 come under low user ra t ing s tra tum . 2 .5 , 3 , 3 .5 come in to med iumra t ings s tra tum . H igh ra t ing cons is t rev iews w i th 4 , 4 .5 and 5 ra t ings . So , rev iews w i th h igh use r ra t ings a re g rouped in to one s tra tum and s im i la r ly rev iews w i th med ium and low ra t ings are g rouped in to respec t ive s tra tum . Th is he lped to ach ieve propo r t iona te amoun t of rev iews across the three s tra ta and draw mean ing fu l ins igh ts from each . Then the rev iews in each s tra tum are se lec ted random ly .A l though the rev iews are se lec ted random ly in each s tra ta , measures a re taken to induce d ivers i ty in samp le . D iverse samp le w i l l ensure w ide coverage of the targe t popu la t ion . The s teps taken to induce d ivers i ty in the samp le are :

 Se lec t rev iews from API produc ts w i th vary ing doma ins .

 Se lec t rev iews from API produc ts tha t are bo th commerc ia l and open sou rce .

A p i lo tsamp le is chosen and the rev iews areana lyzedto tes t the su i tab i l i ty of samp l ing me thod to th is resea rch . The p i lo t s tudy produced re l iab le resu l ts thereby vouch ing for the adop ted samp l ing me thod . Tak ing lessons from the p i lo t s tudy , the resea rch is scaled to larger samp le . The samp le s ize i .e . number o f rev iews is no t f ixed in i t ia l ly . The ana lys is is performed un t i l the resu l ts ach ieve sa tura t ion . The s ta ts re la ted to the number of rev iews , API produc ts , Doma ins chosen to reach sa tura t ion po in t are d iscussed in resu l ts .

3 .7 Data Extract ion:

Da ta on g2crowd webs i te i .e . produc t rev iews a re ex trac ted to pe rfo rm ana lys is . The sec t ions

of da ta ex trac ted from the webs i te a re p roduc t rev iews , ra t ing , produc t name and doma in of

produc t . Th is da ta is cop ied in toM icrosof t word shee t to perform in i t ia l part o f the ana lys is .

La te r exce l shee ts are used for fur the r ana lys is .

3 .8 Data Ana lys is:

Th is sec t ion as a who le dea ls w i th the ana lyses pa r t o f research . I t de ta i ls abou t the process fo l lowed to conver t the da ta s i tt ing in word and exce l shee ts in to mean ing fu l ins igh ts , wh ich leads to resu l ts o f th is resea rch .

3 .8 .1 Why conten t ana lys is?

Con ten t ana lys is and g rounded theory are w ide ly used and mos t feas ib le me thods to pe rfo rm qua l i ta t ive da ta ana lys is[30]. Grounded theo ry is used genera l ly to genera te a theory , by es tab l ish ing re la t ionsh ips be tween ob ta ined ca tegor ies and codes . Qua l i ta t ive con ten t ana lys is is used when the goa l o f resea rch is to unders tand the da ta and express i t in fo rm of ca tegor ies andthemes[30 ]. As men t ioned in sec t ion 3 .1i .e . resea rch a im , th is resea rch exp lo res the cr i te r ia tha t deve lopers use when choos ing a produc t . S ince the emphas is is no t on f ind ing re la t ionsh ips be tween ca tegor ies and genera t ing theory , Qua l i ta t ive con ten t ana lys is is used .

Con ten t ana lys is can be perfo rmed w i th three var ious app roaches[24 ], bu t conven t iona l con ten t ana lys is is chosen . Conven t iona l con ten t ana lys is is used when ca tegor ies and themes are to be ex trac ted d irec t ly from the da ta w i thou t impos ing prev ious know ledge on da ta . In th is resea rch , rev iew da ta pos ted on webs i te is co l lec ted and ana lyzed . As users a re no t imposed to f i l l the rev iew w i th curren t research agenda in m ind , there is h igh chance tha t da ta is genu ine ly g rounded in the ir expe r iences . Th is enab les access to the da ta tha t is pure ly g rounded from the ir exper ience and no t d i lu ted w i th resea rch agenda o r any pre conce ived ca tegor ies . Th is scenar io prov ides poss ib i l i ty to ex trac t new ca tegor ies o r sub ca tegor ies from the da ta . Hence conven t iona l con ten t ana lys is is chosen .

3 .8 .2 Ana lys is Procedure

Conven t iona l con ten t ana lys is or induc t ive approach o f con ten t ana lys is is fo l lowed for the research . Con ten t ana lys is is used toana lyzequa l i ta t ive da ta and the reby ex trac t ca tegor ies or themes from the da ta[24]. Conven t iona l con ten t ana lys is is used when the research l i tera tu re is l im i ted . Th is approach encou rages researcher to app roach the da ta w i thou t any pre conce ived theo r ies or l i tera tu re . Th is he lps in es tab l ish ing ca tegor ies and themes tha t accu rate ly rep resen t the da ta in hand[24]. The s teps invo lved to perform t h is ana lys is as desc r ibed by[24 ]are :

 Read ing the da ta to ge t fam i l iar and immersed . Th is he lps to bu i ld a sense of unders tand ing abou t wha t the da ta represen ts .

 H igh l igh t ing the keywords tha t seemed re levan t to answer the research ques t ion . Ass ign codes to these keywords .

 Depend ing on the s im i lar i t ies and d ifference of in i t ia l codes , they are c lass if ied in to ca tegor ies and sub ca tegor ies .

 Th is p rocess is repea ted i te ra t ive ly un t i l a f ina l l is t o f mu tua l ly exc lus ive and exhaus t ive ca tegor ies and sub ca tegor ies are ob ta ined . Def in i t ions o f the each ca tegor ies and sub ca tegor ies are a lso desc r ibed .

 The resu l ts can be presen ted form of tree d iag ram . Exemp lars fo r each ca tegory can

be prov ided from da ta to repor tthe ana lys is .

Overall analysis procedure is depicted in picture below.

Figure 1- Analysis process followed in this research

4 ^RESULTS

4.1 Summary of sample:

A total of 480 reviews are obtained as results for the study. These reviews are extracted by following the stratified random sampling approach as described in section 3.6. The reviews are collected until saturation is reached i.e. till no new criteria would emerge from analyzing the reviews. All reviews are collected from g2crowd website for software reviews. Reviews are collected into three sub groups divided according to ratings. Low rated reviews i.e.

ratings 1, 1.5, 2 are strata 1. Medium rated reviews i.e. 2.5, 3, 3.5 are grouped into strata 2.

High rated reviews are i.e. 4, 4.5, 5 ratings come under strata3. Each strata has 160 product reviews. The reviews are collected from various API products belonging to 8 different domains. The domains and the API products are chosen randomly.

Domain Number of reviews

Total Strata 1 Strata 2 Strata 3

Databases 20 21 20 61

Web frameworks 18 20 17 55

IDE (integrated

development environment) 22 20 19 61

Version control software 17 22 20 59

Bug Tracking software 20 19 21 60

Test management software 23 19 21 63

Application Performance

management software 18 16 20 54

SAAS(software as a

service) 22 23 22 67

160 160 160 480

Table 2 - Distribution of reviews across various domains

The summary of products chosen in each domain are mentioned in the table below.

Domain API products

Databases MongoDB, CouchDB, OrientDB, Cassandra, Neo4j, Redis

Web Frameworks AngularJS, Django, Backbone, RubyonRails, Symfony

IDE Eclipse, Netbeans, ColdFusion, Visual studio, Code::Blocks, IntelliJ, Webstorm VCS Microsoft Team foundation, Subversion, CVS, Clearcase, Starteam, Mercurial,

BitBucket, Git,

Bug tracking Jira, BugZilla,Assembla,Trac,LightHouse, Test management software ApacheJmeter, Qtest, HP Quality centre, HP

ALM, TestTrack,

Application Performance software Splunk, DataDog, IBM management, Foglight, Logentries, NewRelic Software as a service OpenStack, Linode, Microsoft Azure,

Rackspace, Hellion, Heroku, Amazon EC2, GoogleAppEngine, Amazon S3, Table 3 - Domain and the API products from which reviews are extracted

A screenshot of review by the developer in G2crowd website is shown below. It has product rating given by the reviewer, shows LinkedIn authentication and the review is comprehensive with reasoning for liking and disliking the API product.

Figure 2 Screenshot of a typical review on G2crowd

4.2 Findings from content analysis

Step1: Get to know the data

This step deals with getting familiar with the data and the context. Before proceeding to coding process, the reviews are read and understood to get a sense of what aspects the data holds and how the relevant results could be drawn. This process also includes studying about the product and its domain on which the reviews are written. This helps to understand the review better when user is talking about the functional aspects of the product.

Step 2: Initial coding

The step describes how the preliminary codes are formed from the data. The data in reviews are present in the format of two questions, what do you like best? And what do you dislike?

User responses to these questions consisted of different aspects like users mentioning about their likes and dislikes with product, users briefing about their use cases of the product and users providing suggestions to other users. Out of these aspects, the data that reflect users liking and disliking is only relevant for the research and so other aspects are ignored while coding. When coding is done, research objective is always kept in mind to identify the relevant data and channel out the unnecessary noise out of the data.

The text in word sheet that is reflective of users liking, is highlighted by adding a comment next to it. This comment or code represents the segment of highlighted text and it commonly contains words that are extracted from the raw text. An instance from word sheet with highlighted text and comment is shown below. This step helped to find the initial codes from the raw text. All the reviews of a strata are subjected to initial coding before proceeding for next step.

Figure 3 Snapshot of Word document at initial analysis stage

Step 3: Forming sub categories

The comments (codes) and the highlighted raw text are exported into excel sheet for further analysis. Columns of excel sheet contain the raw text (highlighted), comment added to it (code), product rating given by user, product domain and a label that indicates user liking (L) or disliking (DL) any part of product. In this step, the commonalities observed in codes are mapped together to form subcategories. Raw text and the attached comments (codes) are studied row by row to find patterns in the data. Data contributed to certain pattern are merged under a sub category. This process is repeated across all excel sheets to extract the sub categories.

Step 4: Deriving Categories

The data from previous step contains excel sheet with raw text, codes and sub category. The excel sheet with codes is studied extensively to check for any emerging categories. The categories are formed by aggregating the sub categories that exhibit a similarity. The categories formed at this stage are high level, very abstract and do not fully represent the final categories. These initial categories helped to organize the data and group the codes into clusters. This facilitates for easy analysis in further steps to merge categories if needed.

Below shows a snapshot of the excel sheet at this stage.

Figure 4- Snapshot of analysis process in excel at step 4

Step 5: Finalizing Categories and subcategories

This step involves presenting the final set of categories and sub categories along with their definitions. The sub categories obtained in step 4 are studied to check if it fits the existing category or need to be moved. This process enables to form a new set of categories that accurately represent the sub categories. The categories and sub categories are constantly revisited, updated and merged. The revisions are carried out till the emerged categories and sub categories are exhaustive and become mutually exclusive. The example provided below depicts the process of raw text emerging into category and sub categories.

Table 4 - The journey of raw data emerging into category

The overall coding scheme is described in the table below. Codes are the initial comments made to the raw text. Sub categories are similar codes clustered together. Categories are abstract representation for group of sub categories exhibiting the similar properties.

Codes Sub category Category

there is substantially less documentation

Completeness Limited documentation. Needs

to be more detailed.

Raw text Code Sub category Category

“Although Django provides us quality examples, only has few examples,”

Quantity of code examples

Learning resources

Documentation

“

Really good documentation and tutorials help a new developers to easy dive in and focus on business logic and not on framework details

Helpful tutorials

“And if you don't need all the features it's hard to trim

down the

documentation to find your way through it based on your needs”

Hard to find the needed documentation

Finding artefacts

DOCUMENTATION Hard to navigate through

documentation.

Finding artefacts Difficulty finding information

Tutorials and online resources

Learning resources Source code available to solve

issues

Understandable and concise

Understandablility Misleading documentation

Bugs created in GITHUB solved quickly

Support Online

COMMUNITY SUPPORT Questions in StackOverFlow

not answered quickly Rather small community

Size and Popularity Less popular and hence small

ecosystem

Active community, providing

more plugins Rate of activity

Contributors develop framework rapidly Can be up and running in minutes

Ease of getting started

LEARNABILITY Easy to go from idea to

implementation

Difficult to understand for ppl

with less technical knowledge Prerequisites Requires skill to understand

the software

Need time to leverage software

its fullest capability Time to Master Need to learn lot to use

advanced features

unintuitive, design causes

distraction Intuitiveness

USABILITY Clear and friendly UI

Better visual features Appeal

Visually not appealing

Complex and outdated Freshness

Old and doesn’t follow trends

Impossible to find all features Navigation Hard to navigate

Too much information Complexity

Unnecessary options

problem Innovation

FUNCTIONALITY First of its kind

Suits the user needs Suitability

Efficient implementation

Functional Efficiency Bugs in the software

Plenty services offered

Diverse Features Missing many features

Many built-in functions

Minimal coding Need to write boilerplate code

Integrates well with other tools

Integration

INTERACTION WITH OTHER COMPONENTS Issues with integration

Works well on target platform

Compatibility Compatible with other tools

Hard to migrate to other

environment Portability

High loading time Speed and responsiveness

PERFORMANCE Slow and unresponsive

Hangs and crashes Reliability

Loses data, unreliable

Heavy and memory hog Resource consuming

Unable to do much

customization Customization

FLEXIBILITY Customize with minimal efforts

unopinionated Developer control

Imposes a strict structure

Complex pricing plan Pricing strategy

PRICE Hidden charges

Better alternatives for similar

pricing Competing alternatives

Table 5 - codes, sub categories and categories

5 ^ANALYSIS

In this section, the obtained results are analyzed to systematically answer the research question. The first section provides an overview of the answer and he second section describes the details about how the results provide answers for the research question.

5.1 Answering the Research Question:

Research Question: What are the criteria that developers use in reasoning about the acceptability of API products?

Analyzing the results obtained in table 5 in above section, the criteria can be extracted. The categories in that table emerge as criteria and the sub categories emerge as sub criteria to answer the research questions. These criteria and sub criteria are found to be used by developers when making a decision about selection of an API product. The results i.e.

criteria, sub criteria along with the descriptions are mentioned in the below table.

CRITERIA SUB CRITERIA DESCRIPTION

DOCUMENTATION

UNDERSTANDABILITY If the documents are legible, unambiguous and easy to understand

LEARNING RESOURCES If there are effective tutorials, walkthrough or if source code is available.

FINDING ARTEFACTS The ease through which user could navigate and find the help he needs.

COMPLETENESS If the information on topic is detail and not misleading.

LEARNABILITY

EASE OF GETTING STARTED

The ease with which user can work on system with no training or help. How easy it is to setup and configure, and get the basic app running.

PREREQUISITES If the software demands more knowledge (or previous experience) to leverage it to full extent.

TIME TO MASTER If it takes more or less time to learn the overall system i.e. To move from basic to advanced user of the system.

COMMUNITY SUPPORT

SUPPORT ONLINE

The amount of help received on community blogs or developer oriented websites

SIZE AND POPULARITY The size of community and

popularity of product RATE OF ACTIVITY How often does

community contribute and hence active or inactive.

USABILITY

INTUITIVENESS

The ability of interface to help user carry the tasks with no delays or distraction caused by design i.e., doesn’t slow down the user in his/her pursuit.

NAVIGATION

The ease with which the user could navigate (or explore) through the software and find the information that user is looking for.

APPEAL The degree of

attractiveness and visually stimulating

FRESHNESS If UI following latest trends or still lurking in old and outdated designs.

COMPLEXITY

The complexity due to information overload or taking many steps to achieve a task or easiness of UI (simple and understandable.

FUNCTIONALITY

INNOVATINESS Product offering innovative features or first of a kind or outstanding features.

SUITABILITY If the products meet the user needs/ or lacks any features that user would like to have.

FUNCTIONAL EFFICIENCY Users liking specific features and also impressed with the hassle free implementation. Presence or absence of bugs in software

DIVERSE FEATURES Has many features built-in or provides many services that extend product capabilities.

MINIMAL CODING

Availability of many built- in functionalities that substantially reduces the coding effort, no need to write boiler plate code or reinvent the wheel.

CUSTOMIZABILITY The ability of product to

adapt according to the

FLEXIBILITY

needs of user.

DEVELOPER CONTROL

The amount of freedom developer possesses to use the system i.e. if the software imposes any structure upon user.

INTEGRATION WITH OTHER COMPONENTS

INTEGRATION The feasibility of interaction with other components to collaborate and work together. Support for other tools.

COMPATIBILITY If the product able to work on the target platform(s) or environment without issues.

PORTABILITY The ease of migrating to other environment.

PERFORMANCE

SPEED AND RESPONSIVE If the software is slow or fast when operating (ex:

page loads, lags)

RELIABILITY If the s/w properly responds to the user activity or doesn’t respond well due to hangs or crashes.

RESOURCE CONSUMING If the software is heavy, bloated or if it’s lightweight. Takes up many resources.

PRICE

PRICING STRATEGY

If features are worth the price i.e. cheap or expensive, if pricing model is easy (making charges explicit and avoid hidden tax). If provided Flexibility in pricing plan.

COMPETITIVE ALTERNATIVES

Presence of alternatives that provide cheap and better solutions.

Table 6 - Main findings of the study

From the graph below i.e. figure 5 it can be observed that Functionality was the most liked criteria by the developers in high rated reviews i.e. Strata 3: 4-5 ratings. Usability was the second most liked criteria in high rated reviews. Similarly in figure 6 it can be observed that Usability was most disliked criteria in low rated reviews i.e. strata 1: 1-2 ratings. Next to usability, functionality was mentioned as most disliked criteria in low rated reviews.

According to this analysis the order of importance among criteria cannot be drawn but it can

interpreted that Functionality and Usability criteria are relatively important and they could

have considerably more impact on developers’ liking towards API product.

Figure 5 most liked Criteria in Reviews with High Ratings: Functionality

Figure 6 most disliked criteria in reviews with Low ratings: Usability

5.2 Detailed analysis of each criteria

This section provides the details of each criteria and their sub criteria with descriptions. The reasoning of developers for liking and disliking each criteria in an API product are described.

Analyzing these developer reasoning, practices to follow for API product providers are

discussed. These suggestions are supported by the chain of evidence provided in each section

i.e. true statements mentioned by developers. In this section user implies the user of API products i.e. developers.

5.2.1 Criteria: Documentation

Documentation refers to the artefacts available around the product eco system, which help user to get started with the product and resolve any problems faced. Through Documentation developers expect many things like help for setup or configuration, sample examples, tutorials and also source code availability. The sub criteria that are important for assessing the documentation criteria are Understandability of the artefacts, availability of learning resources, detailed and complete documentation, and ease of finding artefacts.

Figure 7 – No. of times Documentation mentioned by reviewers across 1-5 ratings

Figure 8 – Percentage (%) of reviewers who liked and disliked Documentation across

1-5 ratings

Figure 9 – Occurrence percentage of each sub criteria in documentation

Figure 4 shows that documentation criteria is mentioned most in reviews with 4-5 ratings.

From figure 5, it could be understood that reviews with high ratings have liked the documentation and reviews with low rating have disliked the documentation. But the difference between likes and dislikes is not much significant. It means that documentation doesn’t tend to have much impact on the rating of the product. From Figure 6, we can understand that Learning resources and Completeness sub criteria are most mentioned by the developers when they refer to documentation.

Understandability: The documentation of the product should be easy to understand. It should be readable and should not confuse the developer with ambiguity or misleading information. Users also prefer the documentation to be interactive and not boring. A simple and easy to go through documentation should not make user turn to external help to understand it.

”I also found the documentation extremely boring and old school and the books I found do not seem interesting.”

“The help documentation is not very helpful so it is best to have someone teach users in person.”

“Documentation is something that is really confusing to understand and contain misleading information at times”

Learning Resources: The resources available to facilitate the learning of product. These resources mentioned by users are official documentation and tutorials, walkthroughs, sample code examples and also source code of the product. The quality and quantity of resources to learn the product has affected the liking of the users.

“

Although Django provides us quality examples, only has few examples,

“

Really good documentation and tutorials help a new developers to easy dive in and focus on business logic and not on framework details

“

Whenever I don't understand how a method works in backbone I can read the source code

and easily understand and get back to work