O PEN S OURCE S OFTWARE
3.5 Validity Threats
busi-ness environment. We also employed descriptive evaluation where we obtained detailed scenarios to demonstrate the utility of the CAP model, see guideline three by Hevner [81].
To improve the external validity of the CAP model, we conducted exploratory case studies at three different case firms (see Section 7). In these case studies, we used static validation [71] where we presented the CAP model to participants from the respective firms and applied it in a simulated setting as part of the interviews.
In two of the cases, semi-structured interviews were used with one representative from each firm. In the third case, a workshop setting was used with eight par-ticipants from the firm. When collecting feedback from the three case firms, we focused on applicability and usability of the CAP model.
threat to internal validity is taken into consideration during the estimation process and therefore is not in the direct focus of the CAP model. Moreover, the CAP model does not prevent additions of new factors that support these estimates.
Triangulationrefers to the use of data from multiple sources and also ensur-ing observer triangulation [158]. In this study, our data analysis involved inter-pretation of qualitative and quantitative data obtained from Sony Mobile. We applied data triangulation by using Sony Mobile’s internal artifacts repositories, documents related to contribution strategies and consultation with relevant experts before proposing the CAP model. There were risks of identifying the wrong data flows and subjective interpretation of interviews. In order to mitigate these risks, concerned multiple experts with different roles and experiences (see Table 1) were consulted at Sony Mobile. We ensured observer triangulation by involving all researchers who authored this manuscript into the data collection and analysis phases.
External Validity
External validity deals with the ability to generalize the study findings to other contexts.
We have focused on analytic generalization rather than statistical generaliza-tion [60] by comparing the characteristics of the case to a possible target and pre-senting case firm characteristics as much as confidentiality concerns allowed. The scope of this study is limited to firms realizing OI with OSS ecosystems. Sony Mobile represents an organization with a focus on software development for em-bedded devices. However, the practices that are reported and proposed in the study has the potential to be generalized to all firms involved in OSS ecosystems. It should be noted that the case firm can be considered a mature firm in relation to OSS usage for creating product value and realizing product strategies. Also, they recognize the need to invest resources in the ecosystems by contributing back in order to be able to influence and control in accordance with internal needs and incentives. Thus, the application of the proposed CAP model in an other context or in other firms remains part of future work.
The CAP model assumes that firms realize their products based, in part, on OSS code and OSS ecosystem participation. This limits its external generalizabil-ity to these firms. At the same time, we believe that the innovation assessment part of the CAP model may be applied to artifacts without OSS elements. In this case, the CAP model provides only partial support as it only helps to estimate the innovativeness of the features (as an innovation benchmark) without setting con-tribution strategies. Still, this part of the CAP model should work in the same way for both OSS and non-OSS based products. Finally, the classification of software artifacts has a marked business view and a clear business connotation. A threat remains here that important technical aspects (e.g. technical debt, architectural complexity) are overlooked. However, throughout the static validation examples,
we saw limited negative impact on this aspect, especially in a firm experienced in building its product on an OSS platform.
The meta-model was derived from Sony Mobile’s software artifact reposito-ries. We believe that the meta-model will fit organizations in similar characteris-tics. For other cases, we believe that the meta-model can provide inspiration and guidance for how development organizations should implementing the necessary adaptations to existing requirements management infrastructure, or create such, so that contribution strategies for artifacts can be communicated and monitored. We do acknowledge this as a limitation in regards to external validity that we aim to address in future design cycles.
Construct Validity
Construct validity deals with choosing the suitable measures for the concepts under study [159]. Four threats to the construct validity of the study are highlighted below.
First, there was a risk that academic researchers and industry practitioners may use different terms and have different theoretical frames of reference when ad-dressing contribution strategies. Furthermore, the presence of researchers may have biased the experts from Sony Mobile to give information according to re-searchers’ expectations. The selection of a smaller number of experts from Sony Mobile might also contribute to the unbalanced view of the construct.
Second, there was a potential threat to construct validity due to the used inno-vation assessment criteria based on business impact and control complexity. Both dimensions can be expanded by additional questions (e.g. internal business per-spective or innovation and learning perper-spective [97]) and the CAP model provides this flexibility. One could argue that also technical and architectural aspects should be taken into consideration here. At the same time, the static validation results at Sony Mobile confirm that these aspects have limited importance at least for the studied cases. Still, they should not be overlooked when executing the CAP model in other contexts.
Third, a common theoretical frame of reference is important to avoid misinter-pretations between researchers and practitioners [158]. In this study, the Kraljic’s portfolio model is used as a reference framework to the CAP model. However, the horizontal and vertical dimensions of Kraljic’s portfolio model were changed to control complexity and business impact respectively. Both industry practition-ers and academic researchpractition-ers had a common undpractition-erstanding of Kraljic’s portfolio model [104] before discussions in the study. Furthermore, theoretical constructs were validated by involving one of the experts in the writing process from Sony Mobile to ensure consistent understanding.
Fourth, prolonged involvement refers to a long-term relationship or ment between the researchers and organization [158]. Since there was an involve-ment of confidential information in the study, it was important to have a mutual
trust between academic researchers and practitioners to be able to constructively present the findings. The adequate level of trust was gained as a result of long past history of collaboration between academic researchers and experts from Sony Mobile.
Reliability
The reliability deals with to what extent the data and the analysis are dependent on the specific researcher and the ability to replicate the study.
Member checkingmay involve having multiple individuals go through the data, or letting interviewees review a transcript [158]. In this study, the first two authors proposed the meta-model after independent discussions and reviewed by the third author. Furthermore, the model was validated by a team lead, software devel-oper, and senior manager at Sony Mobile, involved in making contributions to OSS communities, were consulted to ensure the correctness of the meta-model and associated data.
Audit trailregards maintaining traceability between collected data during the study [158]. For this study, the first two researchers kept track of all the mined data from the software artifact repositories as well as the email and informal communi-cation between researchers and Sony Mobile representative. Results were shared with Sony Mobile for any possible misinterpretation or correction of data.