E-Learning as a tool to support the integration of machine learning in product development processes

(1)

EXAMENSARBETE INOM TEKNIK, GRUNDNIVÅ, 15 HP

STOCKHOLM, SVERIGE 2020

E-Learning as a tool to support the integration of machine learning in product development processes

ANTON EDIN

MARIAM QORBANZADA

KTH

SKOLAN FÖR INDUSTRIELL TEKNIK OCH MANAGEMENT

(2)

(3)

Examensarbete Integrerad produktutveckling grundnivå, 15 hp, kurskod MF131x

E-Learning som ett verktyg för att underlätta in- tegration av maskininlärning i produktutveckl-

ingsprocesser

Anton Edin Mariam Qorbanzada

Examinator

Sofia Ritzén

Handledare

Magnus Eneberg och Gunilla Ölundh Sandström

Sammanfattning

Detta forskningsarbete fokuserar på tillämpningar av elektroniska utlärningsmetoder som alternativ till lokala lektioner vid integrering av maskininlärning i produktutvecklingsprocessen. Framförallt är syftet att undersöka om det går att använda elektroniska utlärningsmetoder för att göra maskininlärning mer tillgänglig i produktutvecklingsprocessen. Detta ämne presenterar sig som intressant då en djupare förståelse kring detta banar väg för att effektivisera lärande på distans samt skalbarheten av kunskapssprid- ning.

För att uppnå detta bads två grupper av anställda hos samma företagsgrupp, men tillhö- rande olika geografiska områden att ta del i ett upplägg av lektioner som författarna hade tagit fram. En grupp fick ta del av materialet genom seminarier, medan den andra bjöds in till att delta i en serie tele-lektioner. När båda deltagargrupper hade genomgått lekt- ionerna fick några deltagare förfrågningar om att bli intervjuade. Några av deltagarnas direkta chefer och projektledare intervjuades även för att kunna jämföra deltagarnas åsikter med icke-deltagande intressenter. En kombination av en kvalitativ teoretisk ana- lys tillsammans med svaren från intervjuerna användes som bas för de presenterade re- sultaten.

Svarande indikerade att de föredrog träningarna som hölls på plats, men vidare kodning av intervjusvaren visade på undervisningsmetoden inte hade större påverkningar på deltagarnas förmåga att ta till sig materialet. Trots att resultatet pekar på att elektroniskt lärande är en teknik med många fördelar verkar det som att brister i teknikens förmåga att integrera mänsklig interaktion hindrar den från att nå sitt fulla potential och därige- nom även hindrar dess integration i produktutvecklingsprocessen.

Nyckelord: Maskininlärning, artificiell intelligens, kunskapsdelning (i multinationella or- ganisationer), E-learning, kunskapscenter

(4)

(5)

Degree project in Integrated Product Development First cycle, 15 cr, course code MF131x

E-Learning as a tool to support the integration of machine learning in product development pro-

cesses

Anton Edin Mariam Qorbanzada

Examiner

Sofia Ritzén

Supervisor

Magnus Eneberg and Gunilla Ölundh Sandström

Abstract

This research is concerned with possible applications of e-Learning as an alternative to onsite training sessions when supporting the integration of machine learning into the product development process. Mainly, its aim was to study if e-learning approaches are viable for laying a foundation for making machine learning more accessible in integrated product development processes. This topic presents itself as interesting as advances in the general understanding of it enable better remote learning as well as general scalability of knowledge transfer.

To achieve this two groups of employees belonging to the same corporate group but working in two very different geographical regions where asked to participate in a set of training session created by the authors. One group received the content via in-person workshops whereas the other was invited to a series of remote tele-conferences. After both groups had participated in the sessions, some member where asked to be interviewed. Additionally. The authors also arranged for interviews with some of the participants’ direct managers and project leaders to compare the participants’ responses with some stakeholders not participating in the workshops. A combination of a qualitative theoretical analysis together with the interview responses was used as the base for the presented results.

Respondents indicated that they preferred the onsite training approach, however, further coding of interview responses showed that there was little difference in the participants ability to obtain knowledge. Interestingly, while results point towards e-learning as a technology with many benefits, it seems as though other shortcomings, mainly concern- ing the human interaction between learners, may hold back its full potential and thereby hinder its integration into product development processes.

Keywords: Machine learning, artificial intelligence, knowledge sharing (in multinational organizations), E-learning, center of excellence

(6)

(7)

Preface

This degree project was written in the course MF131X, integrated product development at KTH Royal Institute of Technology, spring term 2020.

The subject of integrating machine learning and other innovative technologies into current product development processes through coding workshops is of interest to any party aiming to stay on the cutting edge of rapid technological advances.

We want to thank our supervisors, Magnus Eneberg and Gunilla Ölundh Sandström, for their continuous support, feedback, and help whenever it was needed. Without them this degree project would not have been possible.

Finally, we want to thank all the participants and the companies involved in the training sessions and interviews for their time and support.

Anton Edin and Mariam Qorbanzada

Kungliga Tekniska Högskolan, Stockholm, July 2020

(8)

1 Introduction

The following section introduces the background of the study was well as the research questions and a purpose and limitations.

1.1 Background

To meet ever more challenging customer demands companies apply different product development processes. These vary in complexity and can often be standardized to increase the scalability of operations. Newly launched products also struggle with grow- ing requirements for technological changes and innovation (Nagy & Szalai, 2011). Prob- lem solving therefore becomes a key factor in most product development processes. To solve complex problems, data, from which solutions can be derived, is collected, and analyzed. This is where Machine Learning, among other optimization algorithms, becomes an interesting alternative.

Machine Learning is commonly defined as a variant of artificial intelligence, where a computerized process learns to produce a desired output based on a set of questions and answers that the system was previously shown (IBM, 2020). In short, instead of programming exact rules to solve a problem with a computer, Machine Learning focuses on data analysis rather than code writing (TensorFlow, 2020).The programmer provides their model with a set of examples, and the model extracts and learns patters from these. A useful and very illustrative quote from TensorFlow (2020) states that:

“You can think of machine learning as ‘programming with data’”.

Organizations and companies across many industries implement, or make use of, machine learning. About 37% [of organizations] claim to utilize artificial intelligence (“AI”), which is widely regarded as a sub-category of the aforementioned machine learn- ing (Gartner, 2019). Examples of applications include computer assisted diagnostics, healthcare, video games, autonomous driving, and customer support. Research indicates that lack of subject matter expertise is one of the central bottlenecks holding back even more widespread adoption of the technology. Intel Corporation, a world leader in computer processor design, also notes that 98% of organizations claim analytics are important to driving business processes (Intel, 2020). Intel also states that only 40% of companies today make use of AI or similar analytics methods.

One way to go about mitigating these challenges can be new approaches to knowledge sharing (Hong, et al., 2011). Knowledge sharing is a vital part in developing working processes, and when it comes to complex technologies these is imperative.

The ability to transfer knowledge can add value to multiple stakeholders and develop company processes across an entire corporation. While these findings have been largely accepted by organizations, the actual transfer of knowledge between departments and professions remains problematic (Guerra, et al., 2009). In an attempt to tackle this, some organizations have established so called centers of excellence (“CoE”) in order to support knowledge distribution in departments throughout multinational corporations (Adenfelt & Lagerström, 2006).

(12)

3 In the context of Machine Learning, some organizations approach this [knowledge distribution] through investing in training programs for employees. Alternatively they can also choose to hire consultants to execute the technology roll-outs on their behalf (Gartner, 2019). Both options can easily balloon expenses, as consultants only provide a high cost, temporary access to the technical expertise.

To implement innovative technologies into product development processes, organizations need to find ways to make them accessible for employees. As described by Aden- felt and Lagerström (2006), companies already have established centers of excellence to support subsidiaries. Therefore, one option is to also use these as a channel to dis- tribute technological expertise. For example: the application of machine learning. How- ever, as mentioned earlier, machine learning is a complicated topic and therefore application of this technology can be a challenge.

1.2 Purpose

This research aims to evaluate a use-case of E-learning methods in multinational corporations. The purpose is to study if e-learning approaches are viable for laying a foundation for making machine learning more accessible in integrated product development processes.

1.3 Research questions

1. How can remote e-learning sessions support the integration of a new technology, specifically machine learning, into the product development process?

2. How do the results from (1) compare to the same training resources administered in person (and not via digital remote sessions)?

1.4 Scope and limitations

It is not this research’s intention to acquire new technical insights into the field of machine learning. The focus is that current machine learning algorithms are to be made accessible to various subsidiaries by increasing the effectiveness of support structures, rather than simplifying the tools and algorithms with technical changes or high-level user interfaces.

(13)

4

2 Theoretical Framework

Here, a theoretical framework that supports the study is provided. This framework focuses on machine learning/artificial neural networks, knowledge sharing, and the product development process in multinational organizations.

2.1 Neural Networks and Machine Learning

This section provides a very brief description of a common machine learning method:

neural networks. It also covers some of their applications and relevant limitations.

2.1.1 General information

An artificial neural network is a mathematical model often referred to under a large variety of names, including “AI”, “NN”, “neural nets”, “machine learning algorithm”

(Hastie, et al., 2008). While there has been quite a bit of hype and misinformation re- garding neural networks, they are just nonlinear statistical models that can approximate many, albeit not all, mathematical functions when presented with enough diverse cases of the functions inputs and outputs. There is a wide variety of neural network

“types” that have been developed over the years to adapt to various problem types such as regression, classification, and time-series prediction among many others (Hastie, et al., 2008).

2.1.2 Real world applications

Some applications of neural networks, while feasible in theory, have been shown difficult to execute in practice (Beede, et al., 2020). Examples of this are seen in healthcare applications, where the technology promises to optimize clinician workflows and provide better patient outcomes. However, while machine learning models have show- cased excellent results on test-data, when applied in real clinical workflows the model’s performance did not live up to expectations. In the research cited above, the team of Google researchers designed a neural network to diagnose diabetic retinopathy, but the solution was described by the clinics nurses as unreliable, frustrating, and even detrimental to the patients’ experience. One of the nurses is even quoted in the paper saying:

“I’ll do two tries. The patients can’t take more than that.” – Nurse commenting on the per- formance of the Google AI implementation (Beede, et al., 2020).

Still, analysts predict that the range of real world applications is predicted to grow by orders of magnitudes in the near future (Gartner, 2019). This is largely due to almost all companies (98%) identifying machine learning and AI as important, whereas only around 40% of businesses state that they currently make use of such technology (Intel, 2020).

Less complicated applications of neural networks can however already be found working in production environments today, such use-cases commonly include optical character recognition for invoice and letter scanning, email spam detection, and financial

(14)

5 analysis of companies (Hastie, et al., 2008). Such applications help automate previously manual processes and reduce processing errors.

2.1.3 Some limitations of neural networks

What makes neural networks of interest is their ability to approximate solutions to abstract problems such as image recognition (Szegedy, et al., 2014). However, it is difficult to evaluate the exact workings of neural networks, as they sometimes behave in unexpected ways. An example given by Szegedy et al. (2014) is when giving a network an adversarial example. Adversarial examples are input data specifically chosen to maxim- ize the error of the neural network’s prediction. Oftentimes such adversarial inputs are, to a human, indistinguishable from inputs that the network can label correctly, indicat- ing that neural networks do not learn a higher-level understanding of a concept. Figure 1 shows an example of this using images. In the left column, three images that the pub- licly available AlexNet neural network manages to classify correctly. Once the changes shown in the middle column are applied, the image is (to a human) still perfectly recog- nizable, however AlexNet classifies all images in the right column as ostriches (the bird!). Similar results have been replicated with other neural networks.

2.2 Knowledge sharing in multinational corporations

This part of the theoretical framework introduces knowledge sharing in multinational corporations and how it is linked to the product development process.

2.2.1 Knowledge sharing

The importance of knowledge sharing in organizations is widely recognized because the competitiveness of a business is dependent on knowledge (Arkam, 2019).

Knowledge sharing is linked to innovative processes and a core practice of knowledge

Figure 1: Images in the left column are cor- rectly classified, right column images are all pictures of Ostriches as far as the AI is con- cerned (Szegedy, et al., 2014)

(15)

6 management. Arkam (2019) states that it directly affects the level of initiative from employees, and innovation in their product development processes. Knowledge adds value that is hard to quantify but easy to perceive, therefore it is imperative that companies utilize the expertise that employees possess to the highest level.

In multinational corporations the different branches develop their own skillsets depending on their workflows and environments (Adenfelt & Lagerström, 2006). The difference in skillsets is explained by branches having different focuses and purpose. How- ever, when it is possible to share knowledge between subsidiaries of a company the employees can acquire skills and apply the knowledge they now possess in new and creative ways. Knowledge that has been utilized by a subsidiary in one way can be applied in another by a different subsidiary. For this to be possible corporations need to integrate knowledge sharing in their development processes (Graeme, et al., 2003). Ex- amples of such integrations is the usage of routine descriptions, seminars, field trips and education. According to Graeme et. al, e-learning in combination with knowledge management can positively impact inquiry and competence.

2.2.2 Center of excellence (“CoE”)

Multinational organizations create centers of excellence to help with knowledge sharing (Adenfelt & Lagerström, 2006). A Center of excellence is a subsidiary, or part of one, that shares knowledge between a company’s branches and other subsidiaries. Accord- ing to Adenfelt and Lagerström (2006) such a center of excellence can share knowledge through different channels and using a variety of methods, depending on the nature of what is being communicated.

2.2.3 Knowledge management and the product development process

As mentioned previously, knowledge sharing adds value to multiple stakeholders within an organization, but the transfer of knowledge between subsidiaries can be an obstacle in the product development process (Guerra, et al., 2009). According to Evers- heim & Rozenfeld: “Product development is essentially a knowledge creation process.

Thus, a factor of success for the leverage of this process is based on knowledge manage- ment.” (Eversheim & Rozenfeld, 2002).

Figure 2: Knowledge management and organizational performance (Singh, et al., 2019)

(16)

7 Furthermore, to enhance development efficiency, companies are required to exercise knowledge sharing in each stage of the product development process since these processes require extensive knowledge and expertise from both managers and employees (Chen, 2010). An organization needs to analyze each step of their product development process and the knowledge that is used in each task to develop their processes.

Knowledge sharing practices, as well as company culture, directly impact the level of open innovation (Singh, et al., 2019). Open innovation, in this case, is when an organization promotes external and internal knowledge sharing practices that drive process and product innovation and thereby generate revenue. Moreover, open innovation is a key factor in organizational success and directly influences performance as presented in figure 2.

There are a variety of tools which organizations can implement to support the knowledge sharing process, for example: standardized documents and libraries (Eversheim & Rozenfeld, 2002). The purpose of knowledge management is to manage a system that records and structures organizational knowledge whilst also enabling a milieu that supports knowledge sharing.

In order to enhance development efficiency companies can exercise knowledge sharing in their product development process, where expertise from both managers and employees is required (Chen, 2010). For optimal results, an organization needs to analyze every step of their product development process and the knowledge that is used for every task within each step.

2.2.4 Remote learning and E-learning

Often referred to as web-based learning or remote learning, E-learning is utilizing com- municative technologies to convey knowledge across large distances (Henneke &

Matthee, 2012). Examples of such technologies are Skype, VR-platforms, chat rooms, and message boards (DeRouin, et al., 2005).

There is an ongoing discussion in scientific literature regarding the effectiveness of e- learning (Njenga & Fourie, 2010). Critics point to e-learning initiatives that in the past have overpromised their capabilities and claim that the technology is overhyped. Pro- ponents on the other hand state that deploying e-learning in an organization can lower costs and create opportunities for employees, and that it makes it easier for large organizations to provide a more consistent learning experience as e-learning does not re- strict the student to a place and time (Montebello, 2017).

2.2.5 Trainings, employee retention, and competitiveness

The effectiveness of trainings is limited by the amount of turnover that an organization experiences, mainly because the trained employee must stay long enough after com- pleting the training for the investment to provide a positive return for the employer (Colarelli & Montei, 1996). However, it has been found that increased training quantity correlates negatively with employee turnover and training quality correlates positively.

(17)

8 Furthermore, Hayes and Stuart (1996) found employee commitment to be stronger when the employee’s managers actively expressed themselves in support of employee trainings.

Competition is often said to provide a company with the motivation to invest extra resources in employee skills to gain an edge on their competitors (Colarelli & Montei, 1996). Yet, Colarelli and Montei’s research shows no correlation between the competitiveness of a company’s operating environment and their quantity and quality of trainings. This is attributed to technological complexity, meaning that companies which operate in more technologically complex environments invest more in their employee training programs, whereas other companies may choose to invest differently.

2.3 Business continuity and Covid-19

It is important for organizations to have well thought-out contingency plans in case of emergencies. Business continuity is a critical success factor if an organization wants to survive during, for example, a pandemic (Wong, 2019). Multinational organizations with subsidiaries that operate in different industries feel the importance of a strong business contingency plan to reinforce the corporate capability, as well as secure their competitiveness. For example: due to restrictions because of the coronavirus pandemic, employees are unable to travel to work (Dingel & Neiman, 2020). It is also important for businesses to acknowledge that an individual’s productivity may differ significantly when working from home, as not all jobs are equally well suited for remote working.

(18)

9

3 Method

In this section the approach, and the course of action of the study are detailed.

3.1 Corporations and Locations

As part of this research, two subsidiaries in different regions belonging to the same multinational corporation were studied. Both companies operate in the automotive financial services sector and specialize in offering loan, lease, and insurance contracts to cus- tomers and businesses. In person interviews and workshop sessions were conducted in the Stockholm region (Sweden) and the remote team joined in via telecommunication from Haryana, India. These companies where chosen as automotive companies and their subsidiaries belong to the largest multinational companies and thus have the most to gain from increasing the quality and scalability of training sessions.

3.2 Training Sessions

To educate and introduce employees to the topic of machine learning, a set of training sessions covering the subject was designed in collaboration with one of the subsidiaries. The subjects that participated in these sessions were not expected to have any prior knowledge in the areas covered. Content was limited to simple, but useful applications that can be run on systems with limited computing power to assert the accessibility of the training sessions. Five lessons, each approximately one hour in length were conducted with a remote and a local team. These lessons were planned and administered by the authors and covered the basic principles of coding and training a simple machine learning classifier for optical character recognition. That means the lessons were loosely based on what one would find in a beginner’s tutorial on TensorFlow or in a first course in deep learning at college. Table 1 details the content that was covered in each of the sessions.

Table 1: Training Sessions

Session # Content

1 Installing and using Python and the required machine learning libraries. Execution of example scripts and proof of concept.

2 Constructing datasets, how to spot good data, realistic expectations of AI

3 Tensorflow/ Keras – training a neural network (1/2) 4 Tensorflow/ Keras – training a neural network (2/2) 5 Further Work with AI / Machine Learning

(19)

10 The content of the onsite training sessions was identical to the remote session to optimize the accuracy of the comparison between the remote lessons and the local ones.

Any digital material that was created for the remote group was also shared with the participants of the onsite training sessions.

3.3 Course of action

At first a set of research papers were studied, thereby making it easier to approach the subject. The study was then applied as a basis for the introduction, purpose and research questions. When the thesis and research questions were set, a theoretical study went underway to create a framework based on the current state of scientific research.

Company employees were asked to participate in training sessions at this stage. These sessions were held both remotely and locally. Employees that had participated, as well as some of their project leaders and managers were then asked to be interviewed. The interviews were conducted remotely and in person based on the interview guide, as seen in appendix 1. The result of which was then coded and analyzed.

3.4 Data Collection

Questions regarding the effectiveness of the training content were mostly standardized, with only minor differences between the versions used for the remote participants. It was assumed that remote interviews need more direct questions, as the interviewer’s ability to read facial expressions and body language are limited during a digital interview. The manager and project leader interview guide had a different main body of questions that was more closely tied to their everyday working tasks.

3.5 Interviewees

The choice of respondents for interviews was detailed in Table 2 below. Remote and onsite participants were interviewed to collect insights and opinions from the group that was meant to gain skills and experience from trainings. Additionally, a project leader and a manager were interviewed to adjust for possible conflicts of interest that may exist between employees and management. By independently interviewing both, the likelihood of finding such differences was maximized.

(20)

11

Table 2: Respondents

Professional title Type of interview Avg. Time

Project leader (1) In person 30 minutes

Operations Manager (1) Skype 25 minutes

Participants of remote training sessions (2) Skype 30 minutes Participants of onsite training sessions (2) Skype 20 minutes

The questions asked were categorized into three different classes: A, B and C (as seen in Appendix 1).

Class A referred to general questions about the topic that all interviewees were asked to answer. This established a common foundation between all parties and allows for easier comparisons between the different classes. Category B was only concerned with questions for session participants. There are two slight variations present, Bv1 and Bv2, which were meant for the remote team and the local team, respectively. The last cate- gory, C, was centered around questions for project leaders and managers.

3.6 Choice of organization and participants

Since the authors of this study participated in creating the sessions for the organizations a unique opportunity arose where it was possible to research and evaluate the participants in action. This allowed for a multifaceted view on the trainings, as well as extra insight into the individual participants responses.

3.6.1 Bv1: Participants of onsite training sessions

The group that participated in the onsite training sessions were employees located in the Swedish branch of the organization located in the Stockholm area, Sweden. Inter- viewing the participants of the onsite training sessions was a given as their experience of the training sessions was valuable to this research, especially in comparison with the rest of the interviewees.

3.6.2 Bv2: Participants of remote training sessions

The group that participated in the remote training sessions was based in Haryana, India.

Their interview responses could be contrasted with the feedback from the Bv1 group to find support for answers to the research questions.

(21)

12 3.6.3 C: Project leader

The project leader that participated in this study oversaw business development projects, process development projects and quality management. Getting the perspective of a project leader that has worked in an international setting for a long time, and who also has seen other projects similar to this study, was helpful and gave important insights especially in contrast to the results from the interviews with the participants of the sessions.

3.6.4 C: Manager

An operations manager was also asked to be interviewed. This added a managerial perspective on the training sessions.

3.7 Data Evaluation

The data was mostly evaluated on a qualitative basis, to draw high level, indicative conclusions, and interpretations. The recordings collected were processed making them easier to compare to each other. Then they were then coded to uncover recurring themes. When the results were compiled, a complete analysis and discussion was built upon the foundation provided by the theoretical framework. Finally, with all pieces in place, conclusions were drawn, and possible further developments suggested.

3.7.1 Quality of the Evaluation Process

Due to the, at the time of writing, ongoing societal situation with the spread of the coronavirus, participants that were originally supposed to participate in the onsite training sessions could only attend three out of five sessions. In contrast, participants of the remote sessions managed to complete all five of them. This may have had an unintended effect on both general responses, as well as the overall outlook on e-, and remote- learn- ing that employees of a company expressed when interviewed.

Moreover, the responses were subject to cultural differences between Sweden and In- dia. The described method did not adjust for such differences as more than two regions would have had to been included in the study.

(22)

13

4 Results

This section covers the collected data that is relevant to the purpose of the research. It is split along the different themes that were uncovered. To further facilitate the reada- bility of this chapter, it will refer to the different respondents by their category when necessary. These are as presented below:

• C: Manager and project leader

• Bv1: employees that participated in the remote training sessions

• Bv2: employees that participated in the onsite training sessions

4.1 Training Sessions

Overall, according to C-respondents, the organization has an aspiration to automate large portions of their in-line functions. To do this a strongly integrated product development process is required. People in managerial roles, both the project leader and manager, were therefore already very interested in possible future solutions and generally answered positively to questions regarding the outlook of machine learning, and the training sessions. As for how sessions like these could support the product development process, all the participant’s’ answers also pointed towards a positive outlook.

However, responses were not as clearly united with regards to how much effort it would take. All remote participants and the operations manager, saw the e-learning sessions as something that could relatively easily be scaled up with the help of a center of excellence. The onsite participants, and the project leader, suggested that a large rollout of such sessions would require very large investments. Still, all respondents agreed that such a setup would provide a tangible benefit to the product development process. In further detail, respondents believed that the training sessions were a great tool for the organization to expand their knowledge base.

“I realize that I am becoming more passionate about this subject and right now; all knowledge that I can obtain is valuable for me as an employee.” – A participant of the

onsite training sessions

Participants of these sessions, both remote and local, also expressed that they felt val- ued by the organization since, according to their responses, the training sessions are an investment in their skills and their development.

4.1.1 Onsite training sessions

Both subjects that participated in the onsite sessions believed that their sessions were more effective onsite than if they had been remote. The reason that was given was that it is easier to ask questions and engage with both the teacher and the other students in a real physical environment. Furthermore, when participants in workshops are physically present, they expressed being able to interact more effectively as body language

(23)

14 and eye contact enhanced communication. Both Bv2-participant noted that it was easier to engage socially, and that they really enjoyed the training sessions since they became a social platform within the workplace.

4.1.2 Remote training sessions

All interviewees of Bv1 agreed that e-learning sessions are a great tool for the organization to share knowledge between subsidiaries that are not geographically or organi- zationally close. Like the onsite sessions, these were referred to as a social platform because employees from different parts of the organization, and the world, could engage with each other and learn. When the Bv1 and Bv2 answers were compared it seemed as though Bv1 [e-learning session] did struggle a bit more with communication.

For example, the Bv1 group felt it was more difficult to detect body language, and the conversations were perceived as less natural. Nevertheless, the overall responses re- mained positive and they did not believe that they would have had a significantly easier time grasping the content of the sessions had they been onsite. According to the authors, the level of subject matter content understood did not differ noticeable between the Bv1 and Bv2 participants.

Both the project leader and manager expressed that the remote training sessions and seminars are inevitable for a multinational corporation in their future knowledge sharing process. The main reason given for this was that they are easier to scale up interna- tionally without excessive use of resources. They were also attributed with the ability to increase cooperation within the organization’s different subsidiaries and as a great way for employees to develop their professional network within the corporate group.

4.2 Machine learning

The Bv1 and Bv2 respondents expressed a change in their perspective on machine learning and neural networks. The training sessions had, in their opinion, made the technology more accessible to them. They felt that they had formed an understanding of basic machine learning algorithms, and the type of problems that can be solved with them. Some (1 local, 1 remote) even felt confident enough to try and investigate possible uses in their everyday tasks.

Both the C-respondents were quick to point out that they thought there exist a lot of processes which were suitable for machine learning, however no detailed examples of potential processes could be provided. When asked to elaborate, C-respondents expressed that repetitive tasks in particular posed a good starting point for machine learning based automation.

“It is about making time and working on improving the quality and expand the business through automation instead of putting unnecessary man hours on tasks that are primi-

tive and repetitive.” – Project leader

(24)

15 4.3 E-learning and working from home

Because most interviews were being held when the local respondents had been issued instructions to work from home¹, some interesting insights with regards to remote learning and working from home were made. All respondents stated that it felt very strange and isolated to work completely remotely, even if they engaged in virtual team meetings every day. Generally, all interviewees indicated that they expected to become used to this condition, and that the technical aspects of the shift to remote working had functioned surprisingly well. However, they also implied that prolonged remote working could become detrimental to departments’ team spirits in the long run. When asked specifically about how this impacted their thoughts about widespread e-learning some indications were given by the operations manager, that perhaps a mixture of remote and onsite learning would be ideal. In doing so one could realize some of the efficiency gains from e-learning, while not completely abandoning the team building aspects of in-person workshops.

4.4 Product development process

All responses expressed how these types of training sessions were value adding to the product development process, especially when applied in a very large organization.

Generally, all interviewees also responded positively to the statement that such training sessions, both remote and in person, can be applied to other subjects such as Excel- or computer programming. One Bv2 respondent highlighted that the organization had of- fered other educations for their employees in a similar fashion to the training sessions, and that educating employees is a way for the company to add value to all processes.

Ultimately, respondents unanimously expressed that they see a future where they have integrated machine learning in the product development process. Especially processes where there is a large quantity of data that needs to be analyzed. One Bv1 respondent was keen to several times point out that digitalized educational tools are a key-factor for integrating new technologies and educating employees. The type C respondents stated that for an organization to easily integrate and apply new technologies in their development processes they need to create an environment that is flexible and allows such changes.

4.5 Competitive Advantages

Although there were some differences in opinions with regards to how one should pro- ceed with e-learning and/or training sessions in the organization, there was a clear consensus among all respondents that tools like these are key for securing the competitiveness of an organization. Managers expressed that for the company to move forward and compete they need to adapt and not only integrate innovative technologies in their de-

1 Due to the spread of the coronavirus (covid-19)

(25)

16 velopment processes, but also other functions. They believed that other companies already deploy similar tools for their employees, and that any company needs to conform if they want to be able to continue to deliver results that on par with their competition.

Between Bv1 and Bv2 responses the consensus was that for them to develop new skills and as keep up with current trends they are dependent on obtaining knowledge through trainings of any kind (remote, onsite, mixed).

4.6 Participants visions of the future

As previously mentioned, most respondents in all categories had non-concrete ideas for what type of business processes that could possibly be automated with the help of machine learning, specifically some areas that were mentioned included contract manage- ment and document processing. When asked to elaborate on how machine learning would be integrated into such processes, the C-responses were that this had to be subject of a feasibility study. They also pointed out that other skills can be taught through training sessions like the ones in this study. Specifically, they mentioned excel-mac- ros/VBA² and traditional computer programming.

2 Visual Basic for Applications, a programming language used for building more advanced formulas and calculations in Microsoft Office apps.

(26)

17

5 Analysis and discussion

The following section presents thoughts and arguments surrounding topics that were especially noteworthy, as well as the answers to the research questions.

5.1 Perception of machine learning

One can observe that there exists a very positive sentiment around machine learning in the industry (Gartner, 2019). However, while Gartner claims that a the biggest bottle- neck is the lack of talented workforce, this does not explain why even high profile projects from industry leaders such as Google have problems moving complicated machine learning algorithms from testing to production (Beede, et al., 2020).

From this we deduce that perhaps the issue at hand is not the lack of engineering talent, but a fundamental problem with misinformation surrounding AI, machine learning, and their capabilities. Meaning that clients perhaps request overly ambitious projects, while failing to identify the processes where machine learning can realistically work (and have a large impact). Something that we claim can be addressed, not by training more engineers, but by raising the general skill level among all employees who come in touch with machine learning and AI based solutions. For companies with tens of thousands of employees this is a significant investment, which is why we believe that e-learning can make a difference.

We also find support for this argument in our own study as the manager and project leader were quick to point out that they already knew possible applications for machine learning. They did not, however, point to any current or planned implementations of the technology. This indicates that they perhaps are falling into a common perception problem described by Hastie, Tibshirani, and Friedman (2008) where one sees machine learning as a magical and mysterious jack of all traits, rather than just the nonlinear statistical models that they are. Furthermore, we think that such issues, if left un- addressed, open the doors to overly ambitious projects that end up underperforming as was the case with Googles trials into machine learning powered medical diagnoses (Beede, et al., 2020).

5.2 Machine learning in product development

One of the central focus points of this research is the sharing of knowledge across an international corporation. In existing research (Arkam, 2019), it is widely realized that such knowledge sharing activities are of great importance for a company’s success. Gen- erally when this topic is discussed it is assumed that the “knowledge” that one wants to share is well defined, examples often include routine-descriptions, seminars and document libraries (Eversheim & Rozenfeld, 2002; Graeme, et al., 2003). What really stood out to us about this way of looking at knowledge sharing is that, in our opinion, you cannot apply them for machine learning in the same way. We argue that this stems from technical properties of neural networks which are still not fully understood, as described by Szegedy et. al, (2014). Their paper discusses the counter-intuitive properties of neural networks, specifically how a trained network can be fooled with what they

(27)

18 name adversarial examples – data that is misclassified by the neural network without apparent reason. When looking at this from a product development perspective, we believe that not being able to explain how or why a software arrives at a solution is a major challenge. Especially since literature recognizes standardized documentation and solution libraries as an important aspect of knowledge sharing (Eversheim & Rozenfeld, 2002).

Furthermore, because (as described above) neural networks sometimes misbehave in unexpected ways, we believe that it is difficult to apply them in product development without sufficient technical knowledge. As mentioned earlier, contrary to other products or software solutions, engineers cannot just handover a detailed description of how the network figures out its answers because they will not know it either. For an efficient product development process to take place we argue that engineers, project workers, as well as project managers need to be able to speak on the same technical level to avoid both misinterpretation of the technologies abilities and miscommunica- tion between different parts of the project. This was also expressed by both the project leader and the manager in their interview responses. Moreover, they [project leader, manager] also claimed that to integrate machine learning into product development successfully, an organization needs to foster an environment where employees are in- centivized to propose process improvements, provided they have sufficient knowledge surrounding the process. This is also supported in literature; Singh, Gupta, Busso, and Kambojd (2019) state that when top management support knowledge sharing and knowledge management practices it creates an environment where employees ex- change ideas (across and inside functions) and nurtures employee commitment. This in turn enhances open innovation and boosts employee performance.

Published research states that organizations need to analyze knowledge used in every task to construct efficient product development processes (Chen, 2010). We see it as reasonable to involve employees in such process analysis when dealing with machine learning powered automations. This because, employees should be most familiar with the processes that are to be automated and can therefore maximize the technologies benefit, provided they have sufficient knowledge about both the process and the technology.

Looking at the responses from interviews with the training session participants, both remote and local, we can further ground our claims in examples where participants responded with confidence boosts for applying and working with machine learning. It is noteworthy that both the local and the remote group independently commented on the above described confidence boost. We think that perhaps this is an implication that they would have an easier time driving product development processes featuring neural networks than employees without any prior subject matter knowledge. With this in mind, and recalling both the issue of neural networks behaving unpredictably in production, as well as the importance of documentation; we also think that if project managers (and project workers) have a stronger theoretical background of neural networks then perhaps the lack of explicit documentation can be remedied with more abstract descriptions of how a specific neural network was designed and trained.

(28)

19 5.3 Training sessions and e-Learning

It can be argued that a lot of the work that we put into the creation and execution of the machine learning workshops would, if integrated into a company’s everyday business, fall under the responsibility of a center of excellence for either machine learning or for knowledge sharing. Adenfelt and Lagerström (2006) define a center of excellence as a company subsidiary that shares lessons learned with other subsidiaries from within the same company, a definition which can be stretched to also apply to the method of this study. Many businesses have already realized the benefits of using such centers of excellence, namely, to standardize product and process quality across large regions as well as to share best-practices among subsidiaries (Guerra, et al., 2009). As such, it seems reasonable to us that one would group training sessions under their everyday tasks. Mainly due to the availability of a wide variety of e-learning methods, something that was left out of the scope of this study. As mentioned previously, our research ex- clusively focused on remote vs onsite workshops. These are generally present in literature, albeit defined as just a subcategory of e-learning together with web-based applications, message boards or even chat rooms (DeRouin, et al., 2005). One prominent benefit of these alternative methods is customization, which implies that learners are able to exert a greater amount of control over their learning by not being restricted to a place and time, and can therefore get an even better understanding of the subject (Montebello, 2017). We acknowledge this and think that such e-learning methods would have added another dimension to this research. But as their absence does not lessen the strong positive interview responses regarding e-learning, they should not be considered a shortcoming of this study.

5.4 Remote vs. onsite training sessions

A trend that was encountered during the interviews was how the expectations of the participants affected their belief in how effective the various training sessions were.

The participants who received the training in person with the instructor present onsite were more likely to answer negatively about the prospect of shifting the entire process online. The remote-only participants on the other hand did not respond in a way that indicated that they would have gained a lot of learning-efficiency by receiving the training in person. We think that this points to a certain bias among employees that remote sessions must be inferior to in-person trainings, which is then removed when participating in a remote session.

Interestingly, we find that some literature confirms the onsite participants view on this topic: in an article published in the British Journal of Education Technology, Njenga and Fourie (2010) make a series of statements declaring many of the perceived benefits of e-learning (such as cost, reachability, and effectiveness) to be nothing but myths. One of their statements specifically concerns itself with the need for human interaction.

Njenga et al. (2010) state that this need cannot be fully satisfied by e-learning methods and therefore puts a fundamental limit on the scalability of e-learning.

We believe that there is truth to both sides here. While difficult to quantify it is (from the author’s and session organizers’ perspective) much easier to “read the room” and

(29)

20 figure out if the content is well understood when one is physically present. E-learning methods seem to have potential, but one should be careful to not overestimate their usefulness and only apply them when they are well suited. However, it should also be noted here that prior to the coronavirus pandemic the onsite participants had the same skeptical views on working from home as they indicated for remote learning sessions.

Because the interviews where held during the outbreak of the coronavirus, we were in many cases faced with employees who seemed very surprised about how well full-time remote working was functioning. This leads us to believe that maybe much of the criti- cism which e-learning faces stems from a similar distrust of the technology and that the generally positive outlook for using e-learning indicated by the remote session group from this study is both valid and noteworthy. However, research by Dingel and Neiman (2020) points to the effectiveness of remote working differing significantly depending on the type of job that the employee in question is performing. This could imply that the participants of our study experienced the shift to remote working in the way that they did, not because their preconceptions about remote work were unjustified, but because they work in positions which are well suited for remote working.

5.5 The management perspective

Trainings of all sorts, and their effect on employee retention is still a poorly understood area of research, with training quantity increasing employee retention, while higher quality trainings seem to increase turnover (Colarelli & Montei, 1996). Contrary to this, the responses gathered in our study were one-sided and both the manager and the project leader indicated that trainings were positive and that they were not worried about employee turnover because of more technical trainings being available to employees.

From our other results we gather that employees felt appreciated by their company since they viewed the training sessions as an investment in them and their development. Interestingly, the very fact that management is supportive of training efforts might be a key success factor of said trainings, as indicated by Hayes and Stuart (1996) who found employee commitment to be stronger when managers actively expressed themselves in support of employee trainings.

According to the manager’s and project leader’s responses in this study, another major factor of trainings is company competitiveness. We notice that the session participants also indicated in their responses that they perceive trainings as very valuable for securing the companies competitiveness. This is particularly interesting as Colarelli and Montei’s research opposes this as they find no correlation between the competitiveness of a company’s environment and training quantity or quality (Colarelli & Montei, 1996).

While their research does not specifically disprove trainings to influence competitiveness it at least indicates that this is was not the accepted industry consensus at the time that Colarelli and Montei published their article. Note that this was before some major technological milestones, such as widespread smartphone adoption, and as such might influence their results, were they to repeat the same study today.

(30)

21 5.6 Answers to the research questions

Looking back at the research questions, the answer to (1) seems to be pointing towards there being great potential for e-learning in a corporation. Continuing off the arguments presented in section 5.2 we believe that both employees and managers need some level of basic understanding of the technologies that are being integrated in the product development process. E-learning could be a viable option for this. In practice we think that these e-learning measures would then be administered by a center of excellence, as discussed in section 5.3. Large organizations, which already have subsidiaries that have accrued expertise in machine learning, could appoint one of these subsidiaries with the task to standardize subject matter knowledge across the whole organization. While Adenfelt and Lagerström do not reference machine learning in particular, the above approach would be very similar to their general definition of a center of excellence (Adenfelt & Lagerström, 2006).

This leads into question (2), comparing the effectiveness of remote and local workshops. Here it cannot be left unsaid that employees who had a chance to experience the training in person found them to be akin of a team building activity. As mentioned in the discussion, while the remote and local sessions seemed comparable with regards to the knowledge passed onto the participants, the long-term benefits of personal local sessions may be significant. Especially since literature indicates a negative correlation between physical employee trainings and employee turnover (Colarelli & Montei, 1996). How e-learning compares to this is still an open question, some critics of e-learning argue that the need for human interaction in learning is difficult to replace (Njenga

& Fourie, 2010). The results from our own study indicate that e-learning for new tech- nologies, such as machine learning, is not only feasible, but also beneficial to a company’s product development processes, mainly because e-learning seems to support the scalability of large training initiatives.

(31)

22

6. Conclusions

In the following section, our conclusions, as well as possibilities for further research are presented.

6.1 General conclusions

We identify the following limitations that possibly hold back widespread adoption of machine learning:

1. Technological limitation of current machine learning algorithms which lead to overly ambitious projects underperforming, even when managed by industry leaders.

2. Insufficient subject matter knowledge among employees, making the handover of solutions from engineering departments to business users difficult.

3. Misconceptions about the potential of machine learning that leads to unrealistic management expectations, and the wrong use cases being prioritized.

Therefore, we conclude that there is a need for making machine learning knowledge more accessible to the people who are meant to work alongside such algorithms, as well as those working with developing products and processes that make use of machine learning. While not being able to provide definite answers, our research seems to indicate that at least point (2) and (3) can be addressed with large scale training initiatives.

Due to the scale of such training efforts, e-learning can be a very valuable tool to a company that intends to follow through with such trainings. Thus, e-learning is a viable option for helping to make machine learning more accessible to employees working with product development.

6.2 Further research

While there is great potential for e-learning as a tool to share knowledge in a multina- tion corporation, further research needs to be conducted on the effectiveness of different e-learning methods. For instance, we would be interested in the effect that learner self-pacing has on the time and effort needed to understand various concepts, as well as the quality of the retained knowledge.

We also believe that further research into the psychological effects of remote working and learning should be conducted. While our study’s participant’s performance seemed unimpacted by remote working vs onsite, all participants of the onsite session praised the social aspect of learning and working together in-person. Especially considering the coronavirus pandemic we would like to see the emotional impact of long-term remote working analyzed, as we caution businesses and academics from purely looking at performance metrics when evaluating how successful the shift to remote working was.

Managers, directors, and researchers must not forget the humans behind such metrics.

(32)

23 Furthermore, we make the claim that e-learning sessions such as the ones in this study would be best suited as the responsibility of a center of excellence. It would be highly interesting to see further research into what a professional center of excellence can do when actively pursuing a large-scale e-learning initiative in a multinational corporation.

6.3 Contribution to research/industry

This study indicates how a multinational organization might gain a competitive edge by integrating innovative technologies in their product development processes. Further- more, it gives clear indications as to where more research is needed, both regarding machine learning in product development as well as remote learning in general.

With the spread of the coronavirus, organizations and institutions are in demand of quick solutions for working and learning from home. This study analyses the impact of one such solution and discusses the difference between remote and onsite knowledge sharing in an organization. Finally, it also cautions about the still unknown social and psychological implications that large scale remote learning and working may have on the individual.

(33)

24

References

Adenfelt, M. & Lagerström, K., 2006. Knowledge development and sharing in multinational corporations: the case of a center of excellence and a transnational team.

International Business Review, Band 15, pp. 381-400.

Arkam, 2019. The impact of organizational justice on employee innovation work behavior: Mediating role of knowledge sharing. Journal of Innovation & Knowledge, pp.

1611-1613.

Beede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P. &

Verdoulakis L. M., 2020. A Human-Centered Evaluation of Deep Learning System Deployed in Clinicts for the Detection of Diabetic Retinopathy. Honolulu, CHI 2020 Paper.

Chen, Y.-J., 2010. Knowledge integration and sharing for collaborative molding product.

Computers in industry, 61(7), pp. 659-675.

Colarelli, S. M. & Montei, M. S., 1996. Some Contextual Influences on Training Utilization.

Journal of Applied Behavioral Science, 32(2).

DeRouin, R. E., Fritzsche, B. A. & Salas, E., 2005. E-Learning in Organizations. Journal of Management, Band 31.

Dingel, J. I. & Neiman, B., 2020. How Many Jobs Can be Done at Home?, Cambridge, MA 02138: University of Chicago.

Eversheim, W. & Rozenfeld, H., 2002. An Architechture for Shared Management of explicit Knowledge Applied to Product Development Porcesses. CIRP Annals, Band 51, pp. 413-416.

Gartner, 2019. [Online] Available at: https://www.gartner.com/en/newsroom/press- releases/2019-01-21-gartner-survey-shows-37-percent-of-organizations-have

[Accessed on 25 01 2020].

Graeme, M., Massy, J. & Clarke, T., 2003. When absorptive capacity meets institutions and (e)learners: adopting, diffusing and exploiting e-learning i organizations.

International Journal of training and developement, pp. 228-244.

Guerra, P., Lugli, V., Parra, F. & Mario, A., 2009. How to improve the knowledge sharing within a MNC: The case of PROAC GROUP. Uppsala University, department of business studies.

Hastie, T., Tibshirani, R. & Friedman, J., 2008. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Second Hrsg. Stanford: Springer.

Hayes, J. & Stuart, M., 1996. Does Training Matter? Employee Experiences and Attitudes.

Human Resource Management Journal, 6(3).

(34)

25 Henneke, M. & Matthee, M., 2012. The adoption of e-learning i Corporate Training Environments: An active theory based overview. Proceedings of the South African Institute for Computer Scientists and Information Technologists Conference, pp. 178-187.

Hong, P., Doll, W. J., Revilla, E. & Nahm, A. Y., 2011. Knowledge sharing and strategic fit in integrated product development projects: An empirical study. International Journal of Production Economics, 132(2), pp. 186-196.

IBM, 2020. Data Science and Machine Learning. [Online] Available at:

https://www.ibm.com/analytics/machine-learning [Accessed on 18 02 2020].

Intel, 2020. Machine Learning | Automate and Optimize Decision Making. [Online]

Available at: https://www.intel.com/content/www/us/en/analytics/machine- learning/overview.html

[Accessed on 18 02 2020].

Montebello, M., 2017. Next Generation E-learning. Tokyo Japan, s.n., pp. 150-154.

Nagy, K. & Szalai, J., 2011. Modeling and optimising alternatives suitable to advance the product development process. Periodica Polytechnica Mechanical Engineering, 55(2), pp. 111-115.

Njenga, J. K. & Fourie, L. C. H., 2010. The myths about e-learning in higher education.

British Journal of Education Technology , 41(2).

Singh, S. K., Gupta, S., Busso, D. & Kambojd, S., 2019. Top management knowledge value, knowledge sharing practices, open innovation and organizational performance. Journal of Business Research.

Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. & Fergus, R., 2014. Intriguing properties of neural networks. s.l., ICLR Conference submission.

TensorFlow, 2020. Why TensorFlow. [Online] Available at:

https://www.tensorflow.org/about [Accessed on 18 02 2020].

Wong, W. N. Z., 2019. Transforming corporate performance: A business continuity management approach. Organizational Dynamics, 48(2), pp. 29-36.

(35)

26

Appendix

Interview structure guide Legend

A: General Questions

B: Questions specifically for participants of the technical trainings v1 = remote

v2 = in person

C: Questions aimed at managers or employees with strategic stakes

Interview questions

A1 introduction

A1.1 giving some background and explaining the aim of the interview A1.2 Assert recording consent

A1.3 What is your current professional title?

A1.3 Do you have any prior experience with working across cultural borders? If so, please tell us a little bit about your experience.

A2 Machine learning

A2.1 Explain, in one or two sentences, what machine learning is to you.

A2.2 Did you have any experience with machine learning before this project? And if that is the case what kind of experience?

A2.3 Do you think that computers in the future will be able to fully automate entry-level jobs, if so, how long until then?

B3 Training sessions

B3.1 Why did you decide to participate?

B3.2 What did you hope to achieve by participating?

B3.3 As you became involved, did you discover other benefits/reasons for participating?

B3.4 In what way has the training sessions met your needs/expectations?

B3.5 In what way has the training sessions failed to meet your needs/expectations?

B3.6 If given the option would you participate in future training sessions?

B3.7 Do you feel confident enough to apply the training material on other problems? If that is the case which current workflows/processes would you automate?

(36)

27 B3.8 How did the training sessions alter your view of machine learning?

A3.9 After hearing the idea of remote e-Learning session, what was your initial reaction.

B3.10 Did you experience any challenges that kept you from participating at the level that you would have wanted to?

A4 Product development

C4.1 Would you be able to utilize these types of training sessions (in machine learning applications) in your product development process (or/and in other departments)?

C4.2 Do you think you would be able to teach other subjects (e.g. Excel) with these types of training sessions?

C4.3 How does your existing product development process work?

A4.4 What, according to you, is the most innovative technology that you are applying in your current (product development) process?

C4.5 What are the challenges of integrating innovative technologies into the product development process?

C4.6 Where do you see the most value added in integrating innovative technologies into the product development process?

C4.7 Which current workflows/processes do you think the company will be able to automate with these types of innovative technologies?

A5 Possible improvements

B5v1 Do you feel that the training would have been more effective if held onsite instead of remote?

B5v2 Do you feel that you would get similar results from the training if held remotely?

A6 Possible further developments

A6.1 Do you think e-Learning of technical and business skills is scalable to a worldwide approach?

A6.2 Are there any comments that you would like to share?

(37)

28

(38)

29

TRITA TRITA-ITM-EX 2020:80

www.kth.se

E-Learning as a tool to support the integration of machine learning in product development processes