How to Assess and Map Employees Competencies

(1)

Linköpings universitet

Linköping University | Department of Computer and Information Science

Master thesis, 30 ECTS | Datateknik

2019 | LIU-IDA/LITH-EX-A--19/081--SE

How to Assess and Map

Employees Competencies

Hur man bedömer och kartlägger anställdas kompetenser

Christoﬀer Johansson

Supervisor : Christer Bäckström Examiner : Peter Jonsson

(2)

Upphovsrätt

Detta dokument hålls tillgängligt på Internet - eller dess framtida ersättare - under 25 år från publicer-ingsdatum under förutsättning att inga extraordinära omständigheter uppstår.

Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka ko-pior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervis-ning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säker-heten och tillgängligsäker-heten ﬁnns lösningar av teknisk och administrativ art.

Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsman-nens litterära eller konstnärliga anseende eller egenart.

För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/.

Copyright

The publishers will keep this document online on the Internet - or its possible replacement - for a period of 25 years starting from the date of publication barring exceptional circumstances.

The online availability of the document implies permanent permission for anyone to read, to down-load, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility.

According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement.

For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: http://www.ep.liu.se/.

(3)

Abstract

The need to manage, map and develop a companies employees competencies is con-stantly increasing in today’s rapidly growing market. It is therefore incredibly important to have a broad understanding of what competencies one’s employees possess and which competencies the company potentially lacks. This thesis investigates how to assess, eval-uate and map employees competencies in the best possible manner, followed by the po-tential effects this can have on the individuals undergoing the assessment. Lastly the the-sis identifies how a competency management system could be implemented and which functionalities said system should contain. This study began with a systematic literature review, for me to get a better understanding of the competency concept and competency as-sessment. The literature study was followed by a questionnaire distributed to all Etteplans employees in Linköping and Norrköping, and lastly four interviews were conducted with employees at the office in Linköping. To get a clearer understanding of the employees thoughts regarding competency and the implementation of a new competency manage-ment system. One of the most important factors to take into consideration when applying a competency management procedure is that these implementations takes time and must be carefully planned and executed to generate the best possible results. All companies are unique and require their own competency management procedure to succeed.

(4)

Acknowledgments

First and foremost I would like to express my gratitude to Etteplan for giving me the chance to write my master thesis with them. I would also like to thank my external supervisor at Etteplan, Jacob Bergvall, for supporting me during this thesis and constantly providing me with valuable comments and feedback. Furthermore, I would like to thank my supervisor Christer Bäckström and my examiner Peter Jonsson at Linköpings University for the help and feedback during this thesis. Lastly, I would like to thank my opponent, Sebastian Parmbäck, for both wise and rewarding comments regarding this thesis.

(5)

List of Figures

2.1 The Knowledge Discovery from Databases process . . . 5

2.2 Competency iceberg model . . . 9

4.1 Distribution of where respondents are located . . . 24

4.2 Department distribution of the respondents . . . 25

4.3 Questionnaire respondents amount of years working at Etteplan . . . 25

4.4 Distribution of how important people think it is to map skill based competencies . 26 4.5 Distribution of how important people think it is to map knowledge based compe-tencies . . . 26

4.6 Distribution of how important people think it is to map behavioural based com-petencies . . . 27

4.7 Respondents definition of competency . . . 27

4.8 Distribution over whether the respondents believe implementing a CMS will ben-efit their personal development or not . . . 29

4.9 Distribution over whether the respondents believe using a CMS would change the way they acted when making different career choices . . . 30

(8)

List of Tables

4.1 Keywords used when creating query strings . . . 17

4.2 Query strings used when searching for articles . . . 18

4.3 Inclusion and exclusion criteria for article selection . . . 18

4.4 Data extraction form . . . 19

4.5 Result from literature search . . . 19

4.6 Primary articles used in the literature review . . . 20

(9)

1 Introduction

The term "competence" first appeared in an article written by R.W White [47] in 1959 as a con-cept of performance motivation. However, the term first gained traction in the early 1970’s when David McClelland wrote a seminar paper with the title "Testing for Competence Rather Than for Intelligence" [7]. In this paper McClelland questions the great amount of power that intelligence and aptitude tests has when it comes to choosing applicants for college entrance or for jobs. He later discusses if this truly is the correct approach, or if people should be judged on the basis of some other criteria, such as competence within a field. Yet, it was not until 1978 that the concept was used in relationship to individual performance development. When Thomas F. Gilbert used the term in his book entitled "Human competence: engineering worthy performance" [16].

The usage of the term "competence" varies widely, and the definition of the concept differs from person to person. Some scholars see competence as a synonym to skills, while some argues that the term is a combination of theoretical and practical knowledge, skills and be-havior. Mapping, analyzing and developing all of these factors can make a huge difference for the growth of a company. Managing the competencies within the organization in a struc-tured and coordinated manner leads to a certainty that the company consists of employees with the correct competencies and skills. Furthermore, mapping each role leads to the hiring and promotion of the correct people.

Competency and skills management has for several years been suggested to utilize employ-ees competencies more efficiently. By using competency maps, employers and employemploy-ees can identify their skills and competencies, and the level of said competencies. In addition to this the maps can also be used to present in which fields each person is in need of develop-ment [1, 27, 49]. By having all the employees competencies mapped, and by comparing these with the required skills for the position, it becomes easy to see if there is anyone within the company that is qualified for the job. Or possibly if anyone is close to having the required competencies, and would become qualified after some training. Competency mapping has several benefits, it increases employees self-awareness and aids in career management, it be-comes easier to find potential leaders within the organization and it assists in job evaluation. However, competency mapping can be applied to more than just individuals. It can also be used to identify key competencies within an organization, as well as to show required com-petencies to complete a specific job. This could be helpful when there is a vacant position in the organization that has to be filled.

(10)

1.1. Motivation

1.1 Motivation

As technology evolves, the importance of hiring people with the right competencies and de-veloping the necessary competencies within your organization increases. With this follows the need to map and assess each individuals competencies in a correct manner. The use of competency mapping has grown significantly during the last couple of years and is now used by Toyota [39] in order for all employees to evaluate themselves and to develop their competencies. As companies are getting more interested in having multi-skilled employees the need to map competencies increases, so that training can be applied to the fields where improvement is necessary [49]. Performing competency management within companies and organizations have become an important process to embrace, in order to keep up in the on-going technological evolution.

This thesis is written in collaboration with Etteplan1. Etteplan is currently using a human re-source management system containing some competency management functionalities. The currently used system is not mainly built for competency management, which is the main reason for why Etteplan are looking into developing their own competency management system. I have decided to exclude the name of the currently used system in this thesis. This thesis will be conducted at Etteplans office in Linköping.

By starting off with a literature study, the knowledge needed to better understand "compe-tencies" as a concept and competency management in general will be gained. An empirical study will also be conducted, where a survey will be used to gain knowledge of what the employees at Etteplan thinks about competency in general, competency management and the potential implementation of a new competency management system. Followed by inter-views with individuals from each department at the office in Linköping. This information can hopefully later be used when deciding on how Etteplan should manage the competency as-sessment and mapping of their employees. In addition to how the competency management system should be used and which functionalities the system should contain.

1.2 Aim

The aim of this thesis is to analyze how one actually assess an individuals competencies, and what impact this has on peoples development. Information that will be used to identify how and why a competency management system should be developed, and later to propose how said software should be built.

1.3 Research Questions

This thesis will try to answer the following two questions regarding competency develop-ment and competency assessdevelop-ment of a company’s employees. As well as answering how and why a new competency management system should be developed. The research questions are presented below, followed by short motivations of why each question is of interest.

• How do you evaluate an individuals competencies? And how does this affect the per-son being evaluated?

This question is important to answer in order to fully understand the necessity of a compe-tence management system and how this system should be used and developed. To evalu-ate someones competencies might be complicevalu-ated, and it can be difficult to decide how this should be done. Two individuals might have the same area of expertise, but their skill might be on different levels. It is also important to understand that different people might react in different ways when being evaluated.

(11)

1.4. Presentation of Etteplan

• How should a competency management system be built, and how should the interac-tion between the system and the employees work?

The systematic literature review and the empirical study should shine light upon how a new software should be developed and which functionalities said system should contain. This is important since the development takes time and costs money, and by making a thorough study these two factors could be reduced to a minimum.

These questions will be answered through a systematic literature review and by conducting a survey at Etteplans offices in Linköping and in Norrköping. The survey will be followed by interviews conducted at the office in Linköping.

1.4 Presentation of Etteplan

Etteplan is a consulting firm specialized in global engineering. Etteplan was founded in 1983 in Finland and have more than 3400 employees spread across seven countries, of which around 500 of these are based in Sweden, divided over 15 offices. Etteplan work towards the world’s leading companies in the manufacturing industry, and provides design engineering services, technical documentation solutions and software solutions. I will be a part of the software solutions department during this thesis.

Etteplan is currently using a human resource management system. The system also consists of some competency management and competency development functionalities.

1.5 Delimitations

The term "competency" is very broad and can be interpreted in several different ways. There-fore, this thesis will not answer the question "what is competence?". This thesis will have its own interpretation of the concept and the study will be conducted considering this. The method I will be using will be presented later in this thesis.

1.6 Thesis Outline

The remainder of this thesis will be outlined as followed. Firstly there will be a theoretic section that will provide the knowledge needed in order to follow along in this paper. Next follows a section presenting the methodology used in this thesis. After that there will be a section presenting the results acquired during the study, followed by a section discussing said results. Lastly, the result will be concluded and summarized, in addition to answering the research questions once again, in a more concise manner.

(12)

2 Theory

This section feature all the underlying knowledge and terms needed in order to comprehend the results presented later in the thesis.

2.1 Competency Management System

Competency management systems (CMS) [13] are used by companies in order to map, man-age and develop their employees competencies. Competency manman-agement is closely related to learning management, and functionality regarding learning is often implemented in com-petency management systems. For example tools to schedule training sessions and courses, and reports covering employees development are some functionalities that developers of competency management systems often implement in their software.

A CMS often includes tools such as competency management, succession planning, skill-gap analysis, in addition to competency analysis and mapping [37, 41]. Competency manage-ment is useful within a company when they want an overview of the competencies and skills of its employees. This information can later be used to manage the company and fill vacant positions. However, there might not be anyone employed that have enough knowledge in order to be promoted, and this is where skill-gap analysis is useful. By identifying how big of a skill-gap there is, a development plan can be constructed and applied to minimize the gap. Thus making it possible for an employee at the company to grow and take a new step in their career, and possibly get one step closer to their career goals.

One of the main usages of a CMS is to map employees competencies, discussed in 2.3, in or-der for individuals to easier plan their career and see where they are, and where they want to be in a couple of years. The goal for the companies using competency management systems is to expand and develop the expertise and knowledge of their work force. In order to stay relevant and at the cutting edge of their area of expertise.

A typical CMS offers the following features [43]:

• A pre-populated database of competencies, most often categorized by industry. • A database with employees and their skills and competencies.

(13)

2.2. Data Mining

• Performance reviews for each employee.

• The possibility to upload and disseminate reports consisting of so called "360-degree feedback". Which is feedback from everyone that have been in direct contact with the employee during an assignment, for example the employee oneself, supervisors and customers.

Competency management systems are designed to make it easier to grasp and analyze this information. The software should therefore have an interface that is easy-to-use and also easy to master for new users.

2.2 Data Mining

Data mining is the process of discovering patterns in large sets of data. It is most easily under-stood as a technology that allows you to take data as an input, and thereafter extract interest-ing patterns or knowledge from said data [48]. One could say that the goal with data mininterest-ing is to find and extract knowledge from data [36]. Data mining and the following technologies presented in this section could be used when initially filling up a new CMS with data and in-formation. Instead of having to manually enter users competencies and knowledge. Fayyad, Piatetsky-Shapiro and Smyth provided one of the earliest and most cited definitions of the data mining process in 1996 [17]. Where they defined the process as “the nontrivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data.” The term "data mining" originally referred to the analysis step in the Knowledge Discovery of Databases (KDD) process. Where specific algorithms are applied for extracting patterns from data. However, since then this definition have been dropped, and these days "data mining" may refer to either the entire process or just the algorithmic step in the procedure.

Figure 2.1 depicts how Fayyad et al. [17] envisioned the entire KDD process. The process consists of the following five stages: Selection, Preprocessing, Transformation, Data mining and lastly Interpretation/Evaluation.

The three first steps in the process deals with preparing the data for mining. It is important that the relevant data is picked from a large and diverse set of data, as data mining only can discover patterns that actually occur in the data set. The data is then preprocessed, i.e. cleaned, where findings containing so called noise and missing data are removed. Thereafter the data is transformed into a representation that is suitable for the data mining algorithm that is applied in the fourth step in the process. This step involves the application of data mining algorithms used in order to find patterns in the transformed data sets. The generated patterns can take many different forms, depending on what algorithms that were used. Fi-nally the patterns are thoroughly interpreted and evaluated. It is important to not accept the patterns right away as sufficient. It could potentially be better to search for patterns using some other metric, in order to find the patterns that are of most importance.

(14)

2.2. Data Mining

The data mining process is an iterative process. Meaning that it is possible to redo the entire process once it is completed. After the initial run is completed, the user has to evaluate the identified patterns thoroughly and make a decision on whether further work is necessary, or if the result is sufficient. As mentioned above it is of most importance that the correct and relevant patterns are used, in order for the result to be as good as it possibly can be. The results from the first run of the data mining process are often not adequate. Either the results are not acceptable or the user believes that further improvements can be done. The process is then repeated after some adjustments have been done. These adjustments can be done at any time during the process. For example additional data sets may be acquired or new data mining algorithms may be selected during the data mining step. When the user is satisfied with the identified patterns the process is completed.

2.2.1 Frequent Itemset Mining

Frequent itemset mining is a data mining technique where data mining algorithms are used in order to find sets of information that occur frequently together in data. Frequent itemsets are patterns that frequently occur. This is often used on shopping websites. When inspecting an item the site often shows items "other shoppers also looked at". This is possible thanks to frequent itemset mining. The algorithms identifies patterns from searches done by other users and are therefore able to display items you as a user could possibly be interested in [2]. There are several methods that can be used to mine frequent itemsets. Some of the most well-known algorithms are [42]:

• Apriori algorithm: The Apriori algorithm uses a so called breadth-first strategy in order to count the support of each itemset. This method uses a generate-and-test approach, which means that the algorithm generates candidate itemsets and tests if they are fre-quent. However, generating candidate itemsets is expensive, both in time and space. Making this method more or less useless when working with larger data sets.

• Eclat algorithm: Unlike Apriori, the Eclat algorithm is a depth-first search algorithm. This strategy is based on set intersection, meaning it uses groups of itemsets that share a common prefix.

• FP-growth algorithm: The FP-growth algorithm, where FP stands for frequent pattern, is the most common depth-first approach to date. This algorithm is also a bit more complex then the other two, and will therefore be discussed more in depth below.

2.2.1.1 Frequent Pattern Growth Algorithm

Unlike the Apriori algorithm, the frequent pattern growth (FP-growth) algorithm allows fre-quent pattern discovery without generating candidate itemsets. The FP-growth method is essentially composed of two steps: Creating a FP-tree, and extracting frequent patterns di-rectly from the tree.

The frequent pattern growth algorithm starts of by counting occurrences of items in the data set. Only the frequent items are then stored in a "frequent items table". Thereafter the FP-tree is built by inserting instances from the frequent items table. Items in these instances are sorted in descending order, as this makes it quicker to process the tree. The algorithm then checks the items in each instance and discards the items that does not meet the required minimum threshold [42, 22].

The main benefit with the FP-growth method is that the algorithm searches for shorter fre-quent patterns recursively and removes these, instead of searching for longer ones. This reduces the search cost considerably.

(15)

2.3. Competency Mapping

2.2.2 Cluster Analysis

The term cluster analysis, or clustering, is a term used to describe the process of grouping a set of objects [33, 45]. Where the objects in the same group are more similar to each other than they are to those in other groups. These groups are called clusters. The grouping of the data objects is only based on information given in the data that describes each object.

This technique can be applied in many different kinds of fields, and does not only apply in the world of data. Clustering is useful in several fields, such as: machine learning, bioinformatics and information retrieval. Cluster analysis is the main task in exploratory data analysis. Which is an approach of collecting and summarizing data sets.

Cluster analysis is related to several other techniques of dividing data into different groups. One could say that cluster analysis is a type of classification as it is easy to label each cluster. Since the objects in each group has something in common.

There are several different types of clusterings, where a so called "clustering" is an entire group of different clusters. The four most common cluster analysis techniques are:

• Hierarchical clustering: The goal with hierarchical clustering is to build a hierarchy of clusters. There are two types of strategies when we talk about hierarchical clustering, and these are agglomerative and divisive clustering. Where the former is a a "bottom-up" approach and the latter is a "top-down" approach.

• Partitional clustering: Partitional clustering is used when the goal is to not have any overlapping clusters at all. Such that each data object is in only one cluster.

• Fuzzy clustering: Fuzzy clustering is also called soft clustering, where each data object can belong to more than one cluster.

• Density-based clustering: Density-based spatial clustering of applications with noise (DBSCAN) groups together data objects that are close to each other. DBSCAN is one of the most commonly used clustering algorithms.

2.3 Competency Mapping

Competency mapping is the term for describing the process of identifying strengths and weaknesses of an individual. The goal with mapping individuals competencies is to make it easier for them to understand themselves and to pinpoint where career development needs to be directed. In order for employees to develop and get to the point in their career where they want to be. It is also useful to have mapped the competencies of the employees when the company have a job position to fill. Instead of hiring someone new, the company can look at their own work force to find someone suitable for the job [1, 27, 37].

There is no correct way to map competencies and it can be done in several different ways. Although there are some guidelines that could, or possibly should, be followed when com-paring the present and desired levels of knowledge and skills, when trying to perform a task effectively. These are [49]:

• Start with identifying the required skills for the job.

• Map the identified skills needed and compare these with the skills of the employees. Identify minimum required level of competencies.

• Map possible skill gaps. • Create a development plan.

(16)

2.3.1 Competency Classification

There are two different classifications of competencies: core competencies and profes-sional/functional competencies. Core competencies are competencies that each individual within a company is expected to possess. These competencies are seen as crucial for a com-pany to succeed, and the goal with core competencies is for individuals to be able to perform in different positions throughout the company. Meaning that each employee should be able to handle different assignments without additional training. Core competencies are meant to be flexible and can change as the company changes, evolves and adapts to the ever-changing technology [1, 27, 37, 41].

Professional competencies and functional competencies are the technical knowledge needed in order to perform a job. The goal with these competencies is to optimize each individuals performance when performing a specific job. These competencies are grouped for each job within the company, and they are grouped in three different categories [1]:

• Behavioural competencies: These competencies refer to the behavioural knowledge and traits needed by an employee in order to complete a task.

• Threshold competencies: This refers to the characteristics needed by an employee in order to perform an assignment efficiently.

• Differentiating competencies: Characteristics that differentiate superior performers from average performers are classified as differentiating competencies.

2.3.2 Competency Iceberg Model

N. Anisha [1] recommends visualizing competencies in terms of an iceberg. Technical knowl-edge and skills are at the top, above the waterline. These competencies are visible, and there-fore easy to identify and appraise. While behavioural competencies are beneath the water-line and more difficult to assess. These competencies can also be more difficult to develop. As these competencies constitutes the actual essence of a person. Behavioural competencies include what values the individual has, how the person views him or herself (self-image), how he or she typically behaves (traits) and what motivates a person to perform a specific task (motives).

(17)

Figure 2.2: Competency iceberg model

• Skill: An individuals ability to do something well. For example being a great Java developer.

• Knowledge: Information that a person uses in a specific area. This could for example be what differentiate an experienced developer from a recently graduated developer. The experienced developer could possess more information and therefore knowledge in the specific area.

• Values: How an individual sees the world and the people around him or her. It could be anything from political values to personal values. For example that all individuals are equal.

• Self-Image: How persons view themselves. Identity, personality and worth are all different aspects each individual takes into account when assessing themselves. For example seeing yourself as a leader.

• Traits: How a person typically behaves. An example of a trait could be how good an individual handles stress

• Motives: What motivates a person to perform a specific task. It could be anything from a need of acknowledgement to feeling powerful.

The competencies that lie beneath the waterline directly affects the competencies above the surface. For example, the things that motivates a person influences how he or she uses their skills and knowledge in order to complete a task. Either deliberately or instinctively. Further-more, the behavioural competencies could be problematic to assess and map from an ethical point of view. People might feel uncomfortable knowing their characteristics are being as-sessed and evaluated, and not just their practical skills.

(18)

3 Method

This section presents the methodologies and concepts used in order to conduct the study and write this paper.

3.1 Systematic Literature Review

The first part of this thesis will consist of a Systematic Literature Review (SLR) based on pre-vious studies regarding competency and competency management systems. The systematic review will follow the guidelines presented by Kitchenham and Charters [25]. The reason for choosing these guidelines is because they are suited for software engineering studies. Which is a category where this thesis fits perfectly well.

Kitchenham et al. discusses the reasons for conducting an SLR, the importance of it, as well as both advantages and disadvantages with a systematic review. They argue that one of the biggest advantages is that the results from the review are unlikely to be biased, thanks to the well-defined methodology of the process. However, the significant disadvantage with systematic literature reviews is that they require substantially more time and effort than a traditional literature review [25].

Kitchenham et al. have summarized several different guidelines for conducting a system-atic review and have concluded that the majority of studies are in an agreement on the main stages in the review process. These steps are planning the review, conducting the review and reporting the review. Below is a short presentation of the process.

3.1.1 The Review Process

The review process consists of three major parts, where each phase covers the stages per-formed during the SLR. The purpose with the first phase is to do a detailed plan of the re-view. Afterwards, the review is conducted, followed by a presentation of the results in the third and final phase.

(19)

3.1. Systematic Literature Review

3.1.1.1 Planning the Review

It is crucial to plan the review in detail, since a thorough planning leads to an increased chance of ending up with a successful result. The planning phase consists of the following five sub stages:

• Identify the need for a review. • Commissioning a review.

• Specifying the research questions. • Develop a review protocol. • Evaluate the review protocol.

The process might appear linear. However, many of these steps involve iteration, which also applies to the sub tasks in the following two sections. Firstly it is important to confirm the actual need for a review. It happens from time to time that systematic reviews are commis-sioned, and in that case a commissioning report has to be written. None of the steps in this phase are mandatory. However, Kitchenham et al. states that the two most important steps in the planning process are defining the research questions that the review will address, and creating a plan over the review procedure. An evaluation of the review protocol is needed if the review was commissioned. This is otherwise not necessary.

I have decided that I will skip the commissioning a review step, since I will conduct this sys-tematic review myself. Furthermore, the last step will partially be skipped seeing that I will skip the commissioning report. However, an evaluation process is constantly ongoing and I will revise the review protocol if I find any possible improvements.

3.1.1.2 Conducting the Review

As soon as the protocol has been accepted, the actual review can begin. In this phase the primary objective is to find as many studies as possible that relate to the research questions stated during the planning phase. The magnitude of this process is what differentiates a systematic review from a traditional literature review. Kitchenham et al. recommends that anyone who performs an SLR tries all of the steps presented in this phase. This phase consists of the following five sub tasks:

• Identification of research. • Select a primary study. • Study quality assessment.

• Extraction and monitoring of data. • Data synthesis.

The first step in this phase is to find as many potentially relevant studies to the research as possible. It is possible to find relevant studies in multiple ways. However, Kitchenham et al. states how necessary it is to devise and follow a research strategy. Once the preliminary search for relevant studies have been completed, the studies have to be assessed and analyzed further. Their actual relevance have to be determined. Using a selection criteria, declared during the planning the review phase, makes it possible to find studies with a direct connection to the research questions. Inclusion and exclusion criteria should be based on the research questions. These are used when deciding on which papers to include and which to exclude in the review. It is considered essential to study the quality of each obtained primary study.

(20)

3.2. Empirical Study

This is done in addition to the inclusion and exclusion criteria. The next step in the process is to extract and monitor data the researchers obtained in each of the primary studies. This is done by designing data extraction forms, which are used to collect data that is relevant to the research questions. The result from the studies is finally compared and summarized in the Data synthesis.

3.1.1.3 Reporting the Review

The final step in the process consists of writing down the results from the review, and dis-seminating it to interested individuals. The following three sub tasks are a part of this phase:

• Specify ways of dissemination. • Formatting the main report. • Evaluating the report.

It is crucial to specify how the SLR should be distributed to individuals that potentially could be interested in taking part of the review. This could be by publishing a paper in an academic journal, or presenting the results during a conference. Therefore it is important to format the main report correctly, since journal and conference papers usually have restrictions regarding size and content. The last step of the entire literature review process concerns evaluating the report. Journal and conference papers are peer reviewed and PhD theses are reviewed by experts as a part of the examination process.

3.2 Empirical Study

The results collected from the systematic literature review will then be used to perform an empirical study. I will start off by conducting a survey that will be answered by the employees at Etteplans offices in Linköping and in Norrköping. Followed by interviews, in order to get some more detailed answers regarding the employees thoughts around competence and the implementation of a CMS. This will hopefully give me a better understanding of what the employees wish to gain from a competence management system. All this information will eventually lead to a proposition on how a CMS should be developed, implemented and used.

3.2.1 Survey

As a first step in this empirical study I will conduct a survey. Thus, getting a better under-standing of the employees thoughts on competence as a concept and their thoughts regarding the implementation of a competence management system within the company. A major ad-vantage with a survey is that it is possible to collect a considerable amount of answers in a short period of time. However, the questions in a survey have to be easy to understand and basic, and it is not possible to ask supplementary questions. Which is a major disadvantage. This will be solved by conducting interviews with a few selected employees. Another dis-advantage is that people in general find it difficult to assess themselves [15], which could turn out to be an issue. Seeing that the survey is partially about the employees thoughts on competence, both as a concept as well as their own. The entire survey can be found in Ap-pendix A.

The survey will be distributed through an email list to all employees at Etteplans offices in Linköping and Norrköping, hopefully receiving a response rate of at least 80%. Everyone will answer the questionnaire anonymously and the only question that possibly could relate answers to a specific group of people is the question where the respondent has to state in which department it works.

(21)

for choosing Glasgow’s guidelines was because they were among the first I found and they fit perfectly with the study I were to conduct. Glasgow breaks down the process into three phases: survey design, survey instrument development and survey execution. Glasgow also argues that data analysis and reporting of results is so important that it should be considered its own phase in the process. All the phases in the survey process is presented below.

3.2.1.1 Survey Design

I will be designing the questionnaire myself and my supervisors will review the question-naire. Making sure the questionnaire is complete and do not lack any necessary questions in order to obtain enough knowledge about peoples thoughts concerning competence. The first step in this phase is to develop a sample plan [20]. This sampling plan is used to describe the approach in which a sampling group will be selected from a population. The sampling plan also states how a sufficient group size will be determined, as well as in which choice of media the survey will be distributed through. Survey media involves approaches like face-to-face interviews, telephone interviews and written surveys. This phase also covers how to choose the best possible target group for your survey and the importance of picking the sampling group randomly.

Glasgow states the importance of this phase, in order for the survey to stay unbiased. How-ever, this is not a phase that I will spend a lot of time with. Seeing that this thesis is conducted in collaboration with Etteplan, and I am only interested in the opinions of the employees at Etteplan. Making it impossible to stay fully unbiased.

I will be conducting a written survey distributed using an email list to all the employees at Etteplans offices in Linköping and Norrköping. 121 employees will receive the questionnaire. Glasgow mentions the importance of choosing a correct sampling size to get a reliable result. I believe that it is reasonable to involve both offices in the survey, since the two are closely connected and work closely together. The survey will be distributed to everyone working at the two offices and not only a few selected departments. Hopefully involving as many different thoughts as possible, increasing the depth of the answers. Involving just a couple of departments could make the answers biased, as there is a possibility that people in the same department have the same views and experiences regarding competence.

3.2.1.2 Survey Instrument Development

The second phase in the survey process discusses the linguistics and what actually catego-rizes a good question. Glasgow states that it is crucial that the survey questions are easy to understand and consistent with the educational level of the respondents [20]. The questions should be as easy to understand as possible, avoid misinterpretation, avoid any undefined abbreviations and avoid any double negatives. Another crucial thing to consider when con-ducting a survey is to avoid combined questions. Where the respondent may wish to answer positively on one question, but negatively on the other. An example of such a question could be "I enjoy my job assignments and I like my colleagues." where one might wish to answer differently on these two questions. The questions must be ethical as well. It is necessary to avoid questions asking the respondents for information they are not allowed to share, or do not have.

Glasgow argues that it is necessary, yet difficult, to avoid questions that could invoke any kind of biased answers. Either formulating a question in a specific way, or even the order of questions in the survey. An answer on a previous question could influence the answer on the following question. Nevertheless, this is important to avoid and only strengthens the importance of reviewing the survey before distributing it. The questions in this questionnaire will be a combination of both open-ended and closed-ended questions. Simple closed-ended questions should provide enough data in order to proceed with the study. However,

(22)

open-3.2. Empirical Study

ended questions will provide more detailed answers regarding the employees thoughts and opinions on competence.

3.2.1.3 Survey Execution

The third phase in the process regards the actual execution of the survey. One essential ethical aspect when executing the survey is to maintain the confidentiality of individual answers. It should not be possible to separate answers from one another and associate answers with specific individuals. Survey results should only be presented in total. Another important ethical aspect to remember is that survey participation is voluntary. Respondents are not obliged to answer the questionnaire, and the researcher should avoid being obstinate when it comes to receiving answers. Furthermore, respondents should not feel pressed to answer the survey, and there should be a reasonable deadline for when the answers has to be submitted. My plan is to give the respondents two weeks to answer the questionnaire. However, this might change depending on the response rate after two weeks.

3.2.1.4 Data Analysis and Reporting Survey Results

The fourth and final phase includes analyzing the data and reporting the results in a correct manner. A step that is crucial in order for the survey to be taken seriously and for the results to be seen as reliable. The results from the questionnaire will be presented in Section 4.2 and later discussed and analyzed in Section 5.2.2.

3.2.2 Semi-structured Interviews

One person from each department at the office in Linköping will be participating in a semi-structured interview, in order to collect more detailed answers. The interviews will follow the guidelines presented by Runeson and Höst in [35]. There are several different guide-lines regarding semi-structured interviews, but they are all very similar. Runeson and Höst’s guidelines were chosen because they are adapted for studies within the field of software en-gineering.

In a semi-structured interview the interviewer does not follow a strict list of questions and a strict plan. This interview technique can somewhat be seen as a combination of a regular interview and a conversation. The interviewer writes a couple of open-ended questions, but leaves room for supplementary questions that might emerge during the conversation. The questions and interview plan are only used as a guideline, and the interviewer can decide to steer away from the plan whenever [35]. A semi-structured interview is suitable in this thesis as it is a follow-up on a questionnaire, to gain more detailed answers and thoughts regarding the topic.

The interviews will be qualitative and not quantitative. Meaning that they follow a specific topic, but the questions are open-ended and might not be asked in the same order or in the same way to each interviewee [5]. The goal is to hear what the respondents think themselves, which is hard when conducting a quantitative interview. Where the questions have more of a "yes or no" character.

My plan is to interview four individuals at the office in Linköping. One from each of the three departments and one executive, since it is interesting to gain as many different thoughts and ideas as possible. The three employees will hopefully be interviewed first, followed by the executive. Being able to analyze the answers from the employees before talking with the exec-utive hopefully generates some new questions that could be asked during the interview with the executive. I have chosen to only interview persons from the office in Linköping, even though the questionnaire will be sent to the employees in Norrköping as well. The reason for this is that I find it relevant to talk to people where I am based.

(23)

have to be assured that everything that is said during the conversation will stay anonymous, unless the interviewee want otherwise. However, one problem is the number of persons be-ing interviewed. It is difficult to keep everyone anonymous when there will only be four persons interviewed, and all the interviews will be conducted at the office. Keeping individ-uals separated from their own answers might be easy, but the interviewees may be grouped together and accused of some answers as a group.

Another ethical dilemma is how to document the conversations. Hopefully everyone will accept being recorded, in order to avoid potential breaks in the interview when I have to take notes. Taking notes may ruin the flow of the conversation, and the interviewee could pos-sibly forget something that he or she was thinking of. Recording the dialogues also ensures that no information is lost during the discussion. Therefore, assuring the interviewee that I am the only one who will be working with the recording is crucial, in order to maintain the confidentiality and anonymity. Longhurst [30] discusses the importance of transcribing the recordings as soon as possible after the interview. Hearing the taped conversations when it is still fresh in your memory makes transcription easier and reduces the time consumed con-siderably. However, this might be difficult to sustain if more than one interview is scheduled during a day.

Several studies discusses the importance of having a lot of experience when conducting terviews [23, 34]. However, I do not have any experience when it comes to conducting in-terviews, but I believe that this may be more important when interviewing people that you have no kind of connection with. When as in my case interviewing people that I at least have spoken to before makes it much easier to maintain a conversation and a flow in the interview. Without much experience at all.

The questions below are the main questions that I will be asking during the interview. How-ever, as mentioned above, the questions may be asked in a different order to each person and in a different way. Depending on in which direction the interview is heading and depending on potential supplementary questions that have arose during the dialogue.

• What is your opinion on the "competence" concept?

• It is impossible to avoid the ethical aspect when it comes to mapping individuals com-petencies. What are your thoughts regarding mapping the three different competency categories: skill based, knowledge based and behavioural based competencies? Are there different ethical aspects to consider for each category?

• How are you using the CMS that is used at Etteplan today?

• Do you have any personal preferences or requests regarding a competence management system? When it comes to how it should work, how it should work, how it should be implemented etc.

(24)

4 Results

This section presents the results gathered during this thesis. The first part of this section presents the results from the systematic literature review and the second part presents the results from the survey and the interviews.

4.1 Systematic Literature Review

The literature review process began with a planning of the review, performing each sub step the phase consists of. Followed by the actual execution of the review. Lastly the findings from the literature study was recorded and reported.

4.1.1 Planning the Review

The first step in the systematic literature review process was the planning of the entire review.

4.1.1.1 Identifying the Need for a Review

Ever since 1978 when the concept of competence was first used in relationship with individ-ual performance development, the term has been a subject of discussion. No consensus exists regarding how the concept should be defined [40] and researchers define the concept in many different ways [11]. Some sees the concept as a synonym to skills, while others define the con-cept as a combination of theoretical and practical knowledge, behaviour and skills. Possibly there will never be a mutual definition of the term. However, there is an agreement that the need to map and develop competence within companies increases, in order to stay relevant and grow as an organization [24, 29].

The necessity to map and utilize employees competencies and knowledge have never been as important as it is today. Competency mapping have grown considerably in recent years and the method have become crucial for companies to adapt. This literature study should lead to a conclusion regarding how Etteplan should perform the competency mapping and competency assessment of their employees.

(25)

4.1.1.2 Defining Research Questions

The main purpose with this systematic literature review is to obtain a deeper understanding of how you evaluate and analyze an individuals competencies. Along with how this poten-tially could have a negative or positive impact on the people being evaluated. Which will be done by answering the following research questions:

• How do you assess and evaluate an individuals competencies?

• What are the biggest challenges with assessing and evaluating individuals competen-cies?

• How could this potentially affect the person being evaluated?

Answering these questions should present a clear picture on how to assess and evaluate an individuals competencies the best way possible. Being aware of the common challenges and how the assessment potentially can affect individuals will help when deciding upon how the assessment process should work, and later when presenting a proposed approach on how a competency management system should work and be implemented.

4.1.1.3 Developing a Review Protocol

A review protocol presents the methods which were used when conducting the systematic literature study. In addition to important decisions that was taken during the construction of the review protocol. I will be using three common databases for scientific papers, primarily in the fields of computer science, to find relevant studies. These three digital libraries are:

• IEEE Xplore Digital Library • ACM Digital Library • ScienceDirect

In addition to these databases, Google Scholar and CiteSeerX will be used for so called chain sampling. Which is the term for describing the process of finding additional studies of interest from the reference lists of the articles found during the literature study. The plan for the data extraction is to firstly gather as many different studies and articles as possible. Which will be done by using different query strings, presented in table 4.2, constructed from a set of keywords, presented in table 4.1.

Primary keywords Secondary keywords Supplementary keywords

Competence assessment Companies Competence Competence evaluation Challenges Competencies Competency classification Behavioural Consequences

Competency mapping Threshold Company

Differentiating Consequence

(26)

Query strings

Competence AND (assessment OR evaluation) AND (Companies OR Company) Competency AND (classification OR mapping) AND (Companies OR Company) Competence AND (assessment OR evaluation) AND (Behavioural OR Threshold OR Differentiating)

Competency AND (classification OR mapping) AND (Behavioural OR Threshold OR Differentiating)

Competence AND (assessment OR evaluation) AND (Challenges OR Consequences) Competency AND (classification OR mapping) AND (Challenges OR Consequences)

Table 4.2: Query strings used when searching for articles

The articles retrieved during the search process will then be filtered through three phases, where different inclusion and exclusion criteria, presented in table 4.3, are applied. Firstly, the broader criteria, such as language and type of publication, are applied on the articles. When the articles have gone through the first step and some have been filtered out, a new criteria will be applied. This criteria will review the titles of the selected studies and filter out the articles with titles that are not of interest to this thesis. Lastly, the abstracts and introductions of the articles that got through the first two phases will be read, to decide which of the articles should be included in the literature review.

Phase Inclusion criteria Exclusion criteria

1st

It is a conference paper or book chapter or journal

and within the scope of competency assessment

Not in English is a duplicate is a summary 2nd Title is related to competency assessment

3rd

Abstract and conclusion states that the articles have studied competency

assessment within companies, organizations and/or different kinds of professions

Table 4.3: Inclusion and exclusion criteria for article selection

When this process is completed there should be a reduced amount of relevant articles. These articles will then be read and summarized using a data extraction form, presented in 4.4. Kitchenham et al. [25] mentions the importance of analyzing and evaluating each article in the same way, reducing the possibility for bias. They recommend constructing a data extraction form to answer a few questions regarding the keywords that have been used, as well as the research questions. This is done in order to decide if the studies are relevant or not. All studies will be classified based on a wide range of questions, such as type of publication, if they contain the stated keywords and if the articles answer the research questions or not.

(27)

Question Yes / No Comment

Name of article Author Type of publication Year of publication Digital library Keywords

How to assess and evaluate competencies? What challenges was identified?

Potential ways the evaluation could affect individuals? Table 4.4: Data extraction form

4.1.2 Conducting the Review

During the initial search the goal was to collect as many articles related to the subject as possible. This was done by using the databases advanced search functionalities, and conduct searches using the constructed string queries listed above in table 4.2. Initially the search gen-erated a total of 78,244 articles, which were filtered down to 15 studies when going through the three inclusion and exclusion phases. The articles chosen for the systematic literature review are presented in table 4.6, and a more detailed presentation is located in Appendix B.

Database Initial Phase 1 Phase 2 Phase 3

IEEE 47,453 139 34 6

ScienceDirect 28,576 244 48 6

ACM Digital Library 2,215 63 19 1

Chain Sampling - - - 2

Total 78,244 446 101 15

(28)

ID Title Reference

A1 Supporting competence management in software projects [12] A2 Competence analysis of IT professionals involved in business services_{— Using a qualitative method} [31] A3 Effects of Automated Competency Evaluation on Software Engineers’

Emotions and Motivation: A Case Study [9]

A4 From Behavioral Indicators to Contextualized Competence Assessment [3] A5 Assessment of the behavioural component of competency

to substantiate practitioner competency claims [10] A6 A Semantic Approach for Improving Competence Assessment

in Organizations [19]

A7 Competence assessment as an expert system for

human resource management: A mathematical approach [4] A8 Robust competence assessment for job assignment [21] A9 The Assessment of Learning: From Competence to New Evaluation [8] A10 The factors related to self-other agreement/disagreement in nursing

competence assessment: Comparative and correlational study [44] A11 Preceptors’ experiences of using a competence assessment tool

to assess undergraduate nursing students [6]

A12 Assessing competencies: An evaluation of ASTD’s Certified

Professional in Learning and Performance (CPLP) designation [26] A13 Evaluating assessment with competency mapping [32]

A14 Competencies and Their Assessment [14]

A15 Challenges to the assessment of competence and competencies [28] Table 4.6: Primary articles used in the literature review

All of the studies above discusses the subject competence management/assessment in some way. However, the majority of the articles discusses the subject within a specific field. For example nursing and teaching. This does not provide a general insight into how to as-sess competence, which requires you to make assumptions. Nevertheless, I believe that the primary studies found during the search will provide enough data in order to answer the research questions.

4.1.2.1 How to Assess and Evaluate Competencies

There is no definite way to assess and evaluate individuals competencies. Different methods suits different occasions. The gathered primary studies present and suggest several different methods, all within their specific field. One reoccurring scenario is within an educational environment. Where the students are asked to assess their competencies at the beginning of a course, and finally at the end of the course. This method is used to evaluate how the students have developed, and to see if what they thought of themselves actually corresponded to their competencies. This method is often used in a educational environment seeing there is a learning element when you have to assess yourself.

The majority of the methods presented in the articles originates from self-assessment and assessment of others. Where one popular method, presented in several of the studies, is the combination of self-assessment and assessment of each other. Essentially, each of the studies have adapted one or both methods to suit the research conducted in each article.

Self-assessment The self-assessment is most often done through questionnaires, where the employees are asked to answer several questions to grade themselves and their competen-cies. The results from the questionnaires could be entirely private, where each participant

(29)

dividuals progress and development. However, the results could also be accessible to a se-lected group of people, for the individuals answering the questionnaire to get help in their development. This method is used by Dorn et al. [12] where a group of university students are participating in a software project management course. The students are asked to use a competency management system to plan and control their future study based on their com-petencies. Primarily, the students answer a questionnaire at the start of the course, where they are asked to estimate and judge their own competencies. The results generated from these questionnaires are available to both the student and their supervisor, making it possi-ble for the supervisor to guide and observe the development of the students competencies. At the end of the course the students are asked to answer another questionnaire, where the final feedback showed which competencies where improved and which new competencies the student acquired during the course.

Bohlouli et al. [4] also presents a method in which the assessment process begins with the participants answering a questionnaire. Each answer is weighted and the results are used together with a mathematical model. This model calculates the persons so called "available competencies" and creates a development plan for each person that answered the question-naire. These plans are then used by each employee in order for them to achieve their goals, and develop their skills the best way possible.

Assessment of Others Interviews and questionnaires are two methods often used when conducting competency assessments [31]. Another popular method often used in conjunc-tion with competence assessment is so called "360-degrees feedback". This method is a pro-cess where feedback from the employees colleagues, supervisors and subordinates, as well as self-assessment by the individuals themselves is collected and analyzed [4]. The feed-back can also, when relevant, include feedfeed-back from external sources who have interacted with the employees during a project or a job. This feedback could be relevant when work-ing as a consultant, where you most certainly interact with several different companies and individuals throughout your career. Making it relevant having your competencies and your performance assessed by the persons you come in contact with at these companies and dur-ing these projects.

A method often used in educational environments, in combination with students assessing themselves mentioned above in self-assessment, is for the supervisors to assess their students performance and competencies at the end of the course [12]. This is done through question-naires where the supervisors get the chance to evaluate and grade each students performance. Drisko [14] mentions how important it is that the one assessing another individuals compe-tencies already possesses the compecompe-tencies being evaluated. Which means that the evaluator possesses the knowledge, skill or competency being examined and have significant experi-ence in the field or profession. Another approach is to use tools that are suited for compe-tency evaluation. Therefore, it is important for the supervisors to evaluate each student as well, since they are experienced and already possesses the competencies.

One important thing to consider when assessing and evaluating competencies is to carry out tests relevant for the work where the competencies later will be used. For example when testing how well a consultant works under pressure, it is important to not put the consultant in a situation where he/she is uncomfortable. Otherwise the results might be tampered, and someone who actually has the required competencies to perform the task might appear un-qualified.

Competencies can be assessed in several different ways, and can be assessed by several differ-ent individuals. The approaches can be both quantitative and qualitative, and the evaluator could be anything from a client to a supervisor. However, Drisko [14] emphasizes how impor-tant it is that the approach is explicit and directly related to the competency being assessed. Furthermore, it is optimal to use several methods at once, to get a result as close to reality as possible.

(30)

Burke et al. [6] presents a mixed method where the assessment process contained both a qualitative phase and a quantitative phase. Each phase respectively consisted of group in-terviews and a questionnaire. The inin-terviews were divided into six focus groups where the purpose was to establish professors experience in using a competency management tool to assess nurses clinical competencies. The questionnaire was used to quantitatively measure the professors attitude towards the competency assessment system.

Kwon et al. [26] also presents a mixed quantitative-qualitative method to identify the relation-ship between a competency model used in an educational environment and the final exam taken by students. The first phase consisted of interviews and a literature study, where the results later was used to construct a first draft of the competency model. Phase two started of by constructing a survey using the first draft of the competency model, and these results were then used to finalize the model.

Collins and Nicholson [10] suggests assessing competencies on a sub-component level. Meaning that instead of assessing and analyzing for example someones leadership compe-tency, you divide it into several competencies needed for someone to be a good leader. These competencies are then assessed separately.

4.1.2.2 Challenges With Competency Assessment

One major challenge with competency assessment is how difficult it actually can be to as-sess individuals competencies [21]. Especially behavioural competencies that can sometimes be hard to define. One specific occurrence that verifies how difficult it actually is to assess your own or someone elses competencies is how often people overestimate or underestimate themselves [12]. People simply misjudge their competencies and what they are capable of accomplishing. Takase et al. [44] states that one reason for this could be individuals person-alities and that it is difficult to be impartial. It is therefore important having as many people as possible involved in the assessment and evaluation of someone. Collecting a large amount of opinions and views should give a picture of an individuals competencies as close to reality as possible.

Some categories of competencies are easier to assess then others. As mentioned in Section 2 competencies can be divided into three categories: skill based, knowledge based and be-havioural based competencies. The first two categories are pretty straight forward to test and assess. Whereas behavioural competencies can be more difficult and somewhat questionable to assess from an ethical point of view.

The majority of the studies found in this literature review are specialized in a specific area of expertise, for example nursing and teaching. Which does not provide an insight into how these methods could be used generally. Therefore, the only thing one could do is to make assumptions on how these methods would work in other environments. Furthermore, the majority of the methods presented in the studies are based on data from interviews and sim-ilar approaches of collecting data. The methods are therefore based on peoples opinions and not very "scientifically proven". “Furthermore, the main challenge and limitation that sim-ilarly addressed in most of the literature is a lack of efficient and generalized competence matching method.” [4].

The need for continuous development of competencies with employees have never been as important as it is today. Individuals have to develop existing competencies along with new once. This calls for new ways of training and learning, since current learning materials most often consists solely of theory. Furthermore, current guidelines on how and when theory should be applied in practice are ambiguous and diffuse [3]. Learning material have to be connected with the actual work process in a better way. Otherwise it will become harder to develop competencies, and with that assessing them.

Burke et al. [6] discusses the challenges that comes with implementing and using a compe-tency management system. It can be anything from linguistic difficulties to how time

How to Assess and Map Employees Competencies

Linköping University | Department of Computer and Information Science

Master thesis, 30 ECTS | Datateknik

2019 | LIU-IDA/LITH-EX-A--19/081--SE

How to Assess and Map

Employees Competencies

Hur man bedömer och kartlägger anställdas kompetenser

Christoﬀer Johansson

Upphovsrätt

Copyright

Acknowledgments

Contents

List of Figures

List of Tables

1

Introduction

1.1

Motivation

1.2

Aim

1.3

Research Questions

1.4

Presentation of Etteplan

1.5

Delimitations

1.6

Thesis Outline

2

Theory

2.1

Competency Management System

2.2

Data Mining

2.2.1

Frequent Itemset Mining

2.2.2

Cluster Analysis

2.3

Competency Mapping

2.3.1

Competency Classification

2.3.2

Competency Iceberg Model

3

Method

3.1

Systematic Literature Review

3.1.1

The Review Process

3.2

Empirical Study

3.2.1

Survey

3.2.2

Semi-structured Interviews

4

Results

4.1

Systematic Literature Review

4.1.1

Planning the Review

4.1.2

Conducting the Review