Software quality characteristics and where to find them : A qualitative study of how developers in Örebro work with design quality

(1)

Software quality characteristics and

where to find them

A qualitative study of how developers in Örebro work with design

quality

Authors: Joakim Cederlöv, 800923 Elinore Hagman, 950308

HT2020

Informatics, Advanced Course, 30 Credits Subject: Informatics

Örebro University School of Business Supervisor: Victoria Wipawee Paulsson Examinator: Kai Wistrand

(2)

Abstract

Achieving satisfactory design quality is essential to software developers everywhere and there are many ways to achieve it. In this study, we examine how developers work in regard to design quality, and how their development affects it. We conducted semi-structured interviews with three different developers that work at different companies in the city of Örebro, Sweden. We have used the software quality characteristics of the product quality model from ISO 25010 to frame our interviews. The same quality characteristics were then used in the analysis of our empirical data. We found that the developers strive to deliver software and services of good design quality, and that they face most of the quality characteristics in ISO 25010 during their work even though the frequency of these encounters differ from characteristic to characteristic. Achieving design quality is governed by the practices established by the developers themselves and their respective companies. Testing seems to be contributing to better design quality in multiple areas, and distributing time-consuming tasks to different types of teams gives the developers more opportunities to work towards satisfactory design quality.

Keywords:

(3)

Key concepts

Developer: Refers to system developers.

Design quality and software quality: Used interchangeably. Refers to quality of software. ISO 25010: Refers only to the product quality model in ISO 25010.

(4)

1. Introduction

1.1. Background

In 2014, Venkat Subramaniam, founder of Agile Developer Incorporated, said that there are deficiencies regarding quality in software development. It is mainly because of lack of discipline and too much focus on functional requirements (Danielsson, 2014). A year later it is argued that developers are getting the job done, but the road to the finished system consists of shortcuts that affect the design quality (Suryanarayana, Sharma & Samarthyam, 2015a). At the same time, there is research arguing for quality control being increasingly important during software development (Haoues, Sellami, Ben-Abdallah & Cheikhi, 2016; Jain, Sharma & Ahuja, 2018; Kashfi, Nilsson & Feldt, 2017; Stevenson & Wood, 2018). This shows an inconsistency when it comes to design quality and its importance versus actual utilization, especially since developers today work with building more and more complex systems (Haoues et al., 2016; Kaliraj & Bharathi, 2019).

When discussing design quality in software development, quality in a more general sense has to be considered. It can be deemed a problematic concept, whose meaning changes from person to person. Oftentimes that person's interpretation of quality can even change when the concept of quality is used in different contexts (Harvey & Green, 1993; Elassy, 2015). This ambiguity has pushed different professions towards trying to define what they think constitutes quality. Within the field of software development this has resulted in the creation of different software quality models (Wagner, 2013). There are numerous practices employed that help fulfill the quality requirements stated in these models, and thus improve design quality. For example, implementing design patterns into one’s code structure helps with maintainability and code comprehension (Wedyan & Abufakher, 2020), and using code reviews help to reduce coding errors (McIntosh, Kamei, Adams & Hassan, 2015).

There is a plethora of research when it comes to what practices are well established as contributors to good design quality, but research about what actual developers use in their daily work is somewhat lacking (Stevenson & Wood, 2018). Geographical area, market sector and product range are factors that affect how software companies operate and what priorities they have (Sánchez-Gordón & O’Connor, 2015). We therefore believe that it is relevant to investigate the topic of design quality from a local level.

(7)

2

1.2. Purpose and research question

This thesis aims to explore how developers work with software development, and how their practices affect the design quality of their developed systems. The intention is to produce a depiction of how developers can work with their development in order to achieve design quality. The thesis is limited to developers in the city of Örebro, Sweden only. Based on this purpose we have created one research question with two subquestions:

● How do developers in Örebro work with system development in regard to design quality, according to ISO 25010?

o What practices do they use in their development?

o How does the practices affect the design quality of their systems?

1.3. Limitations

This thesis has two limitations. These are:

First, we choose to limit ourselves to the city of Örebro, Sweden, for this research. By limiting us to one city and associated developers we have a smaller target group to work with. The limitation is also due to the time limit of this thesis. This way, we can also take in consideration what Sánchez-Gordón and O’Connor (2015) says about companies in different geographical areas operating differently.

Second, to help us explore the concept of design quality in software development we will choose only one of the two models of ISO 25010, the product quality model, to help frame our interviews and base our analysis on. This will also keep our study relevant to our research area and research question.

(8)

3

2. Literature review

Our literature review is arranged into five subsections. The first subsection describes our search process for relevant literature. The second subsection gives an introduction to the concept of

quality. The third subsection continues to talk about quality in relation to different quality

models. The fourth subsection introduces factors that are known contributors to good design quality. The fifth and last subsection presents the framework that we will use in our analysis of the empirical data.

2.1. Literature search process

To find relevant literature we used Primo as a database since it had an easy way to filter the results, like searching for peer reviewed articles only. First, we identified what areas we were going to search for, and since our research question is about design quality in relation to company practices, design quality and software development were our main themes. After an initial search with the phrase “design quality AND software development”, we got over 900.000 results. We could conclude that the subject was well enough researched for us to narrow down the search more. We chose to only include articles that were written between 2015-2020. This filter was made to ensure our study covers information that is up to date to how software development is performed and what kind of practices are established nowadays. We also filtered our searches to only include articles written in English, for language barrier reasons. Lastly, we only included articles that were peer reviewed. To find synonyms and other keywords that could help narrow down the search even more, we used our updated search and looked through the subjects the articles themselves had listed in their presentation on Primo. Based on that we decided to include software quality as a keyword, since it covered the same kind of results we were interested in as design quality. Another keyword that was added to our searches were practice*, which was a word often related to software development and how to achieve design quality.

We did one search using the phrase “(software quality OR design quality) AND practice* AND software development”. This search resulted in roughly 288.000 results, which could still be considered too many, but after looking through the top results we could pick out articles that were relevant to our research question. We continued to identify more keywords to help us narrow down the search, and we found that many articles covered design quality in relation to testing. We therefore included keywords as software test* and code review* in our searches. Another search phrase was “(software quality OR design quality) AND characteristics AND ISO 25010 AND (software test* OR code review*)”, and this resulted in 254 results. After skimming through the results, we could pick out articles from this search that could cover our research question regarding practices, and also the use of ISO 25010. We additionally did multiple different searches where we mixed our different keywords and added the characteristics of ISO 25010 individually. From these searches we could get complementary articles about our framework.

(9)

4 When reviewing our chosen articles, we could identify some articles citing others that also were chosen, mostly Haoues et al. (2016), which could argue for it being particularly central in our research area. Some of the articles ended up being older than 2015, because we wanted to expand to sources we found cited in earlier articles. We also made complementary searches without using publishing year as restrictions to see if we could find any other articles that were relevant to us despite their age. We chose one article written by Harvey and Green (1993) discussing quality as a concept. After confirming the article was peer reviewed we decided to include it in our list of articles. The article is 27 years old, but since it had been cited in other articles 19 times the last two years we decided it still had relevance and currency. When looking through the articles that had cited the original one we picked out a few complementary articles, after checking that they were peer reviewed.

2.2. The concept of Quality

Quality is an abstract concept that has different meanings to different people, and may even

differ to the same person from time to time (Elassy, 2015; Harvey & Green, 1993). Yet the term is still widely used to gauge the overall impression of a product, service or experience. Due to the inconsistent nature of the concept it has been extensively researched, by both commercial and public interests, in order to better define and explain the phenomenon and how it applies to specific areas (Elassy, 2015).

In addition, there have also been attempts at defining the concept of quality itself and the underlying preconceptions, assumptions and expectations that are used alongside it. Some of these attempts tend to categorize quality into different perspectives, where each perspective encapsulates a certain way to approach quality as a concept. Examples of this are quality as

exceptionalism (Harvey & Green, 1993), quality as meeting and exceeding the customers’ expectations (Reeves & Bednar, 1994), or more specific ones like quality for internet of things

(Minovski, Åhlund, Mitra & Zhohov, 2020).

Harvey and Green (1993) states that there are five dimensions to quality, each representing certain values and perspectives. These are:

● Quality as Exceptional

● Quality as Perfection or Consistency ● Quality as Fitness for Purpose ● Quality as Value for Money ● Quality as Transformation

Harvey and Green (1993) concludes that trying to define quality as a single dimension is a lesson in futility seeing as how relative the concept of quality really is, and that one should instead try to identify the criteria that a person uses to identify quality. Elassy (2015) delivers some critique to this type of categorization and states that even though these general categories are helpful, there is still need for different vocations and areas of interest to develop their own definitions of what quality means in their context.

(10)

5

Quality as Fitness for Purpose is of special interest in our study, as it focuses on the intended

purpose of a product or service and measures quality against the extent to which the product fulfills that purpose (Harvey & Green, 1993). This also corresponds well with IEEE’s (2017) definition of quality in a software context:

● “Degree to which the system satisfies the stated and implied needs of its various stakeholders, and thus provides value” (IEEE, 2017, p. 424)

● “Ability of a product, service, system, component, or process to meet customer or user needs, expectations, or requirements” (IEEE, 2017, p. 424)

● “The degree to which a set of inherent characteristics fulfils requirements” (IEEE, 2017, p. 424)

Even though IEEE (2017) defines what quality is in the context of software development, the specifics are still left vague and the specific criteria that constitutes quality is still missing. The definitions provided merely aim to narrow down the possible interpretations of the concept. In order to further concretize the concept of quality, in relation to software development, a number of different software quality models can be used (Wagner, 2013).

2.3. Software Quality Models

A software quality model is a way to deconstruct the concept of quality into manageable and understandable criteria. These criteria can be used to evaluate the product quality in software. The research on this subject goes back decades, with the works of Boehm and McCall during the late 1970’s laying the foundation (Wagner, 2013).

According to Wagner (2013) there are three major types of software quality models:

hierarchical, meta-model-based, and statistical and implicit quality models.

● Hierarchical models try to decomposition the concept of quality into hierarchical structures of quality characteristics. There are overarching general quality characteristics that are split apart into sub-characteristics, sub-sub characteristics and so on. The intent is reaching sufficient granularity in order to find a quality aspect that can be measured. A critique against this type of model is that it focuses on defining

what to measure but not how (Wagner, 2013).

● The meta-model-based quality model tries to make up for the hierarchial’s shortcoming by extending a hierarchical quality model with ways to measure the quality characteristics. Some models accomplish this by translating different quality characteristics into values and thresholds that need to be fulfilled accordingly (Wagner, 2013).

● The statistical and implicit quality models revolve around analyzing the current production process in order to identify future problem areas and reinforce these as deemed appropriate. Usually the criteria that defines these problem areas are based on

(11)

6 established software quality models, hence the implicit nature of the relationship (Wagner, 2013).

2.3.1. Software quality characteristics

IEEE (2017) defines software quality characteristics as a “category of software quality attributes that bears on software quality” (p. 434). These characteristics are the base elements of quality in a software context. Depending on the general purpose and approach of a model these quality characteristics will differ (Wagner, 2013; Samadhiya, Wang & Chen, 2010). A common way to organize the characteristics is to divide them into hierarchical structures. The most general and abstract elements are found at the top, which are then divided into layers that contain more and more concrete elements as you descend. At the bottom the perceived atomic fragments of a top-level quality characteristic are found (Wagner, 2013).

Two historical hierarchical models are the software quality models proposed by Boehm and McCall (Wagner, 2013). Boehm’s model is composed of three top-level characteristics that are divided into five sub-categories. McCall’s model is more granular, where the top-level consists of three major developer perspectives that is further broken down into eleven quality factors. These factors are then further broken apart into 23 quality criteria where some are shared between factors (Samadhiya et al., 2010). Many of the characteristics found in Boehm’s and McCall’s software quality models can also be found in more recent models such as FURPS (Samadhiya et al., 2010) or ISO 25010 (ISO/IEC, 2011).

The aforementioned software quality models share the same goal of further defining what quality is in a software development context, but they differ in how they accomplish this by mapping the characteristics of quality in different ways. Although the intention of the models is to concretize quality into measurable units, the characteristics themselves tend to be too vague and hard to measure. Additional components are required in order to accomplish this (Wagner, 2013).

2.4. Factors that affect design quality

Our literature review has resulted in an indication that the following factors affect design quality. These factors are: (1) Experience as a developer, (2) Testing and reviewing, and (3) Software processes. Each factor is explained one by one below.

2.4.1. Experience as a developer

Clean Code as well as the principles of SOLID are identified as being practices that can improve design quality. There are some practices identified as being misused or overused though, like interfaces and design patterns, and these were mostly connected to low experienced developers (Stevenson & Wood, 2018). The experience level of the system developer is in fact something that is indicated as one major factor in design quality (Stevenson & Wood, 2018; Suryanarayana et al., 2015a). However, there are suggestions that experienced developers are more prone to omit quality practices, due to personal goals (Ghanbari,

(12)

7 Vartiainen & Siponen, 2018). Regardless, there is a need for novices to get support during their development from more experienced developers. The support could be received through different code reviews, which is identified as an important factor for developers when trying to obtain good design quality (Stevenson & Wood, 2018).

2.4.2. Testing and reviewing

Stevenson and Wood (2018) bring up test driven development (TDD) being a practice that may improve design quality. There is also a consensus that decoupling and cohesion are something that makes testing easier, and consequently improves design quality. TDD has time-consuming issues though, since further on in the development cycle more features may be added and therefore refactoring becomes a lengthier task (Rais, 2016). It is also implied that TDD may not be suitable for larger projects, since it could be near impossible to test every function (Nanthaamornphong & Carver, 2017). Due to time constraints and lack of interest for testing activities by stakeholders, testers sometimes have to choose to meet deadlines instead of working towards quality. These factors overall influence the quality of the testing activities that testers perform (Ghanbari et al., 2018).

Code- and design reviews are recognized as practices that are well known and used when developing software, and the practice does help to achieve better design quality (McIntosh et al., 2015; Suryanarayana et al., 2015b). Implementing code reviews are not something that automatically achieves high code quality though. Instead, quality depends on the participation level of the review process. By having a more integrated review process, including active participation and code coverage by experts, software can be developed with fewer defects. These changes can raise the quality of the code (McIntosh et al., 2015). Development where there is a lack of inspection and a lack of quality control tools often results in developers being less motivated to use quality practices. This also creates a phenomenon where systems are deployed in spite of containing numerous faults and buggy code. They are consequently more likely to suffer from different security issues (Ghanbari et al., 2018).

2.4.3. Software processes

Software processes are practices and methods utilized by development teams when developing software, and examples of processes are different plan-driven approaches or agile methods like Scrum. The intention is to use the process for structure, and also to aid the development process so the system can be delivered in a timely manner. At the same time the developers need to keep the quality as high as possible (Sánchez-Gordón & O’Connor, 2015). However, software processes can affect the design quality negatively, depending on how they are implemented or used (Suryanarayana et al., 2015b). Since software processes are something that external stakeholders are not directly involved in, developers also sometimes consciously choose to omit quality practices (Ghanbari et al., 2018). It is, therefore, important to periodically evaluate how the software process is affecting the potential design quality. If it negatively affects the design quality, there should be a discussion whether there should be a change of software process (Suryanarayana et al., 2015b).

(13)

8 Some external stakeholders also sometimes lack understanding of what requirements actually mean when it comes to the project they are involved in. Due to this shortfall, developers usually implement quick solutions, often in the beginning of the project. Mockups are an example of this, and after the customers have tried them out, developers can collect feedback from them. From this activity the developers can then establish more optimal requirements from said feedback (Ghanbari et al., 2018).

2.5. Analytical framework: Product Quality Model in ISO

25010

This section explains the ISO 25010 framework, which is used to investigate the empirical data collected in this thesis.

2.5.1. Suitability of the Product Quality Model in ISO 25010?

To understand what design quality actually consists of, and how we can assess the design quality of our interviewees’ development, we have chosen to base our data collection and analysis on the ISO 25010 standard. More specifically we are going to use the product quality

model, which is one of the two models specified in ISO 25010.

ISO 25010 is a model for assessing and estimating software quality. The standard is relevant for us since it is internationally known and used among practitioners and academic communities alike. Studies have for example used ISO 25010 to investigate what software architecture should be used for different types of software development (Haoues et al., 2016), and how to evaluate the quality of specific software products (Idri, Bachiri & Fernández-Alemán, 2016; Izzatillah, 2019). ISO 25010 is also deemed a generic and broad software quality model that offers flexibility and customization to better fit practitioners’ specific needs (Azouz & Lefdaoui, 2018).

The model has been critiqued for not providing more guidance for the systems engineering aspect of software development. It thereby neglects a large part of the development as a whole (Schneider & Berenbach, 2013). However, both models found within ISO 25010 are categorizations that describe different ways to ensure the overall quality of a product. If the models are used as checklists, it really does not matter if they cover all areas of the development (Wagner, 2013). In this study we will therefore be using ISO 25010 as a checklist to assess what aspects of design quality are prominent in software development. By using ISO 25010 we intend to increase the chance that our focus harmonizes with already established standards within the industry. Our interviews will consequently yield more relevant information.

2.5.2. Components of the Product Quality Model in ISO 25010

ISO 25010 consists of eight top-level characteristics: functional suitability, performance

efficiency, compatibility, usability, reliability, security, maintainability and portability. All

characteristics also contain multiple sub-characteristics to help define it. The characteristics and their respective sub-characteristics are presented in Figure 1. This model is used on

(14)

9 software and computer systems, and its characteristics relate to different properties of a system (ISO/IEC, 2011). These characteristics are the basis for evaluating the software quality, and they are used to give a “comprehensive quality evaluation” (Hovorushchenko, 2018, p. 66). Haoues et al. (2016) argue that software quality characteristics are important to include throughout the whole system development life cycle, since they affect the design quality of the product. It is also relevant to identify the relationships between the different characteristics, since they affect each other in different ways. Sometimes there is a need to prioritize one over the other (Haoues et al., 2016). Security is, for example, implied to have a strong connection to usability. High security practices, like demanding users to have long and complicated passwords, could result in users finding it annoying and thereby lower the usability aspect of the software (Kashfi et al., 2017).

Figure 1. The Product Quality Model in ISO 25010 with its top-level characteristics and sub-characteristics.

2.5.3. Functional suitability

Functional suitability covers how well the system is providing the functions that are requested, and if the functions are performing what they should. It is all about the functions in the system, and how the functions in question are developed in relation to what is required of the system (ISO/IEC, 2011). The use of TDD can improve a system’s functionality, since it lets developers focus on functions one at a time, and therefore have the time to develop them properly (Nanthaamornphong & Carver, 2017). Functional suitability can be seen as a grade of how sufficient the components are, rather than being an either-or situation. There is not always a need for a component to do 100 percent of what is required. The component can still be deemed sufficient for what is needed from it (Kessel & Atkinson, 2016).

2.5.4. Performance efficiency

Performance efficiency represents how the system is performing when it comes to response and processing times when its functions are used. This performance is in relation to the stakeholders and how the resources answer to the requirements (ISO/IEC, 2011). Reaching a

(15)

10 high level of performance efficiency is a demanding task and something that is not easy to accomplish. This is due to many factors that contribute to the overall performance of a system. With the adoption of a more cloud-centric approach to developing services and software, the importance of planning for performance efficiency solutions early on in the development life cycle has increased (Alwadi, Nahhas, Bosse, Jamous & Turowski, 2019).

2.5.5. Compatibility

Compatibility handles information exchange between systems and components, and how well a system can perform while sharing hardware and software environments with other systems (ISO/IEC, 2011). Interoperability, which is one of two sub-characteristics of compatibility, can be regarded as one of the most important fundamentals when integrating multiple systems. Different types of interoperability concerns are for example syntactic interoperability and

system interoperability. Syntactic interoperability approaches compatibility in the context of

data formatting through different data standards, like XML or JSON. System interoperability concerns heterogeneous systems where the use of open source, open standards for data exchange and prototyping are different practics that can positively affect the interoperability, and consequently the compatibility of the system (Vogel, Kurti, Milrad, Johansson & Müller, 2014).

2.5.6. Usability

Usability covers how well a system can be used by its core user, including how different goals can be reached with effectiveness and satisfaction. Factors that affect usability is the system's general ease of use, if the user interface allows fluent interaction for the user, and how the system handles user errors (ISO/IEC, 2011).

Kashfi et al. (2017) define user experience (UX) as how the user perceives the functionality and use of the system. They also mention two different types of attributes that UX consists of, and these attributes cover different parts of how the user is perceiving the system. Pragmatic

attributes handle factors like how useful and controllable the system is. Hedonic attributes are

more about providing stimulation and having the user experience certain feelings, like excitement and interest.

Kashfi et al. (2017) argue that UX in system development is different from usability. Their definition of usability is “removing problem, frustration, and stress (i.e., negative)” and that “UX is based on the idea that removal of dissatisfaction does not necessarily lead to satisfaction and pleasure” (p. 4). Since ISO/IEC (2011) defines usability as partly how well the system is used, for example reaching goals and satisfaction, it corresponds to what Kashfi et al. (2017) define UX as. Our comparison between the two sources of usability definitions from Kashfi et al. and ISO/IEC shows that these two definitions are not similar. We will, therefore, use UX as part of our definition of usability, since we are basing our study on ISO 25010’s characteristics and their definitions.

(16)

11 Usability is generally neglected during software development, since developers see functional issues being more important than quality issues. Developers also have limited knowledge about how to achieve usability and are more focused on the actual product than the usability of it (Kashfi et al., 2017). Caro-Alvaro, Garcia-Lopez, Garcia-Cabot, De-Marcos and Martinez-Herraiz (2018) assert that omitting usability in system development could lead to users having a hard time using the system, and therefore choosing to stop using it. It has become more important with usability, and users do not normally tolerate any setbacks when using software. To minimize the risk of poor usability, user-centered design can be used to support the work towards high usability. User-centered design “focus on the design process, from the point of view of user requirements and goals” (p. 2). It is more about meeting users’ expectations through design prototyping, rather than the software features themselves (Caro-Alvaro et al., 2018).

2.5.7. Reliability

Reliability represents how the system handles the functions that are required. Factors that affect a system’s reliability are how fault prone the system is regarding software and hardware, and if the system is performing as it should under specified conditions (ISO/IEC, 2011). A high level of testing can positively affect the system’s reliability since developers feel more confident in the functions and they work as they should (Nanthaamornphong & Carver, 2017). Unsatisfactory reliability in a system could have consequences for the user environment. This potential shortcoming is especially critical in safety and business systems, and it has generally become more complicated to develop systems with a high level of reliability (Mohammadnazar, Pulkkinen & Ghanbari, 2019).

2.5.8. Security

Security represents how the system is handling and protecting data. Factors that affect security are how well the system handles authorization within the system, and how the system hinders modification of data. Another factor is if the system has components that can be changed without changing other components (ISO/IEC, 2011). Security evaluations is something that is largely done during the later stages of a project. Usually these are done during the implementation phase of development, prompting revisions to the project's code whenever a security concern is found (Morrision et al., 2018).

2.5.9. Maintainability

Maintainability covers how the system is structured and adapted to be modified without it introducing errors. Factors that affect maintainability is how the components in the system are coupled to each other, and to what degree the components are available to reuse in other systems, or building of other components. The level and presence of testing is a factor that also affects maintainability (ISO/IEC, 2011). Moreover, refactoring existing code makes it easier to add new code into an already existing code base. By having refactorization as a concurrent part of the development the developers can get a better understanding of the code, which improves maintainability (Nanthaamornphong & Carver, 2017).

(17)

12

2.5.10. Portability

Portability covers how the system is adapted to being transferred to different environments, platforms and hardware. Other factors that affect portability is how easy it is to install the system, and if it can be replaced by another piece of software with the same purpose (ISO/IEC, 2011). The current emergence of cloud based computing has increased the importance of the concept of portability. For example, when server and computing power is outsourced to external cloud providers a company should easily be able to switch providers without having to make extensive changes to their code base. Potential costs of investing into a system that ultimately locks you into a specific service provider is also something that suggests the importance of planning for portability (Lenhard & Wirtz, 2016).

(18)

13

3. Method

3.1. Qualitative approach

We have chosen a qualitative approach in this study since we are interested in gaining insight into developers' use of different practices, and their thoughts about their own development in regard to design quality. Bryman (2012) pinpoints qualitative research being more of an unstructured approach where there is a possibility to get more details and meanings about a concept. Through a qualitative approach we could gain more information about design quality than what, for example, survey questions can inquire about. However, according to Bryman (2012) it is difficult to replicate a qualitative study, since it is unstructured in nature and that the researcher is being a component that affects the results. This will affect this study’s reliability and validity.

3.2. Semi-structured interviews

For our data collection method, we chose semi-structured interviews. Adams (2015) argues that semi-structured interviews are ideal for scenarios where there are questions that can and should be followed up with a how or why question. Bryman (2012) describes semi-structured interviews being a flexible approach where the interviewees all get the same questions, but depending on their answers, there could be follow-up questions that are not planned. Since we want to be able to get as much information about whatever the interviewee talks about, semi-structured interviews give us the possibility to ask eventual follow-up questions, or to clarify what we mean with different questions. Fully structured interviews are concluded as being counterproductive for our research question. This is because we are aware of design quality being a concept that may differ from developer to developer. We argue that being unable to clarify or ask follow-up questions would somewhat defeat the purpose of choosing a qualitative approach for this study.

Due to COVID-19 and the restrictions that Sweden is under during this study, we had to make the choice to perform the interviews via the internet. We used the video conference software

Zoom because of its ease of use, its popularity, and built-in recording feature.

3.3. Sample

We chose our developers by using purposive sampling. Purposive sampling is a non-probability approach where the researcher chooses their sample so that it is relevant for the research area. This affects the representativeness of the research, and it cannot be generalizable (Bryman, 2012). However, since we are doing a qualitative study, these factors do not affect our goals with the study. By emails, we contacted companies in Örebro that either was a software company or had an IT-department. To get a hold of actual candidates we contacted a manager at the company in question and asked if they had a developer interested in participating in an interview regarding design quality.

(19)

14 Three interviews with developers from different companies were performed. Since we only have three interviewees it can be deemed a small sample, and Bryman (2012) points out small samples affecting the external validity. A reason for having only three interviews was that most of the companies we contacted did not have the time to participate due to the situation with COVID-19. However, we do have a research question that seeks information, rather than being comparative. Therefore, fewer interviews are not a problem for this study.

One company was a smaller product-based company whereas the other two were departments that belonged to larger companies.We are aware the results from qualitative studies can not be generalized to the population, but Sánchez-Gordón and O’Connor (2015) mention that there are significant differences between smaller companies and larger ones. Companies with fewer than 20 employees have different priorities than those with more, and different types of companies also have different problems. These distinctions could be important for us to have in mind when analyzing the interview data. The interviewees’ profiles are presented in Table 1.

The criteria for the developers chosen was to have at least four years of experience in system development. Since Stevenson and Wood (2018) bring up experience being one major factor affecting design quality, we concluded that it would be ideal to interview people with more experience with design quality so we could get more in-depth answers. If we interviewed someone with only one year of experience, there could be topics the person does not know enough about, and design quality may not be something they are well versed in (Stevenson & Woods, 2018). Age and gender were decided not being important to the research question and therefore there was no consideration regarding those factors.

Interviewees Years of

experience Type of company Numbers of developers in the company

Developer 1 7 Product-based 10-50

Developer 2 12 IT department 50-100

Developer 3 22 IT-department 100+

Table 1. Summary of the interviewees’ profiles.

3.4. Structure of interview guide

Our interview guide consisted of 11 questions that covered how design quality is present in different areas in the development process. The interview guide is shown in Appendix 1. The interview questions are based on the eight characteristics and their respective sub-characteristics in ISO 25010. We wanted to make sure we covered all areas of design quality, and therefore we made the characteristics into themes. Based on their description we could establish one to two questions that we thought covered each characteristic. Maintainability was the exception that had three questions. It is a characteristic that can be argued to cover a broader

(20)

15 area than the rest, so we decided it was appropriate to add an additional question. We asked follow-up questions whenever the interviewee mentioned something that we wanted more details about, but we tried to not stray too far from the current topic. The order of the questions were organized so it would be natural to transition from one theme to the next. Before the more specific questions regarding the developer’s practices, we asked an open question about what the interviewee’s view on design quality was. This in accordance to Adams (2015), who mentions that the initial question in an interview can determine how comfortable the interviewee is.

3.5. Data analysis

To analyze our data we used a thematic analysis with predetermined themes. These themes are based on the eight quality characteristics of ISO 25010, since these are the focus of our study. A thematic analysis is done by coding transcriptions and grouping these codes into overarching themes (Bryman, 2012). Coding, in this context, is when a researcher reviews a transcription and labels relevant snippets of the text within. These labels describe the general content of the snippets and how they relate to the current focus of the study. This practice is about categorizing and organizing data, so that a researcher can find themes and indicators of a phenomenon (Bryman, 2012). When using coding, there is a risk of taking information out of context, and the “narrative flow of what people say” can be lost (Bryman, 2012, p. 578), so this is something that we will need to have in mind when analyzing our results.

We used the definitions of each quality characteristic in the ISO 25010 model to code the text within the transcripts from our interviews. These labels were then grouped together based on which quality characteristic they corresponded to. The interviewees talking about user interfaces were for example labeled and mapped to the theme “usability”, since the characteristic’s description includes that area Even though most of the labels did match the definitions adequately, there were labels that could not be mapped perfectly to a characteristic’s description. For example, whenever an interviewee discussed something of interest that indirectly related to something that was defined within ISO 25010, we examined the context in which the snippet appeared and sorted them to the relevant characteristic accordingly. Since we constructed the interview guide using the same quality characteristics as we used in the thematic analysis, we could anticipate where most of the themes would show up in the transcripts. This was not guaranteed however due to the exploratory nature of semi-structured interviews, and the labels helped us to keep track of the themes whenever an interviewee unexpectedly changed subjects or whenever different themes bled into each other.

When the thematic analysis was done we compared the results with what the previous research claimed in order to see if the developers attained design quality or not, based on how they approach the different areas that are listed in the ISO 25010 model.

(21)

16 The following four ethical aspects have been considered in this thesis.

3.6.1. Recording

To be able to process the data with a high level of detail we asked the interviewees for permission to record the interviews, so we could transcribe them afterwards. Recording interviews can make the interviewee feel nervous and they may hold back information (Oates, 2013). This is something we took in consideration when asking for permission to record. We made it clear to the interviewee that they can withdraw their consent by any time, and that the recordings were not to be shared outside of the study.

3.6.2. Evoking pressure

Not only could an interviewee hold back information because of the interview being recorded, there is an issue with questions that “may evoke pressure to give socially acceptable answers” (Adams, 2015, p. 497). Since we are asking people about their development, it was relevant for us to try creating questions that would not inflict feelings of being judged.

3.6.3. Anonymity

Anonymity is also an important factor to consider, and interviewees have the right to be anonymous during their participation (Oates, 2013). Since we asked questions regarding practices that are used in the interviewees’ workplaces, all the participants were anonymous, referred to by gender neutral pronouns. The companies were not disclosed. Our research question is not depending on any specific company, so this was not an issue for our study.

3.6.4. Choice of language used for interview

Since we are interviewing Swedish developers we chose to hold the interviews in Swedish. Adams (2015) mentions that communication can fail if people do not share vocabulary and lingo, so by choosing to do the interviews in Swedish we do not have to worry about eventual miscommunications or misinterpretations during the interview process. We, as the authors, and the interviewees all have Swedish as our native language, so doing the interviews in our shared language was preferred. Afterwards we could take our time translating the Swedish data into English and use it in the results part of the thesis. This translating factor could be an issue if we’re not aware of potential distorting of the meaning of what the interviewees said. Therefore, we were extra attentive to how we wrote about the data in English.

(22)

17

4. Empirical data and analysis

This chapter presents an analysis of the empirical data based on the Product Quality Model in ISO 25010, which is chosen as the main framework to analyze the research question presented in Section 1.2.

4.1. Functional suitability

When working with functions for a system all developers have certain acceptance criteria when the requirements are established. The quality of these criteria differs from situation to situation. Developer 2 said that sometimes there are already clear and settled directives, as well as images or interfaces showing how the customer wants it to look like. However, sometimes the customers do not really know what they want, and in that case a more in depth discussion is needed. Developer 2 said they are trying to not get stuck on details though, and instead work agile where discussion and work intertwines.

Developer 1 said they have different ways to collect requirements for the system they work on. Sometimes their company has events where some of their customers are invited to come test and discuss the system’s functionality. Through these dialogues they can collect wishes from the customers and get ideas of what functions need to be modified. These wishes and suggestions are added to a wishlist, or the development team backlog, and then the developers prioritize from there.

Considering what Suryanarayana et al. (2015b) express about software processes having a risk of negatively affecting design quality, it could be appropriate for developer 2’s development team to consider sitting down regularly and evaluate their agile approach. Especially since they have customers whose knowledge range from having a detailed plan to not knowing what they want. If this kind of evaluation process is present, developer 2 may minimize the risk of omitting quality practices. This is because Ghanbari et al. (2018) state that omitting quality practices is a problem when stakeholders are not included in the software practices per se. The way developer 1 and developer 2 collects requirements can also be connected to what Ghanbari et al. (2018) mention about stakeholders lacking knowledge about requirements. Instead of always trying to get the requirements right away, developer 2 intertwines work and discussions with customers through an agile approach. It can therefore be seen as working towards better overall design quality. Developer 1’s company had events where wishes are collected and discussed, and this could also help with getting relevant information from the customers. Developer 1 also mentioned that they get feedback and suggestions for requirements through their sales department, the developers themselves, and the company’s consultants that are out working at the customers’ own company. When it comes to the developers’ own suggestions, they are often not about the functionalities themselves, but more about how the code can be improved.

Developer 3 talked about how fitness for purpose is the way they see the functions. According to them, one can develop a system, but if that system does not do what is requested it does no

(23)

18 good. The system that they build needs to perform according to the requests made. They also work closely with the customers, where representatives are present during the whole development process. This becomes important when there are projects where larger blocks of functionality are developed, since it becomes a challenge to cover all requirements. To ensure that functions meet the customers’ requirements, checklists are also used:

There is a formal checklist for a project. Like we have these nineteen requirements. Then the test leaders sit and check these off if they are met, like now we have met 90 percent of the requirements, it cannot get better than this. (Developer 3)

Since Fitness for Purpose measures quality against how well the product fulfills its purpose (Harvey & Green, 1993) and IEEE (2017) defines software quality as how the system meets user needs and requirements, developer 3 could achieve design quality through having fitness for purpose as their viewpoint. Developer 3’s attitude could comply with what Kessel and Atkinson (2016) discuss about components not having to be fulfilled 100 percent to be deemed functionally sufficient. What developer 3 says about projects having larger blocks of functionality affects how well they can meet the requirements, agrees with what Nanthaamornphong and Carver (2017) says about TDD. Since TDD lets developers focus and properly work on one function at the time, developer 3’s development team could be affecting the design quality negatively when working with these larger functionality blocks.

4.2. Performance efficiency

Developer 1 employs different strategies to ensure that their software is working as efficiently as possible. Developer 2 and developer 3 are both working with web-based services and they describe a similar approach to measure the performance where their companies use different services to stress test their API and UI. They do this in order to find how well it performs in different environments, to find memory leaks and similar issues. Developer 2 mentions that even though these tests are done, the developers themselves rarely work directly with performance related requirements. It does happen occasionally however and when it does it is treated like any other requirement:

In the last project that I was in there would come queries from within the EU and they had to be answered within 10 seconds, that was the requirement. In this case we had to keep the response time within those limits. Our testers have different tools to help manage this, and in this case we used VPN’s to try response times from different countries, and a few times they just asked the client to make a query from their location to see if it was within the stated parameters. So performance requirements do happen. (Developer 2)

Developer 3 brought up that they try to out-source as many tests as possible, especially those that are more complex and demanding, as well as tests that are not used as frequently. The reason being that they tend to be expensive to license and difficult to maintain: “We employ the help of specialists very often. It is expensive to procure all the tools needed and difficult to find the required know-how to cover all the tests, so we usually out-source as much we can”.

(24)

19 Developer 1’s company builds their own proprietary software that is licensed to customers. These customers will choose whether to host the software themselves on their own servers or if they want the company to host it for them in a cloud computing solution. Developer 1 described that their initial work with ensuring product efficiency was often left to their QA department. They produce estimations on what type of hardware would be needed for the software to run given the client’s conditions and expectations. If the client would want a cloud-based solution the QA department would set up the necessary infrastructure. If they would instead want to host the software themselves the QA department would collaborate with the client to ensure a proper set-up. During this part they would recommend what type of hardware would be needed in order to have the system running according to the specifications. After this initial phase most of the situations that arise concerning performance and efficiency are reported by the clients. Developer 1 said that even though they try to give precise estimations it is up to the individual client to realize them. This occasionally makes it troublesome to troubleshoot perceived problems with the software:

Right now we have a client with performance issues, a pretty large client. Since they host the software themselves we went to their location and examined their hardware and found that it was grossly undersized. Then we planned a meeting and told them that we highly recommended them to get this [equipment] because if they did not provide the proper server environment, the code, the software, would not be able to run that type of load. (Developer 1)

Developer 1 described that if a performance issue could not be traced to hardware related problems they would investigate if it was a fault somewhere in the code that was causing it. If not, they would then study the database and try to find ways to optimize it.

A common theme is that none of the developers are working with performance based issues in their day-to-day work and that an encounter with said issues rarely occurs. When these issues do occur, they are approached in the same way as any other fault or bug, so that the issue might be remedied. This suggests that the company's strategies toward keeping a satisfactory system performance is sound. Having a QA-department handling performance issues can also be linked to what Alwadi et al. (2019) says about performance efficiency being a demanding task to work with. Since developer 1 has a QA-department for this kind of work, the developers can focus on other things. The QA-department may also have better resources to cover the different factors within performance efficiency. This could lead to better design quality.

Alwadi et al. (2019) suggest that it is especially important for cloud-based services to employ comprehensive strategies toward the handling of system performance requirements. The companies of developer 2 and developer 3 have solved this by choosing to stress test their respective systems before releasing it to a stakeholder. In developer 3’s case their company has even gone so far as to outsource some of the more demanding tests to outside parties. This suggests that proactive measures in handling performance related issues is handled at a budgetary level and thus considered to be of significant importance for the company.

(25)

20 All of the developers explained that they do not focus that much on issues regarding system compatibility. In developer 1’s case they explained that although they develop their software for different platforms they all connect to the same database using the same API. Within the system there are a number of different modules each covering a specific area of functionality, and all of these can access the data independently using different interfaces. Developer 1 said “you could say that the common denominator for everything, from subsystems and different projects is the database. It is a very data driven approach”. The details on how to connect to the database API and the general system architecture is decided by a system architect.

Developer 2 describes something similar. Their system is composed of many different modules that cover specific functionality within the system. Some of these systems communicate with each other, and some can only communicate with the database. The communication is handled through different web based API’s. Even though developer 2 and their team has some leeway in creating custom API’s if necessary, they follow established templates and directives. The overall system layout and structure, and how communication should be handled between different modules and subsystems, is decided by the system architects.

The experiences of the other developers are mostly shared by developer 3 as well. Their system is made up of many modules and subsystems that communicate with each other when needed and it is dedicated system architects that map how these lines of communication should be drawn. One significant difference is that developer 3’s architecture involves decades old systems. They said that a major portion of their core services is written in the programming language COBOL, and the system’s creation dates back to the 1970’s. In order to bring this system into a more modern setting they have converted the COBOL code into .NET syntax. The system is still using the design principles and idiosyncrasies of the COBOL language, but it is now accessible using the .NET Framework. Developer 3 also said that since the system has been live for so long it is still dependent on older architectural and design solutions. This has resulted in a system that includes a lot of subsystems that work as translators between the old and the new architecture. Another problem that they had to face was how to translate a system that was not built for the quick responses, which is what a modern system is expected to deliver:

They are not built as online systems, they are built for batch operations where you make a system call, the system writes the output to a file and gives you the results of the operation the next day [...] I cannot open a web page, ask for something and read a sign telling me that the answer will be delivered by the postman within the next 24 hours [...] There are challenges with using different systems. (Developer 3)

Developer 3’s system seems to have a high degree of compatibility seeing how they have worked with integrating decades old technology and code bases into more modern standards. They have been working with syntactic interoperability between the two systems since they have translated COBOL into .NET for their system. Vogel et al. (2014) state that syntactic interoperability is a prerequisite of reaching overall system interoperability, and consequently compatibility. This particular system could therefore be achieving design quality.

(26)

21 A pervading theme is that the process of ensuring good compatibility between the parts of the system is in large part removed from the typical developer. This task is instead assigned to a specialist system architect who drafts the schematics of the system and decides what components and subsystems to include and how these should interact. As systems grow and expand this seems to be an optimal solution with a few designers who manage the architecture and construct a lattice that other developers can build upon.

4.4. Usability

Developer 3 said that they try to make the systems user friendly, but they do not have formal guidelines for how they should implement functions in their internal systems. Developer 2 worked similar with the development as developer 3 when it came to guidelines on projects within the company:

There are pretty strict guidelines and rules at [company name]. As a company within the government sector we need to meet certain requirements regarding contrasts and accessibility in the services provided to the users. So in that area we have provided guidelines, colors and style. The internal projects though... There we have less restrictions. (Developer 2)

Developer 3 also said that when they are doing projects for customers it falls on the project manager to approve the design. When it comes to internal projects for the company itself it is the developers and testers that approve how the functions are presented in the user interface. In most cases this works immensely well, according to developer 3. Developer 3 said that even though they have specialists for the customer projects that make sure the developers follow certain guidelines, it is still hard to answer questions like “does it look good?” and “does it feel good?”.

From what Kashfi et al. (2017) say about pragmatic attributes, developer 2 could work towards good UX since they have guidelines regarding colors and style when it comes to their systems made for the public. Since they work within the government sector they said that they have strict guidelines, and therefore it could show how companies within the government sector strive for good UX, and consequently good design quality. At least regarding pragmatic attributes. In consideration to what developer 3 said about having specialists providing guidelines, but it still being hard to know how the user feels using the system, it could be linked to the hedonic values of UX. Since hedonic values are about the stimulation and feelings the user has with the system (Kashfi et al., 2017), it could show that developer 3’s development team is lacking in hedonic values, but having specialists covering the pragmatic values. It seems that design quality in the sense of UX and its attributes are not fully achieved by the developers.

Two of the developers said UX is somewhat lacking in their work. Developer 1 said that the project managers had the responsibility to collect the requirements from the customer and create a requirement specification from that. Often it is the project managers that need to decide how the system should look like. However, sometimes the developers get the responsibility to

(27)

22 implement the functions however they see fit. Developer 1 said that it would be much more optimal if there could be a designated UX-team that handled that part of the development, so the project managers could focus on other things.

Developer 1 brought up wireframes, a low-detailed design to show the structure of a function, and said that it can be used to make sure the developers cover all areas. It is, however, something that is lacking without a designated UX-team. Developer 1 said that this circumstance affects the finished product in many ways:

You may release something, market it a lot, and the customers buy it and are happy about it... or happy that it existed at least. But then when the users actually used the system they noticed they could not even use it. We then had to start a new project to build the system again, from the ground up, and it took years before we could release the system again. This whole process ended up taking double the time and resources. (Developer 1)

What developer 1 said about developing systems that in the end cannot be used by the intended users is in accordance to what Caro-Alvaro et al. (2018) says about omitting usability. This leads to users having difficulties using the system in question. Since developer 1’s development team does not have a designated UX-team they feel like they lack some practices that can help them with structuring the system. Caro-Alvaro et al. (2018) brings up user-centered design, which could be a way for developer 1’s development team to improve on their systems’ usability.

Developer 1 said that the lack of an UX-team results in unqualified people getting that responsibility. Developer 1 continued to say that there sometimes is not a dialogue between the development team and the customer regarding implementations of functions. If a function is requested by multiple customers, the development team may just create a template and then develop it without any feedback. Developer 1 said it is not because they do not care about the customers, but rather that they do not have the time or priority to focus on the UX and user interface.

There was an expressed concern from two of the developers about the lack of knowledge about UX in the development, and this could be in accordance to what Kashfi et al. (2017) mention about developers having limited knowledge about achieving usability. This factor is prevalent when it comes to developer 1, who said there have been feedback from customers about systems not being usable at all. Together with wireframes, which developer 1 mentions, and design prototyping, which is a way to meet users’ expectations (Caro-Alvaro, 2018), developer 1 could achieve good design quality. However, it seems like developer 1 focuses on developing the functional requirements and doing them well, which is something Caro-Alvaro (2018) implies is the opposite of having a user-centered approach. This focus can be linked to what Kashfi et al. (2018) say about developers focusing more on the actual product rather than the usability of it.

(28)

23

4.5. Reliability

When describing how they work with ensuring that their software works as intended and is as fault free as possible developer 2 and developer 3 share similar practices. Both mention that they work in agile teams where initial testing is done internally within each team. This internal practice is handled differently though. Developer 2 handles testing by handing it over to any other team member whilst developer 3 has a dedicated tester in each team.

In developer 2’s case there were not any established protocols to follow but the intention is “to break the code”. They said that even though each test case is documented, the quality of said documentation is highly dependent on who is doing the testing. There are also built-in automated tests within the code base that can be used should a tester be so inclined, and the expectation to write their own tests if deemed necessary. If a bug or faulty feature is discovered during testing the tester gives feedback to the rest of the team, and if the faults are minor they will try to correct it promptly. Developer 2 said that since they work according to Scrum and work in three weeklong sprints, the discovery of a bug can have different outcomes depending on where in a sprint they are:

If you find a bug on day two of a sprint you try to correct it during day three and usually in a more informal way, but if you are nearing the end of a sprint and you need more time to root it out you will need to document it and plan for the fix to be included in later sprints. (Developer 2)

This documentation is supposed to follow a standardized template but “no one is using them”. Instead developer 2 said that they usually indicate the fault by taking screenshots and writing what is wrong. However, in cases involving more serious and persistent faults the documentation tends to be more formal. When the piece of software is deemed finished a representative for the stakeholder can test it again to see if it fits its specifications.

Developer 3 described their testing process as follows: The developers write their own tests and check to see that their product is working. The function is then turned over to the team’s dedicated tester who performs their own tests. In larger projects there is also the option of using automated tests constructed by the company’s lead testers. If faults are found the function is returned to the developers who then tries to fix it and resubmit it for further testing. If no faults are found the product is then demonstrated to the stakeholder who either accepts or rejects it. Assuming the function is accepted by the stakeholder, it is then implemented in a pre-production environment and submitted to more testing. When deemed stable enough it is transferred to the live production environment.

The testing process at developer 1’s company is similar to those of developer 2 and developer 3 but the most significant difference is that the company employs a dedicated quality assurance (QA) department. Their process is as follows: A developer tests their own functions before shipping it to the QA department, who then runs further tests before releasing it to the client.

(29)

24 If the function is deemed faulty it is returned to the developer, which after repair is resubmitted to the QA department for testing, and so on.

Developer 1 said that they rate having reliable products very high because fixing faults and errors in deployed software is costly in both time and resources that could be better spent developing new features. They also made a comparison of before and after their company employed the QA department:

The difference is huge. A big difference is that we have as many faults today as before, but that is probably tied to that we have QA that works full time with nothing but testing. [...] Earlier most of the errors were reported by the clients but now they are discovered by our QA department instead before it has a chance to reach the clients. [...] The greatest change is that before we had a dedicated QA [the clients] found one or two critical errors per week. Now it is down to once a month. (Developer 1)

They describe critical errors as something that makes the software nigh unusable, like being unable to log in, or the wrong salary being paid to an employee.

Ensuring the reliability of their product is something that is highly rated among all three interviewees. As each new function or feature is finished, it is passed through a testing process that ensures that the submitted code behaves as it should. Developer 1’s case points to the importance of ensuring a high degree of reliability in a product. They described how the discovery of critical bugs have transitioned from being reported by their users to instead being found by their own QA department. This has greatly reduced the amount of critical bugs that reach the live environment. Mohammadnazar et al. (2019) point to the complicated task of developing a complex system whilst ensuring a high degree of reliability. Since the consequences of these critical errors negatively affected the user experience of developer 1’s system, the choice of employing a QA department proved to be an optimal solution to increase design quality. The department could focus solely on developing methods for testing, while freeing up the developers to continue working on their project.

Even though the importance of proper testing seems to be formally recognized by the developer’s companies, the execution of said process differs between the three. Although developer 1 tests their own code, the more formal process is left to the QA department. Developer 2 however, seems to approach the testing and reliability concerns in a more informal way where the testing is arbitrarily assigned to a team member and having more lenient attitudes towards formal documentation. Developer 3 describes a more formal approach with dedicated testers in each team. Even though developer 2 and their team had a more relaxed approach towards testing they did not describe any negative effects resulting from this, and they claimed to be able to differentiate between what cases needed a more formal approach and what cases did not. Comparing developer 2’s situation to that of developer 1 and developer 3, one could argue that developer 2 and their team tread the path of being generalists where every developer needs to fill the role of tester. Thus being forced to invest time into that area in addition to product development. Developer 1 and developer 3 however, can focus solely on

Software quality characteristics and where to find them : A qualitative study of how developers in Örebro work with design quality