NBAP message construction using QuickCheck

(1)

using QuickCheck

Andreas Granberg and

Daniel Jernberg

Master Thesis Project,

Kista, 2006, 2007

(2)

using QuickCheck

Andreas Granberg and

Daniel Jernberg

LITH-IDA

2007-06-18

Master Thesis Project,

Supervised by Graham Crowe.

Examined by Mariam Kamkar

Kista, 2006, 2007

This thesis presents the work done by the two

undergraduate students Daniel Jernberg and

Andreas Granberg at Ericsson Traffic and Feature

SW. The thesis project was to integrate

Quick-Check as an automated tool for software

verifica-tion into the existing test framework. This thesis

presents the purpose, methods, choices made

and the results of this work.

(3)

A

BSTRACT

Traffic and Feature SW is a department based in Kista. At this department the main processor software, or MPSW in short, in Ericssons Radio Base Stations is tested prior to integration of new releases. Traffic and Feature SW, also called MPSW in this thesis, works closely with another department of Ericsson called RBS System I&V which tests the same software but for complete RBS nodes. MPSW uses black- and greybox scripted testing in regression suites that are executed on a daily basis. These regression suites are separated into different subsets of functionality that maps to the capabilities of the Radio Base Station software. The authors of this thesis has performed an imple-mentation of automated test cases for a subset of the Radio Base Station software using an automated software tool called QuickCheck. These test cases were successfully inte-grated into the test framework and were able to find several issues with the main proc-essor software and its subsystems in the Radio Base Station. The commissioner of this thesis have plans to further integrate the QuickCheck tool into the test framework, pos-sibly automating test cases for several subsets of the Radio Base Station software. The authors have therefore analysed and put forth metrics that compares the testing of the Radio Base Station software using QuickCheck with the conventional regression test cases. These metrics covers areas such as the cost related to and the inherent capabili-ties of QuickCheck. The evaluation of these metrics was performed by the authors to give the commissioner decisive information. These evaluations showed that Check was able to generate complex message stuctures in complex sequences. Quick-Check was also able to shrink both the content of these messages and the length of the sequences of messages to be able to provide a minimal counterexample when a fault was discovered. The only metric that QuickCheck failed to support was to inherit func-tionality to support the handling of statistics from executions.

(4)

S

AMMANFATTNING

Traffic and Feature SW är en avdelning av Ericsson som är baserad i Kista. På denna avdelning testas mjukvaran på huvudprocessorn och dess subsystem i Ericssons radio-basstationer innan nya releaser av mjukvaror för dessa. Avdelningens arbete koordine-ras med en annan avdelning i Kista som kallas RBS System I&V som jobbar med liknande testning men mot riktiga radiobasstationer. Traffic and Feature SW, även kal-lad MPSW i denna rapport, jobbar med regressionstestning i en blandning av svartlå-demiljö och grålåsvartlå-demiljö. Dessa regressionssviter körs varje natt mot ny mjukvara i radiobasstationerna från vilka resultat analyseras av utvecklarna av testfallen dagen efter. Dessa regressionssviter är indelade i mindre sviter baserat på vilken delmängd av funktionaliteten i radiobasstationsmjukvaran som testas. Författarna av denna exa-mensarbetersrapport har under detta arbete genomfört en implementation av automa-tiserade testfall för en delmängd av funktionaliteten i radiobasstationen genom att använda ett nytt verktyg för automatiserad testning vid namn QuickCheck. Dessa test-fall integrerades in i testramverket på ett önskvärt sätt och lyckades hitta flertalet intressanta frågetecken med mjukvaran i huvudprocessorn och dess delsystem i radio-basstationerna. Eftersom uppdragsgivaren har fortsatta planer på att använda Quick-Check som testverktyg i sitt testramverk i större utsträckning har författarna även analyserat och postulerat metriker för att ge möjlighet att jämföra testfall av denna typ med testfall av den konventionella regressionstestningstypen. Dessa metriker täcker områden såsom kostnader och duglighet hos testverktyget. En utvärdering av dessa metriker genomfördes också av författarna för att ge uppdragsgivaren beslutsstöd-jande information om utfallet av dessa metriker för QuickCheck. Dessa evalueringar visade att QuickCheck ger möjligheten att generera komplexa meddelandestrukturer såväl som komplexa sekvenser av sådana meddelanden. QuickCheck har också funk-tionalitet för att krympa sådana meddelanden och sekvenser till minimala motexempel när ett fel upptäckts. Den enda metriken som QuickCheck ej lyckades uppfylla var att ge användaren funktionalitet vad gäller möjligheten att kunna hantera statistik från körningar.

(5)

A

CKNOWLEDGEMENTS

We would like to show our appreciation to the following people that made this the-sis work possible and that have helped us in so many ways.

We would like to thank the executives, Erik Backlund, Mike Williams and Roger Holmberg, at the department for giving us the opportunity to perform this thesis work.

We would also like to thank our company supervisor Graham Crowe for his exper-tise in the area as well as his effort to give us all the valuable information needed to be able to perform the thesis work.

We would also like to show our appreciation to our examiner, Mariam Kamkar, for her support and professional approach.

We would also like to thank John Hughes and Thomas Arts for their lectures on QuickCheck, their interest in our work and the great support they have provided for us during the thesis work.

We would also like to thank all the employees at the department. Their accommo-dating manners and professionalism as well as their interesting and funny discussions have helped us maintain a healthy approach to the work being performed.

Finally we would like to show our appreciation towards our friends and families for them putting up with us during these intense months.

Andreas Granberg, Kista, 2007 Daniel Jernberg, Kista, 2007

(6)

T

ERMS AND DEFINITIONS

• 3GPP, 3:rd generation partnership program: A collaboration agreement that brings together a number of telecommunications standards bodies to provide standards and specifications.

• ASN1, Abstract syntax notation one: A standard notation in specifications that describes data structures for representation of encoding, decoding and transmitting data.

• ATM, Asynchronous Transfer Mode: The data link layer protocol describing end-to-end logical connections where fixed cells are transferred.

• BP, Board Processor: The processor on each interface board in the RBS responsible for handling the board signalling.

• Bp, The Bp interface: The interface on which the Main processor communicates with the BP on the interface boards.

• CCM, Cell configuration Management: The common and dedicated procedures speci-fied into a functionality suite by 3GPP for Cell Configuration Management.

• CDMA, Code division multiple access: The method of transmission in the WCDMA network using shared channels.

• CR, Change Request: The Change Requests written in response to issues discovered with the Radio Base Station software or the test framework.

• CBR, Capability Report: A report extracted from the model to reflect a capability of the SUT.

• CRNC, Controlling Radio Network Controller: The mode of the Radio Network Con-troller in which it is responsible for maintaining control of connections.

• CTCM, Common Transport Channel Management: The common and dedicated proce-dures specified into a functionality suite by 3GPP for common transport channel management.

• EP, Elementary Procedure: The basic NBAP procedures for calls against the SUT. • EUL, Enhanced Uplink: The new feature of WCDMA that provides an enhanced

uplink for faster end user performance.

• HSDPA, High Speed Data Packet Access: The new standardized evolution of WCDMA that will enable downlink speeds of up to 14 Mbps.

• IE, Information Element: The most elementary information elements in an NBAP EP message.

• IFHO, Inter Frequency Handover: A handover of a UE communication between differ-ent RAN frequencies.

• Iub, The Iub interface: The interface defined by the NBAP protocol which connects the Node B and the RNC logically.

• MBMS, Mobile Broadcast/Multicast Service: The RAN feature regarding broadcasting and multicasting to UEs.

• MO, Managed Object: The Managed Objects in the SUT used to reflect the internal state of the SUT.

(7)

• MPSW, Main Processor Software: The software for the Main Processor in the RBS, which the implementation attempts to test.

• NBAP, Node B Application Protocol: The protocol which the RNC adheres to in its communication with the RBS.

• Node B: The 3GPP name for the base station in the WCDMA RAN architecture. • PDU, Elementary Procedure Definition: The definition in the NBAP specification of an

EP’s content.

• RAN, Radio Access Network: The telecommunications network architecture between the handset and the core network.

• RBS, Radio Base Station: The Radio Base Station is the unit in the RAN on the Radio Network Layer that performs the main connection handling with the UE. The Radio Base Station is controlled by the RNC.

• RLM, Radio Link Management: The 3GPP subset of functionality for handling radio links.

• RNC, Radio Network Controller: The Radio Network Controller is the unit in the RAN responsible for controlling the RBSs in the RAN and providing connection to the core network.

• SRNC, Signalling Radio Network Controller: The mode of the Radio Network Control-ler in which it is responsible for the transmission between handsets and the core network.

• SUT, System under test.

• TR, Trouble Report: The Trouble Reports written in response to found issues with the Radio Base Station software or the test framework.

• TSG, Technical Specification Group: The groups in 3GPP responsible for suggesting, maintaining technical specifications for 3G and GSM networks.

• UE, User Equipment: The user equipment in the RAN, usually a cell phone.

• UTRAN, Universal Terrestrial Radio Access Network: The communications network commonly known as 3G.

• Uu, The Uu protocol: The protocol used by the RBS to transmit and receive data from a User Equipment.

(8)

T

ABLE OF

C

ONTENTS

1 I

2.1 I

NTRODUCTION TO

WCDMA . . . 3

2.2 I

UB SPECIFICATION

. . . 8

2.3 R

ADIO

L

INK

M

ANAGEMENT

. . . 9

2.4 E

RLANG

. . . 11

2.5 T

EST

S

ETUP

. . . 12

3 T

S

PECIFICATIONS

. . . 15

3.7 S

PECIFICATION BASED TESTING

. . . 16

3.8 A

UTOMATED TESTING

. . . 16

3.9 R

EGRESSION TESTING

. . . 16

4 Q

UICK

C

HECK

4.1 P

ROPERTIES

. . . 18

4.2 G

ENERATORS

. . . 19

4.3 S

TATE MACHINES

. . . 20

4.4 S

HRINKING

. . . 23

4.5 C

OUNTEREXAMPLES

. . . 25

5 A

NALYSIS

5.1 C

HOOSING FUNCTIONALITY TO TEST

. . . 27

5.2 E

LEMENTARY

P

ROCEDURE DEPENDENCIES

5.9 S

TATISTICS

. . . 30

5.10 G

ENERATION

-

TIME AND RUN

-

TIME

. . . 30

5.11 F

INDING METRICS

. . . 32

6 I

MPLEMENTATION

6.1 I

NITIAL DEVELOPMENT

. . . 34

6.2 M

ODULARISATION

. . . 34

6.3 G

ENERATORS

. . . 37

6.4 G

ROUPING OF FUNCTIONALITY

. . . 38

6.5 E

VOLUTION OF

S

TATE

. . . 39

6.6 S

YMBOLIC CALLS

. . . 40

6.7 R

ETURNING PROPER VALUES

. . . 40

6.8 S

TATISTICS

. . . 41

6.9 S

IGNAL

V

ERIFICATION

E

STIMATES FOR

MBMS

IMPLEMENTATION

. . . 46

(11)

L

IST OF

F

IGURES

F

IGURE

2-1:

T

6-1:

C

ODE BASE EVOLUTION

. . . 35

(12)

O

UTLINE

In the first chapter the reader will be introduced to the assignment that this thesis is based upon, the task as presented by the commissioner, the purpose, goals as well as the limitations.

In the following chapter the reader will be introduced to the basics concepts of Uni-versal Terrestrial Radio Access Network for Wideband Code Division Multiple Access and especially the subset of the Radio Base Station functionality that the thesis work is based on. This chapter will also describe the standardized interfaces in play and how these have come to affect the thesis work.

In the third chapter the reader will be introduced to the general concepts of testing that are necessary to be able to comprehend the subsequent chapters that discusses and relates the methodologies used in this thesis work to common testing methods.

In the fourth chapter a fairly detailed description of QuickCheck will follow in which the reader will be presented with enough information about the software tool to be able to comprehend the choices being made while implementing the test cases for the thesis work.

The fifth chapter contains an analysis of the problems that we were presented with during the thesis work. We will in detail describe the problems and will argue why the chosen solution is the preferred one.

In the next chapter the reader will be able to follow our implementation of the issues described in the analysis chapter. The reader will also be able to understand the incremental growth of the code base for the implementation and the decisions being made with regards to that implementation.

In the seventh chapter we will describe the metrics put forth that are needed to be able to compare testing using QuickCheck with conventional regression testing. We will in this chapter argue whether or not these metrics are of value or not.

In the eight chapter we will follow up with an evaluation of these metrics and, in the ninth and final chapter, the reader will be able to discover our conclusions about the thesis work and the end result.

(13)

1 I

NTRODUCTION

This chapter will introduce the reader to the thesis work assignment. What the pur-pose of the assignment was, the main task as presented by the commissioner, the limi-tations, the goals as well as the deliverables will be described in detail.

1.1 P

URPOSE

Comissioner of this thesis work is Traffic and Feature SW, a division of Ericsson, based in Kista, Sweden. This division of Ericsson is responsible for the design, systemi-sation, integration and verification of the software in Ericsson’s RBS’s. This division is separated into two smaller departments, one in Kista and one in Umeå which works in parallel to systemise, test and verify the software for the RBS. These two departments also works closely with another department called RBS System I&V which performs similar testing but on actual machines and not in the simulated environment that the aforementioned department does.

As of today the testing of the MPSW at Traffic and Feature SW is performed using scripted regression test case suites that are run during nights at the department. The nature of the test environment will further be explained in a later section of the thesis called “Test Setup” on page 12. In this regression testing process the design of test cases is based on use cases and functional specifications which often covers test cases in a one-to-one fashion without much code reuse which makes the scripted regression test suites quite repetitive and hard to maintain.

By using QuickCheck as a tool to automatically generate the elementary proce-dures, or EPs in short, from the Node B application protocol, NBAP in short, and sequences of NBAP EPs it might be possible to better model the system under test and be able to automate a greater deal of the testing being done today. By this, QuickCheck might be able to provide a richer variety of parameterization of these procedures, in theory increasing the possibility of finding faults that were not possible to find, prior to using QuickCheck.

Traffic and Feature SW is therefore interested in a evaluation of QuickCheck to receive more information about how effective and cost efficient an integration of this tool can be in their test environment.

1.2 T

ASK

We were given the assignment to implement test cases for a subset of the MPSW functionality using QuickCheck and to integrate the resulting test cases into the test framework. By doing so the thesis work would prepare for a possible deployment of QuickCheck in the testing framework as well as provide best practice methods for dealing with issues that might occur during such a deployment.

As part of the task we were to identify metrics to support the comparison between test cases implemented using QuickCheck and the scripted test cases used in the regression suites. The thesis would result in an evaluation of these metrics which should be able to provide the commissioner with enough information to be able to make an informed decision about whether or not to go forth with further integration of QuickCheck as a tool for testing several subsets of the RBS software.

1.3 L

IMITATIONS

The thesis work included implementing test cases using QuickCheck as a tool in the scope that was necessary to be able to provide a basis for the best practice document that was to be provided as a deliverable of the thesis work. The best practice report is more thoroughly described on page 2. This means that the subset of the functionality of the RBS that is the basis for the implementation using QuickCheck in no way needed to

(14)

be covered fully by our implementation. Only that the implementation as such covered enough of the functionality to be able to provide the metrics as well as providing enough knowledge about test case development using QuickCheck to be able to pro-duce the best practice document. This also meant that it was not implied that further existing functionality that could be covered by test cases written using QuickCheck needed to be covered.

1.4 G

OALS

This subsection describes the goals that were established as part of the thesis guide-lines. These goals have been established in coordination between us, the company supervisor and the company section supervisor. The goals are that we, after completion of the thesis work:

• Have delivered a best practice report that is sufficiently extensive to give Ericsson a proper guide for transititioning their test-environment towards using QuickCheck for testing subsets of the MPSW functionality.

• Should be able to observe whether the result of test cases implemented using Check have equal or greater test coverage than earlier test cases not using Quick-Check.

• Have gained enough knowledge about QuickCheck to be able to integrate it in the test suites given by the supervisor.

1.5 D

ELIVERABLES

Apart from this thesis, the work also resulted in a number of deliverables, internal to Traffic and Feature SW, these will be described below.

1.5.1 B

EST PRACTICE DOCUMENT

The analysis section of the thesis describes particular important issues when using QuickCheck to implement automated test cases for complex system testing. Several of the solutions to these problems and other commonly occurring issues have been for-malised into a best practice document that we will provide the commissioner after completion of the thesis work. These solutions also provides the reader with code examples and use cases that can help him or her in their work when using QuickCheck.

1.5.2 I

MPLEMENTATION

The result of the implementation being done during the course of this thesis is in part included in this thesis to provide important complementary code for the discus-sions. Apart from this, all code will be delivered to the commissioner after the comple-tion of the thesis work.

For confidentiality reasons the full implementation will not be included but the implementation excerpts and intellectual property included is sufficient to cover all aspects of the implementation discussed in the subsequent sections.

(15)

2 C

ONTEXT OF APPLICATION

This chapter introduces the reader to the technologies and concepts that are rele-vant for this thesis work. The reader will also be made aware of the larger picture, that is, the bigger system around which the rather focused subset of functionality this thesis work take place.

2.1 I

NTRODUCTION TO

WCDMA

Upon trying to understand the basis and implications of the thesis work it is of interest to identify the areas of telecommunications that this thesis work correlate to and is dependent upon. Therefore this chapter focuses on the architectural aspects of WCDMA from a larger perspective after which we will narrow down upon which sub-set of the telecommunications systems the thesis work is related to as well as the stand-ards established for describing this subset.

2.1.1 3GPP

The 3GPP, short for third generation partnership program, is a collaboration agree-ment that was met december 1998 [15]. 3GPP’s initial focus was to bring together a number of telecommunications standards bodies and their main focus was to function as the WCDMA specification body which meant that they should work to provide glo-bally available reports and technical specifications for the third Generation Mobile Sys-tem. These were based on the evolved Global System for Mobile Communication, or GSM in short, core networks and the radio access technologies that they support (i.e., Universal Terrestrial Radio Access (UTRA) both Frequency Division Duplex (FDD) and Time Division Duplex (TDD) modes). The 3GPP committees are divided into sev-eral groups of which the TSG groups are of most importance for this thesis. The TSG groups have as their responsibility to prepare, approve and maintain the specifications for the 3G and GSM network functionality. The technical specifications provided by 3GPP standardizes the interfaces used in telecommunications products which there-fore have a large impact on all aspects of Ericsson’s telecommunications systems. Eric-sson have officials that are part of the impact on these specifications and also internal TSG groups to determine Ericssons position on planned specifications as well as deter-mining the internal impact.

(16)

2.1.2 B

ASIC

WCDMA A

RCHITECTURE

As we can see in the picture below, the WCDMA Radio Access Network is con-nected to the Core network to provide radio connections to the User Equipment [11]. The Radio Network Controllers, or RNCs in short, are responsible for controlling the Radio Base Stations, or RBSs in short, which handles the connections to the handset. The RNCs communicates with the RBSs using the Iub interface and with each other using the Iur interface. In the WCDMA network RNCs are connected to the Core net-work using the Iu interface. Furthermore, the Uu radio interface is used by the RBSs to transmit and receive information from the User Equipment, or UE in short.

Figure 2-1: The WCDMA architecture

CORE NETWORK WCDMA RAN RNC RNC RBS RBS RBS RBS Uu Iub Iub Iur Iu Iu

(17)

2.1.3 R

ADIO

N

ETWORK

C

ONTROLLER

The RNCs as nodes in the RAN have two distinct roles, to control or to serve [11]. The serving RNCs are the RNCs that are responsible of connecting the subscribers handsets to the core network, thus providing the radio connectivity to the core net-work. The controlling RNCs takes on the role to control a number of cells in the RAN and their associated RBSs. In this case which is the most important "mode" for our the-sis the RNCs maintains the resources in the network and controls the requests and con-firms being sent between the serving RNCs and the controlling RNCs in the network. This is the type of operation that provides the possibility to perform softer handover of handsets between different cells in the RAN.

Figure 2-2: RNC roles in the architecture

2.1.4 R

ADIO

B

ASE

S

TATION

The RBS is the node in the RAN that is responsible for handling the radio transmis-sion to and from the handset. RBSs come in hundreds of different configuration types ranging from the smallest indoor micro RBS 3303 that provides micro cells in the net-work to the largest outdoor macro RBS 3206 which provides high coverage and capac-ity at the radio access level [12]. The transmission of the RBSs is performed using the Uu protocol over which different types of connections can be established and removed depending on a lot of factors such as the handsets location, subscriber needs, number of users connected and more. This is closely related with the concept of Radio Access Bearer types, or RAB types in short, which will be described in a following section. The RBS is controlled by its controlling RNC using the Iub interface on which the RNC can issue elementary procedures to maintain just about everything in the RBS, this includes for example power control, connections to handsets, cell control and several other areas of functionality. One Radio Base Station can control one or more cells in the net-work. CORE NETWORK WCDMA RAN Controlling RNC Serving RNC RBS RBS RBS RBS Uu Iub Iub Iur Iu Iu

(18)

2.1.5 H

ANDOVER

Handovers are what makes a mobile telephony network mobile. A handover is required when the UE is moving between cells while communicating as depicted in figure 2-3.

In the seamless network different type of handovers are at play [11]. One such type of handover is called inter system handover where the handovers can be from GSM to WCDMA depending on the load of the system or the other way around. This type of handovers are regulated using service differentiation and load control based on poli-cies, load control mechanisms or based on the type of subscription of the subscribers.

Figure 2-3: Handover between cells

2.1.5.1 S

OFT AND SOFTER HANDOVER

Soft and softer handover enables the UE to be connected to multiple cells and possi-bly to multiple base stations [11]. This enables the UE to travel through the network form one cell to another without any apparent loss of connection. While soft handover means a transition of the UE from one cell in one base station to another cell in another base station, softer handover takes place within the different cells of one single base station as depicted in figure 2-4. To be able to perform a soft or softer handover the cells are required to use the same frequency.

Figure 2-4: Soft and softer handover

RBS _Cell1 _Cell2 RBS

RNC

(19)

2.1.5.2 I

NTER FREQUENCY HANDOVER

Inter frequency handover, or IFHO in short, is required when a handset is moving out of coverage of one RAN frequency and needs continuation of service on another RAN frequency, which means that the cells have different WCDMA carrier frequencies [11]. This means that a new connection will be set up in the new cell allowing for con-tinuation of communication as in the soft- and softer-handover cases.

2.1.6 C

HANNEL TYPES

In the WCDMA network there exist different types of transmission channels for which the subscriber can transmit information depending on the type of information and the content type the subscriber needs to transmit [11]. The two basic types of trans-mission channels are called common channels and dedicated channels. The dedicated channel is used when the subscriber is currently transmitting high loads of data such as in a voice conversation or when downloading data from a web page whereas the common channel is used when transmitting lower loads of information. There exists channel type switching for this as well in which the subscriber is switched between the two types of channels depending on type of information being transmitted. This is of lesser importance for this thesis but the subject of channels will be mentioned in a sub-sequent chapters in this thesis. The dedicated channels are of particular interest for this thesis since the thesis work are part of testing the establishment and removal of such channels between the handsets and RBSs.

2.1.7 R

ADIO ACCESS BEARER TYPES

One important service of the WCDMA network is its RAB types that carry the sub-scriber information between the handsets and the core network. A RAB type is a pre-defined set of services which determines the properties of the uplink and downlink set-tings of the connection on the Uu protocol as well as properties related to the dedicated channels. This includes information about the properties of the transport settings for the uplink and downlink for the radio link as well as several important parameters concerning the dedicated channels to be used for that RAB type.

There is as of today four different RAB types for the WCDMA network of which all are of importance for the thesis work [11]:

• Conversational: A Conversational RAB that is used for voice telephony • Streaming: A streaming RAB that is used for streaming data such as video

• Interactive: A interactive type of RAB that is used for web surfing and similar activ-ities of the end user

• Background: A background type of RAB that is used for transferring files in the background of the other RAB types.

All of these types can be combined in several predefined combinations to provide even richer service configuration of the radio access bearers. We could for example have a Conversational Speech + 2 * Interactive Background + 1 * Streaming Unknown RAB type that combines three of the above listed types which would be a configuration for an end user using voice and video telephony while for example transferring a music file to the same or another end user.

(20)

2.2 I

UB SPECIFICATION

As we mentioned in the previous chapter about "Introduction to WCDMA" the 3GPP TSG groups have worked together to form specifications for the architecture of current and future telecommunications networks. Each telecommunications develop-ment company that are part of the agreedevelop-ment impledevelop-ments their architecture based upon these specifications. This contributes to the standardization of the industry at large. What this means is that the architecture previously described in Ericsson’s case have its functionality based upon several of these 3GPP technical specifications. What we mean by "based upon" is that Ericsson have internal TSG groups that decides how to interpret the specification by looking at the larger picture and determine what parts of the specifications that are of interest for them. The Iub interface that we mentioned in the previous section is of much importance for this thesis work and it is therefore important to consider the implications of 3GPP specification on this part of the archi-tecture.

The 3GPP specification for the Iub interface is called the NBAP specification and it specifies the radio network layer signalling to be used for the Control Plane over the Iub interface. This sounds rather complex but what it means is that the specification specifies the protocol to be used by the RNCs to control the RBSs over the Iub interface.

2.2.1 NBAP S

PECIFICATION

The NBAP specification is quite an extensive specification, about 850 pages long which is separated into several high level sections.

2.2.1.1 NBAP E

LEMENTARY

P

ROCEDURES

The first section of the NBAP specification specifies the NBAP Elementary proce-dures, or EPs in short, which are the basic procedures that can be invoked by the con-trolling RNC towards the Node B.

Each EP is determined by an initiating message called a request and possibly two associated messages. The first associated message is called the response message which gives the calleé information about the results of the outcome of the request. The second associated message is called the failure message which in case the request failed gives the calleé detailed information about the causes of failures for the request.

The EPs are separated into common procedures and dedicated procedures [9]. The difference between the two types of procedures is that a common procedure is a proce-dure that the RNC can invoke that is not dependent on a previous common proceproce-dure being called that have initiated a communication context. What happens after a com-pletion of an invocation of a common procedure that creates a communication context is that the communication context is established in the Node B for a specific communi-cation with a UE. This communicommuni-cation context is on the RNC side determined by the CRNC communication context ID and on the Node B side determined by a NodeB communication context ID. It is this ID that is used to invoke dedicated procedures that deals with resources established within such a communication context. For each of these procedures, a description of the successful outcomes, unsuccessful outcomes and the possible fail causes for the procedures are described.

2.2.1.2 A

BSTRACT

S

YNTAX

N

OTATION

O

NE

S

PECIFICATION

The next part of the NBAP specification is the ASN1 specification of the possible content of NBAP messages.

The ASN1 specification of NBAP messages is specified in three layers.

The first layer is what is called the Elementary Procedure Definition specification, or PDU specification in short, which specifies the content of the NBAP message for each of the NBAP EP’s. This implies an ordering of the Information Elements, or IEs in short, in the message and presents information about whether or not an IE is consid-ered mandatory or optional. When considering the example of the ASN1 specification for RadioLinkAdditionRequestFDD on page 55 we can see that the EP message being

(21)

defined is called RadioLinkAdditionRequestFDD and that the top-level PDU is a sequence of two IE containers, namely the protocolIEs-container and the protocolEx-tensions-container of which the protocolExprotocolEx-tensions-container is considered optional. For each of these IE-containers it is specified which type it is, for example the protoco-lIE-container is to be of the type called RadioLinkAdditionRequestFDD-IEs.

The second layer is what is called the IE specification which specifies the IEs or IE containers that can be part of the NBAP message PDUs or in IE containers themselves. An IE container is a container for IEs, that is represented as a sequence. There are many types of IE-containers such as protocol-single-IE-container, private-IE-container and many more.

The containers are specified similar to the PDUs in that it implies an ordering of the IEs and the specification specifies presence information for each IE. As in the example of the ASN1 specification for RadioLinkAdditionRequestFDD on page 55 we can see that the PDU definition contains a ProtocolIE-Single-Container called RL-Informa-tionItemIE-RLAdditionRqstFDD which is specified using an ID and a definition as a sequence of single IEs.

The IE’s themselves are the most basic elements of an NBAP EP message and are in the specification defined as ordered elements with an identifier and a value. An IE ele-ment can have different presence depending on the context of application. It can be specified as mandatory, it can be specified as optional or it can be specified as condi-tional which means that its presence is dependent on the presence or value of another IE in the same context of application.

The value of an IE can be of different types such as integer, enumerated, octet string, bit string or a special constant value reference calledASN1_NOVALUEthat is the ASN1 representation of a null value. If we look at the example of the “IE Definition for Diver-sityControlField” on page 56, we can see that DiversityControlField, which is an IE that is part of the IE-container mentioned in the previous paragraph, is defined as an enumeration meaning that the DiversityControlField can have one of the values in the enumeration, that is, either may, must or must-not.

The third layer of the NBAP ASN1 specification contains common definitions, con-stant values for IE names, EP concon-stant codes and other common information.

2.2.2 A

PPLICATION

O

F

NBAP S

PECIFICATION

The previous section described in general what is included in the NBAP specifica-tion. How this is used in the thesis work will be described in a later section of the thesis but another question begs to be answered and that is how the NBAP specification is used by the different development companies of telecommunications systems. What one has to consider is that all development companies that are part of 3GPP have been represented in the process of developing and maintaining the specification. All these companies are free to perform their own internal application given the specifications and their purpose is to maintain the multi vendor operability.

2.3 R

ADIO

L

INK

M

ANAGEMENT

As we mentioned in the section on “NBAP Elementary Procedures” on page 8 in the NBAP specification the EPs are divided into common procedures that creates commu-nication contexts and dedicated procedures that manages these commucommu-nication con-texts. These procedures are by Ericsson divided into several subsets of functionality that Ericsson have decided are relevant with each other. This can be based upon local-ity of functionallocal-ity or whether it is related to different parts of the hardware. One of these collections of common and dedicated procedures is called DRL.

DRL is the internal name for the subset of the RBS functionality related to control-ling radio links towards the user equipment in the RAN. This functionality has many implications on several areas of the RAN, ranging from the controlling and serving RNC’s down to the actual user handsets. This thesis focuses on the 3GPP RLM subset of the NBAP functionality, and since Ericsson’s division of the NBAP functionality into

(22)

DRL that includes other functionality such as compressed mode and dedicated power measurement as well we have decided to follow the group name RLM as set by the 3GPP standard.

The elementary procedures that are related to RLM can basically be divided into two sections. The first part is the collection of EPs that are specified as common and the second part is the collection of EPs that are specified as dedicated. For more informa-tion about these EP’s, see “NBAP Elementary Procedures” on page 8.

The EP’s that are part of RLM of which are of importance for this thesis are the fol-lowing: • RadioLinkSetupRequest/Response/Failure • RadioLinkDeletionRequest/Response • RadioLinkAdditionRequest/Response/Failure • RadioLinkReconfigurationRequest/Response/Failure • RadioLinkReconfigurationPrepare/Ready/Commit/Cancel

2.3.1 R

ADIO

L

INK

S

ETUP

R

EQUEST

FDD

The RadioLinkSetupRequestFDD is the EP in the NBAP specification that estab-lishes a communication context between the CRNC and the Node B for association with one or more radio links between the serving RNC and the handset. This EP estab-lishes one or more Dedicated Channels, or DCHs in short, on one or more radio links and can also include a High Speed Downlink Shared Channel, or HS-DSCH in short, for one radio link and Enhanced Dedicated Channels, or E-DCHs in short, on one or more radio links.

2.3.2 R

ADIO

L

INK

D

ELETION

R

EQUEST

2.3.7 R

ADIO

L

INK

R

ECONFIGURATION

C

ANCEL

The RadioLinkReconfigurationCancel is the EP in the NBAP specification that is used to cancel a previously prepared reconfiguration and releasing the allocated resources in the communication context towards the UE.

2.4 E

RLANG

This section introduces the reader to the concepts of Erlang that is needed to be able to comprehend the following chapters of this thesis.

2.4.1 R

ECORDS

Erlang records are a way for the programmer to store a fixed number of elements [7]. It has named fields and similar to structs in the C programming language. Erlang records can be used by the programmer to ensure cross module consistency of data structures by using record definitions in a common include file for the modules using these data types. A record definition in Erlang is declared with the record name, its named fields and possible default values such as:

-record(name, {field1=Value1, field2, field3=[]}}.

Accessing record fields is a simple task. If the record is bound to a variable as in:

MyRecord = #name{field1=1, field2=3}.

the record fields can be accessed by:

MyRecord#name.field1 MyRecord#name.field2 MyRecord#name.field3

If the record is matched by a clause, the field values can be bound by local variables as in:

myFunction(#name{field1=BoundVariable1, field2=BoundVariable2}).

The fields of records not having a default value defined will be initialised with the atomundefinedwhich is the value thatfield3would have got in the record in this example.

2.4.2 E

RLANG TUPLES

A tuple in Erlang is a compound data structure with a fixed number of terms where any term can be of any type [7]. The Erlang representation of a tuple is on the form

{term1, term2,...,termn}where each element in the tuple is called an element. The number of elements in the tuple is said to be the size of the tuple.

2.4.3 E

RLANG

L

IST COMPREHENSIONS

Many programming languages including Erlang provides a functionality in the lan-guage that in Erlang is called list comprehensions [7]. This functionality provides a notation for the programmer that enables him or her to generate elements on a list

(24)

con-ditionally. List comprehensions as language constructions are analogous to set compre-hensions in Zermelo-Frankel set theory and are called ZF expressions in Miranda. They are analogous to thesetof andfindall predicates in Prolog [7].

In Erlang a list comprehension is written with the following syntax:

[Expr || Qualifier1,...,QualifierN]whereExpris an arbitrary expres-sion, and eachQualifieris either a generator or a filter. This means that the expres-sion to be evaluated can conditionally be evaluated depending on the qualifiers to the right hand of the list comprehension. Thus the language construction provides the pro-grammer with both the ability to determine what to generate, or if to generate anything at all, depending on conditions external to the construction of the list. This language construction is nothing that can not be implemented using functions together with sim-ilar language constructions but it provides good readability and traceability since we can provide functions for the qualifiers that traces the generation.

A generator for a list comprehension in Erlang is written as:

Pattern <- ListExpr.

thus providing the possibility to choose which values that should be generated for either a shadowed variable in the surrounding function clause or for a free variable in theExpr.ListExpr is an expression that must evaluate to a list of terms.

A filter is an expression which evaluates to true or false. This enables us to filter out unwanted values in the generation.

One important issue to be aware of when it comes to list comprehensions that might not be apparent at first is that all the free variables in the generator patterns are shad-owing variables in the function clause to which the list comprehension belongs. This means that the programmer as such cannot use values external to the list comprehen-sion in the generator patterns which can result in some needs of special solutions if the generated values are dependant on external values.

2.5 T

EST

S

ETUP

This section describes how the test setup of our SUT looks and how this might effect our implementation. The aspects that are described here are the ones that are important to be able to understand some of the decisions made.

Figure 2-5: Test setup

LOST

SUT

Iub

Bp

(25)

The SUT shown in figure 2-5 consists of the MPSW that is running on the RBS. Additional to this software are the stubs taking care of the functionality that we are not testing in this setup, such as stubs for emulating other hardware in the RBS and the software running on this hardware as well as stubs for the interfaces. The most impor-tant one is LOST, this is the stub for the emulation of the subsystem signalling. The hardware signals are sent over the Bp interface. In this test setup the LOST stub is responsible for receiving the signals usually sent to the subsystems and to send back proper responses to the MP. The way that the LOST stub is implemented today poses some problems for some of the aspects of testing in a greybox manner. The signalling performed by LOST pre-define the content of the response signals that are to be sent from the subsystems back to the MPSW. In some use case scenarios where the content of these signals need to be modified to achieve the proper behaviour of the MPSW one can not rely on LOST since this functionality is not implemented. These modifications need to be performed by the test case developer by hand after a short-circuit of the LOST stub.

The Iub interface is the interface we are using to invoke the elementary procedures described in “Radio Link Management” on page 9. The Mub interface is another inter-face that can be used to read managed object in the SUT to verify results of invocations. This contributes to the greybox testing method described in the following chapter.

(26)

3 T

ESTING AND TEST FRAMEWORKS

The testing of complex system software is often a tedious task and the effort to implement test cases for these complex systems will in many cases require a substantial amount of effort equal or greater than the effort needed to design these systems [4]. There are many different methodologies and aspects of software testing that are related to this thesis work. These methods will be described in this chapter as well as other aspects of testing that will be important for subsequent chapters. If not specified, all concepts described in this chapter refers to system testing.

3.1 F

UNDAMENTALS OF TESTING

As is understood, softwares with no defects are a utopia [1]. In practice, it is impos-sible to ensure that even a small software application is free of faults. This is based on several facts. For one, computer systems are complex entities with complex design. Secondly, the development processes in place can always fail and as always, when human beings are involved, faults can happen. As a result of this, adding the fact that the resources in development projects available for testing often are quite limited, per-forming adequate testing becomes a substantial problem. This means that test case developers not longer can focus solely on the test case development but also need to focus on improving the efficiency of the testing process to enable them to discover as many defects as possible in the limited time frame set for testing. Ultimately, the time frame for testing must eventually end when the given system is acceptable for its intended purpose.

Software testing as such can on a high level be sorted into two fundamental para-digms [2]. The first is called scripted testing and the other is called exploratory testing. In the scripted testing paradigm, the scripted testing is a sequential paradigm where the examination of requirements, followed by the design and documentation of test cases is followed by the execution and examination of the results of execution. The sec-ond paradigm, called exploratory testing is in essence orthogonal to scripted testing in which the emphasize is on simultaneous learning of the system, test design and test execution. The test case developer following this paradigm designs and executes tests while learning the behaviour of the system.

3.1.1 S

TOPPING STRATEGIES

As we mentioned above, the time frame set for testing must eventually end. This is one of the most difficult questions when it comes to testing since there is no way to know that the last found fault was the last remaining in the system.

Many organisations incorporates different strategies for testing and the stop strate-gies for testing can be very different [2][4]. Economics, of course, dictates that testing eventually must stop but the completion criterias typically used in practice can differ substantially. Many test case developers often adheres to the first category in which the executives at the company tells the test case developers when it is time to release the software, often after the scheduled time for testing expires. In other cases the halt of testing and releasing of the software can be dependent on the rate of found faults, when the marginal cost for finding a "new" defect exceeds the expected loss from that defect or when the test cases in test suites completes successfully.

Stopping when the scheduled amount of time for testing expires can be quite a use-less criterion since it can be met by doing exactly nothing [4]. Stopping when the test cases completes successfully is also useless since this stop criterion does not measure the quality of the test cases being involved. The second criterion is also counterproduc-tive since this criterion might give the test case developers an incencounterproduc-tive to write test cases that have a low probability of encountering faults.

(27)

3.2 P

OSITIVE AND NEGATIVE TESTING

Positive and negative testing are two complementary testing techniques [1].

In the process of positive testing it is verified through the test cases that the system conforms to its stated specification, that is, that the system fulfils its contract. In this process the positive test cases are derived from the system specification documents.

In the process of negative testing, test cases are designed to outside of the scope of the stated specifications. In this process, the test cases are designed to show that the system does not do what the system specification declares that it should, or to show that the specification of the system is incomplete or faulty.

3.3 W

HITEBOX TESTING

Whitebox testing refers to tests where the system under test is completely transpar-ent for the test case developer. This means that the test cases can be written based on the internal paths, structure and implementation of the system to completely cover all aspects of the functionality that are economically viable to test, under this testing cate-gory such methods as control flow testing and data flow testing come into play [2].

Whitebox testing can apply at all levels of system testing and in all situations the testing methodology implies that the test case developer has great knowledge about the implementation [4][1].

3.4 B

LACKBOX TESTING

Blackbox testing refers to testing when the system under test is completely opaque for the testing code. This means that the test code can not make any assumptions of the implementation of the system under test and must solely resort to the functional speci-fications and or functional descriptions that are available for the system. In these cases the testing is often referred to testing Application Programming Interfaces, thus testing at the functional level. Most blackbox test cases focuses on a single function and tests it with "middle-of-the-road"-values. Such tests are highly credible and easy to evaluate but not particularly powerful [14]. In such cases, such methods as equivalence class testing and boundary value testing come into play [2].

3.5 T

ESTING WITH SHADES OF GREY

Many systems are semi-transparent which means that the system exposes some of its functionality externally [2]. Testing such systems is often referred to as testing with shades of grey. This means that test code can utilize some of these semi-transparent implementation details in the design of blackbox tests by "opening" the box to look at internal details of the system to allow for more efficient blackbox testing.

3.6 S

YSTEM

S

PECIFICATIONS

Functional specifications are a key part of many software projects and they came into existence when the waterfall process model of software development became fash-ionable in the early 70’s [2]. The functional specification provide the basis for func-tional testing and the specification as such often describes an external view of an object or a procedure indicating the options by which a service could be invoked. The test case developer uses this specification to write test cases from a blackbox testing per-spective. Besides the fact that it provides them with the ability to implement test cases in parallel with the implementation of the system it introduces them to what the sys-tem really is [6].

(28)

3.7 S

PECIFICATION BASED TESTING

Specification based testing focuses on validating the functionality against a specifi-cation of the system [14]. These tests are highly motivated since many companies takes their specifications quite seriously and therefore the verification of such specifications becomes important. This is due to the fact that since the specifications often are parts of a contract, conforming to those specifications becomes very important. One can draw parallels to other products such as cars or mobile phones which must adhere to their advertisements or else the customers could feel betrayed.

Specification based testing can often lead to groups inclining towards focusing nar-rowly on what is written in the specification [14]. To them, designing test case suites that includes an unambiguous and relevant test case for each claim made in the specifi-cation becomes the definition of a good test case suite.

3.8 A

UTOMATED TESTING

The use of automated test tools provides the possibility to run tests unattended as well as the possibility to run more tests than would otherwise be possible. The greatest benefit comes from the inattentiveness which is closely related to regression testing described in the following section.

Most testing tools for test automation creates tests by recording behaviour of the SUT [1]. This recorded behaviour can then be replayed on a later version of the system allowing the behaviour to be compared between the two versions. Differences in behaviour can point out plausible problem areas. Other tools provide the possibility to test the functional requirements of the SUT, one such tool is QuickCheck which is the focus of this thesis. There are also tools that test non functional aspects such as per-formance and stress testing. John Watkins argues that automated testing tools, although having many potential benefits, are not appropriate for all testing tasks and that they have to undergo a thorough evaluation before being introduced. One implica-tion of this is that the tool must be integrated with the existing process and will not replace it.

3.9 R

EGRESSION TESTING

Regression testing is focused on designing test cases that are highly reusable with the intent of reusing the test cases whenever the system under test changes. When con-sidering regression tests the focus is on reuse and proper documentation.

As John Watkins mentions, "regression testing is a particularly candidate for sup-port by means of an automated testing tool, especially if frequent new builds and releases of the system under test are anticipated" [1].

A problem with regression tests are that one case might have been highly credible and powerful at the time of creation, but as time goes by, and the test case have passed many times it is not likely that the test case will find any new defects the next time [14]. Thus, most of the time regression tests carry little information value.

For regression suite test cases the documentation aspect becomes very important [3]. These cases have to evolve together with the software it is testing and down the line someone else might become responsible for these suites and need to be able to quickly understand them.

3.9.1 R

EGRESSION TESTING TODAY

The regression testing at MPSW today is mostly performed by the employees being given functional descriptions and functional specifications from which they derive test cases that covers the use case scenarios in the specifications and descriptions for a given version of the software in the RBS. These are then gathered into test suites based on the functionality being tested. The test suites are re-compiled after updates and run on special RBS nodes during the night. The results are gathered by different log

(29)

capa-bilities into a web based framework that allows the test case developers to analyse pos-sible faults and gather statistics from the runs. The framework surrounding these regression suites are based on a behavioural model that provides initialize and cleanup callback functions that ascertains that there are no dependencies between the individ-ual test cases as well as between the test suites.

(30)

4 Q

UICK

C

HECK

This chapter will introduce the reader to the software testing tool called Quick-Check that is the focus of this thesis. The chapter will describe the fundamental con-cepts of QuickCheck as well as some intricate details. After reading this chapter the reader will have gathered enough information to be able to follow the QuickCheck-related decisions being made to solve the problems found during the initial analysis as well as during implementation.

QuickCheck is a software testing tool produced by a recently founded company called Quviq. The QuickCheck software is the result of many years of development by the two founders of Quviq namely professor John Hughes and associate professor Tho-mas Arts. The tool is a testing software that provides the ability to test software in an entirely different way than previous tools. QuickCheck allows the user to write test specifications that includes test properties that ought to hold for the system under test as well as generators that provide the test case data. QuickCheck gives the test case developers a second alternative approach to testing their system. Instead of writing test cases that maps to use-cases in a one to one fashion, QuickCheck enables an automa-tion of the test case generaautoma-tion and execuautoma-tion. With QuickCheck the user can write the specifications by specifying several properties that ought to hold for different aspects of the system under test as well as generators that generates a multitude of the behav-iours that the system might inhabit. The outcome of these generators can be checked against the properties that ought to hold. The important difference to notice here is that QuickCheck enables the test case developer to specify the expected behaviour of the system in a concise way that is not spread out in a thousand different test cases but instead is captured in a small amount of testing code.

To make concise and readable specifications QuickCheck uses qualities from func-tional programming languages. Examples of such qualities will be visible in the code presented throughout this thesis. The advantages of writing specifications instead of a number of small scripted test cases include that less code is required for testing which therefore leads to better readability and maintainability.

Not only does QuickCheck provide the ability to test in a many-to-one fashion but it also provides a quick way of analysing errors in the software by shrinking the sequence of commands leading to the failure to a minimal sequence to guide the user towards the real cause for the failing test case.

4.1 P

ROPERTIES

In the QuickCheck domain, a property is a logical description encapsulated in a function. These properties provides a means to specify which behaviours the SUT should inhabit. Let’s start with a simple example. Assume we are the providers of a library for the Erlang programming language, lets call it ListIT. In this library we have implemented our own list structure that behaves as any lists would do in functional programming languages. In the documentation of ListIT we specify all the library func-tions for operating on lists. For example we could state that:

listIT:delete(elem, list): Deleting an element from a list removes the ele-ment from that list, if the eleele-ment is on it.

That would be a reasonable documentation of a library delete function on a list. But how would we test it. One approach could be to generate a sufficient amount of test cases, perhaps using the boundary principle and try to delete an element from the empty list, doing the same with a slightly larger list and perhaps deleting a empty ele-ment and so on. What we do in QuickCheck is to write a property that says what ought to hold for this list structure in the context of deletion of an element from a list. In QuickCheck we write:

(31)

property_listIT_delete()-> ?FORALL(I, int(),

?FORALL(List, list(int()),

not lists:member(I,listIT:delete(I, List)))).

So this is a property in QuickCheck. Basically it reads: "For all instancesIthat are of the type that is generated by theintgenerator and for all lists that are generated by thelistgenerator using the intgenerator it should hold that if you delete an ele-ment from the list, that eleele-ment should no longer be a member of that list.

So a property is a logical description of what ought to hold in the system under test. The property uses the generators,intandlistin this case, to generate test cases for testing this property. These generators will be explained in the following chapter. A property is tested in QuickCheck by generating 100 test cases from the generators and applying the property on these cases. There are a lot of issues to deal with for the user, since among other aspects, the user is responsible for defining generators that inhabits a good data distribution.

4.2 G

ENERATORS

In QuickCheck all testing is driven by the generators that provides controlled ran-dom test data. The problem with totally ranran-dom generation is that it in many cases results in poor test data distribution. Examples where randomly generated data always or almost always would lead to invalid, not good or in the worst case failing test cases are easy to discover. One such example could be that, given a name, the system should return the persons age provided that the name is in the following database:

As you understand the chance of hitting any of the names with a random length string with random characters is rather small, even when running millions of tests, which does not test the functionality of the system. It would be better to be able to for example say:

"Choose either one of the names Erik Fransson, Rickard Granfeldt, Johan Holmqvist or Patrik Höglund or possibly choose from a list of any valid name or, in for example one case out of seven, choose a random string."

QuickCheck provides just this, a powerful tool to generate random test data with any complexity with full control.

So what is a QuickCheck generator? A generator provides means to randomly gen-erate data for the test case as well as providing possibilities to have built-in shrinking behaviour. This is further described in the section about “Shrinking” on page 23.

The QuickCheck generator defines a set of values for test data and a probability dis-tribution over that set. The test data generators are defined by the test case developer using basic generators provided by QuickCheck. The generators in QuickCheck are encapsulated in symbolic functions such as thelambdafunctions in LISP or thefun

concept in Erlang, which provides an important fact. The return value of a generator is not a value but an encapsulated test data generator that at run-time will provide the user with an actual value. More on this later on.

As an example of a built-in generator we could look at theint generator.

Theintgenerator returns a test data generator for small integers. Another built-in generator is the list generator that given an element generator returns a random length list of elements from the given generator.

Table 4-1: Example database

Name Age

Erik Fransson 24 Rickard Granfeldt 25 Johan Holmqvist 27 Patrik Höglund 22

(32)

As another example of generators, let’s assume that we are testing a system that maintains a huge database of records of persons that live in a city. If we want to test such a system we would want to create a generator for generating names of persons. But since no such generator is provided with QuickCheck we would have to create our own. What we could do is to use a built-in generator that randomly selects an element from a list, such aselements. We could then use a dictionary or a hard coded struc-ture of names to provide us with the raw data to choose from. Let’s assume a diction-ary filled with names. Then our generator would be as follows:

name()->

elements(dict:getAllNames()).

What this code does is to specify a user defined generator called name that as the return value returns a new generator that uses theelementsgenerator to encapsulate a test data generator choosing random elements from a list of names, provided by the

dict:getAllNames() function.

4.3 S

TATE MACHINES

If random sequences of test data is generated, more sequences can be tested than can possibly be written by hand, hopefully leading to better test coverage. With auto-mated testing of such command sequences it is of importance to maintain control of the execution of sequences and the generation of test data. If the SUT have side effects relating to an internal state, the execution of sequences must also reflect the effects of such changes. In QuickCheck this is performed by using a state machine. The Quick-Check state machine has a predefined behavioural model which is made up of a number of required callback functions that defines a specified control flow for the state machine which will be described shortly. A QuickCheck state machine also contains an internal state, determined by the user, that gets modified when running the state machine. This together with the internal QuickCheck control algorithms is what consti-tutes side effect testing using QuickCheck.

During what we call generation-time the command sequence for the state machine run is generated. This starts with a call toinitial_statewhich sets up the initial state by for example getting information from the SUT. Then the command function selects a call either directly or in some cases depending on the current internal state of the state machine. This call is, together with the state passed to the precondition

function to determine whether the call should be in the sequence or not, if so the

next_statefunction is called which determines what state the state machine should be in when command is called the next time. After each generation-time run, the gener-ated sequence is run in what we call run-time. In this scenario the QuickCheck state machine verifies the calls in thepostconditionfunction against the specification for the SUT. If the verification is correct, the state will be updated to reflect the new state of the SUT. All of the calls generated from thecommandfunction during generation-time are symbolic and on the form{call, Module, Function, [Args]}and the value returned from the call is of course also a symbolic value in the form of a tuple{var, Value}. This will be further described in a subsequent section.