Institutionen för datavetenskap
Department of Computer and Information Science
Master thesis
Improving the test environment with focus on
computer cloning, hotfixing and simulation software
by
Robert Jacobsson
LIU-IDA/LITH-EX-A--08/042--SE
2008-09-30
Linköpings universitet
SE-581 83 Linköping, Sweden
Linköpings universitet
581 83 Linköping
Institutionen för datavetenskap
Department of Computer and Information Science
Master’s Thesis
Improving the test environment with focus
on computer cloning, hotfixing and
simulation software
Robert Jacobsson
Reg Nr: LIU-IDA/LITH-EX-A-08/042--SE Linköping 2008
Supervisor: Johan Lissing qa, Sectra Imtec AB
Examiner: Petru Eles
ida, Linköpings universitet
Department of Computer and Information Science Linköpings universitet
Abstract
When restoring a test environment, the key to success is that the process is fast and automatic. The goal is to have as little user interaction as possible.
This master thesis evaluates solutions for handling the restoration of the test environments and investigates a couple of possible approaches. It turns out that the most effective way of solving the problem is to elaborate a customized software. The interface and workflow of this software is described in the thesis.
Testing of the new software is performed and the thesis ends with some con-clusions regarding the new solution and proposals for future development.
Acknowledgments
I would like to thank:
Johan Åtting, QA, Sectra Imtec AB - for initiating the thesis Petru Eles, IDA, Linköping University - for the examination Johan Lissing, QA, Sectra Imtec AB - for supervising me Jonas Bergman Laurila - for the opposition
Joel Petersson, QA, Sectra Imtec AB - for answering a lot of questions All employees at the QA department at Sectra Imtec AB - for all their help Erik Mellström, former thesis worker, Sectra Imtec AB - for help with the com-puter and TEX
Johanna Gistvik - my support in life
Contents
1 Introduction 1 1.1 Background . . . 1 1.2 Objectives . . . 1 1.3 Method . . . 1 1.4 Thesis structure . . . 1 2 Theory 3 2.1 Software testing . . . 32.1.1 Software testing in general . . . 3
2.1.2 Software testing at Sectra . . . 5
2.2 Computer cloning . . . 6
2.2.1 What are clones? . . . 6
2.2.2 Why use clones? . . . 7
2.3 Hotfixes . . . 7
2.4 Simulation software . . . 8
3 Evaluation of different solutions 9 3.1 Questionnaire . . . 9
3.2 Computer cloning . . . 11
3.2.1 Computer cloning at Sectra . . . 11
3.2.2 Alternative solution 1 - Clonezilla . . . 11
3.2.3 Alternative solution 2 - WAIK . . . 13
3.2.4 Alternative solution 3 - Savepart . . . 13
3.2.5 Other alternative solutions . . . 13
3.3 Hotfixes . . . 14
3.3.1 Hotfixing at Sectra . . . 14
3.3.2 Alternative solution 1 - WSUS . . . 14
3.4 Simulation software . . . 15
3.4.1 Simulation software at Sectra . . . 15
3.5 Conclusions and decisions . . . 15
4 Implementation of new solutions 17 4.1 The implementation in general . . . 17
4.2 Workflows . . . 18
4.2.1 Program . . . 18 vii
4.2.2 Clone creation . . . 18 4.2.3 Clone recovery . . . 18 4.2.4 Hotfixing . . . 21 4.3 Database design . . . 21 4.4 User interface . . . 25 4.4.1 Start tab . . . 25
4.4.2 Recover clone tab . . . 26
4.4.3 Create clone tab . . . 27
4.4.4 Log tab . . . 28
4.4.5 Administration tab . . . 29
4.5 Functionality . . . 30
4.5.1 DOS tools . . . 30
5 Testing & evaluation of new solution 33 5.1 Functionality testing . . . 33
5.2 Performance evaluation . . . 33
5.3 Overall evaluation . . . 34
6 Conclusions and discussion 35
7 Future work 37
Bibliography 39
Chapter 1
Introduction
1.1
Background
This thesis was written at Sectra Imtec AB as a master thesis at Linköping Univer-sity. It was initiated by the Quality Assurance (QA) department at Sectra Imtec AB.
1.2
Objectives
The objectives of this thesis were to evaluate how the QA department handles the setup of the test environment, to find better solutions and implement them. Another part of the thesis was to evaluate and improve a test tool for simulating modalities and workstations.
1.3
Method
The major part of this project was spent on developing and testing a software to fulfill the goals of the thesis. In addition already available solutions were invseti-gated and evaluated.
1.4
Thesis structure
This first chapter describes the thesis in short and presents the motivation and objectives.
Chapter 2 covers the theoretical background and preliminaries about software testing.
Chapter 3 contains an evaluation of the solutions I have chosen and the reason for my choice.
Chapter 4 describes how the selected solutions have been implemented in the Sectra test environment.
Chapter 5 is a short summary of my conclusions and also contains a discussion part.
Chapter 2
Theory
The theory regardingtest environments is not really well documented and is mostly just mentioned in a small chapter when talking about software testing. One of the most important things about test environments is that they have to be as controlled as possible in order to eliminate errors not related to the software that is being tested.
2.1
Software testing
2.1.1
Software testing in general
Even though the actual testing phase is not a part of this thesis, I believe that it is necessary to include a short description of what testing is and why testing is used, to understand how this thesis fits.
Software testing is an area that is growing quickly. More and more software is used in our daily life and developing software becomes more demanding with this increasing importance of correctly working software. Sometimes even lives depend on the software. This makes quality issues more and more important since a failure can have catastrophic effects. The reason for software testing is therefore to decrease the risk of software failures.[1]
Ideally the test environment resembles the real-life environment where the soft-ware is to be used. This can be accomplished in many ways, but often it is impos-sible to simulate the exact real-life situation which makes some errors imposimpos-sible to find.
Testing strategies
There are two basic testing strategies, black-box testing and white-box testing. For a schematic overview see Figure 2.1.
In black-box testing, the tester doesn’t know anything about the software code, just functional specifications and quality requirements. The system is viewed as
it would be in the real-life system. In this type of testing it is important to have given input values with corresponding output values.
In white-box testing, knowledge about the software code and system structure is used to find errors. Program description and technical design are also important when performing this kind of testing. White-box testing can also be referred to as glass-box testing or structural testing.[2]
Figure 2.1. Black-box testing (unknown struncture) and white-box testing (known structure).
Testing methods
There are a lot of methods for testing and it is not possible to list them all here. The methods can be classified along five dimensions called the Five-fold Testing
System.[3]
These five dimensions are:
• Testers. Focus on who does the testing, for example a test team, developers
or end-users.
• Coverage. Focus on what is tested, for example function testing where you
2.1 Software testing 5
• Potential problems. Focus on why the testing is done, for example test
how extreme values are handled.
• Activities. Focus on how the testing is done, for example exploratory
test-ing.
• Evaluation. How to tell if the test passed or failed, for example, comparing
with a known result.
All methods cover one or more of these dimensions, leaving the test executor to choose on the other dimensions. For example, a tester gets the assignment to test extreme values. This only includes the dimension of potential problems. The tester must decide who should do the testing, which variables to test, how they should be tested and how to evaluate the result.[3]
2.1.2
Software testing at Sectra
The software produced at Sectra is used at hospitals to handle images taken through different modalities (i.e. x-ray, CT, MR) and to diagnose patients. This means that a software error can have dire consequences, where the death of a patient is a worst case scenario.
Workflow
An approximate picture of the workflow for test iterations at Sectra is shown in Figure 2.2 and a short description of each step follows below.
1. The project management, together with the persons responsible for the test environment, decides which computers will be included in the test iteration and how they are connected to each other.
2. The computers that are to be used in the test iteration are recloned. This eliminates every trace of the previous test iteration, so that errors due to old configurations or software are minimized. How recloning is done at Sectra is described in subsection 3.2.1.
3. Microsoft hotfixes are installed on all Windows computers. This makes sure that the software works with certain hotfixes. How hotfixing is done at Sectra is described in subsection 3.3.1.
4. The software that is about to be tested is installed (both server and work-station software).
5. In the testing phase, the software is tested in an environment and usage that is as close to "real-life"-usage. The testing can roughly be divided into four subareas wwhich quite often overlap:
• Testing on servers. • Testing on workstations.
Decision of test environment setup
Recloning of
computers Installation of hotfixes
Installation of the software to be tested Testing On workstations On servers Simulation of high load Simulation of modalities New test iteration
Figure 2.2. Approximate workflow of the test iterations at Sectra. Areas covered in this thesis are highlighted.
• Simulation of modalities. • Simulation of high load.
6. When testing has finished a new test iteration starts and we are back at the beginning, unless it is the final test iteration, after which the product is released.
Most testing at Sectra is done by following a test case record with clear in-structions and an expected outcome, covering the dimensions of coverage, poten-tial problems and evaluation. This is a quite strict testing and leaves little, or no room, for the tester to explore the software on his/her own. To deal with this, some testing is exploratory where the tester gets an area to test and uses his/her imagination and experience to test if the software behaves as expected.
2.2
Computer cloning
2.2.1
What are clones?
Cloning is generally referred to when talking about making exakt duplicates of something. In the computer case this is almost true. Cloning a computer is about taking all the information on a hard drive and copying it to another computer, either directly or by using an image file as an intermediate step. This creates a
2.3 Hotfixes 7
problem since machines with the same name are not allowed on the same network. Also, computers with identical security IDs (SIDs) are not allowed in the same Active Directory domain. A disk cloning software therefore has to change this information before applying the clone again or the information has to be altered in some other way.
2.2.2
Why use clones?
Cloning disks has many uses and there are many different softwares available for making and restoring clones. Some of the major uses for clones are:
• Restart and restore. This technique is often used on public computers,
such as computers at cybercafes, schools and libraries. By restoring the computer, they can keep the computer free from virus, inappropriate content and programs. How often this is done depends on the administrators of the computers. It can be done on a regular basis, for example every night or even every log off, or in the case of a malfunction of the computer.
• Pre-installed applications. By having a clone that has pre-installed
ap-plications you do not have to waste time to install them. This is often used at larger companies.
• Hard drive change. If the user needs a new hard drive, a disk clone can
be used to move all the data and settings to the new drive.
• Backup. A backup of the entire system and data can easily be created.
• Recovery. Nowadays, a new computer often has a system recovery disc
attached, which can be used to restore the computer to the original factory settings.
In the testing scenario, clones are mostly used to get a clean installation of the computer back fast. This makes sure that no parts of the tested software remains from a previous test iteration, which could inflict errors in the next test iteration.
2.3
Hotfixes
A hotfix is an update that fixes a bug or a security flaw in a software. When dis-cussing hotfixes in the context of test environment, we often talk about updates from Microsoft that contain security updates or other improvements to the perfor-mance of installed components. By updating the computer it gets less vulnerable for trojans and other attacks that use the flaws in Windows. Since these updates make changes on the computer it is important to test that the software still works after installing a hotfix before making it official that the software supports the hotfix.
In a regular environment the hotfixes are often installed automatically at night. This is a standard Windows option that can be found in the Control Panel. Since
the computer is constantly up-to-date, it doesn’t matter if it takes some time before installing the hotfix or if it has to reboot after applying the hotfix.
In the test environment, we want to spend as little time as possible when ap-plying the hotfixes and preferably it should be done instantaneously after recloning the computer to be able to start the test process as soon as possible. With that in mind, one cannot use the normal hotfix procedure in the test environment, but has to find some other solution.
2.4
Simulation software
Sometimes the software interacts with complex devices, for instance x-ray-machines, or workstation set-ups that are almost impossible to have in a test lab. To be able to test this it can be good to have another software that can simulate the be-haviour of the other devices or workstation set-up. Although this probably will not be sufficient to find all errors, it will hopefully help discover most of them. Since we want to test our own software, it is not very likely to find a simulation tool that fits from another company. This means that this software has to be developed by the testing team, possibly with the assistance of the developers.
Chapter 3
Evaluation of different
solutions
To get a better and more objective point of view of the current solutions, I handed out a questionnaire to all employees at the QA department. My intention was to use the answers from the questionnaire to better decide what was most important to focus on and get an idea of how they want things improved. The questionnaire can be found in Appendix A and the summed up answers in Figure 3.1. The answers are presented in section 3.1 and will be commented on throughout this chapter. I will use these answers when deciding on how to move forward with the project.
For evaluating the existing solutions, I have used the existing test environment in real work situations and some free testing. I have also set up a virtual domain, to use as my own personal test environment, on my computer with one server and two workstations. My virtual domain can also be used to make the inital tests of the alternate solutions. This way I did not have to interfere with the regular work to test the solutions before I knew exactly how it was to be configured. It also saved a lot of time, because the workstations did not have to have anything installed (except operating system) which made the clones rather small and therefore faster to create, copy and restore.
3.1
Questionnaire
As mentioned in the beginning of this chapter, the employees at QA filled out a questionnaire to get their thoughts about the test environment by marking how much they agreed on ten statements about the test environment. I also asked some question about how long they have worked at the department and at other companies with these issues just to see if that affected the responses. The sum-marized answers to the statements can be found in Figure 3.1. The value zero is used when a person answered ”Don’t know” to a statement.
As can be seen in Figure 3.1, most low scores were obtained for statements 9
regarding hotfixing (statements four and five) and setup of an entire test iteration (statements six and seven), meaning that this probably should be the areas to put the most focus on. Regarding the simulation software (statements eight and nine), most of the employees did not know how to use it. This may be due to the fact that it is mostly the same persons that do the performance tests where this is used.
3.2 Computer cloning 11
3.2
Computer cloning
3.2.1
Computer cloning at Sectra
When this thesis begun, the QA department at Sectra used a custom made pro-gram named TestLabManager (TLM), that has been developed at QA, to admin-istrate the test environment. A screenshot of this can be found in Figure 3.2. This program handles clone creation, clone recovery and hotfixing, which means that this program is the core in setting up the test environment.
When it comes to clone handling, this program is actually quite good. The creation and recovery of clones is automatic. Just press start and no more user interaction is needed. TLM uses Norton Ghost to create and recover clones. Nor-ton Ghost is a fast DOS-program for saving and restoring hard drive partitions to/from image files, but costs quite a lot of money and is no longer available for new purchases.
Image files cannot be created for the active partition. Therefore, each computer has an extra partition with FreeDOS on it, which is activated by TLM when starting a clone process. The image files are stored on a fileserver and copied from and to the computer when they are used. Here, I find one thing that could be greatly improved. Every time a computer clone is about to be recovered it is copied from the file server to the local hard drive of the computer even if it already exists locally. Also, the copying is verified afterwards by checking the file byte for byte. This takes a lot of time and the verification may be ”unnecessary”, since it has never failed during my tests and no one at QA could recall such an event. Even if we want to verify the file, there is probably a more effective way of doing it.
Another improvement can be done after the computer has been recovered. When the computer has been recovered it has lost the trust relationship with the server since its SID has changed. This means that you have to remove the computer from the domain and rejoin manually. If this could be automatized, a lot of work time could be saved.
Overall, TLM seems to fill the need for clone handling at QA. It is easy to learn and use and is quite fast and automatic. This can also be seen in the answers to my questionnaire where the statements regarding clone handling (statements one to three) got high grades, as can be seen in Figure 3.1.
3.2.2
Alternative solution 1 - Clonezilla
Actually, Clonezilla has two interesting solutions to the clone issue. They have a small version, similar to Norton Ghost for booting from CD/DVD or USB and running the image creation and recovery. They also have a server edition for supplying computers with image files over the network.
The smaller version runs on a linux system, which also is very small, and all of it fits on an USB stick. This is an advantage since not all servers have CD/DVD-drives, but almost always there is an USB-connection. The use of linux also has its advantages. Linux systems support most network drivers which would enable clone recovery over the network. The downside of this Clonezilla version is that
3.2 Computer cloning 13
it seems very difficult to make the procedures automatic. Automatization would require unpacking of the linux package, making the appropriate changes and then repack it and distribute to the computer.
When reading about the server version, I got the opinion that this had its benefits when distributing the same clone file to many computers. Since most computers in the test environment are different from each other, I did not install this to try it out.
3.2.3
Alternative solution 2 - WAIK
WAIK is an abbreviation for Windows Automated Installation Kit. It is a free package supplied by Microsoft for deploying Windows systems to computers through the local network.
Basically, this program initiates a network boot with a smaller version of Win-dows called WinWin-dows Preinstallation Environment (WinPE for short). From this mode, the prefered Windows version is installed to the computer. All hardware drivers for all computers have to be stored in WAIK for them to be installed cor-rectly. This means that if you have numerous computers with different hardware, you have to add quite many drivers and this can be quite cumbersome. This is not the only downside. After trying to get this to work properly I consulted some forums for system administrators and found out that this solution demands a lot of administration and configuration to work properly. Preferebly, you should also attend some courses held by Microsoft.
To sum this solution up, I would say that this is probably the most sofisticated solution and most likely the best, but only if we had the time and money to really learn all about it.
3.2.4
Alternative solution 3 - Savepart
Savepart is the closest to the present solution, Norton Ghost, that I found. It has almost an identical design of the program and functionality. The big difference is that Savepart is freeware and does not have all the features that Ghost has. Initial testing of Savepart also shows that it may be a bit faster when creating and recovering clones. If it turns out to be faster in the real test environment to, it would be a great improvement.
3.2.5
Other alternative solutions
In the search for cloning software I came across some other solutions that I intended to try, but did not succeed with it. This was because of troubles when installing and that I had no time to fully test them. Even so, I think they are worth mentioning anyway.
First on the list is Altiris, a program similar to WAIK developed by HP. This I could not even install since the installation package was corrupt, but from what I’ve read this should be easier to use than WAIK and almost as powerful. This
could be worth investigating in the future if a working installation package is obtained.
The second solution is the Symantecs Ghost Solution Suite. This is another program that is similar to WAIK and Altiris. I never got it to work properly and I would guess it was due to the same reason I did not get WAIK to work the desired way. Massive configurations and administration made it hard to get a hang of.
3.3
Hotfixes
3.3.1
Hotfixing at Sectra
As mentioned in subsection 3.2.1, the same program used for clone handling is used for hotfix installation, TestLabManager (TLM) and as could be seen in Figure 3.1, this is the area that got the lowest grade from the employees at QA. This I can understand. The hotfix installation is quite problematic and time consuming. For every computer that you want to hotfix, you have to find out a number of different things, for example which operating system is running, which service pack, versions of Outlook Express, Internet Explorer and many other things. After finding this out, you mark the hotfix package (more about them when I describe the administration of the hotfixes) and fill in the information for all computers, one at a time. Since this is manual work, it would be a great improvement if this could be automated in some way.
The administration of the hotfixes is even more time consuming. All hotfixes are entered in an Excel spreadsheet with information about where on the file server the executable file is located, which operation system, service pack and versions of programs the hotfix applies for. For some hotfixes, there are different executables depending on all these variables, so for a single hotfix there could be anything from one and up to eight entries (possibly more, but eight is the most I’ve seen). This also means that for each entry you have to download the executable from Microsofts website and structure them in the file system tree. This solution is very sensitive for user errors and there are many different errors that can be made. To decide which hotfixes should be used in a project you create a hotfix package in TLM where you select which hotfixes that should be included. This part is actually quite good and easy to use, but in my opinion it requires more work than necessary to get to that point.
3.3.2
Alternative solution 1 - WSUS
WSUS is an abbreviation for Windows Server Update Service and is a free software supplied by Microsoft which is used to set up your own Windows Update server on your domain.[4] This is the only alternate solution that is tested because it is quite certain that there exists no other good softwares to administrate Windows Updates. When searching for such on the internet, nothing interesting pops up. Also, the QA team had a good idea about what this program could do even before this project started.
3.4 Simulation software 15
WSUS is installed on a server on the domain and computers are set (using Group Policies) to use that server to find the hotfixes instead of Microsofts server. The system administrator can then use WSUS to ”Approve” or ”Decline” updates from Microsoft to be installed on the computers that connect to the server looking for updates. WSUS can be configured to download the hotfixes to a network share where the computers can get them instead of connecting to Microsofts server. This possibly saves a lot of time since it is probably faster to download from a local share than downloading from the internet.
WSUS does not solve the problem with instantaneous installation of the up-dates, but when searching in forums used by system administrators it was quite easy to find scripts that trigger the update sequence on a computer, which would probably solve the problem.
3.4
Simulation software
3.4.1
Simulation software at Sectra
The simulation software used at Sectra is produced at Sectra and is named WISE-load. WISE is the name of the server part of the software developed at Sectra and is an abbreviation for Workflow Information Service Engine. WISE-load is mainly used to simulate load from many workstations to see how the server handles high load. It records what the user does in the software and remembers it. After that you start multiple simulations of the actions with some waiting time in between. This way all simulations do not try to access the same data at the same time. The simulations are run like a service on a computer and can be started remotely, which is quite smooth since it means that the person using WISE-load can sit at just one computer to start simulations at many computers. WISE-load is a bit tricky to learn, but once you got the hang of it, it works like it should. I think the difficult learning step is a reason why so many of the employees could not answer the statements about WISE-load.
3.5
Conclusions and decisions
After evaluating the different solutions I came to the following conclusions:
• The current clone solution fits the needs at QA.
• The current hotfix solution does not fit the needs and could be greatly
im-proved.
• No available solution to replace the current are found.
• The simulation software seems to work satisfyingly.
• If the new solution has to be coded from scratch, as much as possible should
With these conclusions in mind and after discussions with the test environment responsibles at QA, Johan Lissing and Joel Petersson, we came to the following decisions on how to move on with the project:
• A new software for handling the test environment should be written from
scratch.
• The new program will be named Test Environment Organizer (TEO).
• The user interface has to be easy to learn and use.
• Ghost should be replaced with Savepart for creation and recovery of the
image files.
• As much as possible should be automated. The goal is to be able to recover
a project’s whole test environment with as few clicks and with as little user interaction as possible.
Chapter 4
Implementation of new
solutions
4.1
The implementation in general
When starting the implementation of the new software I decided to write it in C# because that is the coding language used at Sectra.
The program should use a database to keep track of all the information needed. For clone creation and recovery, Savepart should be used. To be able to use Savepart, all the computers have to have an extra partition with FreeDOS installed to be able to create image files of the Windows partition. This is already the case for all computers in the test environment since this is a part of the existing solution.
The software development I performed can be divided into four parts:
1. Design of workflows for the program and for the processes that it should perform.
2. Design of the database. 3. Implement the user interface. 4. Implement the functionality.
I will not describe the implementation of WSUS in the test environment since it already is a complete program. WSUS was installed and configured so that the computers get their hotfixes from the local server instead of Microsofts. The aim of using WSUS is to make it easier to administrate hotfixes.
The workflows will be used as an assistance when implementing the functional-ity of the program. That way it will be easier to get the program to do the correct things. The design decisions made in the first parts can of course be changed if it turns out to be impossible to code or if another solution turns out to be better to use.
4.2
Workflows
4.2.1
Program
When designing the workflow for the program my main focus was that it should be easy to use. Since the existing program (TLM) was found easy to learn and use, I thought it was a good idea to try to copy the layout of the program and just make some improvements. This would also simplify the learning of the new program for the employees.
With this in mind I decided to design a program with no top menus, just using tabs to navigate through the different possibilities. These tabs should be named:
• Start
• Recover clone & hotfix
• Create clone
• Log
• Administration
Descriptions and screenshots of these tabs can be found in section 4.4.
4.2.2
Clone creation
The workflow for the clone creation process is quite straight-forward and can be seen in Figure 4.1.
1. An options file describing that a clone creation should be done and which options should be used, is copied to the target computer and the active parti-tion is changed from Windows to FreeDOS. The computer is then rebooted. 2. After the reboot of the computer, the clone creation is started automatically
via a startup bat-script.
3. The process waits for the computer to return to Windows after finishing the creation.
4. The clone file(s) are copied from the local hard drive to the file server and an md5 checkfile is created.
5. When copying is done, the process is complete.
4.2.3
Clone recovery
The clone recovery process is a bit more complicated than the creation process. The workflow can be found in Figure 4.2.
1. The program checks the consistency of the clone files that exist on the target computer by checking their md5 checksums.
4.2 Workflows 19
Switch active partition
Start creation
Creation done?
(Windows started again) Wait No
Copy the clone file to the fileserver
Copying done?
Creation process complete!
Yes
Wait No
Yes
Does the clone file exist on the target computer?
Copy the clone file to the target computer? No
Yes Verify that the clone file is the
same as the one on the file server
Copying done?
Verification ok? Wait Yes
No No
Yes
Reclone
Recloning done? No Wait
Remove the computer from the domain and rejoin
Reboot the computer Yes
Reboot done?
Clone process complete!
Yes
Wait No
4.3 Database design 21
2. If the files are not correct, the correct ones are copied to the computer. Otherwise, the process moves on to the next step.
3. When the correct files are present, an options file, describing that a clone recovery should be made and which options should be used, is copied to the computer. The active partition is changed from Windows to FreeDOS and the computer is rebooted.
4. After the reboot of the computer, the clone recovery is started automatically via a startup bat-script.
5. The process waits for the computer to return to Windows after finishing the recovery.
6. After the computer is restarted, the computer rejoins the domain to ob-tain the trust relationship with the server. This is done with the help of a Windows command named netdom.exe.
7. After another reboot, the process is complete.
4.2.4
Hotfixing
The workflow for the hotfix process can be seen in Figure 4.3. 1. A visual basic script, which triggers an update, is run.
2. If there exist applicable updates, the computer downloads and installs them and then reboots to complete the installation. If it has just been rebooted because of hotfixing and no additional updates exist, the process is complete. 3. The process waits for the computer to return to Windows and then runs the
script again.
4.3
Database design
The database contains all needed information about the clones, computers and configurations.
By first creating an entity-relationship (ER) diagram, it is easy to convert it to actual tables in the database. Designing the ER diagram was no easy task and some fields were added during the development process, when it turned out to be needed, and the (ER) diagram in Figure 4.4 is the final version and not the initial version. The fields that were added during the development were only simple values in single tables and did not affect the relationship between tables.
The ER diagram converted to a database schema can be seen in Figure 4.5. This schema was used when creating the tables in a Microsoft SQL 2005 Express database. This type of database was chosen because it is free to use and easy to install and administrate. It is also easy to code the connection between my program and the database.
Are there applicable hotfixes
to install? Download and install hotfixes
Hotfix process complete!
Has the computer just been rebooted due to hotfixing?
Yes No
Run hotfix script
Reboot the computer Yes No Reboot done? Wait No Yes
4.3 Database design 23 ComputerId Name Status Computer Own Standard Clone CloneId Name Created Comment 1 M Filename Archived Belong Project 1 M ProjectId Name Job Use 1 0,1 1 JobId Name Starttime Aborted Log Logging M 1 LogId DateTime Message Config ConfigId Description Value Installed ImageServer ImageServer /fs WISE WiseId Version SQL SQLId Version Operating System OSId Version
Service Pack SPId
Version 1 0,1 0,1,2 0,1 0,1 Utilities (Clone type) TypeId Type
ImageServer/fs ImageServer ComputerId Archived Created Computer ComputerId Name ProjectId Status StandardClone Project ProjectId Name Clone CloneId Name Comment OS_SP OS SQL_SHDB WISE SQL_WISE Config ConfigId Description Value Filename SQL SQLId Version WISE WiseId Version OS OSId Version ServicePacks SPId Version Logging LogId ComputerId DateTime Message CloneType TypeId Type Job JobId ComputerId CloneId Name Aborted Starttime
4.4 User interface 25
4.4
User interface
As mentioned in subsection 4.2.1, the user interface should be easy to learn and use and should use tabs to navigate instead of a menu. In this section I will provide a screenshot of each tab and a description of how it is supposed to be used.
4.4.1
Start tab
Figure 4.6. A screenshot of the start tab.
This is the first tab seen when starting the program and gives an overview of the project. At the top you can choose which project you want to have a look at. To the left, all computers are ordered into different statuses. Each computer has
either a green, yellow, red or grey backcolor which makes it easy to find out the status of the project. Green means ”Ready”, yellow means ”Work in progress”, red means ”Error” and grey means ”Unknown”. The ”Ready” status can be set either automatically when the process is completed or by the user by pressing the button ”Set Ready To Use”. This button is supposed to be used in case the process fails and the user have to fix the computer manually. In the top right there is a textbox where the log for a marked computer is shown. In the bottom you can see the processes that currently are running or are scheduled.
4.4.2
Recover clone tab
4.4 User interface 27
From this tab you choose which computers should be recovered and/or hotfixed. Like the start tab you choose the project at the top. To the left all computers that belong to the selected project are listed with their clones. The standard clones are pre-selected for all computers that have one. At the top right you select how much recovery should be done (Full / Reclone / Hotfix). You can also choose to schedule the process to start after up to 10 hours. At the bottom right, the clone information is displayed for the clone that is clicked.
4.4.3
Create clone tab
Figure 4.8. A screenshot of the clone creation tab.
state you want to save you choose the computer in the list to the left. Fill in the information about what is installed and press ”Create clone”. From this tab, clones can be moved to the archive. This is used if a clone should be replaced, but we do not want to delete the old one before we know that the new one works. Archived clones can also be deleted by pressing ”Delete”.
4.4.4
Log tab
Figure 4.9. A screenshot of the log tab.
The log tab is a really simple tab. You select a computer from the list to the left and the log for that computer is displayed in the textbox.
4.4 User interface 29
4.4.5
Administration tab
Figure 4.10. A screenshot of the administration tab.
The administration tab is password protected because changes made under this tab could make the program unusable. The screenshot in Figure 4.10 shows the administration tab after a successful login.
After logging in, another set of tabs is displayed enabling the user to change basic configurations of the program, manage projects and computers and configure the additional data values (available operating systems, service packs and SQL versions).
4.5
Functionality
After completing the user interface it was time to get my software, Test Environ-ment Organizer (TEO), to follow the workflows for clone creation (Figure 4.1), clone recovery (Figure 4.2) and hotfixing (Figure 4.3). This part of the master thesis is quite hard to write something about since it is all about coding. My approach was to code one workflow at a time and solve the problems I ran in to. I had a good idea of what the problems could be and how to solve them before I started coding because some issues and ideas popped up during the evaluation of different solutions.
4.5.1
DOS tools
TEO uses some DOS tools to be able to perform some of the required actions. These have to be copied to the computer to be able to use all functions in TEO. Below is a description of these external tools.
In these descriptions I have not included commands such as copy, move and mkdir since I consider them to be basic DOS commands.
Diskpart
Diskpart is a built-in tool available for the Windows versions used in Sectras test environment. Diskpart is command-line based and is used for handling disk administration and can be scripted to perform a certain task. TEO uses Diskpart to change the active partition on the remote computer before clone actions can be made, because these actions cannot be done to the active partition.
FCIV
FCIV is an abbreviation for File Checksum Integrity Verifier and it is available for free from Microsoft. FCIV computes and verifies cryptographic hash values of files, such as MD5 or SHA-1. TEO uses FCIV to verify that the clone files on the remote computer are the same as the ones on the file server. Using hash values is a much faster way to verify files compared to verifying files byte for byte.
Netdom
Netdom is, as FCIV, a free tool supplied by Microsoft and is used to handle computers in the domain. TEO uses Netdom to renew the trust relationship between the domain server and the remote computer. This is done by removing the computer from the domain and then re-adding it. This is followed by a reboot of the computer.
Psexec
Psexec enables execution of commands on a remote computer. Psexec is a part of an utility kit for Windows NT/2000/2003 developed by Sysinternals and is
4.5 Functionality 31
available for free download and usage. TEO uses Psexec to run commands on remote computers to, for example, copy files, restart the computer or use diskpart.
Chapter 5
Testing & evaluation of new
solution
5.1
Functionality testing
Before taking the program into full use it had to be tested on a smaller part of the test environment. During these tests some smaller user interface adjustments were made to TEO. The main problems concerned the database connection and access to the computers remotely, but they were easily fixed and the real trouble was not in TEO, but configurations for the SQL server and remote computers.
5.2
Performance evaluation
To see if the goals with TEO are achieved I made some comparisons with the cur-rent system and measured the times to create and restore clones, taking both time requiring user action (manual time) and waiting time into account. I had access to four computers when testing this, three workstations and one server. Since one of the improvements was to make the program check if the clone already exists on the computer before copying, I had two testing scenarios for TEO, one where the clone files did not exist on the computer (TEO-1) and one where they did (TEO-2). In all recovery scenarios, hotfixing is performed directly after recloning since that is the most common workflow. I calculated the average times for the four computers and the time gain I got with TEO versus TLM. The results can be found in the table below. All times are in minutes.
Action Program Manual time Wait time Total time Create clone TLM 0.5 30.5 31 TEO 0.25 27.5 27.75 TEO gain 0.25 (50%) 3 (10%) 3.25 (10%) Recover clone TLM 1.5 33.5 35 TEO-1 0.25 22 22.25 TEO-2 0.25 19.5 19.75 TEO-1 gain 1.25 (83%) 11.5 (34%) 12.75 (36%) TEO-2 gain 1.25 (83%) 14 (42%) 15.25 (44%) Hotfix TLM 5 19.75 24.75 TEO 0 15.5 15.5 TEO gain 5 (100%) 4.25 (22%) 9.25 (37%) It should be noted that TLM requires two user interactions in the clone recovery process and TEO only requires one. Hotfixing directly after a recovery also requires an extra, quite long, user interaction with TLM, but none with TEO.
I also had a quick look at the file sizes for the clones and found that the clones made by TEO were 10-25% smaller than the ones made by TLM.
5.3
Overall evaluation
One of the main goals for TEO was to decrease the time for user interaction and if it meant raising the waiting time a bit, it would still be worth it. As it turned out, both the manual time and the waiting time was decreased, but the big gain when recovering an entire environment was that no additional user interaction is necessary after starting the process until the computer is ready to use. This means that the time gain is actually even larger than in the table above, since TLM requires more user interactions throughout the process. If the process is not under observation all the time, these additional interactions are not made directly after the previous process is completed and therefore more time is spent.
When it comes to disk space it is always good to save some space. The smaller files also make the copying faster and could be a reason for a small part of the time gain.
Chapter 6
Conclusions and discussion
TEO seems to be a lot faster and more effective than TLM, but has not been put through any hard tests yet. All the clones have to be recreated to the new system and it will take some time until all of them have been converted. Therefore, it cannot be said that TEO is much faster and better than TLM, even though all the tests point in that direction. More testing in the real test environment should be made before drawing any conclusions about the improvement.
Saving manual time at hotfixing is very important since this is work that can not be done simultaneously. This means that the whole restoring process could be halted for many computers while performing this step. The impact of removing this manual time from the time for restoring the whole environment is quite big and I would guess that the restore time for the whole environment is probably lowered by one hour. This might not sound like a lot of time, but considering that there could be 4-5 persons waiting for the environment to be restored, it could potentially save some money. This is time saved every time a large environment is restored, so in the long run it will add up to a lot of time and money.
With the new solution it is much easier to administrate the hotfixes. An approximation is that each hotfix took about 30 minutes to administrate before and now it only takes about a minute. Considering that we handle about four to five hotfixes a month it will probably be noticeable in the long run, but the big improvement is that almost anyone can do the administration instead of just some key people.
Problems with licenses for Norton Ghost are also eliminated after switching to a freeware version called Partition-Saving (or Savepart for short). It also decreased the disk amount needed and the time for creation and recovery of clones.
I still think there exists a better alternative for restoring the test environment, but it seems to require a lot of administration and have a big learning threshold before being able to be used in a correct way. If a lot of time and money is put into learning that kind of system I think much more of the recovery can be automatized, but the question is if it will pay off. I think that the solutions proposed in this thesis is a good compromise between full automatization and user interaction.
To fully evaluate if the new solution is working better, the questionnaire should 35
be handed out again after TEO has been used for a while. More time measurements would also be of great help for a final evaluation.
Chapter 7
Future work
There is a lot of functionality that was considered to be put into this program and some aspects can be improved by making fundamental changes in the code. There was, however, not enough time available to include all functionality and providing a working program was of higher priority than adding all features.
Below is a list (in arbitrary order) of features that can be added to Test Envi-ronment Organizer (TEO):
• Automatic recovery of clustered servers.
• Automatic installation of FreeDOS.
• Create an installation package that automatically induces the database and
user.
• Make it possible to list all computers.
• Make it possible to list computers that are not assigned to a project.
• Store more useful information about the computers and clones.
If time and money are available in the future I would recommend to investigate WAIK a bit closer. WAIK seemed to be the best alternative of the ones investi-gated in this thesis, but as mentioned in subsection 3.2.3 it would require a lot of education before it can be used.
Bibliography
[1] Ilene Burnstein. Practical Software Testing. Springer, 2003. ISBN 0-387-95131-8.
[2] Martin Pol, Ruud Teunissen and Erik van Veenendaal. Software Testing. Pear-son, 2002. ISBN 0-201-74571-2.
[3] Cem Kaner, James Bach and Bret Pettichord. Lessons Learned in Software Testing. Wiley, 2002. ISBN 0-471-08112-4.
[4] Susan Norwood Microsoft Windows Server Update Services 3.0 Overview. Microsoft Corporation, 2007. Microsoft webpage.
Questionnaire
Questionnaire about the test environment
I’m doing my masters thesis at QA and will evaluate and try to improve the test environment with focus on the setup of the environment (i.e. cloning and hotfixing) and simulation of modalities and workstations (i.e. WISE-load). To get some input from the people working with this I would appreciate if you could answer this questionnaire. On some questions you answer on a scale 1-5, where 1 is ”totally disagree” and 5 is ”totally agree”. After each question is a couple of lines for additional comments. Questions may be answered in english or swedish.
The survey is totally anynomous, but you can add your name to make it easier for me to follow up things I don’t quite understand.
/Robert Jacobsson
1. It is easy to administrate the computer clones.
1 2 3 4 5 Don’t know
2. It is easy to create clones.
1 2 3 4 5 Don’t know
3. It is easy to reclone a computer.
1 2 3 4 5 Don’t know 40
41
4. It is easy to administrate the hotfixes.
1 2 3 4 5 Don’t know
5. It is easy to apply the hotfixes to a computer.
1 2 3 4 5 Don’t know
6. It is easy to setup an entire test environment for a test iteration.
1 2 3 4 5 Don’t know
7. It is quick enough to setup an entire test environment for a test iteration.
1 2 3 4 5 Don’t know
8. It is easy to use the modality-simulator in WISE-load.
9. It is easy to use the workstation-simulator in WISE-load.
1 2 3 4 5 Don’t know
10. It is easy to learn how to use the cloning/hotfixing tools (if you disagree, please specify).
1 2 3 4 5 Don’t know
11. The most important things to improve in the test environment is:
12. Other thoughts
About you
Name (optional): I’ve been working...
at Sectra (QA-department or similar) for:
< 1 year 1-2 years 2-3 years 3-5 years > 5 years at other companies with these issues for:
