Project topics and materials
Get Complete Project Material File(s) Now! »
Theoretical framework
Software testing
There are many definitions explaining testing but according to IEEE Standard 610.12-1990, « IEEE Standard Glossary of Software Engineering Terminology », Testing is « The process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component.” Above definition incorporates the whole testing process. Testing constitutes running a system in controlled environment and observing behavior of system and depending on behavior, verdict is given about the system. [9]
Lee Copeland in his book “A Practitioner’s Guide to Software Test Design” described testing as “testing is the process of comparing « what is » with « what ought to be.” [3]
“comparing What is with what ought to be” refers to comparing actual results with expected result. Software testing is a process in which the system under test is analyzed whether is it doing what customer wants from it. The aim of software testing is to help designers in making a product that is capable of performing desired operations. Testing techniques are selected depending on which software aspect is of more importance for the customer. For example for web sites, customer wants that his website should run on all web browsers. So for this, compatibility testing will be performed. If the requirement of the client is to perform compatibility testing and the testing team performs usability testing then the end product will probably be something which client does not want. So, choosing right testing technique depending on the requirement of the customer is extremely vital in developing a high quality product.
Following are some of the most important testing techniques. Among bellow described testing techniques, some were used in the thesis work depending on the customer’s requirements. For more details about different testing techniques, please refer to reference. [21]
Testing levels
Unit Testing
In unit testing, a smallest testable code from the whole system is selected, it is segregated from the system and its behavior is analyzed. Each unit is tested before integrating them to form a complete system. Units can be imagined as blocks, when they are tested and combined, a complete system is formed. For unit testing, Stubs and Drivers are needed to be written. If top down approach is used then stubs are written and if bottom up approach is used then drivers are written.
If a system has for example two units and these units are not tested before integrating them together then problems can arise in anyone of the two units and finding the root cause of that problem will be difficult because tester has to look into the whole system. On the other side, if each unit is separately tested then any bug in unit one will be isolated and fix without taking care of unit two and same will be the case for unit two. So unit testing allows isolating bugs and fixing them separately. Finally, after all units are independently tested, they can be combined to form a complete system. Integration testing will be done of that whole system. For more information about unit testing, please refer to reference. [13]
Pros and Cons
Unit testing enables a tester to find such classes or methods that are not behaving according to the specifications, providing the information about correctness of code. As aim of unit testing is to find bugs that are lurking on low level so this testing approach does not consider the whole system or how these units will communicate with other units, leaving defects that can occur when different units intercommunicate. So it can be said that unit testing does not test the design of a system. Testing a small unit of a big system is very simple as compared to testing how different units are working together to achieve the final task.
Stubs and Drivers
The stubs are usually written by testers and are dummy units that act like real units. They only return the value which calling unit needs. Logic is not implemented in the stubs. The stubs are used in Top down integration approach. Whereas the drivers have less throwaway code as compared to stubs [12] and are used in Bottom up approach.
Integration Testing
Integration testing can be said as testing of interfaces of independently tested units. Integration testing is performed after unit testing but before validation testing. For validation testing, please see section 1.4.2. When unit testing is successfully performed and all units are ready to be integrated, units are combined together and their interfaces are tested. The aim behind integration testing is to know that units are interacting with each other properly to complete a collective task. Inputs are given to the integrated system and outputs are analyzed. All units should collaborate with each other as they are intended to do and the system should generate the expected result. There are four ways of integrating a system.
Top down
First one is top down, in top dowwn approach, integration starts from upper level and goes to the lowest level.
Let, there is a system containing seven units. These seven units are independenntly tested. Now, they have to be integrated as shown in Figure 3. In top down approach, unit A will be integrated with stubs of units B and C. So, unit A will be tested by integrating it with the stubs of unit B and C. When unit A will be completely tested, stubs will be gradually replaced byy real units. This process will be followed from toop that is unit A to the bottom that is unit D, E, F, and G. As bottom most units are leaves, so no stubs for them are required. For top down inntegration, nodes-1 stubs are required.
Pros and Cons
Test cases are written keeping in mind the functional requirements of the system under test. So, the defects in design of the system are uncovered earlier in the testing proceess. In top down integration approach, no drivers are needed. But there are also some draw backs of this approach. As in top down approach, high level problems are uncovered but there is a greeat probability of leaving technical details which reesults in uncovered low level defects. One other limitation is that if the unit for which stub is to bee written is very complex that is having a lot of conditions than writing stub for that unit will be very hard.
Bottom up
Second is bottom up approach, in this approach, integration starts from bottom and goes to the top most level. In this approach, drivers are used instead of upper level units. If a system shown in Figure 4 is under bottom up inntegration, then first, units D and E are integrateed with the driver of unit B. After testing units D annd E integration will move to next level. This will continue until top most unit is integrated. If top down and bottom up integration approaches arre compared then it can be said that less drivers are needed than stubs. As in Figure 3, six stubs are needed to test the whole system. But only threee drivers are needed to test the same system using bottom up approach. For bottom up integratiion approach, nodes – leaves drivers are required.
Pros and Cons
When bottom up development is used then using bottom up integration testing approach is more worthy. Low level details are foocused more which results in uncovering moree defects on low level. But on the other hand, loww level components are usually available off–the shelf. Usually bottom up integration testing appproach is very useful when there is a systemm with real time requirements. But limitation of bottom up integration testing approach is that as testing is started from low level so the user feedbaack about the system is postponed which can ressult in developing a system which user probably has not asked for. In bottom up integration, drrivers are written which are more complex and hardder to write than stubs. [16]
Big bang
Third approach is Big bang, in this approach, all units are integrated at once and then whole system is tested. Integrating the whole system at once saves time but also introduuces the difficulty in fault isolation. If a system has many units then tracking fault becomes extremeely difficult.
Pros and Cons
Big bang integration testing appproach is perhaps useful for small systems. The units and their interfaces should be well-defined to be tested using big bang integration testing approach. As the whole system is integration at once so no stubs or drivers are needed but this also introduces a problem of fault isolation which makes it very hard to find whether the bug is in a unit or in the interface of the unit. This also incorporates the chance of skipping extremely important bugs that should be uncovered during testing of the system. Integrating the whole systtem at once also makes it difficult to confirm test case coverage.
Sandwich
The last approach is sandwich approach. This is a combination of top downn and bottom up approaches. In Sandwich approacch, a middle level is identified, Top down approoach is used from top most level to the middle leveel and bottom up approach is used from bottomm level to middle level. In some cases, a sub tree is integrated and tested using big bang and otther sub trees are integrated using top down or boottom up integration approaches. By using thiis approach, less number of stubs and drivers are needed to test the whole system. For example, in Figure 5, two stubs and two drivers are needed. But fault isolation is compromised. [17]
Pros and Cons
In sandwich integration testing approach, the whole system is tested in a gradual manner. If the system crashes, newly integrateed component is analyzed. The testing progress can be easily verified against the decompositioon tree. The limitation of sandwich integration teesting approach is that it is assumed that the structture and units are correct. So testing can only be performed on correct structures. As sandwich integration testing approach is a combination of both top down and bottom up integration testing approaches, so both stubs and drivers are requiired to be written. If any change occurs in any unit, the whole system has to be retested. [17]
Pair wise integration
Pair wise integration is performed using a call-graph instead of using decomposition tree as used in top down, bottom up, big bang and sandwich integration testing techniques. The main benefit of pair wise integration is that no stubs or drivers are needed in this type of integration. Real units are used instead of investing efforts in developing stubs are drivers. In pair wise integration, one integration session is used to integrate one pair of units. In Figure 6, six sessions are used to test whole system.
Pros and Cons
There is increase in number of sessions but extra effort is saved that is consumed in writing stubs and drivers. The drawback of pair wise integration testing is that if a bug appears in a unit, let’s say unit B in Figure 6, it can be seen that unit B is used in three different pair wise integrations. Bug will be fixed but those three pair wise integrations will have to be repeated and retested.
System Testing
System testing is performed on a complete system after it is integrated. System testing is carried out after integration testing. The tester does not have to have knowledge of internal structure of the system under test. Tester will give inputs and analyze outputs. The aim behind system testing is to check whether the whole system is producing right results. The System should implement all specified functional requirements. The whole system is considered as a single unit. System testing includes, functional testing and performance testing. Functional testing validates functional requirements whereas performance testing validates nonfunctional requirements. For more details about system testing, please refer to reference [15]and reference [12]
Acceptance testing
Acceptance testing is done to validate the requirements. It involves end user evaluation about the end product. There are special tests that are designed for acceptance testing. These special tests are called benchmark tests. Bench mark tests are test cases that are executed on different products from the same category to have comparison of new product and its competitors.
Pilot testing involves installing the system for experimental purposes and testing it against daily working. In some cases, pilot tests are done primarily in house before deploying it in real environment for real pilot test. This in house pilot testing is called alpha testing. The pilot testing performed by end user is called beta testing. One other approach in acceptance testing is to deploy the new system in parallel with old system. The advantage of this approach is if new system fails to meet user requirements then user can immediately switch to old system. User’s everyday working will not be affected in case of any system failure. For more information about acceptance testing, please refer to reference [16]
Testing modes
Black-box Testing
As shown in Figure 7, in black box testing, we do not have any knowledge of code structure, we understand the system only by giving inputs and taking outputs. In black box testing, inputs are given to the system under test and then actual outputs are compared with expected outputs. If actual outputs match with expected outputs then we say that system is performing right function and it has passed the test case. But if actual outputs are not same as expected output then we say that system has failed the test case and correction in system should be made. Black box testing is performed on user requirements and system specification.
To perform black box testing, tester does not have to have programming knowledge as he does not go into the implementation details. Tester should only know what system under test should do. Tester gives inputs and takes records outputs without know how system under test is generating this output. Black box testing can be performed on unit level or on system level. For more details about black box testing, please refer to reference [20]
Pros and Cons
Black box testing technique can be used at any level. As the testing level increases, the size of the system also increases and it becomes difficult to perform white box testing. So at higher levels, the black box testing is more suitable. When using black box testing, the tester cannot be sure of how much code he has covered. Or whether a particular block of code is tested or not because he has no access to the code, he cannot see the code.[3] But on the other hand, black box testing does not require a tester to be good in programming.
White-box Testing
White box testing has different synonyms. White box testing is also called as structural testing or clear box testing or transparent box testing. All terms have almost same meaning that is the code is visible to the tester. In white box testing, we have access to the internal structure of the system as shown in Figure 8. We can peek into the code and analyze how code is working. We can see implementation details of a system. In order to perform white box testing, a tester needs to have good programming skills in order to successfully design and execute test cases. Tester selects inputs that execute all necessary code. Which input will execute which code? This information is gathered by examining the code structure of the system under test. After giving inputs, the tester examines the inputs and behavior of the system. This strategy helps to improve the quality of the code by exposing loop holes in the code.
White box testing can be performed on unit or system level testing. White box testing is also performed on integration testing. In unit level testing, white box testing is done to see different paths within a unit. In integration level testing, white box testing is performed to examine paths between different units. For more details about white box testing, please refer to reference [22] and reference. [10]White box testing has two main sub types, data flow testing and control flow testing. Data flow testing concentrates more on the points where values are assigned to the variables or where these values are used whereas control flow testing concentrates more on code that cannot be tested using inspections and reviews. In control flow testing, the testing is based on internal paths and structure of the system. To test how much code is tested, a criterion called code coverage is used. Code coverage can be done on different levels. For example, line coverage, decision coverage and condition coverage. In line coverage, the aim is to execute lines of the code, irrespective of the decision or condition. In decision coverage, the aim is to test the decision for true and false whereas in condition coverage, the aim is to test each condition within a decision. But the problem in condition coverage is that each condition is not tested for both true and false. To overcome this limitation, multiple condition coverage is used. In multiple condition coverage, each condition within a decision is tested for both true and false.
Pros and Cons
By using white box testing, code structure can be improved. As in data flow testing, improper use of variable values can be detected and eliminated. Limitation of control flow testing is that tester should have good programming skills to control flow of the code. Because of the fact that tester has to understand the code, control flow testing becomes very time consuming.
Grey box testing
Grey box testing is in between black box testing and white box testing. It is also called translucent testing. In grey box testing, tester only knows such details of the code which enables him to understand that a how a particular feature is implemented. It is not necessary for a tester to know all implementation details. While performing grey box testing, the tester prepares the test cases using black box strategy that is preparation of test cases using requirement specification documents and then analyze particular feature of the system using white box testing strategy. For more details about white grey testing, please refer to reference [4]
Pros and Cons
Grey box testing technique has the benefits of both black box testing technique and white box testing technique. But its limitation is that as there is no full code access to the tester so no full code coverage can be assured by the tester.
Testing Types
Functional testing
Functional testing is considered as a sub-type of black box testing because we concentrate more on what the system is doing rather than how the system is doing. In functional testing, we do not peek into the implementation details of the system under test. We provide inputs to the system and see the behavior of the system and record the outputs. Then we analyze that the system is performing the right indented functionality.
For example, for this thesis work, when functionality testing of AddCtcID interface was performed. CTC ID was entered and next button was pressed. Expected result was that the system under test should fetch data regarding that CTC ID and text areas of next interfaces should be populated with that data. So this expected result was matched with actual result. If on pressing next button system under test fetches the right and data and populates the text areas of next interfaces with this right data then system is performing its intended functionality correctly.
There are different types of functional testing. For example, boundary testing and equivalence class testing. Boundary testing focuses on the input boundaries of a system because mostly the bugs lie on the boundaries. These bugs can be either in requirements of a system or in the code. Most efficient way of finding these bugs is inspection [6]. Boundary value testing is performed by first identifying equivalence Classes then by identifying boundary of each equivalence class. For each boundary value, test cases are created. These test cases are created by selecting one value on boundary, one value just above the boundary and one value just below the boundary. The aim behind equivalence class testing is to reduce the number of test cases to a manageable size with keeping reasonable test coverage. Each equivalence class contains data that results in same output from the program.
One very important feature of white box testing is code coverage. Code coverage means that how much code is executed when test cases are run. By using this information, particular code segments can be tested. High code coverage requires more test cases. Writing test cases require effort and time. There are several tools in market for measuring code coverage. One of the tools is BullseyeCoverage. This tool is used to measure code coverage of C++ programs.
Pros and Cons
The equivalence Class testing is effective where system takes set of data within a range. It is assumed that all data in one equivalence class is treated same by the system. Boundary value testing and equivalence class testing can be performed on unit level, integration level or system level.
Non-functional testing
In the non-functional testing, the system is tested against the non-functional requirements. The non-functional requirements define quality aspects of a system. If a system is fulfilling all functional requirements but the system is unsecure or the system is very slow then the customer will probably not want such kind of a system. So only implementing functional requirements does not make a system complete. To test non-functional attributes, non-functional testing is used. Following are some of the examples of non-functional testing.
Compatibility testing
Compatibility testing is an example of non-functional testing. In compatibility testing we test the system under test in different environments. The aim behind running system under test in different environments is to check how it behaviors in different environments. For example for this thesis, system was run in Windows environment and in Linux environment. The difference in behavior was recorded and analyzed. As the system was intended to run in mostly Linux environment and sometimes in windows environment, more emphasize was for Linux environment. It was seen that Functionality was same in both environments only some minor graphical changes were recorded. For example, appearance of text fields, text areas, buttons and alert messages etc were different in Windows and Linux.
Usability testing
The aim behind usability testing is to analyze how much user friendly the system is or we can say that how easy it is for the users of the system to perform their operations correctly. Usability testing of a system is done by giving it to its end users. End users of a system test it and give their feedback regarding the system. Feedback of the users is the input for improvement of the system. To increase the usability of software, usability testing is performed. In usability testing, users are given an opportunity to use the system. They are given some tasks that they have to complete. While they are performing those tasks, usability experts observe users’ behavior. Users are encouraged to think aloud. Usability experts then ask some questions after the completion of each task. These questions are for example, how easy or difficult you found it to complete the task? What options you want to add into the software, what options you think are confusing. How you want them to be? After taking the feedback, usability experts prepare their analysis and give their recommendations to the developers about design changes. After the development of new prototype, the prototype is again tested by the users and same process is followed. Improvements are measured and the process is repeated until expected usability level is achieved.
For this thesis, after development of a prototype, automation team was asked to use the system. After their use, their feedback was taken and improvement was carried out for the next prototype. For example, there was a Execute Query button on AddCtcID interface as shown in Figure 37. When the prototype was given to automation team for usability testing, one of the automation team members gave feedback that this button should be removed and functionality of this button should be put in Next button. This feedback was discussed with automation team lead and after his approval, this change was implemented in next prototype.
Pros and Cons
There is a general perception that usability testing is not necessary, it requires complex and expensive activities. But in reality, usability activities are expensive but they pay off. It is proved in various cases that usability increases the sale, reduces maintenance and redesign costs, it reduces user support costs and improve brand name. [14]
Regression testing
Regression testing is done to uncover new bugs that may have introduced during the implementation of new features. The aim of regression testing is to verify old functionality after a new release or a new prototype. For example in this thesis work, regression testing of all old functionalities was performed in a newly developed prototype. Old functionalities were tested and it was ensured that old functions are working as they are intended to work and new functionality has not affected old functionality.
Table of contents :
1 Introduction
1.1 Problem statement
1.2 Context of study
1.3 Teams interaction in Ericsson
1.4 Approach
1.4.1 Development method
1.4.2 Verification and validation
1.4.3 Tools
1.4.4 Programming Language
2 Contribution
3 Theoretical framework
3.1 Software testing
3.2 Testing levels
3.2.1 Unit Testing
3.2.2 Pros and Cons
3.2.3 Stubs and Drivers
3.2.4 Integration Testing
3.2.5 System Testing
3.2.6 Acceptance testing
3.3 Testing modes
3.3.1 Black-box Testing
3.3.2 Pros and Cons
3.3.3 White-box Testing
3.3.4 Pros and Cons
3.3.5 Grey box testing
3.3.6 Pros and Cons
3.4 Testing Types
3.4.1 Functional testing
3.4.2 Pros and Cons
3.4.3 Non-functional testing
3.4.4 Regression testing
3.4.5 Progression testing
3.4.6 Automation testing
3.4.7 Pros and Cons
3.5 Hardware and tools
3.6 JCAT framework
3.6.1 JCAT Layers
3.6.2 JUnit 3
3.6.3 Subversion
3.6.4 Eforge
3.6.5 Maven
3.6.6 Hudson
3.6.7 Sonar
3.7 Information model
3.7.1 FV Legacy team
3.7.2 Automation team
3.7.3 Work Package
3.7.4 TC suite
3.7.5 TC
3.7.6 CTC
3.7.7 TC Header
3.7.8 Preamble
3.7.9 Post amble
3.7.10 Help methods
3.7.11 Main execution TC methods
3.7.12 Test methods
3.7.13 Signum
4 Implementation
4.1 Overall structure of the application
4.2 Screen shots of application
4.3 Structure of java file
4.4 Class diagram of java file
4.5 Inputs and output of the application
4.6 Manually created java file VS Java file created by Automatic Test Builder
5 Development process
5.1 Semi-automated process
5.1.1 Generation of text file
5.1.2 Reading text file and saving data in local database
5.1.3 TC header generation
5.2 Fully-automated process
5.2.1 Connection to database using SOAP service
5.2.2 Java file generation
5.3 GUI Design
5.4 Working GUI
5.5 Adding main execution TC methods and help methods
5.6 Creation of help methods class hierarchy
5.6.1 Creation of Hardware methods class hierarchy
5.6.2 Creation of System Function Group methods class hierarchy
5.7 Documentation development
5.8 Training of testers
6 Discussion
6.1 What they did before
6.2 What they can do now
6.3 Testing of the system
6.4 Estimations of experts about the application
6.5 Future Improvements
6.6 Reflection on development method
7 References
8 Appendix
8.1 Mockups
8.2 Code Snippets
8.2.1 Reading text file
8.2.2 Saving data into local database
8.3 Main execution TC methods example
8.4 Test method example
8.5 Manually created java file
8.6 Java file created by Automatic Test Builder
8.7 Time plan
