Thoughts about Testing

Why Do We Test?

“Programmers write unit tests so that their confidence in the operation of the program can become part of the program itself. Customers write functional tests so that their confidence in the operation of the program can become part of the program, too. The result is a program that becomes more confident over time – it becomes more capable of accepting change, not less.”

-- Kent Beck, Extreme Programming Explained

Testing processes exist to compare the answers to three basic questions:

1. What should the code do? What are the requirements?

2. What does the developer think the code should do? What is the developer’s understanding of the requirements?

3. What does the code actually do? How has the requirement been implemented?

There is a fourth question that often comes up during testing:

4. Does the code still do what’s expected after a change?

Developer Acceptance Testing

When Kent Beck talks about programmers writing tests to improve their confidence in the operation of the program, what they are doing is comparing the answer to questions 2 and 3. They are trying to determine to what extent they have correctly implemented their understanding of the requirements. As a result, we refer to this process as Developer Acceptance Testing. The objective of Developer Acceptance Testing is to ensure that the developers have correctly implemented their understanding of the requirements.

User Acceptance Testing

Developer Acceptance Testing, by itself, cannot ensure the correct implementation of a program, however. An additional process should exist to test that the developer’s understanding of the requirement is in synch with the customer’s actual requirement. Such a process must necessarily involve someone who is not the developer – usually the customer. This additional process would involve the customer determining the expected output and comparing that expected output to the actual output (in essence, comparing the answers to questions 1 and 3). We refer to such a process as Customer Acceptance Testing, or more commonly, User Acceptance Testing. The key objective of User Acceptance Testing is to ensure that the developers implemented the actual requirements correctly.

Regression Testing

Finally, although much has been written about how to reduce the amount of testing required for a particular change, few organizations have been willing to accept the risk associated with testing only a part of the system. Generally, an organization wants to re-test the entire system to ensure that the change hasn’t resulted in incorrect behaviour in some other part of the application. This process is generally referred to as Regression Testing. Regression testing generally involves repeating all the test cases that we previously executed against the system. Clearly, test case automation is a tremendous advantage in this case.

Testing in Practice

Having defined the above testing processes, let’s turn our attention to the mechanisms by which these processes are put into practice. Clearly each project team must decide for itself how to realize the testing processes, but here are some common practices.

Unit Testing

Objective:	Ensure that code units work correctly. A code unit is a class or small group of classes that work very closely together to fulfill a well-defined responsibility.
Requirements:	Unit tests should run quickly Unit tests should be easy to run and validate Unit tests should exercise each unit in isolation Unit tests should be repeatable
Characteristics:	Unit tests are typically written by and run by developers Unit tests are used in developments, during code merges (the continuous integration process) and during application builds. During unit testing, a code unit calls “mock” implementations of other components it requires to perform its operation. These mock implementations help achieve isolation of the code unit. Test data should be encapsulated within the test whenever feasible.
Best Practices:	Unit tests are usually implemented using the JUnit tool. Typically, each public method of a code unit is tested. JUnit provides clear indication of a test case’s success or failure (using the GUI, one gets a green bar or a red bar). JUnit tests can be automated as part of build scripts using the Ant tool, and Ant’s junit tasks. One can use metrics such as McCabe’s cyclomatic complexity to help determine how many JUnit test cases should be developed. Tools such as Clover can be used to validate adequate test coverage when running an entire suite of JUnit test cases.