Carbon Development

Location

All unit tests should go in a subproject called test (example: com.sapient.framework.config.test). This allows for the ability to easily release versions of the application without the test harness code. Releasing applications with test harnesses included may lead to possible security holes or other unintended results.

During most builds, the test harnesses are compiled as a second step after the main module compile. These are the builds that are used for continuous integration, regression or user testing. Final quality assurance should be done on successful regression builds without the the test harnesses included. The master build file contains a distribution command the builds the appropriate libraries for the purposes of final distribution.

Naming

The major naming standards are:

• Unit test class names should be of the form “{package name being tested}Test” (example: ConfigTest where this class resides in the config.test package). For packages that contain large amounts of functionality, it may make sense to break the functionality tests into multiple test harnesses that are specific to a set of function points or use cases.

• Tests of large scale performance or threading integration tests should be accomplished in separate test harnesses from the main set. The theory is to keep the primary set of test harnesses as quick as possible to encourage developers to use them often during integration. The full set of tests will consequently be executed as part of the continuous integration process on the main build server. These classes should end with the moniker "PerformanceTest".

• Test method names should begin with ‘test’ and end with the name of the function point being tested. An example would be “testGetProperty” to test the functionality of retrieving a property within the configuration service.

JavaDoc

Class JavaDoc should describe what the class is meant to test. The first sentence of each method JavaDoc should be a brief summary of the test. The JavaDoc should also include that prerequisites for the test (test data that must be present, configuration requirements, etc.)

Test harness output

UNDER NO CIRCUMSTANCES SHOULD MESSAGES BE PRINTED TO STANDARD OUT IN SUCCESS CASES.

The guiding principle here is that output of information in success cases hides valuable information written in failure cases. It is also confusing to people running the tests because they can’t tell whether or not the output is expected. With this rule, the only expected output is in the case that something went wrong. Everything within reason should be done to adhere to this rule.

Upon a test failure, the failure, most probable cause and possible fixes should be displayed using JUnit’s assert methods. For test harnesses that require external setup, the failure message should contain a reference to the documentation describing the proper setup.

Results

Where possible, test harness should attempt to return the system to the entry state so that the system is left in the same state as when the test started. This will not always be possible, but is worth the effort to simplify testing and possible alterations in external state that may effect re-executions of test harnesses. A good example of this principle would be to create the tables and test data required to test a database oriented service and then remove those tables and data after the tests finish. Some care should be taken to be able to handle prior cases where the latter half of test cases where unable to run. In the case of a database service, it may mean being able to handle the situation where the tables and data already exist in an inconsistent state before the test starts. It could handle this by catching the creation exception and completing a cascade-drop before retrying.

What to test

The primary types of features to test include:

Main stream functionality

Mainstream functionality is the expected use of each public method for a service. Care should be taken to ensure that the mainstream methods are tested 100%.

Boundary conditions

Boundary condition tests execute methods to the limits of what their expected functionality is. An example would be sending Integer.MAX_VALUE and Integer.MIN_VALUE to a method that is designed to handle all valid integers.

Expected exceptions

This type of test should verify that the proper exceptions are thrown from methods when

Performance tests

A performance test is designed to verify and report on the speed or throughput of a particular service. Performance tests are different from other tests in that they should be kept apart from the primary test harnesses and in that they often take significantly longer than the normal test cycle execution. These tests are not required for every service, but are a good idea for any services that may possibly be used very often or in a highly concurrent manner.

Concurrency tests

Concurrency is the interaction of different streams of execution in a multithreaded environment. Concurrency tests are normally the execution of multiple threads testing the same or different functionality exposed by a single service. These tests are most often considered to be part of performance tests and do not usually run as part of the primary testing structure for a service. These tests are designed to verify the synchronization methods of services that utilize shared resources and should be considered a form of severe stress testing. This type of testing is highly recommended for any service that contains synchronization.

Black box vs. white box testing

Black box testing is the execution and verification of a service's external interface regardless of the internal state of the service. A white box test is the execution of a service's interfaces followed by the internal verification of that service's state. There are benefits to both types of testing, but quite often you'll find that it is a mixture of the two types the makes for the most robust and efficient testing strategy.

Black box testing is normally the default method for testing and is quite often all that is needed for simple services. The advantages of white box testing is that it may help to find problems of consistency in a service or to help narrow down the causes of certain types of failures. White box testing is normally recommended for the types of services that involve large amounts of very complex state. Care should be taken, however, in the design of white box tests on services that involve some form of concurrency as the simple action of internal state viewing may cause a false negative.