Method for Black Box and White Box Tests

Testing Internet applications for scalability and performance is different from testing software applications and code. SOA testing is focused on understanding how a service responds to increasing levels of concurrent requests, message sizes, and response handling techniques. The nature of Internet applications testing is black-box testing; it doesn't matter what happens inside the box.

Code profilers have their place in testing software and software developers often rely on them to learn the location of performance problems. However, black-box testing often yields more actionable knowledge, with these recommendations:

1. Create a baseline performance metric (a Scalability Index) using black-box performance tests showing Transaction Per Second (TPS) results, measured at the service consumer, with a variety of message sizes, message schema complexities, and concurrent request levels

2. Compare performance and scalability between multiple servers, consumers, or brokers by identifying each server's Performance Index and normalizing the test parameters to avoid reporting false slow-performance results. The test is run properly once the test lab has been calibrated in this Calibration step

3. Determine the Performance Index of the service under test and use white-box testing techniques to profile the largest time expensive object operations to handle requests. Optimize the software based on the profile

4. Continue optimizing the service by repeating steps 2 and 3

5. Run the Performance Index and analyze the results

This is the basis for understanding the Appvance method of testing services for scalability and performance