2.9 - Calibration Testing Calibration Testing — Appvance IQ documentation. Determine the Scale and Resources Needed for a Performance Test Contents Linear Scalability in Test Servers The Methodology Transforming Calibration Results to TPS Mitigations Rapidly Determine The Scale Of A Performance Test The Appvance Calibration Testing Methodology is an important step in determining app scalability and performance under load. Performance testing has many phases that take a long time: problem and success criteria definition, use case definition, test scripting, test data preparation, calibration testing, test operation, and reporting. The Appvance methodology for calibration testing saves a tester's time by avoiding unnecessary test operations. Performance tests operate a test script from a test server that acts as a client or consumer of an app. The test server has finite resources: CPU, network, memory, storage, threads, and processes. Drive a test too hot and the test server operating the performance test runs out of resources and falsely identifies performance bottlenecks in the app. In this case, it is the test server that has the performance bottleneck, not the app itself. Testers need to determine the optimal number of concurrently running test scripts on a single test server first. Distributed performance test environments scale up by finding the maximum amount of load a single server generates and then running the test across multiple test servers concurrently to reach the required level of load on the app. If you have any feedback on this topic, please contact us at documentation@appvance.ai Linear Scalability in Test Servers Start a performance test too high and the test delivers results that are a function of a performance bottleneck in the test server itself and not the application under test. Start too low and the results may not hit the upper limits of your app performance and the organization operating the test pays unnecessarily for equipment not used to its potential. Calibration testing determines the optimal level of load by identifying the topmost throughput in a test that achieves linear scalability on a single test server. Applications with linear scalability deliver proportionally increased throughput as the test increases the number of concurrently running users. The above chart runs 10 users concurrently - with each user running a test script over and over-delivers 4 transactions per second (TPS.) A transaction is the successful operation of a test use case, test suite, or test script. With linear scalability doubling the number of users doubles the throughput. In the above example, 20 users of an application with linear scalability deliver 8 TPS. Time for a quick test: If a test at 20 virtual users delivers 8 TPS, how many TPS will 40 Users deliver with linear scalability? 40 users are double the 20 user level; therefore, 40 users deliver 16 TPS with linear scalability. Calibration testing determines the peak throughput possible given the operating environment. A Scalability Index is a measurement of app throughput at several levels of load. In the above example, 40 concurrent users are the optimal level for this application. When the calibration test goes to 80 users the TPS level decreases to 10 TPS. Above 80 users the application throughput continues to decrease. There are many reasons for results as found in the above example: the app backend services may run out of network bandwidth, database caches may have filled, the load balancer may be pegging traffic to a few app servers, the app may have a logical error that prevents it from performing when more than a certain number of other users are on the app. And, of course, it could be the test server has run out of resources to operate the test! Scaling the test to the app and the test servers is time-consuming. If each test level takes 45 minutes to operate (10 minutes to start up, 20 minutes to run, 15 minutes to ramp down) the total test time to build the above Scalability Index is 3.75 hours! With limited performance testing time available, Appvance recommends using the Calibration Testing Methodology to rapidly identify the optimal test levels for the application under test and avoid running levels that do not lead to actionable knowledge. Last updated: June 02, 2023 If you have any feedback on this topic, please contact us at documentation@appvance.ai The Methodology Here are the steps to this methodology: Determine a starting number of concurrently running virtual users. I use the following table from past experiences: Web apps with links and forms, start at 30 virtual usersAjax apps with Javascript, start at 10 virtual usersSOAP/REST services, start at 50 virtual users message Queues (TIBCO Business Works, Oracle ESB, IBM Process Server), start at 100 virtual usersJava objects method calls, start at 200 virtual users Identify the average time to operate one virtual user session. Multiply this value by 2.5. We will refer to this value as calibration test time (CTT.) For example, if the average time to run a virtual user session is 34 seconds set the CTT value at 85 seconds Operate a performance test for the CTT value at 3 levels of virtual users: 50%, 100%, and 150% of the starting number. For example, run a test of Ajax apps at 5, 10, 15 virtual users The above results show linear scalability within 10% from test 2 to test 3. The blue area in test 3 shows where the TPS needs to be for linear scalability from test 2. Test 4 shows a performance bottleneck. TPS at linear scalability in test 4 needs to be at the top of the blue area (14 TPS.) Instead, the test server or the app is seriously constrained to perform. TPS is a measurement of the rate of completed transactions. Since the test time is the same in tests 2, 3, and 4 the TPS value is so low because the time to operate each transaction has grown longer at test 4. Appvance Performance Cloud's (AQI's) Transaction Distribution chart shows the transaction times. See the Results Analysis chapter of the AIQ User Guide for details. Investigate the cause of the performance results in test 4. Look at the test server resources as test 4 operates. Is the CPU running above 90%? Is the network interface above 80%? Are there available network sockets? Is a data source providing the test script operating slowly? Is there free memory? Is there a logical problem in the test script? If you find any of these causes, resolve them and rerun the test in step 1. If test 4 shows linear scalability from test 3, then operate test 5 at 150% of test 4 virtual user level If test 4 performs 90% or less of the linear scalability from test 3, then operate test 1 at 50% of the test 2 level. Continue increasing by 150% or decreasing by 50% the virtual user level until the test reaches 1 virtual user or the test fails to reach 10% of linear scalability from the previous high test measurement Use the required total virtual user count divided by the virtual user count from the test with the highest test measurement on a single test server to determine the number of test servers required to run the performance test. For example, if the total virtual user count is required at 5000, and a single test server produces scalable results at 10 virtual users, then the test requires 500 test servers. With limited performance testing time available, Appvance recommends using the Calibration Testing Methodology to rapidly identify the optimal test levels for the application under test and avoid running levels that are not meaningful. For example, we saved 45 minutes in the above test by not running test 1 when test 3 demonstrates linear scalability. Additionally, you now have the flexibility to run the performance test at lower levels of virtual users per machine, a greater number of machines, or any combination of both. Last updated: June 02, 2023 If you have any feedback on this topic, please contact us at documentation@appvance.ai Transforming Calibration Results to TPS Apps are increasingly used as part of marketing campaigns to engage customers in brands. Brand managers and business managers typically do not talk in terms of virtual users and transactions per second. For example, PepsiCo's brand manager for a Mountain Dew app promoting Halo 4 and Xbox knows how many customers will use the app after running national television advertising. Brand managers typically understand the measurement of user sessions in a given period of time. For example, an advertising campaign will generate peak app usage of 6,000 user sessions in a 10 minute period. Transform the virtual user charts above by determining the average transaction time. Divide by the test time. Then, multiply by the virtual user count. For example, a test with 60 second average transaction time, 5 virtual users, and 10 minutes of test time operate 50 transactions in a 10 minute period. Formula: (test time in seconds / average transaction time in seconds) × virtual user count Understanding TPS Transactions Per Second (TPS) can be measured and may be counterintuitive. Appvance IQ shows a system's Scalability Index chart in the test results, displaying the system's throughput at four concurrent request levels. Test Results of a System's Throughput at 4 Concurrent Requests Levels Figure 1: Test Results of a System's Throughput at 4 Concurrent Request Levels System throughput is measured as the number of transactions a system handles as more requests are received. A perfect information system handles requests at a constant rate regardless of the number of requests; it increases its transactions per second to maintain a constant response time. Charting a perfect system's scalability shows the TPS rate increasing in equal proportion to the number of received requests; it is a linear relationship. Figure 2: Linear relationship of a perfect system For instance, if a perfect system handles 100 concurrent requests in 10 seconds with a 2 second response time, then maintaining the 2 second response time, it should handle 200 concurrent requests in the same 10 second period. Scalability Index for a Perfect System Figure 3: Scalability Index for a Perfect System At each concurrent request level, the system handles requests at a measured rate in transactions per second, yielding the system's response time. As concurrent request levels increase, the system handles requests at the same rate, so the number of transactions increases in equal proportion. TPS keeps going up and to the right in equal proportion to the number of requests. Figure 4: TPS increases in equal proportion to requests For instance, with 100 concurrent requests, a system handling 1,000 requests in a 10 second period yields a handling rate of 100 transactions per second. The same system at 200 concurrent requests handles 2,000 requests in the same 10 second period, increasing the handling rate to 200 transactions per second. That is perfect scalability - the Holy Grail of performance testing. Receiving more requests does not slow down the overall system response time. However, a typical system hitting a bottleneck behaves differently. A Service Exhibiting a Performance Bottleneck Figure 5: A Service Exhibiting a Performance Bottleneck As the system receives a larger number of concurrent requests it slows down when responding to each request. Increasing past 400 concurrent requests, the system would eventually reach zero transactions per second. Many systems checked for scalability have this problem. The Scalability Index helps system managers plan system capacity to achieve the desired throughput needed to keep user efficiency constant while helping developers understand how their design and coding decisions impact performance. Figure 6: Bottleneck behavior A Service Exhibiting a Scalability Problem Figure 7: A Service Exhibiting a Scalability Problem As shown in the first three columns, the system is capable of handling an increased level of concurrent requests. However, the fourth column shows the system hits an upper limit in handling transactions, typically resulting from a database indexing problem, a full data cache, or a saturated network connection. Figure 8: Scalability problem - upper limit reached This simple TPS method can be convoluted, as shown by the results of the following scalability test. Parameter Values Required to Calibrate a Test Figure 9: Parameter Values Required to Calibrate a Test From the results above, TPS increases only slightly even though the test is making 2.5 (25 / 10) times more concurrent requests. The TPS value should have increased by 2.5 times, to 0.775 (0.31 TPS at 10 concurrent requests × 2.5). Free-running threads generate concurrent requests with no sleep time between requests. Their job is to keep making requests to the server during the test period. Yet, the average response time with 25 users is 5.34 times longer (65,344 milliseconds at 25 users divided by 12,234 milliseconds at 10 users). Consequently, there are fewer opportunities to log results and therefore increase the TPS value. Throughput (TPS) Decrease as Service Response Time Increases, as Measured from the Consumer Figure 10: Throughput (TPS) decrease as service response time increases When a test increases the number of concurrent users, one of three things can happen: The server takes less time (on average) to respond than at lower CRs levels. In this condition, each CR finishes sooner, logs a response (a transaction), and makes its next request much sooner. TPS increases from lower CRs levels. The server takes the same time (on average) to respond as at lower CRs levels. In this condition, each CR takes the same amount of time but more CRs are running concurrently. TPS goes up from the lower CRs levels proportionately to the CRs increase. The server takes more time (on average) to respond than at the lower CRs levels. In this condition, each CR finishes later, resulting in fewer opportunities for the server to handle more requests. TPS drops proportionately to the increased response time. How TPS Values Are Calculated in the Scalability Report Assuming the reader has a fair knowledge of creating scenarios and viewing reports, this section explains how the TPS and Linear TPS values are calculated in the scalability report. Below is the scalability report used for the TPS calculation: Figure 11: Scalability report for the TPS calculation The formula for the calculation is: TPS = Test Cases Passed / duration (in seconds) From the iteration summary report, we get the test cases passed and the duration in seconds. Figure 12: Iteration summary report Using the iteration report above, 7.29 TPS is calculated. For the 502 (total users) iteration: 7.29 TPS = 13,259 / (30 × 60 + 18) s Take iteration summary _801 and calculate 10.12 TPS. Figure 13: Iteration summary - 10.12 TPS For the 802 (total users) iteration: 18,463 / (30 × 60 + 24) s = 10.12 TPS To calculate the Linear TPS: Linear TPS = (total users in the iteration summary which has linear scalability) × (TPS of the preceding iteration summary / total users in the preceding iteration summary) 11.65 TPS = 802 × (7.29 / 502) Last updated: February 17, 2023 If you have any feedback on this topic, please contact us at documentation@appvance.ai Mitigations Calibration testing focuses on throughput measured from the client (load-generation) equipment. Monitoring resource usage—including CPU, network, and memory—on both the client and server during a calibration test can help identify and diagnose performance issues within the test itself. If you have any feedback on this topic, please contact us at documentation@appvance.ai