2.8 - Results Analysis Results Analysis — Appvance IQ documentation. Results Analysis Contents Determining the Length of Tests Finding and Solving the Failures Evaluate Applications Ability to Achieve Linear Scalability About Reports Live Results While a Test Runs Determine if the Test System is a Performance Bottleneck Viewing Results from a Previous Test Run What Can Go Wrong Results Analysis and Reporting Unified Test Platform Results Analysis Engine presents test results in several reports. This contains a set of charts showing you just where the app failed, the number of transactions, CSV Reports. This chapter shows how to determine the root cause of functional issues, performance problems, and app service delivery problems. Unified Tess Platform accomplishes this by storing the steps, times, pass / fail results, the reason for failures from every step of every test. Appvance IQ Monitor stores the CPU, memory, network, and app-specific observations from the servers running the web server, application server, database server, and load balancer. Below are the topics that are covered: Determining How Long Do Test Cases Take Run time features Finding and solving the failures Evaluate App's Ability to Achieve Linear Scalability About different Reports Live Results Determine If Test system is a Performance Bottleneck Viewing results from previous test runs What can go wrong Last updated: March 23, 2023 If you have any feedback on this topic, please contact us at documentation@appvance.ai Determine the Right Test Duration Determine the Right Test Duration Choose test duration based on the performance question being answered. A short smoke run can validate a script, while a longer run may be needed to observe steady-state behavior, resource growth, or reliability under sustained load. Common duration patterns Smoke validation: short run to confirm the scenario and data work. Baseline run: enough time to reach steady state and capture representative metrics. Endurance run: longer execution used to observe stability and resource trends. Before interpreting results Exclude ramp-up and ramp-down periods when analyzing steady-state performance. Confirm the run had enough valid samples for the metric being reviewed. Document environment, load model, data, and duration with the result. Last updated: September 18, 2026 Find and Resolve Test Failures Find and Resolve Test Failures Start failure analysis with the first meaningful failure in the execution. Later errors often come from the same root cause and can distract from the actual problem. Failure triage Scenario or locator issue: the application changed or the step targets the wrong element. Test data issue: required data is missing, reused, expired, or invalid. Environment issue: the application, network, browser, or Test Node is unstable. Application defect: the system under test behaves incorrectly under the tested condition. Recommended workflow Open the failed execution and identify the first failed step or request. Review output, screenshots, logs, and consumed test data. Compare with a known good execution when available. Fix the scenario only after confirming the issue is test-maintenance related. Re-run a focused validation before marking the issue resolved. Last updated: September 18, 2026 Evaluate Linear Scalability Evaluate Linear Scalability Linear scalability compares how the application responds as load increases. A scalable system should increase throughput predictably while keeping response times and errors within acceptable limits. What to compare Virtual users or request volume at each load level. Throughput and transaction completion rate. Response time trends, including percentile behavior. Error rate and application or infrastructure saturation signals. Recommended workflow Run controlled tests at increasing load levels. Keep test data, environment, and scenario mix consistent between runs. Compare steady-state periods instead of ramp-up or unstable intervals. Identify the load level where throughput stops increasing or errors rise. Last updated: September 18, 2026 About Reports List of Available Reports Below you will find the list of all the Unified Test Platform Reports. The initial report screen shows eleven reports as shown below. In order to understand better and use these reports it is important to understand some basic concepts: Execution: This is the complete execution of a scenario, it is possible that it can contain several iterations. Iterations: A iteration indicates the number of users who run during a certain amount of time, the below screen-shot shows an execution with 2 iterations. Test Case: Iteration can have one or more Test Cases. Test Case can be a unit test, a selenium file, or any other supported test type by Unified Test Platform. Test Cases are executed in parallel during an iteration. Template Reports Users can create their own Template Reports and create Reports by using these Templates. See the section on Report Templates in the Global Options section of the documentation for information on creating report templates. Server Side Report “Server-Side Only” is a filter where every “row”(step registry) that has a value = 0 in the column “totalRequest” won’t be added in the report. This Report can be used for filtering the Reports. With this not only the “set_value” steps are filtered, but every step where “totalRequest” values are zero gets filtered and becomes a Server Side Report. Functional report Useful for functional testing. Shows first transaction detailed information Scalability Report Scalability Report shows how the application scales during the test. Scaling means the more users use the service, the more transactions will be achieved. Scalability Report is provided in Transactions per second (TPS for short). Good environments should have linear scalability, meaning if X user generates Y transactions, N*X users will generate N*Y transactions. Automatically shows the expected Linear Scalability vs the actual Transactions Per Second achieved. TestCase Execution Count TestCase Execution Count Report shows total successful transactions versus total failures transactions. Each Iteration can be selected from the drop-down. It is useful to quickly see if there are spikes in failures and the overall performance of the service. TestCase Execution TestCase Execution Report shows transactions duration every iteration. Iteration needs to be selected from the drop-down as well the Test Case. Resource correlation Correlated successful transactions vs. resource uses. It is useful to pinpoint bottlenecks on a machine. Several machines can be added for resource monitoring. AIQ recommends adding monitors to all the machines that are used during the test. They might include databases, web servers, load balancers, etc. In order to add machines, a UTPMonitor service should be installed on the desired machine. Appvance IQ support APCMonitor for Windows, Linux, and Mac Operating Systems. Resource Usage Resource usage Report shows machine resource uses during a test. It is useful to pinpoint bottlenecks on a machine. Several machines can be added for resource monitoring. AIQ recommends adding monitors to all the machines that are used during the test. They might include databases, web servers, load balancers, etc. In order to add machines, an APCMonitor service should be installed on the desired machine. AIQ supports APCMonitor for Windows, Linux, and Mac machines. JVM METRICS: Heap Usage This is a JVM metric that shows the used Heap Memory. The heap memory is a run-time data area from which memory for all class instances and arrays are allocated. AIQ by default initializes the heap size limit as 1500 Mb. Garbage Collector JVM Garbage Collector is a process that automatically handles disposing of memory is no longer used. This metric shows the CPU usage of this process. Thread count Shows the current number of active threads in JVM OS METRICS: CPU Usage CPU usage as reported by the operating system. MEM Percentage of memory used (against total memory) as reported by the operating system. NET Net traffic as reported by the operating system. Results on Time Results on Time graph gives you the option of choosing the "Element type" with "Metrics" that are available in the drop-down. Element type contains Test Case, Step, and Request whereas Metrics drop-down contains Size related, Count related, and duration related. Users can select the combination from either 2 and create a graph. In addition to the graph, the user can also download CSV reports depending upon the metrics selected in the "Results on Time" Graph. If the user has selected Duration-related metrics the user can download CSV reports of Avg duration, min duration, and so on. So the CSV reports can be downloaded depending upon the metric selected. The data points for response times and hits/sec are calculated from the total number of executions for a specific time slice (granularity value). The granularity value is automatically chosen, depending on the duration of the test. For example, if the chosen granularity value is one minute, then the system will create an average. response time value, by averaging the response times of Test Case executions occurring every minute. This graph is useful to monitor the state of each Test Case, while the test is still running. TestNodes TestNodes report shows quickly the output and error of the execution TestNodes. It might be useful for debugging purposes. Step Times Step Times Report shows transaction's step information per Iteration. The step information showed is the maximum, minimum, average step time. It also includes the duration standard deviation. Finally, it shows the 85, 90, and 95 duration percentile. This means the maximum duration of 85, 90, and 95% of the steps. It is useful to set limits and confirm them. E.g. 90% of login steps should take less than 10 seconds. Step Times CSV Report Unified Test Platform also provides a CSV report of the step times. It can be opened in almost any spreadsheet editor. Note: In order to capture total throughput in the CSV Report, Error logging should be enabled at the Options panel. Step Summary Report Unified Test Platform also provides a CSV report of the Step Summary. It can be opened in almost any spreadsheet editor. Location-Based Report Unified Test Platform also provides a CSV report of the step times based on each location. It can be opened in almost any spreadsheet editor. There will be a separate link to download the CSV report for each location available. Clicking on the location link will download the location-based report in CSV format as shown below. Note: In order to capture the location-based reports, the User has to enable the locations by going to the Test Cases tab and selecting "Users by Location" and assigning the users in each location. Downloading Graph Unified Test Platform also provides a feature wherein the user can download the step times graph make changes to the graph like annotate etc. Failures report Failure report shows the number of successes and failures from all the iterations (Scenario wise). Just like the step times report, a bar graph is shown with the list of successes and failures. To check the failures in detail, you can click the Download Detailed Failures Report link which will download an HTML report showing all the required data in detail. Iteration Summary Iteration Summary is a CSV document that shows execution information during the iterations. The iteration summary is the most complete report. It contains several rows showing each Test Case with the total transactions achieved, the total failures, and success. It also contains the average, maximum, minimum, and average duration of each Test Case and the Standard deviation. It also indicates the number of users who executed each Test Case during the iteration. Also, it shows the 85, 90, and 95 percentile of each transaction. This means the maximum duration of 85, 90, and 95% of the transactions per Test Case. Percentile is useful to set limits and confirm them. E.g. 90% of Test Case "Working Sequence" should take less than 10 seconds. Finally, it shows all the above information for the iteration as a whole and not only its Test Cases. Compare Summaries The iteration Summary tab also allows the user to compare the reports of the same test that was run previously. This helps to compare the changes in response times. By default, the comparison takes place between the current execution and the execution which was run previously. The screenshot of the UI is shown below. The user loads the baseline and clicks on Compare Summaries and a report in HTML format will be opened which gives the comparison between executions. Security Report The security report tab is only available whenever you execute an App Penetration test type, another kind of scenario will not display this tab since there will be no report available. This report is generated based on the ZAP OWASP open-source tool which is included in AIQ as one of our engines to perform web security tests. It is a formatted PDF file with the list of vulnerabilities of a website returned by OWASP. All about requests and responses Below are some of the details of requests and responses. All the links below a page are requests with size in bytes - This is in the controller window after running a test. Test Case execution tab - Click success or failure transaction - go to request and response tab to see the list of requests and responses. Clicking a request will download the response as seen below: Report - Iteration summary report - provides details in a CSV sheet, This shows the request name, the response codes of each request, a number of occurrences of this request under a URL, what time it took to load that request (Average duration), the minimum and maximum duration is also provided in case of the performance load test. Filter response content: We also have the option to filter out the response contents to be stored in the database or not with this feature. Visit Global Options for more details. Plot Graphs: We can use Results on time graphs to plot graphs based on a Request from a particular step. There are a lot more that we could offer with requests and responses, but again since we have all these data with us, presenting them to the customer as they requested does not take much time for the Appvance team. Assertions: Assertions can be added to Requests as well, please refer to Validation and Store Test Suite - HAR for more details. Last updated: July 10, 2023 If you have any feedback on this topic, please contact us at documentation@appvance.ai Review Live Results While a Test Runs Review Live Results While a Test Runs Use live results to monitor a test while it is running. Live data helps detect invalid runs early, but final analysis should use completed results after the run has ended and all data has been processed. Monitor during execution Active users and ramp behavior. Error rate and repeated failure messages. Response-time trend changes. Test Node or environment health symptoms. When to stop a run Stop if credentials, data, or environment selection is wrong. Stop if the application under test becomes unstable outside the intended test scope. Stop if setup errors make the result invalid. Last updated: September 18, 2026 Determine if the Test System is a Performance Bottleneck AIQ is a distributed test environment where tests run remotely on TestNodes. A TestNode is software running on a server that uses threads to operate multiple virtual users concurrently. As the number of concurrent virtual users increases, the TestNode consumes additional CPU, memory, and network resources. If the TestNode reaches its resource limits, it can become a performance bottleneck and affect the results of the test. To determine how many virtual users a TestNode can support before reaching its resource limits, AIQ recommends using Calibration Testing. The calibration testing methodology consists of three steps: Identify the test use cases. A test use case contains the steps required to operate the application. Identify the optimal throughput of the application and test environment. Gradually increase the number of virtual users and monitor the application's performance and the resources consumed by the test environment. Run the complete performance test and analyze the results. Use the results from the calibration testing to determine whether the TestNode or the application infrastructure becomes a bottleneck during the complete performance test. For detailed instructions, see the Calibration Test Methodology chapter of the AIQ User Guide. Estimating Server Resources Required to Support a User Population Performance testing with AIQ Monitor can be used to identify the server-side resources required to support a population of application users. AIQ Monitor is an agent-based service that observes CPU, network, and memory utilization while AIQ executes a performance test. AIQ stores the monitored values in the Results Repository, normally using MySQL or Oracle. The Resource Utilization report correlates the application's performance with the resources consumed by the monitored servers. This helps identify whether application infrastructure becomes a bottleneck as the number of virtual users increases. Use the Resource panel in the Scenario Editor to identify the AIQ Monitors from which you want to collect results. For example, AIQ Monitor can be configured on: Web servers Application servers Database servers Load balancers Viewing Resource Utilization The Resource Utilization report is available from the Analyze section. To view the resource utilization for a test execution: Open Analyze from the AIQ Start Screen. Select the execution you want to analyze. Select Resource Utilization from the list on the left. Use the report to compare application performance with CPU, memory, and network utilization. This information can help determine whether the TestNode or application infrastructure is limiting the number of concurrent virtual users that the environment can support. Last updated: August 17, 2026 If you have any feedback on this topic, please contact us at documentation@appvance.ai. View Results from a Previous Test Run View Results from a Previous Test Run Use previous test run results to compare baselines, investigate past failures, and track performance trends over time. Confirm that the compared runs used the same environment, scenario version, data assumptions, and load model before drawing conclusions. Recommended workflow Locate the run by project, suite, execution date, or execution ID. Open the completed result and review summary status first. Compare key metrics or failed steps against the current run. Record relevant links or execution IDs in the related defect, report, or maintenance note. Comparison notes Avoid comparing runs that used different environments or data sets unless that difference is the point of the analysis. Use steady-state periods for performance comparisons. Document any scenario or application changes between runs. Last updated: September 18, 2026 What Can Go Wrong During Results Analysis Recommended guidance Results analysis can be misleading when test data, environment, load model, monitoring, or reporting context is incomplete. Review failures and performance results with enough evidence to separate application defects from test or environment issues. Common causes Unstable test data, exhausted accounts, or missing cleanup. Browser, Test Node, network, database, or integration issues. Incorrect assertions, waits, ramp-up, pacing, or load model assumptions. Small sample sizes, high error rates, or incomplete monitoring. Comparing runs from different environments or configurations. Review workflow Start with the first meaningful failure, not the final cascading error. Compare with a known-good run and check environment changes. Use logs, reports, screenshots, metrics, and test data evidence together. Document conclusions, open questions, and retest criteria. Last updated: October 7, 2026