2.8 - Results Analysis

Results Analysis — Appvance IQ documentation.

Results Analysis

Contents

Results Analysis and Reporting

Unified Test Platform Results Analysis Engine presents test results in several reports. This contains a set of charts showing you just where the app failed, the number of transactions, CSV Reports. This chapter shows how to determine the root cause of functional issues, performance problems, and app service delivery problems. Unified Tess Platform accomplishes this by storing the steps, times, pass / fail results, the reason for failures from every step of every test. Appvance IQ Monitor stores the CPU, memory, network, and app-specific observations from the servers running the web server, application server, database server, and load balancer.

Below are the topics that are covered:

Last updated: March 23, 2023

If you have any feedback on this topic, please contact us at documentation@appvance.ai

Determine the Right Test Duration

Determine the Right Test Duration

Choose test duration based on the performance question being answered. A short smoke run can validate a script, while a longer run may be needed to observe steady-state behavior, resource growth, or reliability under sustained load.

Common duration patterns

Before interpreting results

  1. Exclude ramp-up and ramp-down periods when analyzing steady-state performance.
  2. Confirm the run had enough valid samples for the metric being reviewed.
  3. Document environment, load model, data, and duration with the result.

Last updated: September 18, 2026

Find and Resolve Test Failures

Find and Resolve Test Failures

Start failure analysis with the first meaningful failure in the execution. Later errors often come from the same root cause and can distract from the actual problem.

Failure triage

  1. Open the failed execution and identify the first failed step or request.
  2. Review output, screenshots, logs, and consumed test data.
  3. Compare with a known good execution when available.
  4. Fix the scenario only after confirming the issue is test-maintenance related.
  5. Re-run a focused validation before marking the issue resolved.

Last updated: September 18, 2026

Evaluate Linear Scalability

Evaluate Linear Scalability

Linear scalability compares how the application responds as load increases. A scalable system should increase throughput predictably while keeping response times and errors within acceptable limits.

What to compare

  1. Run controlled tests at increasing load levels.
  2. Keep test data, environment, and scenario mix consistent between runs.
  3. Compare steady-state periods instead of ramp-up or unstable intervals.
  4. Identify the load level where throughput stops increasing or errors rise.

Last updated: September 18, 2026

Understand AIQ Reports

Last reviewed: September 22, 2026

Use AIQ reports to evaluate execution health, response timing, failures, throughput, and supporting evidence after a test run.

Validation workflow

  1. Open the completed execution report from Results Analysis.
  2. Review summary status, duration, transaction counts, and failure rates.
  3. Drill into failed steps or transactions before drawing conclusions.
  4. Compare metrics against baseline and acceptance thresholds.
  5. Export or share report links according to project reporting policy.

Note: A single report is evidence, not root cause. Correlate with logs, monitoring, and environment changes.

Last updated: September 22, 2026

If you have feedback on this topic, contact documentation@appvance.ai.

Review Live Results While a Test Runs

Review Live Results While a Test Runs

Use live results to monitor a test while it is running. Live data helps detect invalid runs early, but final analysis should use completed results after the run has ended and all data has been processed.

Monitor during execution

When to stop a run

  1. Stop if credentials, data, or environment selection is wrong.
  2. Stop if the application under test becomes unstable outside the intended test scope.
  3. Stop if setup errors make the result invalid.

Last updated: September 18, 2026

Determine if the Test System is a Performance Bottleneck

AIQ is a distributed test environment where tests run remotely on TestNodes. A TestNode is software running on a server that uses threads to operate multiple virtual users concurrently.

As the number of concurrent virtual users increases, the TestNode consumes additional CPU, memory, and network resources. If the TestNode reaches its resource limits, it can become a performance bottleneck and affect the results of the test.

To determine how many virtual users a TestNode can support before reaching its resource limits, AIQ recommends using Calibration Testing.

The calibration testing methodology consists of three steps:

  1. Identify the test use cases.
    A test use case contains the steps required to operate the application.
  2. Identify the optimal throughput of the application and test environment.
    Gradually increase the number of virtual users and monitor the application's performance and the resources consumed by the test environment.
  3. Run the complete performance test and analyze the results.
    Use the results from the calibration testing to determine whether the TestNode or the application infrastructure becomes a bottleneck during the complete performance test.

For detailed instructions, see the Calibration Test Methodology chapter of the AIQ User Guide.

Estimating Server Resources Required to Support a User Population

Performance testing with AIQ Monitor can be used to identify the server-side resources required to support a population of application users.

AIQ Monitor is an agent-based service that observes CPU, network, and memory utilization while AIQ executes a performance test. AIQ stores the monitored values in the Results Repository, normally using MySQL or Oracle.

The Resource Utilization report correlates the application's performance with the resources consumed by the monitored servers. This helps identify whether application infrastructure becomes a bottleneck as the number of virtual users increases.

Use the Resource panel in the Scenario Editor to identify the AIQ Monitors from which you want to collect results. For example, AIQ Monitor can be configured on:

Viewing Resource Utilization

The Resource Utilization report is available from the Analyze section.

To view the resource utilization for a test execution:

  1. Open Analyze from the AIQ Start Screen.
  2. Select the execution you want to analyze.
  3. Select Resource Utilization from the list on the left.

Use the report to compare application performance with CPU, memory, and network utilization. This information can help determine whether the TestNode or application infrastructure is limiting the number of concurrent virtual users that the environment can support.

Last updated: August 17, 2026

If you have any feedback on this topic, please contact us at documentation@appvance.ai.

View Results from a Previous Test Run

View Results from a Previous Test Run

Use previous test run results to compare baselines, investigate past failures, and track performance trends over time. Confirm that the compared runs used the same environment, scenario version, data assumptions, and load model before drawing conclusions.

  1. Locate the run by project, suite, execution date, or execution ID.
  2. Open the completed result and review summary status first.
  3. Compare key metrics or failed steps against the current run.
  4. Record relevant links or execution IDs in the related defect, report, or maintenance note.

Comparison notes

Last updated: September 18, 2026

What Can Go Wrong During Results Analysis

Results analysis can be misleading when test data, environment, load model, monitoring, or reporting context is incomplete. Review failures and performance results with enough evidence to separate application defects from test or environment issues.

Common causes

Review workflow

  1. Start with the first meaningful failure, not the final cascading error.
  2. Compare with a known-good run and check environment changes.
  3. Use logs, reports, screenshots, metrics, and test data evidence together.
  4. Document conclusions, open questions, and retest criteria.

Last updated: October 7, 2026