Test Generation

Appvance AI quickly learns how real users use an application, learns how to create proper requests, automatically generates correlation data, connects to data sources, and then automatically generates 100’s, 1000s, or 10,000+ scripts at your command, literally in a few seconds. Blazing fast, no-hands, automatic test generation.

This section details options available in Appvance's AI 2.0 implementation. This applies to AIQ versions prior to AIQ 5.0.0.

Tests can be run as functional or performance tests. And developers can quickly surface the best areas to improve application usage, usability, and performance.

  • Allows to rapidly create millions of test scripts without writing a single line of code.

  • Needs a set of server log files and a Blueprint file

  • Server logs indicate what users did in the application

  • Automatic generation of API Level and UX Level scripts

  • Requests in Blueprint are matched against user logs

  • The Blueprint file provides the necessary knowledge to locate which action or actions triggered that request and how to reproduce that request

The below section shows the creation of these tests.

From the left navigation panel, under AI-Driven, click AI Test Generation. There are 6 tabs

  • Logs Directory

  • Blueprint

  • Settings

  • Output

  • Summary

  • Results

Logs

This is the tab where the user browses the logs folder.

Appvance AI chose to utilize any simple server logs (basic breadcrumbs) which track usage. Real users on production systems, or business analysts on QA servers. All servers have production logs. These are often ignored, yet can have valuable information in terms of understanding usage patterns. Of course, many SIEM systems also keep logs (such as Sumo Logic) that are usable by the Appvance AI system.

Appvance AI supports these types of logs:

  • Standard CSV

  • Sumo Logic CSV

  • Server Logs

Parameters:

  • Logs Directory: Browse and choose the folder (that contains all the log files, could be 100's or 1000's of log files), full directory is selected in this step as the name mentioned 'Logs Directory'

  • Max number of scripts to create: Defaults to 100, creates the specified number of scripts or within the specified number.

  • Remove Duplicate scripts: The system can determine if two or more scripts have the same steps or a script is contained in another longer script.

    • Options are:

    • Remove duplicated and contained scripts - Removes duplicated scripts and also the contained scripts. Example: If AI creates a script with steps 1, 2, 3, 4, 5, and if there is another script that has steps 1, 2, 3 which is actually covered in the other script. This is basically a subset that shall be removed as well

    • Remove duplicated scripts - Removes all the duplicated scripts

    • Include all scripts - Includes all the scripts including the duplicate scripts

Error Driven Test

Generate test scripts and scenarios with only log requests that have status 300 or above.

If logs do not have the mentioned status, it will generate 0 test scripts.

Once the option is enabled, you will be presented with the drop-down to choose the response codes.

Below is a couple of examples log file screenshots:

Some common questions that would come to your mind about these logs files.

What are these logs or log files?

Log files contain the user's workflow on a web or other application (login, browsing, actions, filtering data, add to cart...all actions), Also known as production logs which usually contain large sets of data.

How can you get these logs?

On server machine(s), you can install Sumo Logic, or any SIEM system to monitor the apache "acces.log" file or create a Sumo Logic account to monitor the same file. You can use the apache "access.log" file format or change modify this using Sumo Logic. All production machines have production logs usually in W3C format even without installing any 3rd party application.

As all traffic to the server is logged, regardless of being generated from multiple concurrent origins, the session information in each log line is required for log-in apps, unless it is known that the traffic comes from a single session. Apache logs can be configured to include the session number when the sessionId cookie is available: Stackoverflow Reference. But is important to notice that usually, first calls from navigation to a server will usually not have a session cookie (as login is required for having a session id.). This is s regard, SumoLogic can be a better sources for log's on apps that required the separation of the logs by individual users as these tools have more controls.

We do not have access to application servers. Is it possible to collect logs without server access using these tools?

Server access is required to accomplish this task, although we can use a network sniffer tool:

show filter field: click View| "Use quick filter"

save logs: click File | Save all Items, save as CSV file

Once the file is saved, remember to rename the column headers

Host Name: vhost

Method: method

Path: uri

Response Code: status

Server Address: is not needed

How do these logs generate scripts?

Logs by themselves do not generate scripts. The script generator generates scripts using a large data set which may include production logs (W3C or many SIEM logs), a complete application map (called a Master Key File which can be manually generated or automatically generated which was created in the earlier section and will have to be another input in the next step), items to be data-driven, a data source to drive those, correlations, location/address of the QA server for this application, changes in the build, expected responses, etc. The AI system learns from each of these sources and then is able to generate scripts that will represent real-user flows. The system can create 100's or 1000's scripts in order to attain high code coverage.

What should be the file formats of Standard CSV, Server Logs, and Sumo Logic CSV?

Captured logs should be exported to a CSV in the below formats.

Standard CSV

Log files in a CSV format require the following columns: _time, vhost, method, uri, status

Server Logs

Server log files require the following columns: host, ident, authuser, date, request, status, bytes

Sumo Logic CSV

Sumo Logic log files must be in CSV format and require the following columns: method, status, uri, vhost

Sample Search Parse Statement

spree | parse "* - - [*] \"* * HTTP/1.1\" * *" as vhost,time,method,uri,status

Error Driven Tests

This is how Error Driven tests work:

Consider you have logs that contain different statuses that range from 100 to 500

1xx Informational. This class of status code indicates a provisional response, consisting only of the Status-Line and optional headers, and is terminated by an empty line. ...

2xx Success. ...

3xx Redirection. ...

4xx Client Error. ...

5xx Server Error.

Sample logs:

status, vhost, uri

500, http://demosite.com , /path1/path2/page.html

400, http://demosite.com , /path1/path3/page.html

200, http://demosite.com , /path1/path5/page.html

While creating the test scenario, check "Error Driven test" on the logs directory tab, this will allow the AI test generator to catch only the "500" response code and omit everything else like 200, 400, 404, etc, and creates a scenario to check only against these response codes and provides the result to the user.

Blueprint

This is the next tab after Logs Directory, where you will have to browse the Blueprint file (.abp) that was generated in the "Create Blueprint" Section.

Input Format Type: Blueprint and Master Har file.

Appvance AI can generate both UX and API level scripts.

API Level scripts:

  • These are “low-level” scripts because it tests what is “going under” when the application runs

  • Generated scripts are Javascript files .js that contain many GET, POST, PUT, PATCH, OPTION requests with their necessary headers

  • Allows adding assertions for content type and response code

  • Each log file is loaded and passed into the ECB algorithm to generate the most probable path of actions for each log by grabbing the requests of each action

  • Opens browser and Test Designer will execute each request with its headers in the generated scripts

UX Level Scripts:

  • These are “high-level” scripts because it tests the functions and actions of the application available to the user

  • Generated scripts are Test Designer files (.ds) or Javascript files (.js) that contain steps

  • Each log file is loaded and passed into the ECB algorithm to generate the most probable path of actions for each log by grabbing the action string of each action of the path

  • Opens browser and Test Designer will execute each step of generated scripts. Users can see each browser performing actions in the application.

Browse and choose the .abp file. Choose the Scenario Execution Type and the script type.

Click Next.

Settings

Global Setup: Allows to execute of a .ds or .js file before test scripts are run. Useful for performing login or adding custom correlations.

Copy 
// Please copy the code lines below this comment and modify as required necessary.
        
//
        
// LOG IN TO THE APP
        
// In case you need to login into your application before running the logs
        
//
        
// The locators and URLs should be changed to match those in your application.
        
// In the case you don't know this, you can use UTP Record to record your login process
        
// and export the recorded steps as javascript using the menu Export option
        
//
        
// The variables like #username, @password, and $companyid used below, needs to be provided
        
// to the setup script in UTP scenario editor as a data source DPL that read the credentials
        
// from a CSV or a database.
        
// Steps are:
        
//
        
// 01: NAVIGATING TO SIGN IN
        
// _setDomain("https://application_domain.com");
        
// _navigateTo("https://application_domain.com/login"); // navigate to the app login url
        
// _setValue(_emailbox("email"),$username);
        
// _setValue(_password("Password"),$password);
        
// _click(_span("Sign in"));
        
// _selectMainFrame();
        
//
        
// 02: EXTRACTING AN AUTHENTICATION TOKEN
        
// var getRequest = _createGet("/request/path/that/includes/authentication?token"); // need to look for the request that contains the authentication header token and provide it here as long as with the header name
        
// getRequest.addExtractAction("http","CookieValue:authentication.tkt","authentication.tkt"); // first parameter is type, second is target and third is variable name where to store the extracted value
        
// var resp = getRequest.eval();
        
//
        
// 03: ADDING THE AUTHENTICATION REQUEST HEADER TO ALL TEMPLATE REQUESTS
        
// _addGlobalUpdateHeaderAction("CsrfToken","authentication.tkt",true);
        
//
        
// 04: MANAGING DYNAMIC PATHS
        
// Sometimes the requests path can vary on the session logged. In this case, the dynamic part must be identified
        
// and a Regex expression that matches this varying part should be added in the _ignorePathRegex functions as:
        
//
        
// _ignorePathRegex("1[0-9]+");
        
// _addGlobalUpdatePathAction("1[0-9]+",$companyid,false);
        
//
        
// ************************ SAMPLE CONFIGURATION FILE ************************ //
        
_navigateTo("https://application_domain.com/login"); // navigate to the app login url
        
_setValue(_emailbox("email"),$username);
        
_setValue(_password("Password"),$password);
        
_click(_span("Sign in"));
        
_setDomain("https://application_domain.com");
        
var getRequest = _createGet("/request/path/that/includes/authentication?token");
        
getRequest.addExtractAction("http","CookieValue:authentication.tkt","authentication.tkt");
        
var resp = getRequest.eval();
        
_addGlobalUpdateHeaderAction("CsrfToken","authentication.tkt",true);
        
_ignorePathRegex("1[0-9]+");
        
_addGlobalUpdatePathAction("1[0-9]+",$CompanyId,false);
        

Skip Failed Validations:

This check-box can be checked if you want to ignore all the script failures due to failed validations. If the script needs to fail on any failed validations, this check-box should be un-checked.

Browser: A drop-down list with Firefox, Google Chrome, and Edge browsers, you can choose 1 from the drop-down and the test cases that are created in Scenario builder will use this browser for the script playback.

Output

This is the place where you mention the Scenario Name, Scenario Test Type, Location details, and the Output folder to store all the scripts.

Scenario Test Type: Is either functional or performance Test

Selecting Functional Test Type, you can provide the Maximum parallel Virtual Users.

With Performance Test Type, you can provide Global Test Ramp up in Seconds, Total Duration of the performance test (A completion criteria), Concurrent users per test case.

Always save logs (Failures only) is a checkbox checked by default, can be checked to analyze the failures when the scenario is run.

Locations:

This is all about test distribution, Tests can be run locally or on multiple servers or can also be run on Cloud locations as Appvance IQ is integrated with EC2, Softlayer, and OpenStack cloud services.

Add Server Location:

1 or several server locations can be added by using the 'Add Server Location' link.

Add Cloud Location:

Refer to Test Deployment in Cloud chapter to learn more about Adding a cloud location.

Output Scenario and Scripts folder:

Is simply an empty folder to save all the scripts and scenarios created by Appvance AI. It is required to have this folder empty else you will be prompted to overwrite the files in the output folder.

The system will generate a scenario file and the scripts folder with test cases, dpl, and setup files.

Each log file will become a test case which can be seen and edited in the Test Cases tab of the Scenario Editor.

dpl.csv will be generated with variables and values of datasets. This file can be edited in the Resources tab of the Scenario Editor.

An empty setup file will be created if none was provided in the previous step. This file can be edited in the Resources tab of the Scenario Editor.

Summary

The summary is simply the summary of all the previous selections, you can go back to the previous tabs and make necessary changes if required using the Previous button or can directly click the tabs.

Max Duration:

Max Duration is based on minutes, by defining a maximum value you can limit the duration of Generating Scripts process

  • Enter 0 to define an unlimited time

  • Enter 120 to limit the duration to 2 hours

  • You can always define a different Max Duration, before starting Generating Scripts

  • Defaults to 2 hours duration

Hitting 'Generate Scripts' will analyze all the logs from the folder selected and also from the .abp file that was provided as input in the 2nd step and will process all the files and provide us with the scenario file with all the scripts generated (and also based on the selections we made in a different tab)

Results

Results will show the status of the test generation with all the logs processed along with the number of requests processed in each of the log files.

All the details like, Current Action - Running, Not Running, Complete status shall be shown.

Elapsed Overall time shall be displayed.

Max Duration took.

Average time of Script Creation in seconds

Number of Scripts created

Number of Scripts Expected

Number of Scripts Found and also Duplicate Scripts.

It will show all the log files that were processed and a link to the scenario created, clicking which will take you to Appvance IQ Scenario builder.

There shall also be a link to click here, which creates the new flow for the AI Script generation.

A scenario will be created based on all the inputs provided by the user.

Scenario Type

Completion criteria

Number of concurrent users

Duration of the Test

Locations - where to run the scenario

Browsers chosen

and other settings that were provided as input.

From here you can simply click play scenario and the scenario would start playing the scenario with the test cases created and start analyzing the results.

Some common questions about the scripts generated

What are these scripts?

UX Scripts:

It generates designer scripts that are fully data enabled, you could open the scripts folder and open the scripts in test designer to check the created script.

It generates Java scripts that are fully data enabled, you could open the scripts folder and open the java scripts in JS Edit & Play to check the created JS scripts.

API Level Scripts:

The scripts are at the HTTP level and requests are formed with java-script. Note that scripts can be simple ones that might be just 10 steps or complex data-driven 100 step scripts. The system does not care.

Below is an example screenshot of the script that was created.

How do we have to customize the scripts?

These scripts are fully editable, although that is not the normal use case. Since machine-generated scripts better represent human user flows than could be written, they should generally be left alone since any changes would only reduce code coverage versus real users instead of increasing it. Validations can be set in the validation library, though the system can automatically validate against expected responses learned from the production system. So in general, even validations do not have to be human-generated and added (though they can be).

How do we know that test cases are valid that are being generated and created with AI?

Test cases are generated using Appvance’s machine learning and understanding of the Blueprint of the application. The logs then help tell Appvance paths the user takes in the application, and then using the Knowledge Matrix (Appvance’s AI/Machine Learning) with paths to take in the application, Appvance is able to use that information to produce scripts. In essence, the scripts generated can only follow paths that users have already followed in the application - meaning they cannot be invalid.

A Summary of Appvance AI Benefits

  • Appvance AI can rapidly generate 1000’s scripts, run them, and report the results

  • Appvance AI is 1000x faster than manual testing or scripting

  • Appvance AI can improve code coverage over any other manual or automated testing

  • Appvance AI can rapidly find real user bugs and retest and verify to drive rapid response in the dev

  • Appvance AI generates tests that represent real user actions

  • Appvance AI can utilize the same scripts for functional or performance tests

  • Appvance AI can create tests for web, web services, mobile, and IoT applications

  • Appvance AI tests can be launched by popular CI tools and fully supports a DevOps methodology

  • Appvance AI can modify use cases (maintain) automatically as an application changes

  • Appvance AI can validate against any criteria including known response codes, timeouts, or expected responses