ElasticSearch OpenSearch Integration

The ElasticSearch/OpenSearch integration allows you to share the information of steps and/or requests execution during a scenario execution with an AWS OpenSearch managed service or an ElasticSearch 7.x server.

For an in-depth comparison of Amazon OpenSearch and ElasticSearch see OpenSearch vs. Elasticsearch - Dattell.

This integration allows you to leverage Kibana and OpenSearch Dashboards for performance analysis on steps/requests durations executed by an AppvanceIQ scenario.

This integration has two different ways to connect and send data:

  • Using a REST Header Authentication

  • Using AWS Authentication Protocol

Using a REST Header Authentication

The integration relies on sending results data to a RESTFUL API endpoint and a token or apikey exposed by the search service.

This mode of connection is available in AIQ 4.9.2 and above.

This mode requires that you will also create an Authorization token or apikey(apikey you will use when connecting to an ElasticSearch or Elk Stack services, and for AWS OpenSearch you will need the token)

To create the basic authentication token you must to create a user.

Authentication Example

Here is a simple basic authentication approach using Kibana’s internal user database.

  1. Click Internal User Database and create a new user by typing a user name and password. You can limit the role of the user created in the Role Mappings option.

  2. A basic authentication token can be created by appending the kibana’s user name + “:” + password and then converting that string to a Base64 encoded string. You can do this with Chrome’s JS Console.

  3. For the basic authentication scheme you will need above string but with the prefix 'Basic '.

  4. So final token will be similar to this:

    Basic dGVzdC12c2VyOlRoZVChc3N3b3JkSGVyZQ==

    Above is just for using Kibana internal user database. Using a cloud-based Kibana you will need an APIKEY that can be created using the steps in this document and will have a format similar to: ApiKey aVZlLUMzSUJuYndxdDJvN0k1bU46aGxlYUpNS2lTa2FKeVZua1FnY1VEdw==

Setting up Search Integration in Scenario Editor for REST authorization header option

  1. Open your scenario using the Appvance IQ Scenario Editor and select the Integrations tab. You can only have one integration configured per scenario.

  2. Click on the Add Integration link and select the “Search API Connector” integration in the dropdown.

  3. Enter the endpoint URL, the index name, and the authorization token/apike.

  4. In the dropdown select the level of detail that you need to send to the search service (STEPS or REQUESTS).

  5. You can use the Add a string to identify the data for this execution field as metadata to identify the scenario name in the data sent to elastic search.

Using AWS Authentication Protocol

This connection mode uses the AWS Java SDK protocol for authentication. The connection will first attempt to use environment variables with access credentials, but if they are not present it will try to use other methods provided in the AWS authentication protocol.

This mode of connection is only available in AIQ version 4.10.4 and above.

The recommended method for configuring your Appvance IQ system is through an Instance Identity Role. With this mode temporary credentials are provided by the ec2 instance’s metadada and assigned to an ec2 user case role.

This method requires to be able to access the Instance Metadata Service (IMDS) which provides temporary access credentials an a token to the requesting connection.

For more information see: Instance identity roles - Amazon Elastic Compute Cloud in the Amazon documentation.

The setup is detailed in the following two sections, which shows how to create an ec2 use case role and how to apply it to an ec2 instance.

Create a ec2 use case role with permission to index records to the Amazon OpenSearch service

Assign the role to the EC2 machine where the AppvanceIQ controller is hosted.

Select your Appvance IQ controller instance in the AWS EC2 console and select the Actions > Security > Modify IAM Role option.

This authentication protocol requires that the EC2 where Appvance IQ is running has the instance metadata service active which can be tested by connecting to the EC2 server and navigating to the following URL: http://169.254.169.254/latest/meta-data/

The response should be similar to the following:

If the metadata URL does not respond means that you need to restore the Instance metadata service or create a new EC2 instance from an Amazon provided AMI. In the case you get 'Waiting for the metadata service' message please follow the steps here in the Amazon documentation.

For more information on instance metadata see: Retrieve instance metadata - Amazon Elastic Compute Cloud

Configuring the integration in Scenario Editor

To use the ec2 instance role mode just select the 'AWS Authentication' in the Type of authentication select option in the Integration tab in the Scenario Editor.

Setting up an Amazon OpenSearch Service

OpenSearch is an open-source, distributed search and analytics suite derived from Elasticsearch supporting up to the 7.1 version, along with OpenSearch dashboards and Kibana.

  1. To create your Amazon OpenSearch domain, see the "Creating and managing Amazon OpenSearch Service domains" documentation in the Amazon OpenSearch Developer Guide.

  2. Once you have created the domain, you will have a Kibana URL that you will use to create your visualizations and dashboards, and a domain endpoint that you will need to configure your AppvanceIQ scenario editor integration.

Refer to the following Amazon documentation for other safe authentications options that you can configure. These are not covered in this topic.

Importing a Sample Set of Visualizations, Dashboard and Index

As a starting point to visualize your AIQ data within Kibana you can import a JSON file that contains sample visualizations and a proposed dashboard using the attached scripts.

In Kibana, import the JSON file using the option in Management > Saved Objects > Import.

Customizing Data Sent from AIQ to OpenSearch

You can use the metadata for the test scenario to display the JSON to display in a dashboards. This can be accomplished using the test scenario name which is the metadata content defined in the Add a string to identify the data for this execution field when setting up the integration in the Scenario Editor.