How to query and validate Splunk logs in Service Suite

Overview

The splunk.js library provides a reusable integration layer between AIQ Service Suite and Splunk. It allows test scripts to query Splunk indexes using SPL (Search Processing Language), filter and retrieve log data, and assert on the results as part of a functional or end-to-end test validation flow.

Common use cases include:

• Asserting that an audit log event was written after a user action (e.g. login, purchase, payment)
• Validating specific field values in application logs (e.g. transaction status, HTTP response codes)
• Checking that no ERROR or FATAL log entries appeared after a critical scenario
• Waiting for async events to appear in Splunk before continuing a test
• Extracting values from Splunk and storing them in TheLocker for downstream test cases

Prerequisite: The AIQ TestNode executing the script must have network access to the Splunk REST API on port 8089. Verify security group / firewall rules before using this library.

---

Setup

1. Add splunk.js to your repository

splunk.js file: splunk.js 

Upload splunk.js to your scenario repository. A recommended location is a shared folder so it can be reused across multiple scenarios:

your-repo/
  shared/
    splunk.js <-- place it here
  scenarios/
    checkout-flow/
      setup.js
      script1.js

2. Configure DPL variables

Never hardcode Splunk credentials in scripts. Add the following variables to your DPL (Data Provider Library) or scenario data file:

DPL Variable	Description	Example
SPLUNK_HOST	Hostname or IP of the Splunk instance	splunk.internal.mycompany.com
SPLUNK_TOKEN	Splunk service account Bearer token	eyJrIjoiT...

To generate a Bearer token in Splunk: Settings → Tokens → New Token. Use a dedicated service account, not a personal user.

Do not commit credentials to the repository. Always use DPL variables or TheLocker for sensitive values.

3. Include the library in your script

Use the Service Suite include directive at the top of any script that needs Splunk access. The {ds} token resolves to the root of your scenario's data source directory at runtime:

/**
 * @aiq.servicesworkbench
 */
include('{ds}/shared/splunk.js');

---

Initialization

Call splunk.init() once at the beginning of each script before making any queries:

splunk.init({
  host:            $SPLUNK_HOST,    // DPL variable
  port:            8089,
  token:           $SPLUNK_TOKEN,   // DPL variable
  defaultIndex:    'app_logs',      // used when no index is specified in SPL
  defaultEarliest: '-15m',          // default look-back window for all searches
});

Full init options

Option	Default	Description
host	localhost	Splunk hostname or IP
port	8089	Splunk REST API port
scheme	https	https or http
token	null	Bearer token (preferred auth method)
username	null	Basic auth username (alternative to token)
password	null	Basic auth password (alternative to token)
defaultIndex	main	Fallback Splunk index
defaultEarliest	-15m	Default time window start
defaultLatest	now	Default time window end
defaultCount	1000	Max rows returned per search
pollIntervalMs	2000	How often to poll for job completion (ms)
maxPollAttempts	30	Max poll attempts before timeout (30 × 2s = 60s)

---

How HTTP calls work internally

The library uses native Service Suite HTTP commands throughout. No browser-style fetch or external modules are used.

Operation	Service Suite command used
Submit search job	createPost(url, body, "application/x-www-form-urlencoded") + process(call)
Poll job status	createGet(url) + process(call)
Fetch results	createGet(url) + process(call)
Per-request auth header	call.addHeader("Authorization", "Bearer ...")
Read response body	resp.toStringContent()
Read HTTP status code	resp.status
Sleep between polls	java.lang.Thread.sleep(ms)
Base64 encode (Basic auth)	Java.type("java.util.Base64").getEncoder().encodeToString(...)

---

API Reference

Query methods

Method	Returns	Description
splunk.search(spl, opts)	Array	Run any SPL query and return all result rows
splunk.searchOne(spl, opts)	Object|null	Return only the first result row, or null if none
splunk.count(spl, opts)	Number	Return the number of matching events
splunk.getField(spl, field, opts)	String	Return a single field value from the first row
splunk.getFields(spl, fields[], opts)	Object	Return multiple field values from the first row
splunk.getAllValues(spl, field, opts)	Array	Return one field's value across all rows
splunk.waitForEvent(spl, opts)	Array	Poll until at least one matching event appears

All query methods accept an optional opts object: { earliest, latest, count }. waitForEvent also accepts pollMs and maxAttempts.

Assertion methods

Method	Description
splunk.assertCount(rows, n, msg)	Fails if row count ≠ n
splunk.assertAtLeast(rows, n, msg)	Fails if fewer than n rows returned
splunk.assertFieldEquals(rows, field, value, msg)	Fails if the field on row[0] ≠ expected value
splunk.assertFieldContains(rows, field, substring, msg)	Fails if the field on row[0] does not contain the substring
splunk.assertNoErrors(index, earliest, msg)	Fails if any ERROR or FATAL entries exist in the time window
splunk.assertEventOccurred(spl, msg, opts)	Combined search + assertAtLeast(1) in one call

Utility methods

Method	Description
splunk.storeResult(rows, field, lockerKey)	Store a field value from row[0] into TheLocker for downstream scripts via _retrieve(lockerKey)

---

Usage Examples

Example 1 — Assert a login audit event was written

After executing a login API call, verify that Splunk captured the corresponding audit entry:

/**
 * @aiq.servicesworkbench
 */
include('{ds}/shared/splunk.js');
splunk.init({
  host:         $SPLUNK_HOST,
  token:        $SPLUNK_TOKEN,
  defaultIndex: 'audit',
});
var testUser = $TEST_USERNAME;
splunk.assertEventOccurred(
  'index=audit action=login userId="' + testUser + '" result=SUCCESS',
  'Login audit event must appear in Splunk after successful authentication',
  { earliest: '-5m' }
);
log('✓ Login audit event confirmed in Splunk');

Example 2 — Validate field values on a specific transaction

Retrieve a transaction log by its ID and assert that its status and currency fields are correct:

/**
 * @aiq.servicesworkbench
 */
include('{ds}/shared/splunk.js');
splunk.init({
  host:  $SPLUNK_HOST,
  token: $SPLUNK_TOKEN,
});
var txnId = _retrieve('LAST_TXN_ID');  // value stored by an upstream test case
var rows = splunk.search(
  'index=payments txnId="' + txnId + '" | head 1',
  { earliest: '-10m' }
);
splunk.assertAtLeast(rows, 1, 'Transaction ' + txnId + ' must appear in Splunk');
splunk.assertFieldEquals(rows, 'status',   'APPROVED', 'Transaction must be APPROVED');
splunk.assertFieldEquals(rows, 'currency', 'USD',      'Currency must be USD');
log('✓ Transaction ' + txnId + ' validated — status=APPROVED, currency=USD');

Example 3 — No-error assertion after a critical flow

After running a checkout scenario, assert that no ERROR or FATAL entries were written to the application log index during the test window:

/**
 * @aiq.servicesworkbench
 */
include('{ds}/shared/splunk.js');
splunk.init({
  host:         $SPLUNK_HOST,
  token:        $SPLUNK_TOKEN,
  defaultIndex: 'app_logs',
});
splunk.assertNoErrors('app_logs', '-10m',
  'No ERROR or FATAL log entries expected after checkout flow'
);
log('✓ No errors found in Splunk after checkout flow');

Example 4 — Wait for an async event

Some flows write to Splunk with a delay (e.g. payment processors, message queues). Use waitForEvent to poll rather than failing immediately:

/**
 * @aiq.servicesworkbench
 */
include('{ds}/shared/splunk.js');
splunk.init({
  host:  $SPLUNK_HOST,
  token: $SPLUNK_TOKEN,
});
var txnId = _retrieve('LAST_TXN_ID');
// Poll every 3 seconds, up to 10 times (30s total)
var rows = splunk.waitForEvent(
  'index=payments event=PAYMENT_CONFIRMED txnId="' + txnId + '"',
  {
    earliest:    '-5m',
    pollMs:      3000,
    maxAttempts: 10
  }
);
splunk.assertFieldEquals(rows, 'processorCode', '00',
  'Processor must return approval code 00'
);
log('✓ Async payment confirmation found in Splunk');

Example 5 — Extract a value and store it in TheLocker

Retrieve a generated value from Splunk and make it available to downstream test cases:

/**
 * @aiq.servicesworkbench
 */
include('{ds}/shared/splunk.js');
splunk.init({
  host:  $SPLUNK_HOST,
  token: $SPLUNK_TOKEN,
});
var testUser = $TEST_USERNAME;
var rows = splunk.search(
  'index=sessions userId="' + testUser + '" | sort -_time | head 1',
  { earliest: '-5m' }
);
splunk.storeResult(rows, 'sessionId', 'SPLUNK_SESSION_ID');
// Downstream scripts: var sid = _retrieve('SPLUNK_SESSION_ID');
log('✓ Session ID extracted from Splunk and stored in TheLocker');

Example 6 — SPL aggregation: validate counts across all hosts

Use a | stats pipe to check that no host served 5xx errors on a specific endpoint:

/**
 * @aiq.servicesworkbench
 */
include('{ds}/shared/splunk.js');
splunk.init({
  host:  $SPLUNK_HOST,
  token: $SPLUNK_TOKEN,
});
var rows = splunk.search(
  'index=api endpoint="/checkout" | stats count(eval(status>=500)) as server_errors by host | where server_errors=0',
  { earliest: '-30m' }
);
log('✓ ' + rows.length + ' host(s) had zero 5xx errors on /checkout in the last 30 minutes');

Example 7 — getAllValues: check for unexpected HTTP status codes

Retrieve all HTTP status codes from recent API calls and fail if any 5xx appears:

/**
 * @aiq.servicesworkbench
 */
include('{ds}/shared/splunk.js');
splunk.init({
  host:  $SPLUNK_HOST,
  token: $SPLUNK_TOKEN,
});
var statuses = splunk.getAllValues(
  'index=api endpoint="/api/order" | head 200',
  'status',
  { earliest: '-5m' }
);
var badStatuses = statuses.filter(function(s) {
  return parseInt(s) >= 500;
});
assert(badStatuses.length === 0,
  'Found ' + badStatuses.length + ' 5xx response(s) on /api/order: ' + badStatuses.join(', ')
);
log('✓ All ' + statuses.length + ' /api/order responses were below 500');

---

Error handling

All library methods throw a descriptive error on failure. These surface directly in the AIQ test report with context including the HTTP status code, Splunk SID, field name, or assertion mismatch. No additional try/catch is required — let errors propagate so the scenario fails with a clear message.

---

Common issues

Symptom	Likely cause	Fix
HTTP 401 on job creation	Invalid or expired token	Regenerate the token in Splunk Settings → Tokens
Connection refused / timeout	TestNode cannot reach Splunk port 8089	Check EC2 security groups and firewall rules
assertAtLeast fails even though event exists	Log ingestion lag — event not yet indexed	Use splunk.waitForEvent() instead of splunk.search()
Search times out	Broad query or slow Splunk cluster	Narrow the SPL time window or add more filters; increase maxPollAttempts
Field not found error	Field name differs from what Splunk extracts	Run the same SPL in the Splunk UI and verify field names in the results table

---

Best practices

• Always use DPL variables for SPLUNK_HOST and SPLUNK_TOKEN. Never hardcode connection details in scripts.
• Keep time windows short (e.g. -5m or -10m) to keep searches fast and assertions specific to the current test run.
• Use | head N in SPL to cap result sizes when you only need a few rows.
• Prefer waitForEvent over a fixed sleep for async flows — it stops as soon as the event appears.
• Use assertNoErrors as a blanket post-scenario health check in addition to specific field assertions.
• Use storeResult to pass Splunk values into TheLocker rather than re-querying Splunk in every downstream script.
• Call splunk.init() once per script at the top, before any other splunk.* call.