How to query and validate Splunk logs in Service Suite

Overview
The 
splunk.js library provides a reusable integration layer between AIQ Service Suite and Splunk. It allows test scripts to query Splunk indexes using SPL (Search Processing Language), filter and retrieve log data, and assert on the results as part of a functional or end-to-end test validation flow.
Common use cases include:
Asserting that an audit log event was written after a user action (e.g. login, purchase, payment)
Validating specific field values in application logs (e.g. transaction status, HTTP response codes)
Checking that no ERROR or FATAL log entries appeared after a critical scenario
Waiting for async events to appear in Splunk before continuing a test
Extracting values from Splunk and storing them in TheLocker for downstream test cases
Prerequisite: The AIQ TestNode executing the script must have network access to the Splunk REST API on port 8089. Verify security group / firewall rules before using this library.
Setup
1. Add splunk.js to your repository
splunk.js file: splunk.js 
Upload 
splunk.js to your scenario repository. A recommended location is a shared folder so it can be reused across multiple scenarios:
your-repo/
  shared/
    splunk.js       <-- place it here
  scenarios/
    checkout-flow/
      setup.js
      script1.js
2. Configure DPL variables
Never hardcode Splunk credentials in scripts. Add the following variables to your DPL (Data Provider Library) or scenario data file:
DPL Variable
Description
Example
SPLUNK_HOST
Hostname or IP of the Splunk instance
splunk.internal.mycompany.com
SPLUNK_TOKEN
Splunk service account Bearer token
eyJrIjoiT...
To generate a Bearer token in Splunk: Settings → Tokens → New Token. Use a dedicated service account, not a personal user.
Do not commit credentials to the repository. Always use DPL variables or TheLocker for sensitive values.
3. Include the library in your script
Use the Service Suite 
include directive at the top of any script that needs Splunk access. The 
{ds} token resolves to the root of your scenario's data source directory at runtime:
/**
 * @aiq.servicesworkbench
 */
include(
'{ds}/shared/splunk.js'
);
Initialization
Call 
splunk.init() once at the beginning of each script before making any queries:
splunk
.init({
  
host
:
            $
SPLUNK_HOST
,
    
// DPL variable
  
port
:
            
8089
,
  
token
:
           $
SPLUNK_TOKEN
,
   
// DPL variable
  
defaultIndex
:
    
'app_logs'
,
      
// used when no index is specified in SPL
  
defaultEarliest
:
 
'-15m'
,
          
// default look-back window for all searches
});
Full init options
Option
Default
Description
host
localhost
Splunk hostname or IP
port
8089
Splunk REST API port
scheme
https
https or 
http
token
null
Bearer token (preferred auth method)
username
null
Basic auth username (alternative to token)
password
null
Basic auth password (alternative to token)
defaultIndex
main
Fallback Splunk index
defaultEarliest
-15m
Default time window start
defaultLatest
now
Default time window end
defaultCount
1000
Max rows returned per search
pollIntervalMs
2000
How often to poll for job completion (ms)
maxPollAttempts
30
Max poll attempts before timeout (30 × 2s = 60s)
How HTTP calls work internally
The library uses native Service Suite HTTP commands throughout. No browser-style 
fetch or external modules are used.
Operation
Service Suite command used
Submit search job
createPost(url, body, "application/x-www-form-urlencoded") + 
process(call)
Poll job status
createGet(url) + 
process(call)
Fetch results
createGet(url) + 
process(call)
Per-request auth header
call.addHeader("Authorization", "Bearer ...")
Read response body
resp.toStringContent()
Read HTTP status code
resp.status
Sleep between polls
java.lang.Thread.sleep(ms)
Base64 encode (Basic auth)
Java.type("java.util.Base64").getEncoder().encodeToString(...)
API Reference
Query methods
Method
Returns
Description
splunk.search(spl, opts)
Array
Run any SPL query and return all result rows
splunk.searchOne(spl, opts)
Object|null
Return only the first result row, or null if none
splunk.count(spl, opts)
Number
Return the number of matching events
splunk.getField(spl, field, opts)
String
Return a single field value from the first row
splunk.getFields(spl, fields[], opts)
Object
Return multiple field values from the first row
splunk.getAllValues(spl, field, opts)
Array
Return one field's value across all rows
splunk.waitForEvent(spl, opts)
Array
Poll until at least one matching event appears
All query methods accept an optional 
opts object: 
{ earliest, latest, count }. 
waitForEvent also accepts 
pollMs and 
maxAttempts.
Assertion methods
Method
Description
splunk.assertCount(rows, n, msg)
Fails if row count ≠ n
splunk.assertAtLeast(rows, n, msg)
Fails if fewer than n rows returned
splunk.assertFieldEquals(rows, field, value, msg)
Fails if the field on row[0] ≠ expected value
splunk.assertFieldContains(rows, field, substring, msg)
Fails if the field on row[0] does not contain the substring
splunk.assertNoErrors(index, earliest, msg)
Fails if any ERROR or FATAL entries exist in the time window
splunk.assertEventOccurred(spl, msg, opts)
Combined search + assertAtLeast(1) in one call
Utility methods
Method
Description
splunk.storeResult(rows, field, lockerKey)
Store a field value from row[0] into TheLocker for downstream scripts via 
_retrieve(lockerKey)
Usage Examples
Example 1 — Assert a login audit event was written
After executing a login API call, verify that Splunk captured the corresponding audit entry:
/**
 * @aiq.servicesworkbench
 */
include(
'{ds}/shared/splunk.js'
);
splunk
.init({
  
host
:
         $
SPLUNK_HOST
,
  
token
:
        $
SPLUNK_TOKEN
,
  
defaultIndex
:
 
'audit'
,
});
var
 testUser 
=
 $
TEST_USERNAME
;
splunk
.assertEventOccurred(
  
'index=audit action=login userId="'
 
+
 testUser 
+
 
'" result=SUCCESS'
,
  
'Login audit event must appear in Splunk after successful authentication'
,
  
{
 
earliest
:
 
'-5m'
 
}
);
log(
'✓ Login audit event confirmed in Splunk'
);
Example 2 — Validate field values on a specific transaction
Retrieve a transaction log by its ID and assert that its status and currency fields are correct:
/**
 * @aiq.servicesworkbench
 */
include(
'{ds}/shared/splunk.js'
);
splunk
.init({
  
host
:
  $
SPLUNK_HOST
,
  
token
:
 $
SPLUNK_TOKEN
,
});
var
 txnId 
=
 
_retrieve(
'LAST_TXN_ID'
);
  
// value stored by an upstream test case
var
 rows 
=
 splunk
.search(
  
'index=payments txnId="'
 
+
 txnId 
+
 
'" | head 1'
,
  
{
 
earliest
:
 
'-10m'
 
}
);
splunk
.assertAtLeast(
rows
,
 
1
,
 
'Transaction '
 
+
 txnId 
+
 
' must appear in Splunk'
);
splunk
.assertFieldEquals(
rows
,
 
'status'
,
   
'APPROVED'
,
 
'Transaction must be APPROVED'
);
splunk
.assertFieldEquals(
rows
,
 
'currency'
,
 
'USD'
,
      
'Currency must be USD'
);
log(
'✓ Transaction '
 
+
 txnId 
+
 
' validated — status=APPROVED, currency=USD'
);
Example 3 — No-error assertion after a critical flow
After running a checkout scenario, assert that no ERROR or FATAL entries were written to the application log index during the test window:
/**
 * @aiq.servicesworkbench
 */
include(
'{ds}/shared/splunk.js'
);
splunk
.init({
  
host
:
         $
SPLUNK_HOST
,
  
token
:
        $
SPLUNK_TOKEN
,
  
defaultIndex
:
 
'app_logs'
,
});
splunk
.assertNoErrors(
'app_logs'
,
 
'-10m'
,
  
'No ERROR or FATAL log entries expected after checkout flow'
);
log(
'✓ No errors found in Splunk after checkout flow'
);
Example 4 — Wait for an async event
Some flows write to Splunk with a delay (e.g. payment processors, message queues). Use 
waitForEvent to poll rather than failing immediately:
/**
 * @aiq.servicesworkbench
 */
include(
'{ds}/shared/splunk.js'
);
splunk
.init({
  
host
:
  $
SPLUNK_HOST
,
  
token
:
 $
SPLUNK_TOKEN
,
});
var
 txnId 
=
 
_retrieve(
'LAST_TXN_ID'
);
// Poll every 3 seconds, up to 10 times (30s total)
var
 rows 
=
 splunk
.waitForEvent(
  
'index=payments event=PAYMENT_CONFIRMED txnId="'
 
+
 txnId 
+
 
'"'
,
  
{
    
earliest
:
    
'-5m'
,
    
pollMs
:
      
3000
,
    
maxAttempts
:
 
10
  
}
);
splunk
.assertFieldEquals(
rows
,
 
'processorCode'
,
 
'00'
,
  
'Processor must return approval code 00'
);
log(
'✓ Async payment confirmation found in Splunk'
);
Example 5 — Extract a value and store it in TheLocker
Retrieve a generated value from Splunk and make it available to downstream test cases:
/**
 * @aiq.servicesworkbench
 */
include(
'{ds}/shared/splunk.js'
);
splunk
.init({
  
host
:
  $
SPLUNK_HOST
,
  
token
:
 $
SPLUNK_TOKEN
,
});
var
 testUser 
=
 $
TEST_USERNAME
;
var
 rows 
=
 splunk
.search(
  
'index=sessions userId="'
 
+
 testUser 
+
 
'" | sort -_time | head 1'
,
  
{
 
earliest
:
 
'-5m'
 
}
);
splunk
.storeResult(
rows
,
 
'sessionId'
,
 
'SPLUNK_SESSION_ID'
);
// Downstream scripts: var sid = _retrieve('SPLUNK_SESSION_ID');
log(
'✓ Session ID extracted from Splunk and stored in TheLocker'
);
Example 6 — SPL aggregation: validate counts across all hosts
Use a 
| stats pipe to check that no host served 5xx errors on a specific endpoint:
/**
 * @aiq.servicesworkbench
 */
include(
'{ds}/shared/splunk.js'
);
splunk
.init({
  
host
:
  $
SPLUNK_HOST
,
  
token
:
 $
SPLUNK_TOKEN
,
});
var
 rows 
=
 splunk
.search(
  
'index=api endpoint="/checkout" | stats count(eval(status>=500)) as server_errors by host | where server_errors=0'
,
  
{
 
earliest
:
 
'-30m'
 
}
);
log(
'✓ '
 
+
 rows
.
length
 
+
 
' host(s) had zero 5xx errors on /checkout in the last 30 minutes'
);
Example 7 — getAllValues: check for unexpected HTTP status codes
Retrieve all HTTP status codes from recent API calls and fail if any 5xx appears:
/**
 * @aiq.servicesworkbench
 */
include(
'{ds}/shared/splunk.js'
);
splunk
.init({
  
host
:
  $
SPLUNK_HOST
,
  
token
:
 $
SPLUNK_TOKEN
,
});
var
 statuses 
=
 splunk
.getAllValues(
  
'index=api endpoint="/api/order" | head 200'
,
  
'status'
,
  
{
 
earliest
:
 
'-5m'
 
}
);
var
 badStatuses 
=
 statuses
.filter(
function
(
s
)
 
{
  
return
 
parseInt(
s
)
 
>=
 
500
;
});
assert(
badStatuses
.
length
 
===
 
0
,
  
'Found '
 
+
 badStatuses
.
length
 
+
 
' 5xx response(s) on /api/order: '
 
+
 badStatuses
.join(
', '
)
);
log(
'✓ All '
 
+
 statuses
.
length
 
+
 
' /api/order responses were below 500'
);
Error handling
All library methods throw a descriptive error on failure. These surface directly in the AIQ test report with context including the HTTP status code, Splunk SID, field name, or assertion mismatch. No additional 
try/catch is required — let errors propagate so the scenario fails with a clear message.
Common issues
Symptom
Likely cause
Fix
HTTP 401 on job creation
Invalid or expired token
Regenerate the token in Splunk Settings → Tokens
Connection refused / timeout
TestNode cannot reach Splunk port 8089
Check EC2 security groups and firewall rules
assertAtLeast fails even though event exists
Log ingestion lag — event not yet indexed
Use 
splunk.waitForEvent() instead of 
splunk.search()
Search times out
Broad query or slow Splunk cluster
Narrow the SPL time window or add more filters; increase 
maxPollAttempts
Field not found error
Field name differs from what Splunk extracts
Run the same SPL in the Splunk UI and verify field names in the results table
Best practices
Always use DPL variables for 
SPLUNK_HOST and 
SPLUNK_TOKEN. Never hardcode connection details in scripts.
Keep time windows short (e.g. 
-5m or 
-10m) to keep searches fast and assertions specific to the current test run.
Use 
| head N in SPL to cap result sizes when you only need a few rows.
Prefer 
waitForEvent over a fixed sleep for async flows — it stops as soon as the event appears.
Use 
assertNoErrors as a blanket post-scenario health check in addition to specific field assertions.
Use 
storeResult to pass Splunk values into TheLocker rather than re-querying Splunk in every downstream script.
Call 
splunk.init() once per script at the top, before any other 
splunk.* call.