Scalability Index

A Scalability Index is a function of service performance (in TPS) as concurrent request levels and payload sizes change in a test scenario as shown in Scalability Index.

There are three distinct parts to this Scalability Index:

  • View A: Testing at 10, 20, and 30 concurrent requests (CRs) levels shows the TPS level also rises. Imagine the conclusion if the CRs levels if View A were tested, where the assumption that the service will scale to handle the increase in CRs levels would be made. However, at 40 CRs the TPS value begins to decrease.

  • View B: Calibration tests seek TPS values for the 40, 50, and 60 CRs levels. In this range, the service responds to a moderate number of CRs at a TPS rate acceptable to the organization hosting the service.

  • View C: When testing at the higher levels – 70, 80, and 90 CRs – the TPS value does not change noticeably. Imagine the conclusion if the CRs level if View C were tested, where the assumption that the service will not be able to handle an increase in CRs levels would be made. System managers typically buy overly equipped server hardware to contend with situations where the server received too many responses to handle efficiently. The managers could have saved a lot of money if they bought multiple smaller (less expensive) systems and used load balancing to split large loads into multiple smaller loads handled by the multiple servers.

View B shows the optimum where underdriving (too few requests) nor overdriving (too many requests) the service within its CRs levels range occurs. If View B is the Calibration Test results, a range of CRs levels to use in the Full Test of our test scenario is defined. At this point, additional run scenarios that use alternative APIs and products, with the CRs levels used in View B to compare the TPS results.

In this example, the CPU, network, and memory utilization values are used to determine where performance and scalability bottlenecks occur. As a note of reference, CPU and memory bandwidth are helpful in stateless tests. CPU and memory bandwidth is usually meaningless for stateful tests.