Stress Test vs Load Test

People are often mixing Stress Test and Load Test in software development. So, let’s demystify a bit.

Stress test tries to break the application until its limits and have a clear KPI regarding the SLA. In this stress test, you have 2 aspects: Positive test using all the system working optimally and negative test having some system already broken and what is the limit of the system in degraded mode.

Load test is a scale up and a controlled volume loading test. The main aspect here is to define the bottle necks of the system at different level of volume.

Here a few tools that could do both type of tests:

  • Locust (open source system and Python)
  • JMeter (open source system and Java)
  • LoadRunner (commercial tool)
  • NeoLoad (commercial tool)

Samy Kacem

Test Automation Strategy

Before starting to automatize tests, you need to take a step back and define few aspects.

Consider the test team members

Depending on the test team members you need to think what kind of test scripts you are going to adopt. Depending on the tools like Tosca, UFT, Ranorex or Selenium, there are 3 ways of scripting:

  1. Record : This is a an easy way, but could be expensive to maintain. The profiles adopting this way of working are the Business Users.
  2. Script : This a fair quick way to script, and the maintenance is less expensive then using Record. It requires a Test Automation Expert to develop and to maintain them. Any tool is able to achieve the automation.
  3. Data Driven and Workflow : It requires both profile : business user and a test automation expert. This could take more time to implement but the maintenance is way easier and also scripts could be easily reused by the business user without requiring an expert.
    1. With HP, it will require BPT.
    2. Tosca is able to handle it if you use “modules”.
    3. Selenium and Cucumber or any other BDD are able to handle it. See a test factory as exemple.
    4. Ranorex is supporting the data driven part. But no workflow management.

Consider Manual vs Automated testing

You should not automate everything and keep some test cases for manual testing. To define which one should be automated, keep in mind the following:

  1. How much time the same script need to be repeated? If you are doing more than 4 times, you should automate.

    Y = minutes spent and X= time test execution. We base the figure on a small test scenario.
    Y = minutes spent and X= time test execution. We base the figure on a small test scenario.
  2. If your test are UAT (User acceptance test), exploratory test or complex with many asynchronous processes, you should consider manual test.
  3. Also when you prioritize the test cases based on their business criticality. More a test case is critical, more it should be automatized.

Consider Test Data Strategy

Ask yourself these following questions:

  1. Do you need real data with a state data based on a process? That means your data will change from the beginning until the end of the test. You may need a copy from a production database with data anonymization and a way to reset it. This could be a complicated task. You could also use advanced tool for Service Virtualization to avoid to copy database and have a instant “refresh”. But this path is also quiet expensive in license or in man days.
  2. Does your data drive the test? In other words, do you need more a copy of the database, but also “properties” data to drive all the test scenarios?
  3. Define the right set of data that covers all the business risks. When you generate all the test scenarios for the same workflow, you should ensure to optimize the number of scenarios without being systematic, otherwise it could fall in an exponential number of scenarios. See an example below:

    Define data set scenarios - Gender and t-shirt size are the variable to generate the data set. (*) mean critical data from business point of view.
    Define data set scenarios – Gender and t-shirt size are the variable to generate the data set. (*) means critical data from business point of view.

Consider Test Script Maintenance

When you create automated test you will require later to maintain them for 3 main reasons :

  1. “Repair” test cases based on business changes.
  2. Enhance test cases based on issue found in UAT (User acceptance test) or later in Production.
  3. Optimize or correct test based on false positive errors.

Consider the right scripting tool

Based on all your SUT (System under test) technology. You need to find the right scripting tool depending on the features and your budget.

Comparison of test automation tools between Selenium, Ranorex, UFT, Tosca
Comparison of test automation tools between Selenium, Ranorex, UFT, Tosca

Data test strategy and Service Virtualization

Actually I’m digging in different SV (=Service Virtualization) Solution like HP SV, OSV from Tricentis and CA SV. In this context, it allows you to cut from production the real data and play with test data.

What commercial SV can do for you?

  1. Record transactions : You record all the transactions you need for your test scenarios, then the system is playing back all the responses as required.
  2. Scripting the service with a business logic if you need some complex response.
  3. Data driven set : Allows you to fill an Excel spreadsheet or any data container with all the data you require for your test scenarios. Your data set need to be “stateful business process” oriented, that means during the data lifecycle, the object could change state and value.

This last feature will be a good candidate if you are stuck with a legacy or new applications in order to put in place a test data set. The majority of the commercial tools could work with different protocol (http/https, MQ, Filesystem, Ftp/s, etc…). And if the price is a show stopper and you require only http/https, some open source tools are also good candidates. Here you can find a good list of open source and commercial tool.

Samy Kacem