Load test, performance test, end-to-end test, system test – What to choose?

Why performance testing applications?

Many companies acknowledge the importance of assuring that the technical and functional quality of an application or system meets the requirements. But load- and performance tests are often not executed. Verifying nonfunctional requirements such as the maximum through-put and response times is at least as important as functional quality. Working with ‘standardized’ environments, default settings or making wrong assumptions in implementation phase, problems and limitations are introduced that will only appear in performance tests. Think about:

• Configuration of application- and connection pools and queues
• Assignment of memory capacity and memory management strategies
• Running under Citrix, virtualization or Docker

It is therefore a necessity to not only test functional requirements, but also test the non-functional requirements.

Create the right environment

When testing applications, it – naturally – is of importance to perform the right test in the right way to get results that actually reflect the future situation. However, in most cases, applications are created and tested in an environment where the infrastructure is not sized like it is in production and application interfaces are stubbed. After all, a development environment is created based on formal specifications which is an ideal world for the application to live in. In a production environment, conventions may turn out to be interpreted differently or the environment was simply based on different standards than the application was designed for. These differences often introduce unexpected behavior, functional as well as nonfunctional. So before even choosing a performance test, make sure your environment reflects the actual production environment.

End-to-end testing, integration testing and/or system testing

After the development phase all software components involved are connected and tested as a whole in an “integration test”. During the “systemtest” that follows, infrastructural elements are added: hardware, clustering etc. In some organizations it is useful to perform an “end-to-end test” to make sure that the flow of information that propagates through all applications involved, is still valid at the end stage.

Depending on the complexity and risks involved, a selection of these test types should be performed. If only one end-to-end functional test is executed, all kinds of functional and technical faults will come to light and will frustrate the testing process at many points. The more complex the system is, the more test types will be needed to be able to manage the complexity of the quality stack.

Load testing, stress testing, performance testing

These terms are mixed up very often, but they represent different things. In a “performance test”, the speed of the software is measured: the response times as they are experienced by users or other systems. Loading times of (web) pages, user actions or batch processes. A performance test will mostly deliver a set of response times, measured at a selection of most used or most critical actions.

A test is called a “load test” when response times are measured while a certain load is applied. This load can consist of artificially generated (web) requests or batches of data that are feeded to the system. When generating traffic, it is important that the real production load is matched as well as possible. The orchestration of test cases or requests that are applied during a load test should be based on measurements and estimates of the current or expected production situation. This preparation will result in technical parameters with which load generators are configured and test data will be composed.

A “stress test” is in fact a normal load test, intensified by a certain factor. The load generated in the load test is gradually increased until the system under test gets saturated and throughput comes to a limit. This test will prove what is the maximum capacity of the system and if either capacity or configuration will cause the limitation of the level of load it can process.

What to choose?

There are many types of tests to choose from and they partly overlap. Which tests to choose should depend on the risks that apply for specific applications and situations. The more complex the situation is, the more risks are cumulated and should be tested before an application is deployed in a production environment. A small configuration mistake could cause serious limitations and downtime at a production day one.