Performance is very important thing for all types of application because, in this digital era, we cannot challenge users’ patience with slow or unreliable application. With plenty of competitors out there, users will choose what can provide the best user experience over the others. Therefore, its very needed to include performance test in development life cycle.

Note: this article is focused on performance test for Web applications and BE Web services with JMeter (or same) tool. It might not be valid for other type of applications or other type of performance test (with difference tool).

1.  What is performance testing?

Performance testing is a non-functional software testing technique that determines how the stability, speed, scalability, and responsiveness of an application holds up under a given workload. It’s a key step in ensuring software quality

The goals of performance testing include evaluating application output, processing speed, data transfer velocity, network bandwidth usage, maximum concurrent users, memory utilization, workload efficiency, and command response times.

2.  Factors impact to Performance Test results

Performance problems affect all types of systems, regardless of whether they are client/server or Web application systems and could cause by many factors. Reliable performance test results would be very helpful in analyzing and identify root cause of problems. It is imperative to understand the factors affecting system performance and performance test results before embarking on the task of handling them. Let’s go with us to explore those.

Server resource

Server resources means allocated resource like CPU or RAM, along with it is auto scalability (provided by cloud platform like AWS, GCP). High resource server configuration or scalable could handle more users while low one couldn’t. Understand about server’s configuration and monitor it could give us more vison for analyze the test results.

Example issues:

• During the test, the server resource is changed by someone else – we’re not controlling the information or the server under the test is shared with multiple teams.
• Team wanted to check for capability of current application in production, but just provide lower configuration in test server for the test – We cannot multiple the test result from lower resource server for the higher one.
• Team wanted to compare application’s performance before and after deploying new feature. But allocated resource for the test with old version is different with allocated resource for the test with new version – There is no record of the resource information.
• Keep try to increase number of CCU meanwhile currently, usage resource almost reached to 100%

Recommendations:

• Have a clear test plan about resource configurations
• Make sure that the resource configurations are correct before any tests, including resource allocations, scalability….
• Monitoring server resource usage during the test and include information in report. It could help to analyze results and determine to decrease or increase ccu for the test.

Client resource

Sometimes, performance issue would come from client/user themselves. With low resource laptop or computer or an outdate mobile phone model (such us the iPhone 4,6) may cause user to have bad experiences. true that there could be some requirements to test the performance under low resource client but normally, requirement will focus on how server response. Therefore, we should have adaptable resources for the test.

Example issues/problems:

• The test required to test with 2000-3000 CCU (thread) with JMeter. Tester used a low resource computer to run the test and get a very bad result, then complain about application’s performance

Recommendations:

• Make sure that your computer/laptop is strong enough to handle large number of CCU (thread).
• It might hard to handle large numbers of CCU in a single machine. So we could think about building distributed test like mentioned in jmeter_distributed_testing. Beside that, we also can purchase cloud service of Blaze Meter or Azure Load Test make it easier.

Networking

Network-related issues must also be considered as Networking is a bride between user and server. It is easy to understand that bride should be strong enough to connect user-server and for more users. In term of performance test, issues could come from both User and Server. Thus, it’s also an important thing to aware of.

Example issues/problems:

• Tester run test from local machine in their company but connect to the server that very far away (example: Running the test from Viet Nam and make requests to server in US/Europe) – Even both client-server have good internet connections, there will be some delay and lead to high response time.
• Client-servers are nearby, but Tester run test from their company and must go through company’s VPN (or server’s VPN)– which have limited bandwidth. There might be other configuration about limit number of connection to the server (Firewall, Anti DDos configuration). It has high impact to the response time and might lead to invalid test result

Recommendations:

• Make sure that we have good connection between test machine and the server. Try to figure out all limitation in connection in both client and server site and prevent it to make the test more valid
• Mostly, performance test will be executed from a computer/laptop. So, its good to remove configuration about limitation for request come from a client. With that limitation, we cannot reach to high number of requests in our tests. But in fact, it could happen cause real users come from difference locations.
• Buying service from Blaze Meter or Azure Load Test. They allow us to run the test from selected locations with good connection. Additionally, we also can simulate request come from different locations in a single test.

Database

Setup for database in performance test is also an important part. Beside of setup test data for multiple CCU in the test to make requests successful, we also need to make sure that volume of test data in database is reliable comparing to production environment. Database resource, version and connection could also lead to issues.

Example issues/problems:

• In production environment, user’s requests might make a query for millions of records but in the test, tester does not prepare that much data for same type of requests. It makes the request return faster with low response time and impact to analytic activities.
• Tester executes test for several times but number of records in each test are different. Example: the test contains the request to search for Products in system (e-com application), in first round, there’s 50k products but in second round, someone delete records and there’re just 10k products left and make response faster (or longer in other case).
• Database has it own resource as well and in test environments, it might not strong enough to contains large amounts of data or provide bad connection from server to database – which lead to higher response time
• Database version for different environments are not the same, it makes some query become longer (or faster) and impact to response time.

Recommendations:

• Ensure that we have large enough amounts of data in database for the test.
• Ensure that we understand amount of data in each test – which impact to result of our requests
• Ensure that we could understand the configuration, version, or resource of database server

3.  Conclusion

Evaluate performance of an application is not easy. Therefore, we need to make the performance test result more reliable as much as possible by preventing and handling factor that could impact to the result. There will be other factors as well in your actual test and I hope with this article, we – testers could have more view about performance test activities and make bring good results.