Ideally, Performance test environment should be sized with same capacity as that of production to avoid any risk associated in interpreting the system performance characteristics for Production environment. This arrangement helps Performance Tester to primarily focus on the performance & scalability analysis of the system under test rather than spending time, investigating how to interpret or extrapolate performance test environment results to production environment. Many medium to large organizations invest in setting up a production similar environment dedicated for carrying out performance tests. As a Performance tester, we have seen many of our clients are convinced enough to have the budget allocation done, after realizing the associated risks in using downsized test environment.
But unfortunately, there are situations where some of our clients couldn’t afford production similar test environment due to several constraints. In such cases, there are other alternatives available to carryout performance tests, but it comes with added risks. Hence, it is the responsibility of the Performance tester to keep the client stakeholders aware of the risks associated in using any of the alternatives discussed below.
Use of Production Environment (during off peak hours)
This alternative is usually preferred when client stakeholders are convinced that the risks associated in conducting performance tests on scaled down environment is far higher than conducting the performance tests on production environment with proper planning & control. Though the tests are carried with proper arrangements during off-peak night hours, but still, this method is not advisable in sensitive environments as there might be situations leading to performance impact for real end users.
Use of Cloud based Production similar Environment
This alternative is usually preferred when the need for carrying out performance tests on production similar environment is very high, but has challenges in setting up dedicated performance test environment at their on-premise data center. Though it addresses major risks in setting up the environment quickly & test results mapping, this method is not preferred by clients due to security reasons, inspite of doing high investment in setting up the environment.
Use of Scaled down Environment
This is the most preferred method where client stakeholders want to benchmark the system performance on a downsized environment. But the worst part is many a time, client stakeholders are not clearly aware of the risks associated in adopting this method. It is often the responsibility of Performance testers to take complete ownership in educating the client about the disadvantages in adopting this method. Even if you couldn’t convince enough to change the Performance test environment usage strategy, atleast you take enough measures to highlight the differences associated between the TEST versus PROD environment, so that you don’t let your client stakeholders to think, “there is a bigger hardware in production, so the system performance characteristics will be always much better in production than test environment results”.
Key factors to be considered during Test Environment sizing analysis
There are many factors that need to be assessed to compare the differences between test & production environment. Some of the key factors include,
- Deployment Architecture
- Special Infrastructure components like firewall rules / Load Balancer configurations / CDN, etc.
- Software Differences like OS version, application build version, #JVMs, etc
- Hardware Differences like Server model, CPU, Memory & HDD or SAN, etc
- Server Configuration settings like Thread pool / JDBC Connection pool / JVM Heap size / server log modes, etc.
- Network differences
- Test data volumes & DB storage arrangements
Things to Ponder
As a Performance tester, we need to remember the below points while using scaled down test environment for performance testing.
- Do not compromise on replicating the tiers as in Production system architecture / deployment architecture on top of server capacity differences.
- End to End production environment cannot be proportionally downsized equally in the test environment.
- Prioritize the key components to be considered in the sizing analysis for extrapolating the test results to production environment.
- You are assessing & benchmarking the application performance characteristics for the available test environment only. Your Performance test report should mention this clearly.
- For extrapolating the test environment results to production environment, do not always prefer using simple linear regression method, where higher level of accuracy if expected.
- Adopt queuing theory based models & industry benchmarks like TPC/SPEC to extrapolate/map the test environment results to production environment with better accuracy.
- Derive & Report a confidence level index based on the associated risks & techniques you have employed to extrapolate the results to production environment. This quantitative reporting style would add lot of value to the sizing analysis.
- The most simple & effective strategy is to use a single feasible unit from all tiers of production environment to be available at test environment with absolutely no capacity or configuration differences.
- There is no universal strategy that is applicable for all systems to setup a downsized the test environment. The strategy is best derived based on deployment architecture, project constraints & budget constraints, which varies on case by case basis.
Happy sizing & extrapolation!