Is Performance Modelling / Prediction a replacement for Performance Testing?
Never consider Performance Modelling / Prediction as an alternative or replacement to Performance Testing. Rather, Performance Modelling is a smart way to evaluate system performance in early SDLC phases when complete application is not even ready to carryout performance tests. It also plays wonders for predicting system performance for variations like workload changes, hardware upgrades, etc for applications that are already in production for a while.
Happy to share our next online course, “Queuing Theory & its Applications in Performance Testing” is made available for readers in our e-learning platform. You can check it out at http://elitesouls.teachable.com/
What is Performance Modelling?
A model is an abstraction of a complex system which accepts certain inputs & generates specific outputs. A Performance model uses specific number of building blocks to predict how performance will vary under different what-if scenarios like varied set of load conditions, change in workloads, and change in server capacity, etc.
Performance modelling is the process of creating a model of the system to forecast its performance. As the performance model is usually used to evaluate alternatives, it is usually built during design phase and refined throughout the SDLC phases to evaluate how the end system will behave under expected user load conditions.
There could be several objectives to do performance modelling including performance NFR validation, analyzing design or implementation trade-offs, infrastructure capacity sizing analysis, etc. For business critical, high user traffic applications, where performance plays a key role in providing great user experience, it is recommended to include performance modelling activities during early SDLC phases such as Design phase to build high performing & scalable system.
Performance models are usually created for a specific application workload which includes one or more business critical, traffic prone use cases. Usually the inputs to the model are expressed in mathematical quantities such as number of users, arrival rate, response time, throughput, resource utilization, etc.
There are several popular tools like CA Hyperformix, TeamQuest, Metron Athene, etc, available that helps to build analytical models using measurements like response times, throughput, resource utilizations, etc. In addition, BMC, IBM, HP, and SUN have their own modelling tools.
Performance Modelling techniques are broadly classified as
- Analytical (QN) modelling,
- Statistical (Regression) Modelling,
- Simulation Modelling.
Generally, analytical models are easy to use & comparatively less expensive than simulation solutions that are available in market. Statistical models (& machine learning models) are usually preferred to make predictions for systems that are already in production for years (with huge historical data).
Effective performance modelling requires good understanding of the modelling techniques, gathering right set of data to feed the models & benchmarking the model performance against the reality to ensure the model results represents the realistic data.
How to select the right Modelling Technique
Implementation & choice of right Performance modelling technique depends on several factors like criticality of performance, expected outcome, data availability, application complexity, client’s interest-level, time & budget constraints, etc. Ideally speaking, the right time to implement performance modelling is during Design phase & refinement happens during testing phase & thereafter. There is no magic formula that can provide justification on which is the best modelling technique. Hence, the answer is “It Depends” [though i don’t like to use :)]
During Design phase, if you want to choose the right modelling technique, the obvious option that has worked better in my experience is QN analytical model. Analytical or Simulation models are usually recommended at this stage as per industry best practices. As the system is not available, a prototype or Proof-Of-Concept (POC) with key workload needs to be built (which yields appropriate service demand). The model can be refined during development & testing phase through performance test results to make it more accurate. In Agile environment, performance modelling should be thought throughout the SDLC right from the initial sprint based on the sprint scope model. Data collected from every sprint will help to improve the accuracy of the model. In my experience, I have found Performance Modelling activities are very effective & provides a great cost benefit revealing the futuristic system performance much early during development phase.
During Testing phase, combination of QN model, Regression model & Dr. Neil Gunther’s Universal Scalability Law (USL) can yield good results for analyzing & predicting the system performance based on the requirements & application complexity.
During Post-deployment phase, time series forecasting techniques like Moving Average, Exponential Smoothing and ARIMA are usually recommended. Apart from these traditional techniques, several machine learning models have become very popular in recent days to predict system performance anomalies. As these techniques help in forecasting, based on time series based historical data, they are better recommended for analyzing the workload changes, user load changes, etc expected in near future in production.
Performance Modelling is not a rocket science, though there are several high complex models. Performance Modelling should not be considered as an alternative or replacement for Performance testing. It needs to be considered to get a direction in the early life cycle phases or to get a feel of system performance for projected load levels to make necessary infrastructure investments. It forms an integral part of design to answer the most common question, ‘Will the design capable enough to meet the system performance targets?’. Deciding upon the right modelling technique considering the project constraints can yield better results & provide great cost savings.
Based on my experience in Performance Modelling & having used several commercial & free modelling tools, the best 2 tools that I always recommend to have a successful career in this field is
1) Continuous learning & gaining sound practical knowledge on various performance modelling techniques along with its limitations &
2) Knowing how to use MS Excel to create handy tools to support & accelerate your modelling activity.
Do take a look at the course contents of our new online course “Queuing Theory & its Applications in Performance Testing” & share your suggestions on any additional foundation topics that could be included in the agenda.
Happy Performance Modelling!!