Over the last few years I've noticed more and more new load testing services and tools, but especially cloud-based SaaS solutions, as part of the movement towards "everything as a service".
There are so many to chose from: Load Impact, BLITZ.IO, BlazeMeter, loader.io, Load Storm, Load Focus, SOASTA, LoadUI, Locust and much more, each has its pros and cons, but I'm getting really mad with the fact that they are just more of the same.
Why do I say that they are all more of the same? You need to go back to early 90's where an Israeli company was one of the first to come with commercial load testing tool for WEB and few other protocols, this tool was (Win/) Load Runner and the company named Mercury (later purchased by HP).
Since early 90's the technology of load testing was not changed at all, it was and still is about simply sending HTTP requests and measure responses (I'm being focused on WEB/HTTP load testing for simplicity).
Now, over time, Mercury created good stuff to make things easier for getting the work done, mostly with auto-correlation of parameters and predefined macros for known applications and platforms where it was complicated to develop the load testing scripts due to application complexity with massive usage of http parameters being send back and forth between the browser and the server.
(and I ignore for the scope of the post, all of the other good stuff they've done with systems monitoring, application insight monitoring aka diagnostics, and bringing several perspectives into single powerful analysis tool which was, in my opinion, technological break-through)
At some point, somewhere around 2010, where AJAX was getting popular, generating load testing scripts got more complex and one of the ways to deal with it was to try and change the way load testing was done until that point. If up until now it was all about sending HTTP requests and measure responses (with some parsing), the change was to try and run kinda of real browsers, in memory, without UI, so the load testing tool is now starting to run UI-less browsers and manage them (you could imagine Selenium headless browser running multiple times). This makes it easy to deal with frequent changes in the application UI, API and any changes with AJAX calls will simply require no change with the load testing scripts, as the scripts now interacting on the UI level. Sounds great, right? Well..
Basically the main down side of this technology is that while it works great for functional testing where a single user, or maybe few, running from a host to test the application, the problem is that it is extremely Memory and CPU intensive to run a real browser with MANY windows. So in practice when it comes to really load you web application with more than tens or few hundreds of users, such approach is simply too expensive. Indeed memory and CPU in general are getting cheaper, and maybe in few short years we will get there, but you still need about 100MB (gross) of memory for every virtual user, depending on the complexity of the client side (css/js). So for intranet / internal web systems - it might work, you can create few load generator hosts which with 8-16GB RAM can create a load for about 80-160 virtual users, where it might be enough to load test such internal systems.
When it comes to load testing big, Internet facing applications, you would probably want to load test with thousands of users, which based on the gross numbers above, will require 1000GB of RAM for 10,000 concurrent users. Based on Amazon pricing this will cost about 32$ USD per hour (based on c3.8xlarge 60GB instance which is about 1.7$ per hour - I'm rounding up to 2$ as there are other expenses like network and disk usage).
Now, given the fact that you never run one load test and then wrap things up and never do that again.If you are serious, you will run load tests regularly and at least, once before every production release. We are talking about hours of load tests every month.
With one of my clients, we run 8 hours of load tests every day and 24 hours load test every weekend, with a similar scale to the example above - it would cost more than 9000$ per month to run load tests with real / headless browsers (and that's assuming that we use Amazon On-Demand instances and shut them down between load tests).
If you can have such budget spent on load testing infrastructure alone, you should also consider the fact that, such approach with driving a headless browser, is still considered inaccurate and inconsistent. The timing mechanism is still considered immature, virtual browsers may affect each other due to spikes with CPU consumption and thus running same tests may get you with pretty different results from my experience.
So now, I come back again to why I say all load testing solutions are more of the same.
All of those services and tools I've mentioned above, focus on running load tests with the very first technology that Mercury came with in early 90's, not with running real browsers, which is still immature and expensive but with scripts that define what kind of HTTP requests to hit the system with and measure the response, that's all. So they all provide with different scripting languages or ways to control the test. Some allow you to work with UI and modules to create the required script, some with XML, some with coding style in either proprietary scripting engine or wide spread scripting engine, but the very bottom line is that they all ask you to provide with same data to get your load test running.
Go and try few of them, you will see and understand that each solution wraps the same idea with different UI. They all work the same and if they work the same, I urge you to use those who are based on JMeter. Why? Because it is the most popular load testing tool in the world, it is free and open source and most importantly - it has the biggest set of features and it continue to grow with regular releases backed by an awesome core team of commiters which push it forward. If you go with other proprietary scripting engine, you will soon find show stoppers, blocking you from executing a script which should interact with your application due to missing functionality, as all other engines are trying to keep up with JMeter.
So why I mad about all those tools and competitors? They all try to re-invent the same wheel. Not a smarter one, not a better shape nor better material. Just re-creating same old stuff which was invented about quarter of a century ago!!
Last point is that 5 years ago - the idea of driving real/headless browsers was really promising, but so far no real solution is doing that successfully.
I'll create a technical post on this topic in the future to show timing issues with such approach, currently I have in mind to show results from Selenium browser running from by JMeter with the JMeter Plugins, but any other ideas or pointers are welcome here.
There are so many to chose from: Load Impact, BLITZ.IO, BlazeMeter, loader.io, Load Storm, Load Focus, SOASTA, LoadUI, Locust and much more, each has its pros and cons, but I'm getting really mad with the fact that they are just more of the same.
Why do I say that they are all more of the same? You need to go back to early 90's where an Israeli company was one of the first to come with commercial load testing tool for WEB and few other protocols, this tool was (Win/) Load Runner and the company named Mercury (later purchased by HP).
Since early 90's the technology of load testing was not changed at all, it was and still is about simply sending HTTP requests and measure responses (I'm being focused on WEB/HTTP load testing for simplicity).
Now, over time, Mercury created good stuff to make things easier for getting the work done, mostly with auto-correlation of parameters and predefined macros for known applications and platforms where it was complicated to develop the load testing scripts due to application complexity with massive usage of http parameters being send back and forth between the browser and the server.
(and I ignore for the scope of the post, all of the other good stuff they've done with systems monitoring, application insight monitoring aka diagnostics, and bringing several perspectives into single powerful analysis tool which was, in my opinion, technological break-through)
At some point, somewhere around 2010, where AJAX was getting popular, generating load testing scripts got more complex and one of the ways to deal with it was to try and change the way load testing was done until that point. If up until now it was all about sending HTTP requests and measure responses (with some parsing), the change was to try and run kinda of real browsers, in memory, without UI, so the load testing tool is now starting to run UI-less browsers and manage them (you could imagine Selenium headless browser running multiple times). This makes it easy to deal with frequent changes in the application UI, API and any changes with AJAX calls will simply require no change with the load testing scripts, as the scripts now interacting on the UI level. Sounds great, right? Well..
Basically the main down side of this technology is that while it works great for functional testing where a single user, or maybe few, running from a host to test the application, the problem is that it is extremely Memory and CPU intensive to run a real browser with MANY windows. So in practice when it comes to really load you web application with more than tens or few hundreds of users, such approach is simply too expensive. Indeed memory and CPU in general are getting cheaper, and maybe in few short years we will get there, but you still need about 100MB (gross) of memory for every virtual user, depending on the complexity of the client side (css/js). So for intranet / internal web systems - it might work, you can create few load generator hosts which with 8-16GB RAM can create a load for about 80-160 virtual users, where it might be enough to load test such internal systems.
When it comes to load testing big, Internet facing applications, you would probably want to load test with thousands of users, which based on the gross numbers above, will require 1000GB of RAM for 10,000 concurrent users. Based on Amazon pricing this will cost about 32$ USD per hour (based on c3.8xlarge 60GB instance which is about 1.7$ per hour - I'm rounding up to 2$ as there are other expenses like network and disk usage).
Now, given the fact that you never run one load test and then wrap things up and never do that again.If you are serious, you will run load tests regularly and at least, once before every production release. We are talking about hours of load tests every month.
With one of my clients, we run 8 hours of load tests every day and 24 hours load test every weekend, with a similar scale to the example above - it would cost more than 9000$ per month to run load tests with real / headless browsers (and that's assuming that we use Amazon On-Demand instances and shut them down between load tests).
If you can have such budget spent on load testing infrastructure alone, you should also consider the fact that, such approach with driving a headless browser, is still considered inaccurate and inconsistent. The timing mechanism is still considered immature, virtual browsers may affect each other due to spikes with CPU consumption and thus running same tests may get you with pretty different results from my experience.
So now, I come back again to why I say all load testing solutions are more of the same.
All of those services and tools I've mentioned above, focus on running load tests with the very first technology that Mercury came with in early 90's, not with running real browsers, which is still immature and expensive but with scripts that define what kind of HTTP requests to hit the system with and measure the response, that's all. So they all provide with different scripting languages or ways to control the test. Some allow you to work with UI and modules to create the required script, some with XML, some with coding style in either proprietary scripting engine or wide spread scripting engine, but the very bottom line is that they all ask you to provide with same data to get your load test running.
Go and try few of them, you will see and understand that each solution wraps the same idea with different UI. They all work the same and if they work the same, I urge you to use those who are based on JMeter. Why? Because it is the most popular load testing tool in the world, it is free and open source and most importantly - it has the biggest set of features and it continue to grow with regular releases backed by an awesome core team of commiters which push it forward. If you go with other proprietary scripting engine, you will soon find show stoppers, blocking you from executing a script which should interact with your application due to missing functionality, as all other engines are trying to keep up with JMeter.
So why I mad about all those tools and competitors? They all try to re-invent the same wheel. Not a smarter one, not a better shape nor better material. Just re-creating same old stuff which was invented about quarter of a century ago!!
Last point is that 5 years ago - the idea of driving real/headless browsers was really promising, but so far no real solution is doing that successfully.
I'll create a technical post on this topic in the future to show timing issues with such approach, currently I have in mind to show results from Selenium browser running from by JMeter with the JMeter Plugins, but any other ideas or pointers are welcome here.
