- Image via Wikipedia
Cloud hosting providers with any degree of scale should have an application load testing product offering. In fact, I think it has the potential to be a killer app for cloud hosting companies.
Load testing has traditionally been a painful process and inefficient process from a time and cost perspective due to challenges around traffic generation and scaling.
There have been typically two options for the traffic generation aspect of the load test:
- Buy a bunch of servers to make into a traffic generation farm, buy some software (e.g. Mercury Interactive) to automate the testing, and tie it together by hand. This is no fun, no matter whose software you are using.
- Pay a load testing managed service provider (e.g. Mercury or Keynote) to provide both the testing server capacity as well as construction of the test scripts. This is less painful than #1, although you are not able to test on demand and its likely that the price you’re paying the provider is going to be negatively affected by the large fixed cost infrastructure load testing providers tend to own.
Let’s also look at a customer’s objectives during a load test.
- See how much traffic of various types the application as-configured (hardware, software, etc) can handle.
- See how much more traffic the application can handle if configuration changes are made or additional infrastructure components (i.e. more servers, bigger servers, etc) are added.
#2 is painful in a non-cloud world because a customer has to purchase or rent servers just to see how their site responds to scaling. And its even more painful if you work with a hosting provider whose business model is dependent on long term usage of server capacity to cover a large fixed cost base.
Here’s where the cloud comes in…
For a customer who has their app hosted w/a cloud hosting company, scaling is not an issue, as long as that customer is willing to take the variable cost hit of spinning up more servers to see how their app handles higher loads. No need to reconfigure existing servers or purchase new hardware! For a customer who has a traditional hosting environment but has the capability to cloudburst could see similar benefits.
Also, a cloud-based load testing platform would by definition be available on demand and less expensive to operate than a platform that lives on physical servers since the fixed cost base would be drastically reduced in the cloud case.
And here’s the best part: if I’m a cloud hosting provider who offers a load testing service (or resells one), I’m getting paid for the extra capacity that the application owner is using during the load test AND if the cloud load testing platform lives on my cloud, I’m getting paid for their capacity usage during the test as well!
Seems like a beautiful thing to me. App owners get cheaper, more flexible load testing capabilities, load testing companies get paid, and the cloud hosters get paid twice!
Related articles by Zemanta
- HP adds to apps performance management pack (infoworld.com)
- uTest To Launch AppGrader – Free Web App Testing (video) (centernetworks.com)
![Reblog this post [with Zemanta]](https://i0.wp.com/img.zemanta.com/reblog_e.png?w=1200)
18 comments
It is a great app for the cloud. Some other companies in the space include LoadStorm, BrowserMob, and SOASTA which are included in OpenCrowd’s cloud taxonomy, http://www.opencrowd.com/views/cloud.php, under development and testing.
I had an experience with Mercury’s Load Testing as a service application about 5 years ago (I’m sure they have improved the service). They had trouble with loads in the hundreds of thousands concurrent users which caused their servers to crash.
Hey Brad – thanks for the comment. I was also an ex-Mercury and Keynote customer and scaling up to very large loads was frequently a challenge for them. The other issue I recall was that they could only generate load from a couple of locations at once. I guess the other cloud providers would have the same problem but maybe the cloud load testing co’s could do integrations with a few different clouds (GoGrid, Amazon, Slicehost, etc) so that end users could get a more diverse set of traffic sources.
Hi Brad, nice post! I have been making this same argument for Cloud Testing for several years now (www.soasta.com). The only push back that I would give you on this post, is with the tools that you chose to do Load Testing in the cloud, and on this point I’m obviously bias. I personally found Loadrunner to be a wonderful tool for testing client server software over the past twenty years, if you could afford it. My issue is that we are not building CS Apps anymore, we are delivering new breed of web based global Services that are being built with an entirely new set of technologies. Loadrunner is like a great athlete that has slowed down and is now past its prime, the game has moved forward, and it is time to seek newer services that match todays market. This is why we built CloudTest Service as an affordable, scalable testing service for global web services.
Even though they have their own infrastructure and thus don’t fit your “cloud-based” qualifier, Load Impact http://www.loadimpact.com provide on-demand load testing as a service and might warrant a look.
I have no affiliation with Load Impact.
Hello all,
I, however, do have an affiliation with Load Impact. It is correct that we don’t use cloud services as of today. Cloud services are great for being able to scale on demand and generate really large simulated loads, and we might start using them later on but for the time being, we think maintaining our own infrastructure is more cost-efficient. We aim for the small- and medium sized companies out there, and they can run smaller load tests for free using our service, so we need to keep costs down.
But the general idea that cloud-based load testing is a killer app is something I can heartily agree upon. Load testing has been far too expensive and far too difficult for far too long. With new offerings such as ours and Soasta’s, it will be difficult for the traditional players to motivate the pricetag of their solutions.
/Ragnar, Load Impact
I would not say we compete directly with e.g. Soasta as we aim for the small- and medium-sized companies out there and Soasta seems to go for the larger segment. Cloud-based services like Soasta have the advantage of being able to easily and dynamically scale their backend infrastructure and so generate very large simulated loads. We use our own infrastructure, which limits how much load we can generate but also allows us to
Oops, should have checked that text a little better before posting…
There is a lot of room for exploration in the cloud computing sector, obviously. As a long-time web developer, I think that clouds offer a great solution for scalability… but they stop there. To really build a great model, I would like to build a company which implements known systems architecture solutions automatically. Who wants to hire a systems architect to design the system and then continue to pay him just in case you suddenly need to scale to ten times as many users because you got popular before you expected to? How about if your data schema, code, and machine image were kept under version control and you could chose which combination to deploy to a test server then run some automated benchmarks and regression tests against it to determine if it is better or worse than another combination? Wouldn’t you like to roll out PHP 6 with the confidence provided by first having proven it through regression testing the system with a virtual load bigger than you’re currently seeing on live? I’ll bet. What about having someone else deal with CDNs, load balancers, systems performance tuning and security while you deal with your startup idea? Sure, just chose an architecture based upon a simple overview of their benefits and read the simple documentation that tells you what goes where.
Hacker News just picked up the post and there are a bunch of comments there as well:
http://news.ycombinator.com/item?id=527363
Well done. We at CustomerCentrix have struggled with load testing our web apps since 1999. When our CTO, Roger Campbell, figured out how to dynamically allocate load generators from AWS ECS, we knew there would be a tremendous demand for low-cost, massively-scalable tools. LoadStorm is exactly that app. Built on EC2 from the ground up. Free account – not a trial.
While I agree with most reply’s on this page, I would like to add my point of view on Cloud and Load Testing.
I’ve been using Load2Test from Enteros, formally DBA InfoPower and while most of the companies described above are only specialize on HTTP/HTTPS/SOAP, Load2Test can do much more. It got ability to record and create flows for most databases and application servers, etc.
As to it takes long time to create a test, it’s not true. While other companies including LR depends on their GUI, structure of the test, and database integrity, Load2Test does not, which makes pre-production tests just as easy as the first one.
They can do back-end database load testing as well.
Lastly they provide integrated root cause analysis and great reporting which can suite needs of any person from IT to CIO.
As far as I know Enteros Load2Test supports Amazon Elastic Cloud EC2 transparently and this makes it one of the best products to use for complex load testing in cloud.
I’m a little late to the party on this post, but I think you’re absolutely right – the cloud is great for load testing. We’ve been using it for a while on the services side of our business. This week we released direct support for EC2 in our load-testing product, Load Tester 4.0, which lets a tester launch tens, even hundreds, of load generators and have them ready for testing in just a few minutes.
We’ve got a video demonstrating just how easy it is:
http://www.webperformanceinc.com/load_testing/demo/CloudDemo.html
I am still not clear about:
1) The lead testing tool will be hosted as SaaS?
2) The customer can subscribe for an instance of a load testing tool?
3) The customer application may reside on another server (physically apart) from where the load testing tool is hoted?
With all above is it possible to conduct load testing ?
Thanks
Hi Rahul,
Yes it is possible to accomplish all three. Please visit http://LoadStorm.com and sign up for a free account to try it. There is no risk. Also, send me an email to scott@loadstorm.com and we can schedule a time for a phone call to answer your questions.
Hi,
Is it possible to have a load testing tool hosted on server (as SaaS) and the application under load testing is hosted on entirely different servers/locations and still one can conduct load testing for that application?
Hello,
We agree of course on the fact that cloud testing is key for load testing. We’ve just released a new version of NeoLoad able to automatically deploy cloud based load generators in minutes.
We think that depending on the stage of your load testing campaign, you can leverage your own infrastructure or the cloud:
– in-house: initial tests, no firewall to setup, app can be deployed on a small preproduction environment – not accessible from Internet
– cloud: large load tests, test the application’s entire delivery chain (firewall, routers, third party servers, etc) from different parts of the globe
More information at http://www.neotys.com/product/neoload-cloud-testing.html
We’ll be really glad to have your feedbacks
Thanks,
Getting started in Load Testing in the cloud has quite high barriers to entry and can carry an extreme upside risk in costs in terms of having out-of control load farms running. At http://loadzen.com we built the infrastructure to share as much resource as possible to minimise the risk of having runaway load generators.
However, in the end for consumers having a simple, quick and easy way to generate moderate to high load on their servers provides an excellent way to ensure that your site will withstand traffic when that one piece of content goes viral.