Wednesday, January 11, 2012

Why Apache JMeter is better than HP Load Runner?


Few months ago I had a session with a friend of mine on performance testing and discussed the future load testing projects in his organization and on-going performance issues with current deployed systems.
During this session he shared with me his plans to invest on upgrading the current HP Performance Center 9.52 (Enterprise style Load Runner) to the new HP QC-PC 11 Platform during 2012.

It was after about a year or so I was familiar with JMeter and I was really pushing him to try it instead of Load Runner, as I found it much more reliable to work with.
I was trying to convey that instead of investing on licenses he can invest on better HR, better performance engineers.

So after some comparisons we made during this session, we got to a point where any lacking capabilities of JMeter in compare to Load Runner, is something that he can work around or live with.
The bottom line of that session was that there is nothing in Load Runner that he currently can't get out of JMeter (with some customizations and Plugins and aside of other protocols than HTTP/s).

Then he raised a question I never heard before:

What can JMeter give me, which Load Runner cannot?
I was always thinking the opposite way, always trying to compare with the leading product, with the "mature" product which is Load Runner.
I never thought that JMeter can provide something better than Load Runner other than it's being Free and Open Source.

So having this question echoing my mind for few months, I just had an enlightenment:

Community and support
JMeter has a strong supporting and prosperous community, questions asked or bugs found on the product got answers and fixes in a manner of minutes or hours (even on weekends!).
Bugs I found and reported got fixed in few hours and were publicly available on the next build (which is made on a daily basis or so).
With HP support I had a terrible experience, waiting hours for first response.
So whenever I found a bug in Load Runner I knew that if I will report it, I will probably have to wait for the next content pack or minor version for a few weeks in the good case (or usually forget about it and work-around it).
Even more than this – try googling for Load Runner errors / questions / topics and you find almost nothing that really helps you.

Scripting language
JMeter is java based, mostly you don't need to code anything, you only see a (gray) swing-based application and you easily play with test items which control the load test.

Load Runner is C based and you need to learn a propriety functions to generate the desired script.
Now who develops a web application in C? it is much easier to J2EE and even .Net developers (OO languages) to extend or work with a Java based application.

Simplicity
JMeter shows you everything you need in order to control the HTTP flow of the desired scenario, this way you know exactly what you are asking for and you know what response to expect.

Load Runner is hiding a lot in order to have a cleaner script code, this driving you crazy when you need to understand why certain things are working in a different way than what you expect.
Even with the new Ajax supporting engine "TruClient" they are doing the same trick – they hide things in these script "Levels", in order to make the script look nicer and you - a mad performance engineer who doesn't understand why the script behaves as it behaves.

Stupid-prof
JMeter force it's user to understand how web applications works, what is HTTP Request, what are headers, how a browser behaves and so on.

Load Runner is for everyone, so you find yourself trying to understand what is this load test engineer is talking about because he doesn't know what he is talking about.
I saw too many times load tests being executed by people who doesn't know the difference between a cookie and a header, or what is java script and where it being executed.
Do you want someone like this being responsible for you application performance or load stability? I wouldn't.


Conclusion
Of course JMeter has drawbacks which are fully documented on the net when comparing jmeter and load runner.
My point is that you can invest your budgets on utilities and you should, but you must invest some time in comparing the free utility with the expensive one.

In this case, if you are performance testing web based applications, JMeter may be better option for you.

Continue reading

Go to a following post on free jmeter-based cloud services.

37 comments:

  1. Spreading the JMeter love, ten more for your list:

    1. JMeter's interface make building a scenario substantially easier - JMeter's tree view GUI shows absolutely everything, it's super simple. It's also very customizable, everything can be coded if you need to.

    2. LR has impressive analysis abilities but, really, I find that 90% of the time all I need is the data JMeter gives me. I also love how easy it is to take the output from JMeter and: put it in a db, awk it, view it in excel, etc...

    3. I can easily run JMeter from the Command Line! Enough said.

    4. JMeter is lightweight which means I can install it on a new machine in about 5 seconds. I can also automate doing this.

    5. It's portable, at 21Mb you can take it with you anywhere. It's Java, you can run it anywhere.

    6. Regular Expressions. LR has this (finally!) but still only in a limited fashion; in LR you can't get automatic groups of results from multiple matches on a regexp. Sigh.

    7. Capacity. Bit controversial this but, licence considerations aside, I would argue JMeter can be setup to offer higher capacity than LR.

    8. JMeter's Debug Sampler - so simple, so useful.

    9. Using a database in JM is easy. Not so in LR.

    10. If I use JMeter, I don't have to use Windows... Sounds a bit aloof that, but given 95% of everything I test, touch or think about is running on Linux and friends it kind of makes sense to have my test tool do the same.

    ReplyDelete
  2. HI sir i am new to jmeter ... Can u please refer me some links abt scripting in jmeter and what are the areas where we have generate automated scripts .Please send some material for the same to my id parankusampavan@gmail.com

    ReplyDelete
  3. Hi Paven,
    You may find good starting points on:

    1. http://jmeter-expert.blogspot.de/ - a blog with some demos.
    2. http://vimeo.com/3453772 - 5 minute video on start using JMeter (it's a bit old version of JMeter, but for a starter it will do it)
    3. http://community.blazemeter.com/knowledgebase - These guys trying to create a knowledgebase, which actually looking good and up to date.
    4. http://wiki.apache.org/jmeter/ - Look at the "Tutorials and How-to articles" section.
    5. http://jmeter.apache.org/mail.html - Join the users mailing list to ask questions and help other members.

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. Before step in performance testing, what qualities should a manual testing engineers have?

    ReplyDelete
  6. hi i want to learn performance testing which one is better load runner or jmeter

    ReplyDelete
    Replies
    1. Well, for load runner you gonna need both access to HP website where PDF and other resources are available as well as valid license to work with. At the end load runner is being used on big organizations (due to the high license fees) while JMeter is available for free and learning materials and community support are available as well.

      Delete
  7. Does Jmeter supports ERP products like SAP ? If yes - will it support SAP NWBC ??

    ReplyDelete
  8. Hi I have tried loadtesting JMeter with MQ and web services. However I observed that its not scalable enough as per our requirement. For MQ I wanted to load test for 120 tps and for webservices I wanted to test 10 services each at 80 tps. But I couldnt achieve it in Jmeter. You seem to have good expertise on JMeter. Can you tell me if JMeter works best only for web based applications.?

    ReplyDelete
  9. Sure JMeter is best with Web protocols because that's what is being most popular for.
    I still think you should research where this bottleneck is coming from. Maybe your computers is utilizing 100% CPU? Maybe the JMeter JVM reaches it's limits? Maybe the bottleneck is actually the MQ Server which cannot handle more than certain level of tps.

    ReplyDelete
  10. Hi Shmuel thanks for your reply. My CPU uses only 20-30%. I have also increased the JVM heap memory to 2 GB. There cannot be a bottleneck with MQ server because we have achieved the same thing using Loadrunner. I have tried my best to improve the performance by running my load test on 2 machines( client server mode in JMeter) and by also running it through command mode. Still scalability is a problem. I am using a Constant throughput timer to ensure it meets the desired tps rate.After a while tps rates drops and error rate increases. Which makes me want to forcibly but sadly believe if Loadrunner wins when it comes to scalability with MQ.Can you think of any other possible solutions. It would be really useful

    ReplyDelete
  11. I would:
    1. Look the logs for errors, etc...
    2. Use the JMeter mailing lists to discuss this.

    ReplyDelete
  12. @Mary, instead of a constant timer try other ones, you may tweak with the number of agents and scripts, i.e., create repeated scripts inside the same test scenario and distribute the number of agents, also try to run more than 1 JM instance in the same or other servers, it may fix the problem or help you to figure out where the problem is. I have done tests which achieved 1000+ TPS in a same MQ queue with no issues, 2 JM instances and a 64-core server just for me :).

    ReplyDelete
  13. Hi Shmuel,
    We are working on stress testing our POS solution which is Java based on a Client/Server architecture. The goal that we want to achieve is following

    1. Identify the maximum load that a server can handle.
    2. Identify the maximum number of clients that can be hooked to the server.
    3. Identify the network bandwidth load
    4. Identify the Memory/CPU/Thread/Process lists on the server and clients
    5. A dashboard to monitor the above pieces

    My question is whether Jmeter is the right tool for the stress testing on this Multi Tiered Architecture (Client-Server) on Java Swing based product? Please let me know so we can make an informed decisions. Greatly appreciate your feedbacks.

    ReplyDelete
  14. Hi Asu,
    Please direct your question to the jmeter users list.

    ReplyDelete
  15. Hi Samuel,
    Please clarify if JMeter can be used for SAP Gui Based and Web based applications?

    ReplyDelete
  16. Hi Antony,
    Well while JMeter can be used to load anything over the network, you will have to find a way to understand the SAP GUI protocol over TCP and send the required data to the server. You cannot record and play that easily with the proxy recorder.
    Gladly - there's an easy work-around for that, you can use SAP GUI over HTTP by installing additional component into your landscape - the SAP ITS. As JMeter supports HTTP protocol, it allows makes it possible to capture/record all traffic and play it back (with some manual work). You just need to keep in mind that you do not over load ITS itself, causing it to be the bottle neck of your entire installation.

    ReplyDelete
  17. Gents! What a pity but the 'facts' you present are pure fiction :) All the points I see in support of JM and against LR are half knowhow. Without wasting my time much, can any JM lover tell me what their loved tool provides for detailed Root Cause Analysis??? Without an effective RCA all the love JM mumbo jumbo is flat on the floor...please read some perf engineering and we can talk thence.

    ReplyDelete
    Replies
    1. Why don't you share how effective is indeed the RCA Load Runner provides?
      Please be kind and share how easy it is to install and maintain HP Performance Center system with HP Diagnostics system, connected to work together through BSM to get the so called RCA.

      While in fact RCA is good in theory, almost 0 clients of HP are actually use that themselves. This means that wherever someone want's to have such capabilities, it doesn't only require licensing for those frameworks I've mentioned, but also require investment in infrastructure and the worst part - having an HP BSM specialists in place to install and maintain all of that. Only then it comes to have another HP Specialist(s) with knowledge in Performance Center, Load Runner and Diagnostics.

      Simple, isn't it?

      Delete
    2. And BTW Rovy - When someone clicks on "Try Load Testing on SaaS now" on HP website - he expects to actually try it now and not to fill a form to have a meeting with sales.

      Try SaaS JMeter load testing now at:
      https://flood.io/
      http://blazemeter.com/

      Delete
    3. Loadrunner does not have root cause anaylsis capability, no tool does that is the point of having an engineer engage with the project and IT teams to start digging into the system stack when an issue is observed and apply good engineering principles to it.

      Scott

      www.idealtechnology.co.nz

      Delete
  18. Hello,
    I found your blog very useful for newcomer to Jmeter like me.
    I am a QA, new to Jmeter, in fact to Performance testing itself. I have to do performance testing of a data management web application, which requires login, creates records and put the values into database, extract the data from database and update information etc. I don't have much time to do the load testing, as the releasing date is coming closer.
    What I plan to do in this short interval of time is as follows

    1. Create scripts using blazemeter chrome extension.
    2. Export it to Jmeter.
    3. Do some static parameterization for the Login(User name and password- Since I need to have a scenario with 1000+ users accessing the website.)using the CSV file.
    4. Upload the script to Blazemeter and do the load testing from the Cloud by availing the Blazemeter service.
    5. Analyze the result and do the tuning.

    Since I don't have much knowledge in this , you may find my attempt silly.
    Your advice/help on this regard will be highly appreciated.

    ReplyDelete
  19. In QTP or Load runner, when we play a recorded script, it shows visuals of the application with the navigation steps as in the script. Is there some way I can see it in Jmeter?

    ReplyDelete
    Replies
    1. Before starting the proxy recorder, add a sub element "View Results Tree" which will allow you to view the recorded responses.
      Try asking such questions in the JMeter users list or go through the documentation. Both available in the JMeter website.

      Delete
  20. In the 'View Results Tree', I am not able to identify some of the steps after the login steps. The home page has a some images and I guess that 'Response data' with HTML view within 'View Results Tree' should display the images; however I see a whole bunch of characters (with too many special chars); Do you know a way to display the actual image from the application?

    ReplyDelete
    Replies
    1. Not sure exactly what your issue is, generally the View Results Tree doesn't display css, images and doesn't execute JS...
      Regarding characters, I'm not sure neither, please try to get help from the JMeter community.

      Delete
  21. This comment has been removed by a blog administrator.

    ReplyDelete
  22. I ѡɑs recdommеnded this blog throuցh my coսsin. I'm now not certain whetheг or not this publisҺ
    is written via him ɑs nobody else recognise such detailed abput myy problem.
    You're incrеdible! Thank you!

    my blog: bse exchange

    ReplyDelete
  23. Do Jmeter script has provision of Vuser_init,Action and Vuser_end as it was in Loadrunner. This is one of the questions asked to me when I proposed my management to migrate form LR to Jmeter. This is was a legitimate one from them as we record our login part in Vuser_init and similarly logouts in vuser_end function and these get executed once (vuser_init at start and vuser_end at the end). Now the big question is how can I achieve the same in Jmeter. Please reply as it is urgent

    ReplyDelete
    Replies
    1. Hi jaideep umray,

      yes of course that you can imitate the same behivour like in LR by creating a thread group and then adding only once controller and under this controller you put your http reuqest sampler and that's it you are good to go... :)

      I hope that it is still relevant.

      Erez

      Delete
  24. As well if you really wanna use LR and first you want to test it there is a Loadrunner Community edition with free 50 vusers for life, you can achieve it through https://www.pronq.com/ site where you can download Loadrunner and other products.

    Cheers!

    Erez (Load & Performance Engineer).

    ReplyDelete
  25. Hi Guys, Has anyone used jmeter for XMPP chat application load testing?

    ReplyDelete
  26. This is very nice post for Load Runner Learner thanks for share If you need More Help Please visit How To Install HP - Loadrunner on _PC

    ReplyDelete