Wordpress load testing framework

Following my last two posts, about how to create realistic load tests and Wordpress performance I guess that the next post is just on time.

In the last post I've shared that we build up a new Wordpress platform and we continuously load test and improve its performance. In this post I'll share the resources and the concepts behind how we do that, while I hope some people will find it useful and will contribute back to improve it.

I have created a public github repository under https://github.com/jmeter-templates/wordpress about a week ago. In this repo there are all required resources for load testing a Wordpress site.

It is basically focused on the features we use on our own website, such as high rate of authenticated users, viewing blog posts (including AJAX calls interacting with the WP popular posts plugin for view counts and with Shariff for social figures), searching content, publishing new posts, posting comments, liking and unliking posts and comments (interacting with Like posts and comments plugin) and consuming RSS feeds. Anyone can extend this skeleton to have more features covered with his load tests, to reflect the relevant realistic scenario or usecase.

One of the most important parts of the JMeter script here is that it is modular. Each piece of interaction with a feature, is broken into a separate JMeter script, which is then loaded by the load test script and placed on the right place in the user flow. This allows each module to be changed while being focused only on a specific feature and allow re-usability of that module also on the single user script - which is used for base-lining performance or just as functional test.

Other resources in this repository are few drop-in plugins for Wordpress to allow:

1. User impersonation - in case you don't want to create test users or just because you don't want to have all users passwords in a plain text file.
2. Data export - which you can access with the third JMeter script in this repo - to extract all required objects to interact with during the performance test, such as existing users, blogs, posts, comments and tags.

The idea is that these tests should become part of your continuous integration flow and get you with up-to-date performance figures.

One of the most popular approaches is that for tests you should have an isolated "playground" - a predefined dataset where the test is running against, which is maintained (usually this is done automactically) to have predefined users and data to test with your automation and is deleted or recovered back to the same state as before the test was running, right after each test is finished.

While this concept may work for Functional testing, I think this is a wrong approach for load testing, as you either over-populate or over-interact with specific set of items in the application - thus over time they are getting overloaded and change the test results. You may clean them up with every load test, but you loose any natural / realistic growth of data in your load testing environment that way.

The right approach in my opinion is that your load tests should generate close to realistic workloads on the system, thus, they should provide at least the realistic growth and distribution of data along the way, so once you hit a bottleneck - you'll see it on your testing system before you'll get it on the real system. This is why this framework doesn't provide any mechanism for pre-populate the target Wordpress system with any dummy content. It will create it along the way, as you continuously load test the system. This framework should be agile enough to use anything you already have in the system to continue producing realistic load tests.

A final note is that I hope this github organization will become home to other load testing setups for popular frameworks, such as drupal, django, joomla and others.

Wordpress Performance

4 months to go-live, more than 6 months after we started to work on the new Wordpress platform for our Social Platform, viewing a blog post takes about 400ms (median during our realistic load tests).
This is quite fast compared to the Internet in overall, but somewhat slower than other popular WP based websites such as TechCrounch or Time where loading times are about 250ms (TTFB to be specific) and I'm including the network latency in those web sites, which is almost 0 when I'm measuring our new platform, so the diffs here are even bigger.

I'm looking back to the very first days (on the left side) where the project just started, with the default theme, without any code charges or plugins and with almost no content in the DB, a blog post TTFB was about 150ms (again during realistic load test). It means that we have an addition of 250ms or 160% decrease in performance, that's bad!

Looking at the homepage loading times, I see that we started with about 170ms and went to 650ms today, that's about 3 times slower (you can see that we had some major slowdown - about 2 seconds - for a while until we resolved that around test number 145):

There are few things to mention at this point which were done or happened over that period of time:

1. We have changed the theme.
2. We have installed / removed countless WP plugins and upgraded couple of WP versions, including switching from WP 3.x to 4.x.
3. We have added a Load Balancer and additional servers.
4. We have switched to HTTPs and changed apache and php versions and configuration.
5. We have improved the coverage of the load test with more types of activities.
6. We have imported content into the system from our existing blogging platform.
7. Every night our realistic load test is actually creating more and more content, as we expect it to happen in real life.

All of those may and most likely did change our application performance and the observed response times.

What we do to measure and compare our application performance?

1. We use fully virtualized / on the cloud servers. This allows us to get 1:1 relation in sizing between our load testing environment, which we call staging and our production one.
2. We use fully automated construction or build and configuration of these machines - so we are always 1:1 in configuration of all infrastructure components between the staging and the production systems.
3. We have production like content volume and distribution in our staging system.
4. We have built up a comprehensive load test framework for Wordpress, which allow us to define the realistic workload (as well as other workloads, such as stress test) and use that to hit our most recent application builds, on a regular basis (at least once a day, depending on amount of changes that day).

So some of you may notice that we follow the Continuous Delivery moto -  fail fast, fix fast - we use CI framework to always get a valid build version to our staging system and test it with predefined and (almost) constant workloads to verify we will be able to use that on our production system.


There are lots of things to do which can improve the performance and I'll post updates along the way, on the different improvements that we have both tried and rejected or tried and implemented to get to the grail - sub 250ms response times.