Wednesday, May 13, 2015

SAP Blogs performance

I've recently published a new post on the SCN (SAP Community Network) about the new blogging platform we're building there and how it is going to be super fast.

Read on at:

If you're interested on how we do that, read some of my previous posts here and wait for future posts as I'll share more of the improvements we do to build our social blogging platform based on Wordpress.

Thursday, April 02, 2015

Does HHVM really improve performance in real life?

HHVM in one sentence is a relatively new implementation for the PHP runtime, built by Facebook as open source project and focused on improving PHP performance.

For about half a year (somewhere around late 2014) Wordpress is fully supported by the HHVM runtime.

There are plenty of posts around the net claiming you can get a boost of 5 times faster page loading times (reduction of 80% of the original response times) and up to 40 times faster page loading times (reduction of 97.5% of the original loading times) by simply switching from the original PHP runtime to HHVM.

As I'm in the seek to improve overall response times for our new Wordpress based platform, such improvement figures sounds like I must give it a try.

I've created a new application server into our load testing environment, installed it by following installation guidelines from Building-and-installing-HHVM-on-RHEL-7, made few tweaks and got it to work with Apache via FastCGI.

With that I've started to manually browse across the Wordpress system and feel if there's any improvement to performance. Using Chrome developer tools I've noticed a decrease with page response times from around 400ms to 300ms for viewing different blog posts. Meaning a 1.5X times improvement (or reduction of 25% of the original response times).

After validating that the Wordpress application works fine and resolved several issues, I've decided it is time to start load testing the cluster.

At that point - I had a cluster of 3 "normal" PHP app servers (php 5.5 with opcache enabled) and 1 HHVM-based app server, all in the same network, having the same sizing, connected to the same DB, having the same application installed with as close as possible configuration.

Starting the load test while monitoring application response times and comparing between the 3 normal PHP-based app servers and 1 HHVM-based app server (shown as app08 below).
Here are some figures for few Wordpress activities I've been looking into:

If you've been reading my blogs for a while, you would know that my load tests are basically hitting the system with production-like workloads, meaning that the entire setup is not under stress and doesn't suffer from any resource starvation, but rather is kept pretty idle, not using more than 20-30% of CPU across the different components (app/db servers).

What is fairly easy to see here is that also with load test running, the performance is a lot like what I saw with my manual, single user browsing around the system.
Here is a summary:
  • Viewing a blog post improved from 475ms to 300ms (37% reduction / 1.6X times faster).
  • Viewing the homepage improved from 850ms to 600ms (30% reduction / 1.4X times faster).
  • Login improved from ~350ms to ~220ms (37% reduction / 1.6X times faster).
  • Posting a comment degraded from 500ms to 5500ms (-1000% reduction / 10X times slower).
    I've decided to ignore this for now, as it was only evaluation of HHVM.
    Not sure what was the cause for this decrease in performance.
Basically HHVM was tested as is, without any other optimization or configuration changes.
I found that the version I used was already with JIT enabled and with the defaults to JIT code after few executions (cannot recall but I think it was 10 or 12 runs).

Looking back to the original numbers which led me to give that a try, this was somewhat disappointing. I've been looking to give a try with HHVM precompiling but I'm not sure I got it right or just didn't see any significant improvements. Maybe that's obsolete? Anyone knows?

At this point we decided not to use HHVM yet, as we are so close to the initial go-live and using HHVM will also force us doing some infrastructure changes to comply with the HHVM prerequisites, but at least we know what is it that we are missing.

Sunday, March 22, 2015

Wordpress load testing framework

Following my last two posts, about how to create realistic load tests and Wordpress performance I guess that the next post is just on time.

In the last post I've shared that we build up a new Wordpress platform and we continuously load test and improve its performance. In this post I'll share the resources and the concepts behind how we do that, while I hope some people will find it useful and will contribute back to improve it.

I have created a public github repository under about a week ago. In this repo there are all required resources for load testing a Wordpress site.

It is basically focused on the features we use on our own website, such as high rate of authenticated users, viewing blog posts (including AJAX calls interacting with the WP popular posts plugin for view counts and with Shariff for social figures), searching content, publishing new posts, posting comments, liking and unliking posts and comments (interacting with Like posts and comments plugin) and consuming RSS feeds. Anyone can extend this skeleton to have more features covered with his load tests, to reflect the relevant realistic scenario or usecase.

One of the most important parts of the JMeter script here is that it is modular. Each piece of interaction with a feature, is broken into a separate JMeter script, which is then loaded by the load test script and placed on the right place in the user flow. This allows each module to be changed while being focused only on a specific feature and allow re-usability of that module also on the single user script - which is used for base-lining performance or just as functional test.

Other resources in this repository are few drop-in plugins for Wordpress to allow:

  1. User impersonation - in case you don't want to create test users or just because you don't want to have all users passwords in a plain text file.
  2. Data export - which you can access with the third JMeter script in this repo - to extract all required objects to interact with during the performance test, such as existing users, blogs, posts, comments and tags.
The idea is that these tests should become part of your continuous integration flow and get you with up-to-date performance figures.

One of the most popular approaches is that for tests you should have an isolated "playground" - a predefined dataset where the test is running against, which is maintained (usually this is done automactically) to have predefined users and data to test with your automation and is deleted or recovered back to the same state as before the test was running, right after each test is finished.
While this concept may work for Functional testing, I think this is a wrong approach for load testing, as you either over-populate or over-interact with specific set of items in the application - thus over time they are getting overloaded and change the test results. You may clean them up with every load test, but you loose any natural / realistic growth of data in your load testing environment that way.

The right approach in my opinion is that your load tests should generate close to realistic workloads on the system, thus, they should provide at least the realistic growth and distribution of data along the way, so once you hit a bottleneck - you'll see it on your testing system before you'll get it on the real system. This is why this framework doesn't provide any mechanism for pre-populate the target Wordpress system with any dummy content. It will create it along the way, as you continuously load test the system. This framework should be agile enough to use anything you already have in the system to continue producing realistic load tests.

A final note is that I hope this github organization will become home to other load testing setups for popular frameworks, such as drupal, django, joomla and others.

Sunday, March 08, 2015

Wordpress Performance

4 months to go-live, more than 6 months after we started to work on the new Wordpress platform for our Social Platform, viewing a blog post takes about 400ms (median during our realistic load tests).
This is quite fast compared to the Internet in overall, but somewhat slower than other popular WP based websites such as TechCrounch or Time where loading times are about 250ms (TTFB to be specific) and I'm including the network latency in those web sites, which is almost 0 when I'm measuring our new platform, so the diffs here are even bigger.

I'm looking back to the very first days (on the left side) where the project just started, with the default theme, without any code charges or plugins and with almost no content in the DB, a blog post TTFB was about 150ms (again during realistic load test). It means that we have an addition of 250ms or 160% decrease in performance, that's bad!

Looking at the homepage loading times, I see that we started with about 170ms and went to 650ms today, that's about 3 times slower (you can see that we had some major slowdown - about 2 seconds - for a while until we resolved that around test number 145):

There are few things to mention at this point which were done or happened over that period of time:
  1. We have changed the theme.
  2. We have installed / removed countless WP plugins and upgraded couple of WP versions, including switching from WP 3.x to 4.x.
  3. We have added a Load Balancer and additional servers.
  4. We have switched to HTTPs and changed apache and php versions and configuration.
  5. We have improved the coverage of the load test with more types of activities.
  6. We have imported content into the system from our existing blogging platform.
  7. Every night our realistic load test is actually creating more and more content, as we expect it to happen in real life.
All of those may and most likely did change our application performance and the observed response times.

What we do to measure and compare our application performance?

  1. We use fully virtualized / on the cloud servers. This allows us to get 1:1 relation in sizing between our load testing environment, which we call staging and our production one.
  2. We use fully automated construction or build and configuration of these machines - so we are always 1:1 in configuration of all infrastructure components between the staging and the production systems.
  3. We have production like content volume and distribution in our staging system.
  4. We have built up a comprehensive load test framework for Wordpress, which allow us to define the realistic workload (as well as other workloads, such as stress test) and use that to hit our most recent application builds, on a regular basis (at least once a day, depending on amount of changes that day).
So some of you may notice that we follow the Continuous Delivery moto -  fail fast, fix fast - we use CI framework to always get a valid build version to our staging system and test it with predefined and (almost) constant workloads to verify we will be able to use that on our production system.

There are lots of things to do which can improve the performance and I'll post updates along the way, on the different improvements that we have both tried and rejected or tried and implemented to get to the grail - sub 250ms response times.