AB Testing with Google Analytics

Written on January 12, 2010 by george

I love AB testing. I think it is either related to the fact I did a year of Maths at university before switching to Computer Science or because human psychology fascinates me. Either way when I launched 5ft Shelf I was keen to test lots. First on the agenda was the default shelf view. For those not familiar with the site you can view a shelf of books in one of two ways – cover view or spine view as we call it on the site (see screenshots – click to enlarge). Whilst my designer was sure that the cover view would be better I knew the target audience wasn’t necessarily technical, and besides the cool factor of a 3D book shelf would capture attention. It was going to be an interesting AB test.

Traditionally when doing AB testing I’ve added the logic to the rails model (it doesn’t really matter if you’re not a rails developer, the point is it was in the application database). This had a few disadvantages. First up it didn’t keep code or the database as clean as it could be in places. Secondly tools had to be built to then analyse the data and thirdly it couldn’t be easily plotted against relevant variables such as those stored in analytics software.

So when Google announced custom variable support in Analytics I sat up and paid attention. By passing through my AB test number to Google Analytics I can do all the reporting associated with AB testing outside my application database and report against more complex metrics that I don’t store such as bounce rate. Perfect, so let’s get started:

The first step is to assign each user to your site a unique AB test number. I went for a random number in the range 1 to 120. Why so large? Well most AB test I perform are just 2 or 3 options. Depending on traffic you can go much larger but to be honest it becomes harder to understood the factors that lead to the users choice if you do that. Anyway 120 is 2*3*4*5 which means you can have 2, 3, 4 or 5 options (or any product of these numbers – eg 6,8…). This gives plenty of options for the future. So for every visitor that comes to 5ft Shelf they get a random number and this remains with them (unless they clear out their cookies).

I then pass this number through to Google Analytics via a custom variable. For example:

var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
    document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));

    try {
    var pageTracker = _gat._getTracker("UA-11028671-1");
    pageTracker._setCustomVar(1, "shelf_view", "USERS_AB_NUMER", 1);
    pageTracker._trackPageview();
    } catch(err) {}

The USERS_AB_NUMBER needs replacing with the users actual ab number. In ruby I used:

@site_preferences.ab_test_number.modulo(2)

as I was only testing two options. You can substitute this code as appropriate for the number of AB choices you have (and to your language of choice if not using Ruby).

A few notable things:

• The first argument in the custom variable index number. Google analytics provides you with 5 of these so you can feasibly track upto five AB tests at once.
• The second argument is the name. I use this to identify the name of the AB test for easy identification in Analytics at a later date
• The third argument is the value we want to store against the name. I used the ab number modulo 2 as that is what is used to display the different options in the view of the application. If you use more complex logic you should substitute it here. You’re not limited to numbers though – you can also store strings
• The last argument is the the scope of the variable. For AB testing you’ll nearly always want 1 for visitor level. More on this can be found in the Analytics help centre if you need it

Note: you can just pass through the AB test number but you will have to combine each number’s entries into 2 sets in this example. This is a real pain as one set will be 1,3,5,..119 the next 2,4,6,..120, so unless you like maths best let your application do the logic and keep the data in analytics simple.

Now we’ve passed the data through to Google Analytics we need to make application specific logic in your views to render the different AB tests (again substitute for your test setup and language):

  if @site_preferences.ab_test_number.modulo(2) == 0
    # Render option 1
  else
    # Render option 2
  end

Finally we need to create a custom report in Google Analytics. Login and click ‘Custom Reporting’ down the left hand side. Then click ‘Manage Custom Reports’ underneath and then ‘Create new report’. You can then drag the metrics which you wish to measure along the top (think what would constitute as a success in your test – longer page views, lower bounce rate etc) and the dimensions down below (you’ll probably only want the Custom Variables here). Make sure you get the Custom Variable values not the keys. The screenshot on the left should show you how it looks (click to increase size).

Then it’s just a case of waiting however long you want to run your test case for and logging into analytics to see the results. You’ll hopefully end up with a chart looking something like the one shown on the right (click to increase size). Note here the real data you’re after is not in the graph but the table below.

So for those of you that asked that’s why 5ft Shelf switched to spine view as the default, there’s your answer :D Happy AB testing folks.

Posted in Uncategorized