In part 1 we looked at what is involved with allocating different users to different groups. The next stage after allocation is to actually present the test to the user.
There are two basic choices when it comes to presenting the interface changes, and that is client or server side. Generally, if you work with a server side scripting technology such as asp.net or php, you are likely to harness this to load the ab testing data from your database and conditionally present the different versions of the site to the different groups. A sample client-side implementation is optimost, where javascript is used to do the tracking and showing of different versions. Its generally a lot less powerful as you are limited to showing/hiding html rather than on the server where you have complete control over the presentation.
Test storage is usually done with a database, so you would register all your tests in a database. When the page code runs, it looks in the database to see if any tests are assigned to this page type, and returns them all. Now the page knows about the tests that apply to it, individual tests that are coded on the pages can check whether the test is running or not, and whether this user will get the A or the B or the C etc. The code then executes to modify the interface for the user. We then need to gather the key information about that user, which is where your tracking comes in. Traffic tracking is quite a large area on its own, but for the purposes of this, we just need to be aware that the following things are tracked;
Which version the customer saw
Whether the customer converted or not, and how much they spent.
The analysis script will later crunch all this data to find out which test is winning.
There are often dilution effects to consider as well. For instance if some functionality will only be shown for certain product types, yet the test is registering across all users, then the results are diluted. Also, it may be that the test is only shown within an ajax piece on the page, which only a certain proportion of users will actually use. Dilution is not a huge problem, if you can reduce it, it certainly helps.
The next phase we will look at is the reporting and analysis side. We will be looking at power functions, which are used to calculate the sample size we need to test again, a few difference confidence calculation formulas, along with charting and decision making. This is probably the hardest part of ab testing as its a subjective area, and we will soon be there.
Get SEO Elite - for only $47!