Editorial Articles

We’ve been using Multivariate Testing (MVT) in a new way and getting phenomenal results for our clients in the process

MVT is a process by which more than one component of a web page is tested, live, to determine which combination of factors leads to the highest conversion.

Using tools like the basic but free Google Web Optimizer or more sophisticated offerings from other leading technology providers, it is now easier than ever to test multiple versions of different components of a page to get to an overall 'winner'.

We've been evaluating a number of methodologies and platforms recently for our clients and know how to get results quickly, without great expense and with zero risk.

If this is too techie, the impact on conversion is probably all you need to know! In fact, on a recent campaign for BT, we increased conversion by 37%.

MVT in action
BT’s online marketing team are always keen to test new innovations and recently briefed us to help them evaluate a number of MVT solutions.

The objective for each test was to determine how users responded to different copy, imagery and offers, ultimately deciphering which combination generated the most sales.

We started the process by developing a test plan. What areas of the page did we want to test? What insight did we have to inform our thinking and thus what possible versions would we test? What would success look like (e.g. click through or confirmed order)? What types of visitor were we talking to? And finally, what level of statistical significance did we want - how many sales would tell us that a particular route was working?

By placing a single line of code into the page templates on BT's website, we were able to upload alternative creative (copy, imagery and interactive elements). We created a total of four alternative versions, for each of the three areas of the page. Alongside the default (or control) page, this gave us a total of 125 possible combinations.

The (other) clever bit

Using the Taguchi methodology, we simplified the number combinations or recipes down to 16. Put simply, we didn’t need to test every version of every container against every version of every other container. If A+B works and B+C works, then A+C will also work.

We monitored the stats daily to see which recipes were working and which were not. Along the way we took the worst performing elements out of the test as soon as possible to get statistical significant returns faster and make sure all the remaining live recipes were performing as well as, or better than, the control.

The result

Within a few weeks we had a route which we could prove performed 37% better than the control page. And, with one click of a button, it became the new live site. Because we used this adaptive approach to remove recipes which were not working, the number of sales during the test period continually increased as the page evolved.

Scott Hamilton
Crayon