|
|
|
|
|
|
|
|
|
|
URL: http://www.bt.com
|
|
Key Industries:
|
|
Retail
|
|
Telecommunications
|
|
|
Key Sectors:
|
|
Analytics
|
|
Design & Build
|
|
e-commerce
|
|
Pay Per Click
|
|
Usability
|
|
|
BT
|
|
Brief
Crayon is a lead digital agency for BT.com. Our work ranges from website optimisation to marketing comms including social media platforms and eCRM. Because of the sheer scale of the site and products and services offered, volume of traffic and an innovative and enthusiastic online marketing team, we are regularly able to test new innovations.
BT recently briefed us to help them evaluate a number of MVT solutions. MVT is a process by which more than one component of a web page is tested, live, to determine which combination of factors leads to the highest conversion.
The objective for each test was to determine how users responded to different copy, imagery and offers, ultimately deciphering which combination generated the most sales.
Strategy
We started the process by developing a test plan. What areas of the page did we want to test? What insight did we have to inform our thinking and thus what possible versions would we test? What would success look like (e.g. click through or confirmed order)? What types of visitor were we talking to? And finally, what level of statistical significance did we want - how many sales would tell us that a particular route was working?
Using tools like the basic but free Google Web Optimizer or more sophisticated offerings from other leading technology providers, it is now easier than ever to test multiple versions of different components of a page to get to an overall 'winner'.
Execution
By placing a single line of code into the page templates on BT's website, we were able to upload alternative creative (copy, imagery and interactive elements). We created a total of four alternative versions, for each of the three areas of the page. Alongside the default (or control) page, this gave us a total of 125 possible combinations.
The (other) clever bit
Using the Taguchi methodology, we simplified the number combinations or recipes down to 16. Put simply, we didn’t need to test every version of every container against every version of every other container. If A+B works and B+C works, then A+C will also work.
We monitored the stats daily to see which recipes were working and which were not. Along the way we took the worst performing elements out of the test as soon as possible to get statistical significant returns faster and make sure all the remaining live recipes were performing as well as, or better than, the control.
Results
Within a few weeks we had a route which we could prove performed 40% better than the control page. And, with one click of a button, it became the new live site. Because we used this adaptive approach to remove recipes which were not working, the number of sales during the test period continually increased as the page evolved.
MVT continues to be a used as part of our optimisation process and we are using it across other clients in email too.
|
|
|
|
|
|
|