An Introduction to Split Testing in WordPress

Kirk Kaiser

Split testing is a way to experiment with a live site and find which headlines and text are the most effective. Amazon uses split testing to determine which versions of their site convert customers better. We’ll use WordPress and Google’s Website Optimizer to test two different headlines, and find which works best at capturing customer emails. [Más…]

Step 1: Decide on Your Experiment Type

First, we’ll need to decide what sort of an experiment we’d like to run. In Google’s Website Optimizer, we have two choices: Multivariate Testing or A/B Testing.

Multivariate testing is a good way to test a lot of things at once. Big online stores use multivariate testing to figure out which layouts and ad copies work best. For the purpose of this tutorial, however, we’ll use A/B testing.

A/B Testing is a way to test two different types of copy, and see which works best. We start with two different variations of the same page, send some traffic to the pages, and see which converts best statistically. For the purposes of our split test, we’ll want to have at least 100 conversions before deciding which page is the statistically significant winner.

Anuncios

Split testing is a way to experiment with a live site and find which headlines and text are the most effective. Amazon uses split testing to determine which versions of their site convert customers better. We’ll use WordPress and Google’s Website Optimizer to test two different headlines, and find which works best at capturing customer emails. Leer más “An Introduction to Split Testing in WordPress”

A Complete Guide to A/B Testing

A/B testing (also called split testing) is a testing method generally used in marketing to compare results between two samples with the goal to improve conversion or response rates.

In web design, A/B tests are generally used to test design elements (sometimes against the existing design) to better determine which design elements will get the best response from visitors.

A/B tests, by definition, compare only two variables (design elements) at a time. There is also multivariate testing, which compares more than one variable.


A/B testing (also called split testing) is a testing method generally used in marketing to compare results between two samples with the goal to improve conversion or response rates.

In web design, A/B tests are generally used to test design elements (sometimes against the existing design) to better determine which design elements will get the best response from visitors.

A/B tests, by definition, compare only two variables (design elements) at a time. There is also multivariate testing, which compares more than one variable. Leer más “A Complete Guide to A/B Testing”

How to Improve Your Website With A/B Testing

By: Kean Richmond

In recent years, competition on the Internet has grown fiercer, and many businesses are looking for anything that gives them that competitive edge. Online marketing methods such as SEO and PPC are a good way of improving a company’s ability to compete, but increased traffic alone doesn’t make for a successful business. When it comes to a website, it all comes down to number of conversions and thus, revenue generated; this is where a/b testing can help.

Improve Your Website With A/B Testing

What is an A/B Test?

A/B testing is a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates. – Wikipedia

As the Wikipedia quote above describes pretty well, A/B testing is a method whereby an alternate version of an element within a website can be displayed to a group of users in order to track its effect on conversion rates.


In recent years, competition on the Internet has grown fiercer, and many businesses are looking for anything that gives them that competitive edge. Online marketing methods such as SEO and PPC are a good way of improving a company’s ability to compete, but increased traffic alone doesn’t make for a successful business. When it comes to a website, it all comes down to number of conversions and thus, revenue generated; this is where a/b testing can help.

Improve Your Website With A/B Testing
Image credit: Darrenhester

What is an A/B Test?

A/B testing is a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates. – Wikipedia

As the Wikipedia quote above describes pretty well, A/B testing is a method whereby an alternate version of an element within a website can be displayed to a group of users in order to track its effect on conversion rates. Leer más “How to Improve Your Website With A/B Testing”

Marketing Optimization Technology: Be careful of shooting yourself (and your test) in the foot

…) I had the pleasure of learning about an experiment devised by my colleague, Jon Powell, that illustrates why we must never assume that we test in a vacuum devoid of any external factors that can skew data in our tests (and even looking at external factors that we can create ourselves).

If you’d like to learn most about this experiment in its entirety, you can hear it firsthand from Jon on the web clinic replay. SPOILER ALERT: If you choose to keep reading, be warned that I am now giving away the ending.

So after reanalyzing the data and adjusting the test duration to exclude the results from when an unintended (by our researchers at least) promotional email had been sent out, Jon saw that each of the treatments significantly outperformed the control with conclusive validity.


(…) I had the pleasure of learning about an experiment devised by my colleague, Jon Powell, that illustrates why we must never assume that we test in a vacuum devoid of any external factors that can skew data in our tests (and even looking at external factors that we can create ourselves).

If you’d like to learn most about this experiment in its entirety, you can hear it firsthand from Jon on the web clinic replay. SPOILER ALERT: If you choose to keep reading, be warned that I am now giving away the ending.

Computer ChipAccording to the testing platform Jon was using, the aggregate results came up inconclusive. None of the treatments outperformed the control with any significance difference.  However, what was interesting is the data indicated a pretty large difference in performance with a couple of the treatments.

So after reanalyzing the data and adjusting the test duration to exclude the results from when an unintended (by our researchers at least) promotional email had been sent out, Jon saw that each of the treatments significantly outperformed the control with conclusive validity. Leer más “Marketing Optimization Technology: Be careful of shooting yourself (and your test) in the foot”

Multivariate Testing: Can you radically improve marketing ROI by increasing variables you test?

In response, one emerging MVT service model offers getting to a “lift” faster by using adaptive elimination of likely underperformers, in exchange for the test results providing limited information beyond identifying the winner. Such test results are not as useful as their full-factorial brethren for designing subsequent tests because adaptive elimination of treatments makes it difficult to extrapolate the psychological factors and consumer preferences responsible for the test outcome. The immediate business benefits, however, are more immediate.


As I was reading a few LinkedIn discussions about multivariate testing (MVT), I began to wonder if 2010 was going to be the year of multivariate.

1,000,000 monkeys can’t be wrong

Multivariate Testing (MVT) is starting to earn a place in the pantheon of buzzwords like cloud computing, service-oriented architecture, and synergy. But is a test the same thing as an experiment? While I am not a statistician (nor did I stay at the Holiday Inn last night), working at MarketingExperiments with the analytical likes of Bob Kemper (MBA) and Arturo Silva Nava (MBA) has helped me understand the value of a disciplined approach to experimental design.

MonkeyWhat I see out there is that a little knowledge is indeed a dangerous thing. Good intentions behind powerful and relatively easy-to-use platforms like Omniture® Test&Target™ and Google® Website Optimizer™ have generated a misleading sense that as long as a multivariate test is large enough (several hundred or more combinations being tested), at least one of the combinations will outperform the control.

This notion has become the value proposition of a growing number of companies offering services around either the big-name or their own (simpler, and often therefore easier to set up) MVT tools. They are ostensibly betting on the technology, and not on a systematic approach to experimental design or any particular UI/UX (user interface/user experience) optimization theory.

Even though, as Bob has pointed out to me, it is reasonable that an MVT setup with a billion combinations may not yield a lift over the control, my contention is that the risk-weighted business cost of a dissatisfied customer is low. Therefore, little stops the burgeoning MVT shops from safely offering a “100% lift guarantee.” Just like the proverbial million monkeys with typewriters, somewhere among thousands of spray-and-pray treatments their MVT tests are expected to produce one that’s better than the rest.

1 monkey with a stick Leer más “Multivariate Testing: Can you radically improve marketing ROI by increasing variables you test?”