And then I wake up from my dream. Because, of course, that’s not the experience most of us have anymore. For the majority of the country, we step into a big box retailer. The doors open with a cruel, efficient “whoosh.” Sure, we get a “welcome to…” greeting, but then a cold metal shopping cart or the latest circular is thrust upon us. One-to-one marketing is dead…well, in the bricks and mortar world.

Online, it is actually alive and well, thanks to…

Customer Segmentation Analysis

Customizing offers based on customer segments involves a sequence of steps, beginning with identification of meaningful customer segments (which is the objective of the customer segmentation analysis).

Any method of customer-specific message optimization requires knowing “who” your customers are, beyond just their names and email addresses. You must also have some insight into what they “want,” how they think, the words, terms and images that attract and inspire them (as well as those that repel them), and “how” they think.

Discovering these things about all of the distinct groups of your “ideal customers” is the objective of a customer segmentation analysis.

The outcome of the customer segmentation analysis is a set of reports, charts, tables and interpretive text that strive to reveal the important characteristics that distinguish your best and most valuable customers and customer prospects, and to provide actionable recommendations about how best to communicate with them in a way that will serve them and profit your company.

Dynamic Offer-Content Customization

Once we have identified the optimal customer segments, the next step is to develop and test content at each stage in the conversion path designed to communicate your value proposition in a way that optimally conveys the elements that are most compelling to that particular segment according to their motivations, biases and thought sequence.

This is the transition point in implementing Dynamic Offer-Content Customization.

Thereafter, additional gains are achieved through the iterative optimization process that continually seeks to communicate more clearly and effectively with each customer and customer prospect to assure optimal long-term value and customer goodwill.

It should be noted that Dynamic Offer-Content Customization is only one among a host of breakthrough marketing approaches that advanced customer segmentation analysis makes possible – surely it’s far from the easiest or the least expensive.

Still, the electronic experience is the closest virtual representation of our quaint bakery shop antecedent – one-to-one marketing to your customers in a mass specialization world.

Related Resources

One-to-One Marketing at Four Levels: Strategic ways every marketer can enter into an online conversation with customers

Optimize your Email in Three Steps: How one marketer tripled revenue from their house list

Configurator – How the use of a Configurator enabled our test site to achieve an 11.74% conversion rate through its primary order path

What else can I test… to increase email clickthrough rate?

That’s right; we like to turn them into guinea pigs.

We like to see if our blog readers, knowing the basic circumstance surrounding a recent test, can predict the outcome. How good is their online marketing radar? Can they spot a good webpage when they see one? How is marketing intuition performing these days?

But honestly, what really matters is the cheese they will be racing for today – one good-ole slice of free online certification course cheese with a little Twitter-love wine to wash it down.

Leave a comment below to enter and let the games begin.

The Experiment

The Research Partner we were working with provides online consumer brokerage services through a subscription-based model. This page, in particular, was aimed at visitors interested in signing up for the foreign exchange trading (FOREX) solutions.

The Control (click to zoom)

After analyzing the current landing page, we concluded that there were some significant factors contributing to confusion on this page. For one, there were  many competing graphical elements and objectives. In almost all cases, this type of layout negatively impacts conversion. We also believed that the value of this offer could be communicated with a little bit more oomph.

So we tested three designs against the control to address some of these issues.

Treatment 1 (click to zoom)

The first treatment is probably the closest to the control. However, there are some strategic changes.

First, we added a headline that better communicated the value of the offer. The copy also has been reorganized in a clearer, easier to read fashion.

And finally, we added a call-to-action button in the main section of copy.

Treatment 2 (click to zoom)

The next treatment used a more long copy approach than the control. It also incorporated a stronger headline and clearer copy layout similar to that of the first treatment.

It is important to note that some of the visual elements from the control have been removed from the bottom of the page. However, the left-hand column remained the same as the previous two designs.

Treatment 3 (click to zoom)

This version of the page is almost identical to treatment 2’s long copy layout.

The one big change for this version was that the elements in the left-hand column were changed into a simple navigation.

(Update) The Results

If you are reading this post now, the contest mentioned above is over. Congrats to @terryrydzynski, a marketer who’s intuition got him a free seat in one of our online certification courses. If I were you, I’d follow this brilliant guy’s twitter account.

Which one was the winner you ask? All of the treatments outperformed the control, but Treatment 3 had the highest conversion rate with a validated 31% increase over the control. Now the results were not too surprising if you read some of the reasoning behind our designs above. Treatment 3 significantly reduced the amount of friction over the control by removing the competing graphical elements and focusing the visitor on one objective.

So what can we learn from this experiment?

If there’s one thing that we can all take away from this case study, it is that many times we are trying to accomplish way too much with our pages, and if we could just simplify our message and make options clear for our visitors, we would potentially see an increase in response.

But this is just scratching the surface, if you would like more information about this case study and some of it’s implications, you can find a more detailed explanation in the replay of yesterday’s web clinic, which will be available next week. To be notified when the replay is available, feel free to sign up for free research updates from MarketingExperiments.

Heat Map vs. Data Analysis

Breakdown:

When Heat Map enters the ring, he’s very intimidating, and looks great on the surface. He always shows up in a flashy, beautiful silk robe and has crazy music playing when he enters. Heat Map was a wizard at getting attention and he values the things that give him attention the most. He points to the camera flashes and banners and professes his love with a big red kiss.

All of Heat Map’s theatrics are fine. It’s part of the show. However, he gets himself into trouble by making outlandish conclusions about his opponents, like being able to know exactly what punch his opponent would land just by looking at the color of his clothes. Many times when Heat Map thinks someone will punch him in the rib cage, they instead throw a cross to his temple.

For his fight against Data Analysis, he trained mercilessly in the gym – lifting weights, doing crunches, and hitting punching bags. When it came time to fight, he strolled into the ring confident that his opponent’s strength was the left hook. Why? Well because he saw a red and yellow tattoo of a python on his left bicep.

Too bad he didn’t actually watch footage of Data Analysis’s previous fights. If he had, Heat Map would have known that most of his punches come from the right. What a shame to see him knocked out in a matter of seconds.

Sometimes Heat Map is right about his opponent though. Sometimes the key to victory is written on one’s sleeve. But it’s very, very dangerous to rely on just “sometimes.”

Data Analysis on the other hand is subdued and quiet. A stark contrast to Heat Map. But under that calm exterior is a supreme confidence because, unlike Heat Map, he has been studying his opponent’s film for hours. He’s learned that Heat Map throws himself a little off balance every time he attempts a left hook. He knows that Heat Map doesn’t bob and weave, but instead focuses his attention on one thing…in this case it’s the tattoo of the python.

Heat Map vs. Data Analysis

Breakdown:

When Heat Map enters the ring, he’s very intimidating, and looks great on the surface. He always shows up in a flashy, beautiful silk robe and has crazy music playing when he enters. Heat Map was a wizard at getting attention and he values the things that give him attention the most. He points to the camera flashes and banners and professes his love with a big red kiss.

All of Heat Map’s theatrics are fine. It’s part of the show. However, he gets himself into trouble by making outlandish conclusions about his opponents, like being able to know exactly what punch his opponent would land just by looking at the color of his clothes. Many times when Heat Map thinks someone will punch him in the rib cage, they instead throw a cross to his temple.

For his fight against Data Analysis, he trained mercilessly in the gym – lifting weights, doing crunches, and hitting punching bags. When it came time to fight, he strolled into the ring confident that his opponent’s strength was the left hook. Why? Well because he saw a red and yellow tattoo of a python on his left bicep.

Too bad he didn’t actually watch footage of Data Analysis’s previous fights. If he had, Heat Map would have known that most of his punches come from the right. What a shame to see him knocked out in a matter of seconds.

Sometimes Heat Map is right about his opponent though. Sometimes the key to victory is written on one’s sleeve. But it’s very, very dangerous to rely on just “sometimes.”

Data Analysis on the other hand is subdued and quiet. A stark contrast to Heat Map. But under that calm exterior is a supreme confidence because, unlike Heat Map, he has been studying his opponent’s film for hours. He’s learned that Heat Map throws himself a little off balance every time he attempts a left hook. He knows that Heat Map doesn’t bob and weave, but instead focuses his attention on one thing…in this case it’s the tattoo of the python.

Google vs. Yahoo. This one’s scheduled for 12 rounds, but I think it will be a knock out.

And Google Buzz trained for years for a title match but failed a drug test. So for now, Twitter still holds the Light Heavyweight Belt, but eventually everyone meets their match. Anyone remember Buster Douglas knocking out Mike Tyson?

This spectacular entertainment is appreciated by so-called “gurus” of epic proportions. The fight for Internet marketing supremacy is a quality collection of bouts that will satisfy many an Internet guru, but how will it impact the masses in attendance?

Will they follow the hot trends blindly? Will tricks for quick bucks convince them to throw in their monthly budget for an unproven, highly touted secret weapon?

So whether it’s above-the-fold landing pages or live chats, you need to see the results before you invest your wad. There are reasons galore to believe “best practices” or “paradigm-shifting Internet sensation” hype and hope these latest and greatest tactics to generate revenue make a dent into your bottom line, but if you don’t personally see the fight through you are merely gambling. Do you really want to do that?  Before you start paying the piper, make sure you do your own research first so you have a high confidence level to make bets on the entire fight card.

The Undercards

Above “The Fold” Page vs. Sequence of Thought

The Breakdown: The Fold is not new to the fight scene. He originates from a previous generation of fighters who have mostly retired or passed on. The Fold comes from the newspaper ring, where everyone fought for the space on the top half of the page. For people who didn’t read an entire newspaper word for word, the top half of the newspaper page is where you would look for the most important and popular stories. Much of the content below the fold was missed.

The Fold has a new following on the internet. Everywhere I go people are talking about The Fold. He’s the place on the page where a typical visitor’s screen ends. Anything below him takes extra effort to get to either by manually scrolling or using your mouse wheel. Although it only takes a quick twitch of the index finger, many people will not exert this effort.

So what are we supposed to do? Cram every possible image, word, or call-to-action above The Fold? The answer is no, and our tests time and time again prove it.

With over a decade in the game, and 100+ professional fights under his belt, Sequence of Thought enters the ring to take down the champion. At first glance, The Fold appears ready. He’s neatly arranged and he dons a beautiful new robe. The Fold feels this is good matchup for his style. He is typically at his best when he can use his brawn to draw in conversions without much thought. He wants you to press his buttons right away without even doing the pre-fight research.

But consumers are getting smarter, more keen. His tactics have got him this far, but The Fold has met his match.

The reason why Sequence of Thought can defeat The Fold isn’t because of his appearance. The fight for conversions is not won on the page, but in the mind of the user. It’s won before anyone even steps in the ring. It’s won by understanding your visitors’ motivations and anticipating what they will do, what punches they will throw before they arrive on your website.

What’s their thought process? What information are they looking for? Do they need longer copy because the product requires a big decision? Or do you need to utilize the top half of the page to introduce yourself because you’re unproven in the ring?

Not only is it vital to understand the motivations of the user before you build your page, but it’s essential to test your way into an effective page design. Use metrics and data to fully optimize your site. Launch a test for a few weeks then analyze your visitors’ actions in order to identify patterns and find the areas of your site where improvements will produce the biggest ROI gains.

For some product offerings, a short page is effective. But The Fold is not a one-size-fits-all type of fighter. Our research has proven that often a visitor will scroll if you lead them down the page, either through compelling content or product descriptions. If you tap into their motivations and provide them with what they are looking for, page length is of no consequence.

ADAM LAPP’S UNOFFICIAL SCORECARD:

The Fold – 76

Sequence of Thought – 102

Pretty Page vs. Ugly Page

The Breakdown: All of the promoters these days are looking for lean and chiseled with a marketable face. Their goal is to attract attention, put up a visually stimulating facade in order draw viewers to the fight and maximize pay-per-view orders. Pretty Page has lots of colors and images on it. (Do customer service reps really look like the stock photos? Wow!)

There’s lots of flash (along with some Flash). White text on black background. I see at least three twitter icons. Gradients galore. And then there’s that video that I had go out of my way to click pause just so I could focus on reading the actual content of the page.

The Pretty Page looks good. As it should be since you probably paid a small fortune to have it designed and developed.

But does a Pretty Page increase your conversion rate?

Meanwhile in the cellars of the arena, Ugly Page is whaling away at a punching bag. Unlike his opponent, Ugly Page is a grizzled veteran with an acute knowledge of how to win a fight. This comes from years and years of experience. He knows that conversion doesn’t depend on looks, having the biggest biceps, or wearing the most colorful shorts. He knows that conversion depends solely on your strategy. And strategies are not built from the latest trends or by implementing quick hits that you learned at a one-day marketing conference. They are built from testing. From figuring out definitively what works and what does not work.

But the point is not that Ugly Page is a better fighter than Pretty Page. In fact, Pretty Page may eventually become a much better fighter than Ugly Page. The important thing to understand is that strategy is more important than design. You could have the best-looking page in the world, but if it hasn’t been tested, if it hasn’t been optimized based on what you learned from data and results, then it’s nothing more than a pretty page.

The fighter that will prevail is one that tests the layout, tone, copy, and positioning before testing images and color schemes. Ugly Page learns how to communicate his value as a fighter before he adorns himself with a fancy robe. He reduces friction and anxiety before his publicist gives him a makeover.

The bottom line is that the ideal testing sequence is to test key strategies first, figure out which works the best, then add an aesthetic layer.

ADAM LAPP’S UNOFFICIAL SCORECARD:

Pretty Page – 88

Ugly Page – 96

Let’s get ready to rumble

Throughout the week, I’ll be shedding light on more Internet marketing fights right here on the blog. We have Flash Banner vs. Headline, Quantity vs. Quality, Heat Map vs. Data Analysis, Bounce Rate vs. Conversion, and finally…the main card…Tricks vs. Testing.

Related Resources

The Magical Metrics Tour: Demystifying the secrets behind analytical “tricks” to help you drive ROI

A/B Split Testing – How to use A/B Split Testing to Increase Conversion Rates, Challenge Assumptions and Solve Problems

Multivariable Testing – How testing multiple changes simultaneously can save you time, speed up your optimization schedule, and increase your profits

LOGISTICS

When it comes to the “how to test” question, what most people usually want to know is “which tool should I use.” But before you delve into testing platforms, you should really ask yourself a few questions:

• Do I have IT or technical support to build out the new page/path to test?
  • If not, who can I contract to build out the HMTL of the page?
• Do I have FTP or Apache Subversion (SVN) access to modify the page code so that I can add the testing platform code?
  • If not, where does this fit into IT’s project queue?

TOOL SELECTION

Once you’ve documented the logistics to build out the test, now it’s time to choose the testing platform that is best for your business. There are quite a few options out there right now. To help you pick the right tool, I put together the following list with Jessica McGraw, our technical manager of research partnerships…

Google Website Optimizer

The biggest benefit is quite obvious – it’s free! So what do you get for nothing?

Let’s start with support. There is a lot of instructional information about GWO available on the GWO site, including step-by-step directions and FAQs. This makes it easy for just about anyone to implement. And it is fairly straightforward and easy to use with little IT involvement necessary.

Yet it has the capabilities to allow you to conduct everything from simple A/B split tests to more complex multivariate tests (testing different combinations of elements).

Paired with Google Analytics e-commerce tracking, you’re able to report at the transaction and product level.

It can take a few hours for data to begin showing in the reports.

Omniture Test&Target

Test&Target also offers A/B and multivariate testing along with transaction- and product-level reports, but it does so with some really impressive in-depth reporting. For example, you have access to not only aggregate data, but daily data as well. This gives you the ability to spot cyclical changes within your test data and validate tests quicker.

You also gain the flexibility to target content to different groups of people based on defined segments. If you don’t know which segment should get which content, you can find out by setting up a test with different segments and watching the results of each segment individually to determine your new content. Test&Target also includes a feature where offers can be automatically targeted to site visitors based on several different variables including observed visitors’ on-site behavior.

In terms of support, you are not on your own. Your contract likely provides access to an account rep along with support hours to gain some insight into reports and help with complex test setup.

You data will likely appear in reports in mere minutes. And you can integrate reporting into a Site Catalyst account (Omniture’s web analytics product).

And while Omniture is a paid tool, the cost difference compared to a free tool might not be clear cut, because your company may already have a contract with Omniture. If it does, find out the additional cost to use Test&Target.

Unbounce

Unbounce bills itself as having “self-served, hosted landing pages for marketers with A/B testing.” It allows you to not only design the page, but build it and set it up all within the same tool.

This is an easy-to-use service which requires little to no IT involvement. You even get a

WYSIWYG (what you see is what you get) landing page editor, so you can build landing pages with no web designer or without even having any HTML experience.

The tradeoff is that you can only run simple A/B split tests.

Drupal

If your site uses a content management system (CMS) like Drupal, there is likely a split testing plug-in that is available. Drupal itself is free and open source, and if it is your CMS and you use it for testing, you are assured instant compatibility with no need to pay for extra testing platforms.

As with other open-source solutions, “free” usually comes at a cost. Not surprisingly, Drupal requires a lot of IT involvement. Although, new platform updates are coming that will require less from IT.

On the flip side, since Drupal is open source, you can add any and all reporting features your particular tests require.

Build your own

You can always build your own split testing tool. Well it’s free, right? And by free, I mean that there are no outside costs or need to engage a consultant on product questions.

And you certainly have flexibility to build your own reports. You also have the support of in-house staff with the full knowledge of tool capabilities and ability to add extra features as IT has time.

And there’s the rub…”as IT has time.” You’ll have to gauge exactly what those words mean for your particular organization.

BUILDING THE TIMELINE

This last section usually doesn’t fall under “how to test” for many marketers, but rather “why I really can’t test even though I’d like to.” In fact, in a poll we conducted in a recent web clinic, time was identified as our audience’s biggest challenge to begin the implementation of the testing-optimization cycle in their organization.

This might seem like an insurmountable challenge up front, (since most marketers feel like they can barely keep up with their deadlines as it is). But as with any excuse (I have no time to exercise, eat right, learn Swahili, etc), it can be overcome by simply planning ahead and getting a firm understanding of the resources you will need.

When building your marketing campaign timelines, keep in mind that depending on what direction/solution you take with testing… you’ll need to build in extra time for:

• Reviewing your site analytics to identify your biggest opportunity and the best elements to test
• Working with a designer to modify your current design and incorporate the test variables
• Fitting into the development queue (if not using Unbounce). In most cases, marketing is the lowest priority in the development world, so ensure that you have your place in the queue before setting an expected launch date

Related Resources

Technology Blind Spots: How human insight revealed a hidden (and almost missed) 31% gain

A/B Split Testing — How to use A/B Split Testing to Increase Conversion Rates, Challenge Assumptions and Solve Problems

Cost of Delay: How to win approval for your test and test schedule

What to test (and how) to increase your ROI today

How Can You Make Your Web Site Smarter?

Are You Listening Fast Enough? Using A/B and Multivariate Testing to Accelerate Conversion Improvements

LOGISTICS

When it comes to the “how to test” question, what most people usually want to know is “which tool should I use.” But before you delve into testing platforms, you should really ask yourself a few questions:

• Do I have IT or technical support to build out the new page/path to test?
  • If not, who can I contract to build out the HMTL of the page?
• Do I have FTP or Apache Subversion (SVN) access to modify the page code so that I can add the testing platform code?
  • If not, where does this fit into IT’s project queue?

TOOL SELECTION

Once you’ve documented the logistics to build out the test, now it’s time to choose the testing platform that is best for your business. There are quite a few options out there right now. To help you pick the right tool, I put together the following list with Jessica McGraw, our technical manager of research partnerships…

Google Website Optimizer

The biggest benefit is quite obvious – it’s free! So what do you get for nothing?

Let’s start with support. There is a lot of instructional information about GWO available on the GWO site, including step-by-step directions and FAQs. This makes it easy for just about anyone to implement. And it is fairly straightforward and easy to use with little IT involvement necessary.

Yet it has the capabilities to allow you to conduct everything from simple A/B split tests to more complex multivariate tests (testing different combinations of elements).

It can take a few hours for data to begin showing in the reports.

Omniture Test&Target

Test&Target also offers A/B and multivariate testing, but it does so with some really impressive in-depth reporting that gives you the ability to pass through product IDs.

You also gain the flexibility to target content to different groups of people based on defined segments. If you don’t know which segment should get which content, you can find out by setting up a test with different segments and watching the results of each segment individually to determine your new content. Test&Target also includes a feature where offers can be automatically targeted to site visitors based on several different variables including observed visitors’ on-site behavior.

In terms of support, you are not on your own. Your contract likely provides access to an account rep along with support hours to gain some insight into reports and help with complex test setup.

You data will likely appear in reports in mere minutes. And you can integrate reporting into a Site Catalyst account (Omniture’s web analytics product).

And while Omniture is a paid tool, the cost difference compared to a free tool might not be clear cut, because your company may already have a contract with Omniture. If it does, find out the additional cost to use Test&Target.

Unbounce

Unbounce bills itself as having “self-served, hosted landing pages for marketers with A/B testing.” It allows you to not only design the page, but build it and set it up all within the same tool.

This is an easy-to-use service which requires little to no IT involvement. You even get a

WYSIWYG (what you see is what you get) landing page editor, so you can build landing pages with no web designer or without even having any HTML experience.

The tradeoff is that you can only run simple A/B split tests.

Drupal

If your site uses a content management system (CMS) like Drupal, there is likely a split testing plug-in that is available. Drupal itself is free and open source, and if it is your CMS and you use it for testing, you are assured instant compatibility with no need to pay for extra testing platforms.

As with other open-source solutions, “free” usually comes at a cost. Not surprisingly, Drupal requires a lot of IT involvement. Although, new platform updates are coming that will require less from IT.

On the flip side, since Drupal is open source, you can add any and all reporting features your particular tests require.

Build your own

You can always build your own split testing tool. Well it’s free, right? And by free, I mean that there are no outside costs or need to engage a consultant on product questions.

And you certainly have flexibility to build your own reports. You also have the support of in-house staff with the full knowledge of tool capabilities and ability to add extra features as IT has time.

And there’s the rub…”as IT has time.” You’ll have to gauge exactly what those words mean for your particular organization.

BUILDING THE TIMELINE

This last section usually doesn’t fall under “how to test” for many marketers, but rather “why I really can’t test even though I’d like to.” In fact, in a poll we conducted in a recent web clinic, time was identified as our audience’s biggest challenge to begin the implementation of the testing-optimization cycle in their organization.

This might seem like an insurmountable challenge up front, (since most marketers feel like they can barely keep up with their deadlines as it is). But as with any excuse (I have no time to exercise, eat right, learn Swahili, etc), it can be overcome by simply planning ahead and getting a firm understanding of the resources you will need.

When building your marketing campaign timelines, keep in mind that depending on what direction/solution you take with testing… you’ll need to build in extra time for:

• Reviewing your site analytics to identify your biggest opportunity and the best elements to test
• Working with a designer to modify your current design and incorporate the test variables
• Fitting into the development queue (if not using Unbounce). In most cases, marketing is the lowest priority in the development world, so ensure that you have your place in the queue before setting an expected launch date

Related Resources

Technology Blind Spots: How human insight revealed a hidden (and almost missed) 31% gain

A/B Split Testing — How to use A/B Split Testing to Increase Conversion Rates, Challenge Assumptions and Solve Problems

Cost of Delay: How to win approval for your test and test schedule

What to test (and how) to increase your ROI today

How Can You Make Your Web Site Smarter?

Are You Listening Fast Enough? Using A/B and Multivariate Testing to Accelerate Conversion Improvements

In typical Internet fashion, the blogs and Twitter lit up with doom and gloom news that web tracking was dead…run for the hills web analysts. In fact, it was very reminiscent of the reaction people had when Germany announced they were going to investigate the legality of Google Analytics and collecting data on their citizens. But, as with past incidents, people calmed down and life went on.

Personally, I am all for Internet privacy. We as businesses and marketers need to respect users’ wishes if they decide they do not want to be tracked (even if retail counterparts do not honor this). If a significant amount of people are choosing to opt out, then we need to adapt and find other ways to determine what our users want and need. Heaven forbid we talk or engage them more personally (see blog post of different ways to do this online).

But in talking with people that have concerns about being tracked online (especially by Google Analytics), I typically find that they simply misunderstand what the tool does. Most people with concerns feel like it is a Big Brother tool that tells us exactly who they are, tracks them after they leave our site, and relays to us every website they visit.

They lighten up significantly when I tell them that the tool is really used to anonymously look at users of our website and help us understand how to make our process, products, or websites better.

In fact, we cannot even see (at a personally identifiable level) who these people are if we are following Google Analytics’ terms of service. Once they hear this, most skeptics see the value and how it can really make the Internet a better place without negatively impacting privacy.

What I hope Google does

When Google releases this opt-out feature, my desire is they also release an FAQ page or other educational outlet to debunk the privacy myths about Google Analytics. I truly believe it is urban legends, what people do not know (fear of the unknown), or other more invasive measurement tools that cause the privacy scares. Perhaps if people are properly educated they will be less reluctant to opt out.

Whatever it decides to do, Google should err on the side of too much information. When it comes to anxiety, perception is reality. And as with any anxiety, we must overcorrect to overcome it.

But in the end, it is ultimately up to the consumer. And as painful as this is to say as a web analyst, we need to respect our visitors and their desire for privacy, even if it is a poor or wrongly informed concern.

Related Resources

What the “opt-out” option means for Google Analytics

Google adds more flexibility and intelligence to Analytics and Website Optimizer

Friction and Anxiety in your marketing process: defining the difference

1,000,000 monkeys can’t be wrong

Multivariate Testing (MVT) is starting to earn a place in the pantheon of buzzwords like cloud computing, service-oriented architecture, and synergy. But is a test the same thing as an experiment? While I am not a statistician (nor did I stay at the Holiday Inn last night), working at MarketingExperiments with the analytical likes of Bob Kemper (MBA) and Arturo Silva Nava (MBA) has helped me understand the value of a disciplined approach to experimental design.

What I see out there is that a little knowledge is indeed a dangerous thing. Good intentions behind powerful and relatively easy-to-use platforms like Omniture® Test&Target™ and Google® Website Optimizer™ have generated a misleading sense that as long as a multivariate test is large enough (several hundred or more combinations being tested), at least one of the combinations will outperform the control.

This notion has become the value proposition of a growing number of companies offering services around either the big-name or their own (simpler, and often therefore easier to set up) MVT tools. They are ostensibly betting on the technology, and not on a systematic approach to experimental design or any particular UI/UX (user interface/user experience) optimization theory.

Even though, as Bob has pointed out to me, it is reasonable that an MVT setup with a billion combinations may not yield a lift over the control, my contention is that the risk-weighted business cost of a dissatisfied customer is low. Therefore, little stops the burgeoning MVT shops from safely offering a “100% lift guarantee.” Just like the proverbial million monkeys with typewriters, somewhere among thousands of spray-and-pray treatments their MVT tests are expected to produce one that’s better than the rest.

1 monkey with a stick

One major difficulty with testing in general becomes painfully obvious with MVT: the more treatments, the longer the test will run. For most companies, what looks at first like a great test may require a year’s worth of traffic to get statistically valid results.

In response, one emerging MVT service model offers getting to a “lift” faster by using adaptive elimination of likely underperformers, in exchange for the test results providing limited information beyond identifying the winner. Such test results are not as useful as their full-factorial brethren for designing subsequent tests because adaptive elimination of treatments makes it difficult to extrapolate the psychological factors and consumer preferences responsible for the test outcome. The immediate business benefits, however, are more immediate.

So, where exactly is the problem? As marketers, are we in the business of employing the scientific method to design graceful experiments or is our fiduciary duty to get measurable results? I humbly suggest that as marketing professionals, we should neither bet on nor be satisfied with just one test, no matter how successful it is.

The bad news and the good news is that we must design an experimental plan to optimize continually, to learn from preceding test results, and to respond to changes in customer preferences, market conditions, and our ability to segment data and traffic. Expertise in experimental design and understanding how to interpret results simply cannot be replaced by set-it-and-forget-it technology (yet).

Economy of testing

That is not to say that MVT provides incorrect results. The results are mathematically valid, even if they do require a long time to obtain. At the same time, from the business point of view, investment into experimental design expertise is expensive. Understanding volumes of published research consumes valuable time. The 100% guarantee sure sounds good.

And so the “guaranteed lift” offers will appeal to the spendthrift marketers who are yet to delve into the science of optimization. The critical issue in the economy of testing is whether methodical design of experiments is likely to provide greater ROI through an interpretation-driven sequence of test iterations than a successful, but terminal one-off test. Our research supports the former.

2010 may become the year of multivariate, but I hope that it will also quietly set the stage for an upcoming year of ROI-conscious design of experiments.

How do you use multivariate testing? Have you created an experimentation plan or do you rely on a series of one-off tests? Share your triumphs and concerns in the comments section of this post or start a conversation with your peers in the MarketingExperiments Optimization group.

No, not literal squirrels with their cute little acorn-filled cheeks. The squirrels we seek to battle are figurative. That is the name we’ve given to that bad idea your boss has that just drives you nuts (pun intended). The “squirrel” is that red herring that diverts your audience’s attention from the main conversion objective. Of course, since squirrels are more personable and cuddly than herrings (not to mention easier to train), that is the analogy we went with for our videos…

To see how our dynamic marketing duo initially tried to fight the squirrel, please watch Part 1 and Part 2 in our ROI-increasing/squirrel-slaying trilogy.

To help you win the budgetary and executive approval to defeat your organization’s squirrels, watch a replay of our latest free web clinic – The Business Case for Testing: How one marketer convinced her business leaders to start testing and drove a 201% gain in the process.

Related resources

Online Marketing Optimization: Does my 95-year-old Grandmother Understand Split Testing Better than your CMO?

A/B Split Testing: How to use A/B Split Testing to Increase Conversion Rates, Challenge Assumptions and Solve Problems

Multivariable Testing: How testing multiple changes simultaneously can save you time, speed up your optimization schedule, and increase your profits