Experienced online marketers have heard this saying at one point or another: we just don’t have the resources to do what you’re asking. Maybe the resources aren’t there or maybe your project is last in line. Either way, somebody else is determining the destiny of your plans and your message just isn’t being heard.

So, many marketers scramble for decision makers’ attention. They’ll set up big meetings, put together big PowerPoints, big Gantt charts, and stay up all night practicing the perfect pitch – all to get a simple, disheartening response. There is a more effective way to get approval.

The cost of delay

You sell your decisions makers more effectively by presenting them with a simple, relevant metric: the potential cost of delay.

Decision makers need information they can understand, and the language of money (money lost to be more specific) can be one of your most powerful tools. If your company’s website is bleeding and you are tasked to fix it, there is no better way to start the conversation than by showing how much blood is being lost on a monthly, weekly, or in some cases daily basis.

Remember, this is the same blood that helps pay for all those other priorities sitting in front of yours.

The solution is at your fingertips

If you are testing on behalf of your company, you will likely have all the information you need to get started.

1. Start with your web analytics. Every test should have an objective, and knowing the objective from a web analytics perspective is your first step. This could be a sale, a lead, or sign-up on the page or process want to test.

1. Understand its financial impact: Look at each test’s success metric and ask yourself:
   1. How much is that lead worth (over time)?
   2. What is the likelihood that the completed step will become a lead, and then revenue for your company?
   3. What is the average order value of a person completing their order on this page/process?

All of these questions begin to connect your web analytics with your transactional/financial data. Healthy companies keep tabs on what each of their processes is worth, and make business decisions based on these business-level metrics. If you do not have a basic understanding of the financial impact (for better or worse) of what you are testing, you are walking a dangerous path.

1. Connect the dots: For example, you know that step two of your order process is seeing a 70% drop in visits and that the average customer order size is $150. If you were to recover, say, 10% of those visits that are typically lost thanks to the intelligence you gain through testing, how much revenue would those tests drive?

1. Ask the big question: Are we (as a company) willing to risk losing this much money by not getting this test up in time for this season’s peak?

You cannot possibly know all the competing priorities on your decision maker’s list, so you can’t actually do the full analysis for them. Taking this approach, however, gives them the information they need and the attention your test deserves to complete the analysis – comparing the potential loss to the cost of pushing those other priorities back (or acquiring the resources).

Empower yourself and your test schedule by clearly communicating the problem to your decision makers to show how your testing may solve these business-level problems over a set period of time.

Related resources

A/B Split Testing

Welcome Message Sequence Tested

Site Compatibility Tested, Section 1 (Research)

Site Compatibility Tested, Section 2 (Analysis)

As you may recall from my last post, I spent nearly six days in Austin, TX attending SXSW Interactive, a conference which Advertising Age once famously referred to as the bellwether for what lies ahead for digital culture.

The event was a fantastic gathering of ideas, technology and core conversations – the kind of conversations that tend to happen when the Internet takes a moment to meet in person and exchange fancy business cards.

Test everything?

And speaking of fancy business cards, for this event MarketingExperiments ran an A/B split test version of our business cards. For those keeping count, Side B won, although the results are far from being statistically significant.

(click below to see the two treatments)

But what the cards were able to prove is that the future of online testing is brighter than ever! While panels focused on a wide-range of topics from content development and management to social media and design standards, I spoke to a number of Web 2.0 designers, business owners and service providers all fascinated with the prospect of testing their pages, registration paths and email.

Test more?

Even more exciting is that many of these people were already testing on their own sites. One such tester, FanBridge co-founder Noah Dinkin, was especially interested in discussing testing strategies as he has attended several of our webinars here at MarketingExperiments.

FanBridge is a site that allows someone (such as a musician) to manage a list of their fans. In this way, FanBridge must focus not only on attracting new signups to its own service, but to help their current subscribers effectively communicate to their own fans. Noah, who said he tests using Google Website Optimizer, confided in me what many of us already know. He is testing some, but he needs to be testing more.

Test the right thing

This was a common theme in many of my discussions as I met people who were interested in testing and knew they should be testing, but were often unsure where to start. And I think, like any problem, the answer starts with research.

Resources such as this blog or our webinars are a great resource for anyone looking to see what testing is all about and see some real-world case studies.

I think we can all agree that the benefits of testing are such that we cannot continue to implement untested best practices or design standards without measuring the impact of our changes to the bottom line. The question becomes how many conversions can you afford to give away?

Because whether you run a Web 2.0 service application that receives five visits per day or manage an ecommerce site that receives 50,000 visits, your landing pages, cart processes and emails will determine whether or not you end up with a customer or a bounce.

Thanks to everyone I met at SXSW, and good luck testing!

Several companies are offering multi-million dollar rewards to anyone who can pick a perfect bracket in the NCAA Tournament. Sounds like a good deal, doesn’t it? You can enter for free, and the chances must be better than the lottery, right?

Ask yourself…what do you think the odds are? Maybe one in a million. Perhaps one in 50 million.

Or maybe you put a little more effort into it and do a few basic calculations. You figure that in the first round of 32 games, the probability of having a perfect prediction is one in four billion.

So if you’re prone to extrapolate you might think that, for the total 63 games in the championship, the overall chance would be something like one in eight billion, right? Logically speaking there are twice as many games, so half the probability.

I was wondering myself, so I actually ran the numbers. The chance of predicting a perfect bracket for March Madness is one in 9.22 quintillion. That’s one in nine billion billions. In other words, you have a better chance of getting struck by lightning, being hit by a meteor, and winning the Mega Millions lottery.

But wait – I know my college basketball

“But wait,” you say. “I’ve been following college basketball and I know which teams have a better chance of winning. No way Arkansas-Pine Bluff has any chance of beating Duke.”

Fair enough. In the above example, I used a random-result probability model (a 50/50 chance for every game). So I also created an informed-result probability model.

In this model, I assumed that the higher-ranked team had a two-thirds chance of winning in the first two rounds (after that, it’s still anybody’s game). The chance now is one in 9.29 trillion. Much improved, but still amazingly long odds.

But wait – I know my customers

There’s a greater lesson to be learned here for testing. Chance is an intuitive concept but estimating chance is not.

When running an online test, chance has to be accounted for. We can’t rely on intuition or a feeling that we know what our customers want. We can’t just assume that because we got the results we hoped for that the results are significant or a test has run long enough.

We need to implement the appropriate statistical validation to assure that what we are seeing is not just random chance, but likely representative of our market as a whole.

Bad data equals bad decisions

Here is an extreme example to show you what I’m talking about. Let’s assume we turn a test and our treatment page gets four visitors. If three visitors buy, and one visitor bounces, we cannot assume that 75% of our traffic will buy. Because once we get to 10 visitors, we may find that now six have bounced.

While the above example is obvious, not every testing scenario is. Perhaps you’ve run the test for a week and feel like that is long enough. Or the sample size seems quite large. Or, and perhaps the biggest danger which I referenced above, you feel that you know your customers well enough that when a result comes along that you were hoping to see, you’re prepared to stop testing.

Making a business decision based on any of these scenarios is a dangerous thing. And therein lies the power of true statistical validity. We are given an assurance (for example, 95%) that the results from the test we just ran can be relied upon and not attributed just to chance. This way, when we duplicate across the entire population (assuming we tested on a representative sample), we’ll sensibly be expecting similar results.

By doing so we never completely eliminate the chance of an unpredictable event (one in 9.29 trillion is still possible), but we gain a strong enough understanding of it to confidently make bottom-line decisions based on quality data.