The One Metric Dashboard Helping SumoMe Reach 1 Billion Users

A few weeks ago I was hanging with Sean Work of KISSmetrics, drinking hippie gunpowder tea in Venice, CA. I mentioned our company dashboard for our free tool that is used to grow email lists at, and Sean was blown away.

Sometimes simple is better.

Here’s what our dashboard looks like:

sumome dashboard

So what’s going on?

1. One Company Goal

We have only one company goal for the year – one billion unique visitors at

Rationale: As we aligned the company around one priority, it became much easier for all of us to be on the same road traveling in the same direction. We noticed that, during the times we had a common goal everyone was aware of, we were much more focused and likely to achieve it.

This is what Mark did when I worked for him at Facebook. A simple test (I call “the focus test”) used to find out if a company is aligned is to call any employee and ask if they know what the number one goal of the company is. If not, then the goal is not clear. People need clarity for something they can all rally around.

2. A Motivating Sound

We play the sound of an airplane flying overhead every time we reach a goal of 10,000 unique visitors. Click here to listen.

Rationale: While sitting at our computers all day, we lose track of progress, so the audio queue of a flight sound helps us recognize, non-visually, that we are progressing. It gives everyone a chance to cheer. One of the guys on our team was a bit bummed when SumoMe was getting only 10,000 unique visitors a day. I felt the same. But, small wins and the feeling of momentum are powerful motivators that make us want to get more of those wins.

This is something Jeff Bezos did at Amazon where a bell rang for every online order. Eventually, there was too much ringing so they had to stop.

3. Real People

As people subscribe on List Builder at different websites, it’s good to be reminded that we’re dealing with real people.

face of sumome subscriber

Rationale: At Facebook we had a dashboard and we were growing 50,000 new students a day. What this neglected to take into account was that these were real people using our product. Too often, online, we see only the email address, the analytics, or the money that comes in from our customers. The more we can personify those email addresses, the better we can empathize and create better solutions for our customers.

4. Daily Targets

The bottom / second number is our daily target. It gets reset at midnight, and it helps us clearly and visually see how we are doing in real time.

It’s displayed on our wall while we work:

AppSumo Office Wall

Rationale: Our goal is one billion unique visitors this year. That seems like it’ll never happen, which is discouraging and demotivating when times are tough. Since that is the case, we mapped out a daily level of numbers we need to reach. Knowing we are on track on the daily helps make the macro seem more digestible.

Why We Do This

We used to track everything, literally. It was RAINING metrics:


This made us optimize toward too many different goals. It was like trying to cook four dishes at once. We’d start optimizing for the number of orders per day, but then revenue per order would go down, so ultimately, we were spinning our wheels. Therefore, choose just one main metric to target and display on your dashboard.

A good way to think about your dashboard with your team is how actionable the information is. If you saw you were down today, could you instantly make an impact with that information? For us, knowing traffic is low one day motivates us to evaluate what’s off on that day.

Then we evolved to having one goal of 3,333 memberships at So we created a dashboard to reflect that:


Yea, real simple. This is what it shows:

  1. Total goal
  2. Percentage of the goal we have reached
  3. Our daily sales

At a high level, your dashboard should reflect the fact that you have united everyone with a common goal and clear alignment. So, set your company goal, clarify the strategy to get there, and then lay out tactics to accomplish the goal.

Goal -> Strategy -> Tactics

What You Can Do for Your OWN Company Today

1. Don’t worry about developing your own dashboard. We always start with a Google Spreadsheet to set our goal and track it on a daily / monthly basis. There are companies like Geckoboard that do this as well. Here’s our current spreadsheet:

current spreadsheet

2. Pick one goal to target for one year. Anything more is too much, and less is too soon. At AppSumo, we like to choose goals that seem hard to accomplish but are significant. A billion people sounds like a lot and is a motivating number for all of us to strive for. Last year’s goal of 3,333 for Monthly 1K reflected that it would be a million dollar product at that number. We wanted the product to go “platinum.”

3. Remember, your goal has to be something you genuinely want to accomplish. Align everyone on your team around that goal so they are motivated to perform their particular role in order to help achieve it.

4. Break down your goal daily, make sure it’s visible regularly to everyone on your team, and evaluate your progress toward that goal on a weekly basis.

About the Author: Noah Kagan is the Chief Sumo at, which has free tools to help you grow your website. Grab his free strategy for exactly how he grew his email list to 50,000 in 12 months.

Avangate scholarship winners for Business of Software Conference USA 2014

Avangate announced the recipients of its 2014 Business of Software Conference Scholarship today.

The three winners, CloudPress, Effective Computing and Pivotdesk, will receive complimentary passes to the Business of Software Conference USA taking place in Boston September 15-17, as well as the opportunity to attend an Avangate networking dinner with software industry leaders.

“As the new services economy continues to grow, it’s more important than ever for software and online services companies to continue innovating their products and services and their delivery model,” said Michael Ni, CMO/ SVP of Marketing and Products at Avangate. “For several years, we’ve supported promising new software and online services companies with this scholarship, connecting them with the people and resources to help them succeed. We’re excited about the winning companies this year, and we look forward to tracking their success as they grow.”

Reflecting Avangate’s commitment to the new services economy, this year’s scholarship winners focus on providing innovative services.

  • CloudPress, a SaaS product of Extend Studio, offers a set of tools and services designed to eliminate coding and server tasks for WordPress sites.
  • Effective Computing developed VoiceShortCuts, a voice command system that reduces the stress and limitations of computer use, enhancing productivity.
  • Pivotdesk is an online service that is changing the way people find office space: an office sharing marketplace that helps companies offset costs by sharing their excess office space with smaller companies and entrepreneurs looking for a place to grow their business.

It’s always a delight to have Business of Software scholarship winners at our conference. These companies are selected by Avangate in a competitive process and winners have come from all parts of the world over the years. We are excited to have them at Business of Software Conference in Boston as we are always interested in learning and bring fresh ideas to our sessions. Congratulations to the winners.

In addition to connecting these companies with new insights at the conference itself, Avangate is coordinating a networking dinner in Boston on the evening of September 16th where participants will have the opportunity to share ideas with other emerging software, SaaS, and online services industry peers and make valuable connections. To join the Avangate Boston networking dinner, please RSVP on the event’s registration page.

See you in Boston!

6 Advanced Steps for Doubling Your Activation Rate

Imagine you’re living the marketer’s dream.

You’re acquiring hundreds, even thousands, of customers every day because your product and marketing are just that awesome. Your team, spouse (or Tinder match), and family are excited as your company gets more valuable by the day.

However, all this means nothing if you ignore one key metric – activation.

Activation is one of the most important metrics for any SaaS company. It’s a measure of how many users take a specific action to get value out of a product. In Twitter’s case, that action might be following 6 other accounts or sending 2 tweets. In KISSmetrics’s case, an active user probably is one who installs KISSmetrics and runs one report on their data.

This graph shows what happens when you are crushing user acquisition but failing to activate users:


From Andrew Chen’s post Retention is King

Viddy, the formerly popular video sharing app, grew rapidly by getting users to sign up using their Facebook accounts and then sharing Viddy with all their friends.

However, Viddy never nailed activation. Many of the users who signed up never came back. Thus, when Facebook put a stop to their friend-blasting user-acquisition approach, they went into a tailspin from which they haven’t recovered.

This activation problem only gets worse as you venture beyond consumer products. In fact, according to Intercom, 40-60% of users who sign up for a free trial of software will use it once and never come back again.

In today’s post, we’ll cover 6 advanced steps for activating users and customers. But first…

Why is Activation Important?

Most companies are (rightly) concerned with user acquisition but fail to pay adequate attention to activation. What few realize is that activation plays a major role in user acquisition itself.

Imagine that you (and your competitors) can spend $10 on Facebook to get one user signup, and that you make $20 for every customer you have. Let’s also say that only 50% of your prospects who sign up for a trial will activate.

Overall, you’re breaking even on your marketing expense: $20 x 50% activation = the $10 you spent on Facebook. If you can boost activation from 50% to 60%, your marketing channel will suddenly become profitable: $20 x 60% = $12.

Nice work! The increase (thanks to improved activation) means you can – profitably – spend more on advertising, acquire more users, and get more traction! So…

Step 1: Start Tracking!

Set up proper analytics. I highly suggest using a tool like KISSmetrics to track the actions users are taking in your product. Google Analytics just doesn’t cut it when it comes to activation.

Step 2: Map out Your Ideal Customer Flow

At my last company, the first thing we did to improve our lagging activation was measure the actions our most successful customers took.

Airbrake is an error-tracking tool developers use to capture errors that might occur in their software. So, in order for a customer to get maximum value out of the product, they had to track their errors!

This wasn’t just a hypothesis. After looking at the data, we saw that users who captured an error were about 300% more likely to remain customers than those who didn’t. Once we figured this out, we retooled our activation flow to get as many people as possible to capture their errors.

In the ideal customer flow, the user:

  1. Signs up for Airbrake
  2. Indicates which programming language they use (Ruby, Python, PHP, etc.)
  3. Installs and deploys a few lines of code in their app
  4. Captures their first error
  5. Marks their error as resolved in their dashboard

To get an idea of what this looks like for another product, let’s take a look at the example of Bingo Card Creator, a tool that helps teachers create and download customized bingo cards for their lesson plans. Bingo Card Creator’s ideal customer flow looks like this:

Bingo Card Creator

From Bingo Card Creator

In this case, the ideal customer signs up, hits their dashboard, creates a list, customizes their list of bingo cards, schedules a print run, and then downloads the cards.

Take a minute and map out the ideal flow for your customers. What actions do your most active customers take? After looking at your analytics, what is the step with the highest customer drop-off?

Once you figure out where people are struggling to activate, it’s time to find out why.

Step 3: Learn!

Activation can be broken into multiple steps, which lets you figure out the exact areas that need the most work.

In the case of Bingo Card Creator above, you can see that the greatest drop-off occurs when users try to go from creating a list to customizing one. Only 82% of users complete this step, compared with 86%+ across the rest of the funnel.

Why is this?

At this point, you have 2 options. You can either blindly test different onboarding flows and other triggers to improve this activation step, or you can talk to people (gasp!).

Personally, I’d go with the second option and ask users why they’re struggling to create their lists. You can get this data in a few different ways:

  1. Call people who haven’t finished a key step. (Bribe them with gift cards if you have to. You’d be surprised how well a gift card or a t-shirt works for people making $100k+ a year.)
  2. Set up a survey tool like Qualaroo.
  3. Watch users go through your onboarding process. Do this live (if possible), or use a tool like UserTesting or CrazyEgg.

Let’s say you talk to 15 people who have created lists of bingo cards but haven’t customized their lists. You find that 9 of them couldn’t think of how to customize a list and planned to finish it “later.” That’s a huge insight. It means that by showing users how to customize their lists, you should improve your activation.

Step 4: Make Things Simple

To improve your activation, make things simple. Often, this means reducing the amount of friction a user encounters in your product. Common causes of friction are:

1. Unnecessary Form Fields

Compare these 2 forms:




2. Visual Friction

Compare Bonobos vs. Lings Cars. Which one is cleaner?



To see/hear LingsCars in full effect, you’ll need to visit the site

Here are a few more ways to make things simple:

Cut down the number of steps users have to take. For example, if you allow new users to import contacts from Facebook so they don’t have to manually enter their friends’ email addresses, it would make the process a lot easier, which would make it more likely users would complete that action.

Clarify language. Is it possible for you to say what you want to say in fewer words when writing copy on your website? Language should be clear, action-oriented, and concise.

Improve site performance. Load time, scrolling, and page performance all matter, both for SEO and for users.

For more ideas, check out this massive Slideshare on activation.

Step 5: Re-engage Users

Re-engaging users after they’ve fallen out of the funnel is another way to get them to activate.

Email is a great way to do this. Going back to our Bingo Card Creator example, you could send an email to all users who create a list but don’t customize it. The email would show users how to customize lists and cover the reasons why someone would want to do this.

Here are some tools you can use to engage users:

Amazing support. Good support is huge for improving activation and building customer goodwill. One useful trick is to tag users who’ve signed up in the last 30 days and make sure they receive priority support. Not only does it lead to a great customer experience, but it allows you to learn common problems customers run into during their first 30 days. Then, you can write documentation that addresses such common problems.

Automated welcome emails. Colin Nederkoorn of suggests sending a personalized email from the founder within an hour of a user signing up. In the email, just ask how you can help or if they’re confused about anything. This gives new users the ability to ask questions, and it gives your team an understanding of the problems new users face. Some companies take it a step further and try to call every new user who signs up. Often, these conversations are great times to discuss why a user signed up and what they’re hoping to get out of your product. You also can get a sense of how likely they are to upgrade down the road.

Onboarding flow. Creating a first-time-user onboarding tutorial can significantly improve your activation rate. Twitter is a great example here. Your experience when signing up for a new account is completely different from the experience you have the rest of the time you are an active Twitter user.

Lifecycle emails. We touched on this above, but lifecycle emails that introduce new features to a user – especially if you can base them on actions a customer has already taken in your app – are huge for improving your activation rates. These are the kinds of emails you’ll get if you sign up for Twitter but don’t follow anyone. Twitter knows the number of people you follow is a key activation metric, and they will send emails suggesting individuals you may want to follow.

After reducing your product’s friction and setting up systems to engage users who fall away from your product, you’re ready for the last step!

Step 6: Rinse and Repeat

Well, that’s annoying. You see, improving your activation rate is an ongoing battle, but one with huge dividends. After rolling out a new onboarding flow, implementing an email lifecycle sequence, or testing simpler language, be sure to measure and see what’s working (and what’s not). If activation has gone up, roll out the improvements, and begin another test!

There you have it – the 6 steps for improving your activation rate. Hopefully, after reading this post, you’ll start improving your activation, thus helping thousands more customers fall in love with your product.

I’m sure I’ve missed some great ideas for ways companies have improved activation. What are your favorites? Let us know in the comments below.

About the Author: Justin Mares is the co-author of Traction, a startup guide for acquiring customers. Get the first 3 chapters free here.

Bootstrapped, Episode 48, “Gabriel Weinberg of DuckDuckGo”

We chat with special guest Gabriel Weinberg of DuckDuckGo. We cover the full gamut from bootstrapping, VC, competing against Google, Perl, privacy, advertising, remote work, finding customers, biz dev and Gabriel’s new book Traction.


Discuss this episode on the Bootstrapped forum


Behind the Scenes of Behavioral Advertising

Today we’re going to look at some of the little known details and services surrounding the world of behavior advertising. And just in case the term “behavioral advertising” is confusing to you, let’s define it real quick:

Behavioral advertising is a technique used by online advertisers to present targeted ads to consumers by collecting information about their browsing behavior.

Several pieces of data may be used, such as:

  • The pages browsed on a website
  • The time spent on the site
  • The clicks made
  • The recency of the visit
  • The overall interaction with the site

All this data creates a user persona or profile that can be used to segment the audience into certain types. People with similar online behavior can be clubbed together into one segment. Then, these segments are shown ads that cater to their interests.

According to a survey by the Network Advertising Initiative, conducted with 12 advertising networks, targeted advertisements based on user behavior converted 6.8% compared with non targeted ads at 2.8%.

How Behavioral Advertising Works

Lots of platforms work by inserting a cookie into a visitor’s hard drive. Advertising networks have a larger demographic of users at their disposal. Being able to serve different sites, they can easily divide the populace into segments.

The first step in serving behavioral ads is tracking who the users are and what they do online.

Tracking refers to the process of collecting and processing data, while targeting refers to using that data to offer personalized solutions or advertising.

In order to understand how tracking works, we need to define a few terms:

HTTP request: Generally a tracking pixel or JavaScript sends client information to the ad serving company.

The user is identified with the help of a cookie in which unique IDs are stored.

Furthermore, combining HTTP request headers with UserAgent strings can lead to even more precise User Identification.

Flash Cookies: Flash allows advertisers to place Flash cookies known as local shared objects with the ability to store client information. The Information can range for preferred volume settings or even unique identifiers.

One of the notable advantages with Flash cookies is that they are outside the browser. They persist even when the user closes the browser, and they carry through when he/she opens another.

While standard HTML cookies store 4kb of data, Flash cookies can store up to 100kb.

There are mechanisms which keep track of the user even when he/she deletes the cookie. For example, say cookie named 144587 is attached to my IP. Imagine that I delete the cookie. The next time I log in, the server attaches a cookie with a different name, say 8654977, but the software knows that both cookies relate to the same person. This is called cookie respawning.

Etags: Etags are a feature of the browser cache. The cache saves a previously visited website in the memory, and if a visitor is inclined to visit the site again, the saved site is shown. Advertisers use the cache to store unique identifiers pertaining to the user.

Etags are difficult to remove because deleting the cache would make browsing slower.

HTML5: HTML5 makes use of a cookie called HTML5 local storage. Even if the user closes the browser, the cookie persists until deleted willfully. Compared with Flash and simple HTML cookies, HTML5 cookies can store data up to 5 Mb.

It is preferred because users don’t need any plugins like Flash to run it.

The EverCookie: The EverCookie uses the combined features of HTML5 cookies, Etags, Flash cookie, etc. If a user deletes one identifier, say Flash cookie, then the other identifier, say Etags, is enough to activate the EverCookie.

AOL has explained it beautifully with images of Penguin. When you visit a sports site or gourmet site, for example, an ad company sends a cookie to your computer. Later, when you visit some other site, the ad company reads the cookie and displays an ad related to gourmet or sports.

AOL penguin explanation

Online Behaviors That Advertisers Use

Let’s go into more detail on two pieces of data that advertisers use to create targeted ads:

IP address & geolocation: Targeting your audience by where they live is one of the most basic segmentations you can make. You can show discounts and sales targeted to a specific region. You also can blend urgency into the messages, such as: “This offer is only for Utah readers and expires in 10 hours.”

There are even more metrics you can add, like the person’s country, state, region, and city.

Verve, after comparing data from its 2,500 mobile targeted campaigns, found that mobile ads that are geo-aware performed 50% better than the rest of the campaigns.

In another study, Central Market saw a 4.1% click-through rate location based targeting.

Recency of visits & whether repeat visitors or new visitors: A repeat visitor can be shown related content based on what they searched for in the past.

You might have seen yourself that, when you visit some sites to search for certain things, you later see ads of those sites elsewhere.

Amazon does this. They even email users based on the products they browsed earlier.

Personyze is a platform that offers real time personalization based on user data.

How does that happen? It’s called retargeting, and we are going to learn a bit more about it.

How Advertisers Use Retargeting

When a user visits a site, he/she is expressing interest in the site. Later, the user can be shown ads related to the site.

Let me clarify with an example. I was searching for a company called IfeelGoods on Google. Minutes later, I logged into Facebook and found this ad from IfeelGoods in my news feed.


It doesn’t end with that. Days later, when I am reading a random piece on Salon, right there before me is an AdSense ad of IfeelGoods.

salon advertisement

Here are a few KISSmetrics articles that go over retargeting in detail:

Tools for E-commerce and SaaS Businesses

Here are a few tools that e-commerce and SaaS businesses can use for behavioral targeting:

RichRelevance: RichRelevance has served more than 1 billion personalized product recommendations. It has delivered more than $10 billion in attributable sales to its retail partners like Walmart, Target, Sears, Best Buy, etc. It’s the perfect solution for e-commerce site owners.


You can sign up for RichRelevance here.

Related Products Management by CommerceStack: This is a Magento extension that integrates user data into the shopping experience by generating relevant up-sells, cross-sells, and down-sells. It’s currently free.

The below examples show you how the Products Manager can be used to add related products and manage them with a few clicks.

magento admin panel


Visit the CommerceStack installation page here.

Personyze: Personyze is a behavioral targeting tool that collects and analyzes information on the user and makes real time recommendations. It can display targeted banner ads from a range of data like the keywords used for search, click behavior, etc.

An example of a site offering personalized recommendations with Personyze is shown below.

Personyze personalizes the webpage for each user, as follows:

  1. Greets the user with his/her name, “ Welcome Brian,” as you can see
  2. Products shown based on previous searches
  3. Down-sells and cross-sells
  4. Up-sells
  5. User guides
  6. Reduced prices to attract customers

monkey gadget

You can sign up for Personyze here.

Intelligent Cross-Sell by CNET: This tool is a boon for those in the consumer electronics industry. CNET has partnered with RichRelevance to help merchants optimize their cross-sells. The tool taps into CNET’s database of 5 million digital goods and comes up with relevant recommendations.

The example below shows a product comparison table powered with the rich data base of electronic products on CNET.

e-storesite product comparison

Get started with Intelligent Cross-Sell here.

CoreMetrics Intelligent Offer by IBM: The tool generates product recommendations based on past purchases and business rules. It delivers smoothly across email, mobile, and web platforms.



Get Started with CoreMetrics here.

Magiq Dynamic Personalization Software: Magiq builds and maintains long term customer behavior records which can be used to offer personalized content. Magiq records key user behavior like the place they are from, whether they have registered on the site, search keywords used to reach the site, etc. It also categorizes active customers as “ready to buy,” which means if the sales team pitches them with an offer, they probably will buy the product. was able to achieve a 61% conversion on cart abandonment emails with Magiq.

Whenever a user abandons his/her cart, a trigger is fired. The trigger can either send an email offering a 15% discount or show the banner ads of the products on subsequent visits to the site.

Here is a video that shows the entire process of remarketing via Magiq:

Sign up for Magiq here.

Facebook interest based targeting: In a post here, Tommy Walker talks about how a True Value ad got into his news feed. In his own words: “Well … I’m 27, I’m married, I’m a male, I’m a father, and I’ve Liked the pages Saving Money, Super Coupon Lady, and DIYnetwork, among other things that would signal I’m a responsible adult living on a budget and like to take care of things around the house.” He also suggests the perfect tag cloud to target people who’d be excited about a $5 True Value coupon.

facebook interest based targeting

Get started with Facebook Advertising here.

On Facebook, people enter a lot of data that makes it easy to target them with highly personalized offers. They like pages, so that indicates their interest in certain products or businesses, they mention where they live, and so on.

Have You Tried Behavioral Advertising?

As you can see, behavioral advertising helps you provide consumers more personalized offers. It helps you get into their shoes and understand what they need. You can use the tools to better serve your customers. How have your experiences been with behavioral advertising? Leave your thoughts in the comments below!

About the Author: This article was written by George Mathew.

10 Things Every Marketer Should Know About A/B Testing

As customer attention spans are shrinking and online competition is expanding, Conversion Rate Optimization (CRO) is a must-have skill for marketers.

CRO is one of the most important aspects of your digital marketing strategy because conversion rate is the only measurable metric that actually correlates with ROI.

Even if a customer “conversion” on your website is something other than a purchase (such as a newsletter signup), the rules of CRO still apply.

Unfortunately, when it comes to implementing a CRO plan, you can get completely lost in a sea of online resources that tell you to do things like change the colors of your buttons, add social proof, shorten your web copy, include gamification… Stop the madness!

Before jumping into tactical fixes, there is only one thing you need to do to optimize conversion rates on your website, and that is what today’s blog post is all about – A/B testing.

While a leap of faith worked for Bruce Springsteen in 1992, it won’t bring you success in the future. So, rather than take a leap of faith on a set of tactics, use web analytics to get a ton of insight based on real-time user feedback. The data can be used to optimize any area of your website based on the real-life behaviors of real-life customers. What could be better?

Of course, you may already have a hunch about what your users prefer and how they consume content, and that brings us to…

A/B Testing Rule #1: Forget everything you think you know about your customers.

forget about what you think about your customers

It is tempting to make assumptions about your audience based on things like age, gender, location, or income. Resist the temptation when possible! There was a time when customer profiling was the best way (the only way) to target customers; and, yes, it still has its place in marketing.

However, in the digital era we have so many more options! No longer do we have to rely on segmentation to deliver hyper-personalized experiences. We now have the ability to leverage every digital touchpoint as an opportunity to learn about our customers’ preferences on a one-to-one basis.

A/B Testing Rule #2: Always establish a baseline.

Increasing conversion rates is your immediate goal, and if you’re like me, you’re in a hurry. But, before jumping into a high stakes A/B test (or even a low stakes A/B test), it is important to budget time up front to establish a current baseline to measure against. If you don’t know what your current conversion rate is, how will you know if your future tests are successful? (More on that in Rule #5.)

A/B Testing Rule #3: Just because it worked for someone else, does not mean it will work for you.

If CRO were a repeatable process that worked the same way for every website every time, there would be no need for testing at all. Marketers would know the way all e-commerce websites perform, and everyone would follow the same rules.

Unfortunately, this is not the case (and a world full of sameness would be rather boring anyway), which is why you must perform A/B testing on your own unique content with your own unique audience. Sure, you can borrow ideas from other CRO-ers, but don’t expect the same results.

For example, let’s say Company ABC sells shoelaces and Company XYZ sells enterprise software applications. Clearly, the buying cycle looks completely different for these two companies, even if they have customers in common. Company ABC may find that changing its primary call-to-action (CTA) button to green instead of red increases sales by 75%. But, it is not likely that Company XYZ would experience similar results.

A/B Testing Rule #4: Test one thing at a time.

This one is pretty self-explanatory but worth mentioning because it’s important. When performing A/B testing on your website, test one variable at a time so that results are readable at the end. If you change your headline at the same time you change your navigation, how will you know which one of the variables contributed to the most conversions?

Pro tip: If you run a headline test, be sure your test headline works with the rest of your digital touchpoints throughout the sales funnel. Consistency builds credibility.

A/B Testing Rule #5: Do not call a “winner” until statistical confidence is reached.

dont call a winner

In A/B testing, statistical confidence refers to the likelihood that the same results can be expected if the same test is run again in the future. In other words, it tells you how confident you can be of the results of your test.

For example, let’s say you perform an A/B test on your shopping cart page where “A” is the use of radio buttons and “B” is the use of dropdown menus. Let’s also say that “B” produces a 75% lift in conversion rate. Obviously, B is the winner, right?

Not necessarily. There are three more facts to consider:

  1. Sample size: Using the example above, if your sample size is 4 people, that means only 3 people prefer dropdown menus. Sure, it’s a good start, but the likelihood of the results remaining true in a sample size of 1,000 is extremely low; therefore, this test result has a low confidence level.
  2. Percentage: The accuracy of your A/B test results also will depend on your margin for error. If, in a sample size of 500, 99% of customers convert when shown dropdown menus, you can be fairly certain that your margin for error is low. If, on the other hand, only 51% of customers convert when shown dropdown menus vs. 49% who are shown radio buttons, random chance gives you a larger margin for error, and you should continue running the test until a higher confidence level is reached.
  3. Population size: If the size of your entire audience is 250,000 and your sample size is 25, again, this will yield a test result with a low confidence level. To calculate your recommended sample size, check out Raosoft’s Sample Size Calculator.

A/B Testing Rule #6: Walk before you run.

This proverb is true in many aspects of business, and A/B testing is no exception. As customer perceptions and expectations evolve, CRO has always been and will always be a moving target. You will make mistakes. You will learn from your mistakes. With practice, you will become an A/B testing master.

A/B Testing Rule #7: Get a second opinion. Or a third. Or a fourth.

User testing has never been more important, nor has it ever been easier! Even if you do not have the luxury of a User Experience (UX) Department on hand, there are a number of free and low-cost services that offer usability testing on the fly, such as:

Peek User Testing: Peek is a super easy and quick way to gather qualitative feedback on your website.

  • The pros: Feedback is generally unbiased, detailed, and free!
  • The cons: It doesn’t always make sense to test an interface outside of its intended audience. Also, it is difficult to gather a large quantity of feedback using this method due to its time-consuming nature.

Amazon Turk: Amazon Turk allows you to gather feedback from thousands of real-live people in a short period of time through the use of quantitative research methods such as surveys.

  • The pros: Generally inexpensive, scalable, and quantitative, and you can pre-select qualifying criteria for your testers.
  • The cons: This is generally performed via a survey engine, which can introduce artificial filters.

The bottom line: any feedback is better than no feedback!

A/B Testing Rule #8: User behavior data and customer survey data may conflict.

Surveys certainly have their place in marketing, but realize they may not always provide honest feedback the way behavioral feedback captured via your web analytics can. This is because surveys introduce human biases in a way that raw behavioral data does not.

For example, imagine that you are in a hurry to print out important documents on your way to a meeting and, 3 pages into your print job, you find that the ink cartridge needs to be changed. Now, what if I ask you how you would handle this particular situation?

Before reading any further, please pause and think about your honest answer.

You probably said you would change the ink cartridge and continue printing your documents. If this were a survey, I would accept that as your answer.

In a user-testing environment, though, I would note that you kicked the printer 4 times, cleared a paper jam, and hit the cancel button 7 times; and then you changed the ink cartridge. While sorting your documents, you spilled coffee all over your shirt, got frustrated, and had to re-schedule your meeting.

In the survey setting, you didn’t outright lie about what you would do in the situation. You did change the ink cartridge, after all. But in the survey setting, I would have missed out on all the extra behavioral data that happened before and after.

A/B Testing Rule #9: Clearly define your success metric.

Never lose sight of your ultimate success metric. CRO is about conversions. It is not about open rates, click-through rates, tweets, shares, or pins. Unless, of course, tweeting and pinning is the “conversion” on your website. In that case, go crazy with it.

The bottom line: have a goal in mind and optimize your content around that goal. Everything else is a key performance indicator (KPI).

A/B Testing Rule #10: Don’t test whispers.

dont test whispers

This saying dates back to the days of direct mail, and it still holds true for online marketing. Avoid testing miniscule elements that have little chance of driving significant change. Use your common sense, trust your intuition, and focus on high impact tests. For a list of 485 real-life test ideas, check out Which Test Won.

Bonus Hacks

CRO is not just about getting more people to click your buttons. It’s about delivering the right content to the right audience and encouraging them to click the right buttons at the right time. If you’ve A/B tested your entire website, optimized based on the data, and your conversion rates are still lower than you’d like them to be, perhaps you are measuring the wrong set of metrics.

For example, let’s say you own a gourmet cupcakery and your website has a 2% conversion rate. In this example, a customer placing an order for cupcakes is the “conversion.” Here are a few questions to ask yourself:

  1. Is the 2% conversion rate based on all web traffic, or is the 2% conversion rate based on only those who click through to the “How to Order” page?
  2. What are the traffic sources of those who click through to the “How to Order” page?
  3. What are the traffic sources of those with the highest bounce rates?
  4. What are the behavior patterns of those who ultimately convert? Did they watch a video? Browse your photo gallery? Read customer testimonials?
  5. Last and most important question: how can I use this data to better qualify prospects?

By slicing and dicing your web analytics in this fashion, a couple of things may happen:

  1. You may uncover areas of your website and/or your search strategy that need improvement.
  2. You may find that your conversion rate is better than you had originally estimated.

Either way, this exercise will help you prioritize your A/B testing calendar.

A Final Word

Outside of basic functionality like site speed and mobile optimization, there is no single truth or secret sauce to CRO. The only way to know for sure what works with your audience is to run a set of A/B tests and then be willing to implement changes based on the data.

About the Author: Nicki Powers is a Digital Marketing Strategist located in Saint Louis, Missouri, who loves to engage customers and drive sales through the use of emerging technologies. You can follow her on Twitter here: @nicki_powers.

Episode 65: Choosing your First Customers

pick-customerWhat if you could choose exactly who signed up for your offering? This is something we really screwed up with CRELister when we were launching it last fall. We thought "Everyone in the Austin market should love us". And when we launched it, we got nothing. Crickets. 600 outbound emails, 10 phone calls, 1 signup.

The problem was we didn't understand the profile of who should be signing up.

In this episode, Brecht explains how he is only letting the right people sign up for the new version of Bank Prospector, and how that is allowing him to make a lot more revenue per customer, with a lot less hassle.

Building a bad ass software company: Dharmesh Shah at BoS 2011

Dharmesh neatly sums up BoS’s mission in one short talk title.

As a contributor to many editions of Business of Software – in the audience as well as on stage – Dharmesh is one of the most helpful and generous members of the BoS community.

Here he is talking about the most important things you need to consider as your business goes for growth.

Find more great talks from our archive in our talks library. Or to hear them live, register for Business of Software now and meet some of the brightest and nicest founders around.

4 Simple eCommerce Product Page Tests That Take Less Than an Hour

To buy or not to buy is the question. And a product page is where this crucial decision is made.

At (Visual Website Optimizer), we see an array of product pages being tested every day. While some people love to start testing with their “add to cart” button, others change copy, layout, and what not. Below, I’ve listed four super interesting case studies that go beyond the usual product page tests. You can use the insights they provide to boost the conversion rate of your e-commerce store.

Case Study #1: Social Proof Dilemma

Social proof helps to improve conversions. This is why many of you have social sharing buttons on your product pages. Right?

“If so many people are doing it, it must be right,” you think.

Honestly, it’s disheartening to see the mediocrity we sometimes confine ourselves to!

Thankfully, our customer decided to break free from this herd mentality. They called out the rampant trend of displaying social sharing buttons on product pages.

When they tested their original product page with social sharing buttons against the new version with no sharing buttons, their “add to cart” click-throughs improved by 11.9%

remove social sharing buttons

While 39% of digital marketers believe in the effectiveness of social sharing to improve website conversion rate, the data proved otherwise in this case. A couple of arguments can be made to justify why social sharing buttons are most likely a bad idea for product pages:

  1. They compete with the main conversion goal of the page (i.e., adding items to cart) and distract visitors from it.
  2. Unless it’s something exceptionally cool or exclusive, not many people like sharing a product page or landing page on their social profiles. So, the negative social proof actually can backfire in such cases. Who would want to buy a product that no one likes/cares about?
  3. They sometimes increase your page load times. Matthew, an e-commerce site owner, found that his Facebook “Like” button was adding 1.3 seconds to his page load time. (To support this with some data, every second of delay in page load time can reduce conversion by 7%.)

Considering the large number of websites that make this mistake, the humongous popularity of this case study doesn’t surprise us at all.

Takeaway: Chances are, your social sharing buttons on product pages are doing more harm than good. Even if you’re one of the few online brands that do receive social media traction on your product pages, I’d still strongly suggest that you run a quick test to see if it is working for you. Don’t forget to notice the change in your page load time, too.

Case Study #2: Which Matters More – Price or Authenticity?

Express Watches created a valuable test for their product page. Their original page confidently displayed their “Lowest Price Guarantee” badge to assure their customers they were getting the best deal in the market.

The new version they decided to test, however, replaced this price badge with the “Seiko Authorized Dealer” badge. The hunch was that authenticity of the product can be a more important concern for brand-loyal shoppers.

To sum up, it was a direct war between two primary concerns shoppers might have – price vs. authenticity. The question was: which is more important?


Before I tell you the test results, I want you to understand that the reason I decided to share this case study is because I want you to notice how the test result (whatever it may be) will give Express Watches a clear understanding about a crucial customer value.

Curious to know which version won? The version with the authenticity badge increased online sales by 107%

Unfortunately, that’s all we know about this test. I’m not sure if Express Watches utilized this customer lesson to tweak their marketing message. But, ideally, that’s what they should do to get the maximum benefit from their test/customer lesson.

For example, in this case, they probably could try persuading/selling to people using an increased emphasis on authenticity of the product (instead of price) and talk more about their authorized dealership in their PPC ads, emails, and other marketing channels.

Takeaway: A successful testing mindset goes beyond celebrating the highs and lows of test outcomes. Create tests that give you a clear customer lesson. They will give you the most impactful keys to unlock the success of your overall marketing messaging.

Then, use the information wisely in all your marketing channels, and you will reap benefits that go further than a successful A/B test, and for years to come.

If your website deals with branded products, don’t shy away from stealing this test idea. Price vs. authenticity – give it a shot!

Case Study #3: Cut the Clutter

Underwater Audio’s product comparison page displayed a table drawing similarities and differences between their two waterproof headphone sets. The table was followed by a few short paragraphs that listed additional information about the products, along with some links to other pages, like “instructions to use waterproof headphones,” individual product pages of the two headphones, etc.

But somehow the page didn’t seem to engage visitors much. Here’s what Emily from Underwater Audio had to say about it:

“The (rather) unattractive table had information in terse phrases organized in no particular fashion (activity, seal, size, features, warranty, depth). The paragraphs continued below the fold and essentially repeated the table, with only a few unique additions hidden in the text. In short, it was not the most engaging page!”

They decided to remove the unnecessary information (from the table as well as the short paragraphs and links that followed) to keep the page crisp. And, some other interesting comparison points were added to the page to provide better value.

Result? The focused new page with no distractions engaged more visitors and beat the original. This spiked Underwater Audio’s online sales by 40.81%. Here are images of the two versions:


Takeaway: Recheck your product pages. Do they have any irrelevant links? Do they offer too much information that the majority of visitors do not need to make their purchase decision? Such unnecessary information can sometimes distract visitors from your conversion goal or might even overwhelm them. Cut the clutter on your page and A/B test it. Establish a clear eye flow on the page to help visitors easily scan the page.

Case Study #4: Compare with Your Competitors’ Prices

Paperstone is an e-commerce website in the UK office supplies space that competes with some big names, like Viking and Staples, in its industry.

When you’re competing with established brands, it’s important to pay close attention to what makes you a relatively better choice. And, it’s important that your prospects do not overlook it. Paperstone figured that the relatively cheaper prices of many of their products was what they could win with.

They showcased a small price comparison table on product pages, comparing their prices with two known brands in the industry. The table was placed right below the “add to cart” button so that visitors would notice it. This assured visitors they were getting the best deal. It’s important to note that these comparison tables were shown ONLY for products where Paperstone was offering relatively lower prices.

Here’s the new version with a quick-to-scan price comparison table added to a product page:

paperstone example

As expected, this extremely smart move by the company increased their online sales by 10.67%.

Takeaway: If you’re a small business competing with big names in your industry, find the one thing that should make your prospects buy from you instead of your competitors. Place it strategically so that your visitors do not overlook it. And, finally, test it.

If your competitive advantage is price, so be it.


The best part about optimizing this extremely important step of your conversion funnel is the wider reach and the manifold revenue impact when you A/B test them.

Also, site-wide product page template tests cumulate traffic, which is especially useful for low-traffic sites, making it easier for them to get conclusive test results in a shorter time period.

About the Author: Smriti Chawla works at VWO (Visual Website Optimizer), the A/B testing tool that helps you improve the conversion rate of your website. You can create experiments within minutes with their point-and-click editor. Sign up for their 30-day free trial here. For more inspiring conversion optimization examples, visit their case studies section. Follow Smriti Chawla on Twitter.