Uncovering a/b test ideas in your existing data: customer transcripts

If you’re overwhelmed by articles, meetups and peers telling you a/b testing is critical but you’re unsure where to start, we have good news! Chances are you already have much of the data you need to start running quality a/b tests quickly.

In part 1 of this 3-part series on uncovering test ideas in your existing data, we provide a lightweight, step-by-step framework for mining your customer service and sales transcripts to identify “low-hanging fruit” that have a high likelihood of ripening into winning a/b tests.

We’ve detailed the steps below but feel free to check out our sample “problem tracker” template and copy + adapt to fit your organization’s unique needs. (Note: we’ve stripped this version down to the studs to make it less overwhelming for those readers just dipping their toes, but are happy to share the more detailed version we use with our clients if you email us.)

Step 1: Review your existing data

Start by reviewing your sales and chat logs, customer service transcripts, emails, etc. and make a list of the most commonly asked questions. These questions often reveal problems or sources of confusion your customers are experiencing. Common problems to look for amongst the questions are things like “confusion about shipping” or “unclear product pricing”.

Step 2: Identify areas of focus

Once you’ve identified commonly asked questions, document where on your site users are asking them. If your company uses a chat tool like Intercom or Drift, ask your support team to log the page the user was visiting when they asked the question. Were they on your product page? Step 1 of checkout? This will help surface pages that are good starting points for a/b testing.

Step 3: Define some hypotheses

Your high school chemistry class is finally about to come in handy! The next step is to brainstorm “solutions” to the problems you identified in step 2, otherwise known as your hypotheses. As you probably remember, a hypothesis is a statement, not a question. It is an educated, testable prediction about what will happen.

We like to follow the format “I believe if we [x], then [y] will happen” because it states the facts simply and keeps things general enough to eliminate the possibility for emotion and blame should the test lose.

For example, if visitors are consistently erroring on your shipping step, a good hypothesis might be something like: “I believe if we pre-select a shipping option, it will reduce visitor errors on the shipping step and lead to an increase in clicks to the payment page.”

Step 4: Implement tests

Design a/b tests to address your hypothesis and configure them in your tool of choice – even if that tool is free. (Do you need some support with this step?) Our best advice here is not to get too hung up on the design or details – after all, this is just a test.

If you address your customer’s biggest problems, they don’t care if your solution looks like it fell out of the world’s tallest ugly tree and hit every leaf on the way down. They care about having their problems solved.

Step 5: Analyze your results

Once your tests are complete, analyze the results. It should be fairly obvious whether a test has successfully addressed your customers’ problem, but take a look at your customer service trends. If you see a drop in the frequency of questions you’re addressing through a/b testing, you’re on the right track!

To wrap it neatly with a bow, you should be a/b testing because it’s a win-win for your company and customers alike.

For the company, a/b testing is a lightweight way for you to address customer service issues or product confusion, which can eventually lead to cost savings in the form of reduced strain on your customer service and sales teams. In addition, addressing your customers’ frustrations and friction points increases your chances for significant revenue gains by making it easier for them to find what they need and purchase.

For your customers, a/b testing builds trust, credibility and loyalty because it makes them feel like your company “hears” their voices. And in today’s world where it’s all about the lowest prices and the free-est shipping, loyalty and word-of-mouth are things every company could use a bit more of. More on that in a future blog post 🙂