Steering your a/b tests toward impact: customer decile analysis

We love testing a site’s navigation as much as the next bunch of test-and-learners—but is that the problem your most valuable customers are facing? Are we running tests that actually make a difference in the customer lifecycle, or simply eking out wins on the margins? There’s a great method for answering these questions, and it’s called customer decile analysis.

okay, what is decile analysis?

The process of breaking your customers into ten groups, arranged by value. It’s a time-tested technique for answering common business questions like:

• “How much of our revenue comes from our biggest customers?”

• “What are the purchase patterns of our most profitable (or unprofitable) customers?”

• “Should we invest more in our low-margin products like accessories, or lean into higher-priced flagships like customization?”

It’s a great diagnostic for identifying patterns and exploring customer behaviors, but lacks the critical dimension of time.

time, like on my watch?

Yeah, but longer. If we look at customer value and how it changes from year to year, we can glean all kinds of useful insights about customer behaviors and areas of improvement for the business. Here are some examples:

• 80% of our revenue in 2019 came from our top 10% customers, but 90% of those customers didn’t shop again in 2020—why?
  Well, 75% of them bought our hot pink jorts on final sale because we recommended it to them, and they will never forgive us for it.

• Woah, 70% of our customers in 2020 had never shopped with us before, and didn’t make a second purchase?
  Turns out the order management/CRM system isn’t merging customer logs correctly, making it much harder to offer loyalty rewards and significantly under-attributing the ROI of our rewards program.

• We had a lot of first-time customers last holiday season—but how come we didn’t see many of them come back this year?
  We pushed a bad update to the returns page on the site last January so they might have had a hard time exchanging or returning their purchases. We should see if we can send them some goodwill coupons.

how do you actually do it?

Remember percentiles from standardized tests? It’s like that. “Decile” is just a fancy way to refer to a particular 10%-wide chunk of data points in a given data set.

In more concrete terms:

1. We calculate total customer value by year, then sort users by their total value.
2. But what is value? Think total items sold, total dollars sold, how many times they returned or exchanged products, etc.
3. We can use other periods of time that you think makes sense for your business and customers, but annual/yearly usually works well.
4. With our users sorted, we give those users rankings in each year they’ve been active customers. (And if they weren’t active then, we just mark them as an N/A.)
5. Once everyone is ranked, we combine them into cohorts tracking their migrations between groups so we can generate summary statistics like order frequency, average order value, etc.

i’m in! what do I need to get started?

We’ll be honest, it takes more than just Google or Adobe Analytics data. Usually, direct access to your customer data warehouse (or something similar) will enable most of this analysis. At the bare minimum, you or a partner like surefoot will need access to:

• Some kind of durable and reliable user/customer ID to stitch data together.
• An email address can work, though we prefer not to see your customers’ data like that.

• Bonus points if you have some kind of identity resolution or de-duplication system in place for collating customer activities by similar addresses, emails, etc.

• Transaction records including date of sale and order size, which can be tied to customer IDs when they’re available. As much relevant metadata is helpful, but here are some of our favorites:
• What items or product categories were included
• Cost of goods sold, margin, cost per acquisition, etc.
• One-time vs. subscription purchase

Note, a few places to look for this kind of data are your customer data hub (e.g. Tealium, Segment), data warehouse/business intelligence tools (e.g. BigQuery, Looker), your online store’s order management and CMS (e.g. Shopify, Magento).

After that, anything else is gravy! Once your customers are deciled, you can ask all kinds of questions by joining their decile rankings against any other data sets you have.

• What was the average cost per acquisition of our top 10% of customers?

• Were the pink jorts really that common amongst users who shopped last year but didn’t come back?

And perhaps most importantly: Who in the hell actually buys pink jorts to begin with?!

‍