Product discovery: How we measure impact on mobile sites

One of today’s most commonly-faced challenges in ecommerce is the mobile paradox. Businesses are experiencing increases in mobile traffic, but flat revenue on the same device. As the proportion of mobile traffic to retail sites increases, retailers are loosing money - fast. We identified product discovery as a solution to help solve this problem. In October last year, we launched Aura, a solution for product discovery on mobile.

The last few months have been focused on understanding how we should measure and report on discovery. This blog post outlines some of the challenges in approaching the answer.


The role of discovery

Discovery plays an interesting part in the path to purchase. If discovery is taking place on mobile, how sure can we be that the transaction is also on the same device? Google research highlights that 6 in 10 online purchases are multi-device. In fact, if you are a multi channel retailer, the goal of discovery should be to drive cross-device traffic (reducing traffic acquisition costs), encourage customers to come in-store (where you can provide a more differentiated experience) and ultimately be more sure when they make a decision of what to purchase.

Additionally, if discovery is getting someone to stumble across something that they had not yet considered, it might not be reasonable to expect a consumer to be immediately compelled to purchase the product right there and then. Instead, discovery is more akin to upper funnel advertising, where the idea is to plant a seed and let it grow. McKinsey might describe this as increasing the size of the total consideration set. 

McKinsey's visualization of the four phase customer decision-making process.png

Historically at Qubit, we have been very strict when it comes to determining the value of an "online experience", and have strongly advocated statistical frameworks and randomised control trials as the gold standard (as you can see from our 6% more report, and the accompanying academic paper).

However, when it comes to measuring the value of discovery, and Aura, things aren't so straightforward, in terms of what you're measuring, when, and how you should do it.

Randomised control trials (or A/B testing in the strictest sense) expect each person in the experiment is independent, and has no impact on any other person. But that is not how consumers operate. Shopping is social. Customers talk, they discuss and share. Aura includes a feature that enables customers to WhatsApp an item to their friend - who may then go on to purchase. Statistics are a lot more complicated for measuring the impact of mobile site personalization than for a randomized control trial on desktop.


Measuring discovery on mobile

To account for these complications, we have taken a different approach to measurement, which provides for a range of attributed revenue values to account for the difficulty in providing a definitive answer.

In this range, we provide a lower and upper bound. The lower end of the range provides a very strict, conservative boundary that only considers revenue from transactions after an item was specifically discovered through Aura. The goal here is to represent that this is the very minimum that discovery has supported in driving revenue.

However, an increase in product discovery can have a positive impact for the retailer. The very act of scrolling through the catalogue can prompt another outfit or gift idea—that wasn’t part of the initial reason for coming to the site—or open up the customer to view entirely new product categories, which the customer may go on to buy. The upper bound accounts for this type of behaviour and measures the revenue from transactions from customers who have engaged and interacted with Aura.

On both of these measures, we are limiting the window for discovery to 28 days, and and are only looking at a single, mobile device. No ‘halo’ effect is calculated or assumed, although it is very possible there is a longer term benefit to customer lifetime value, which we will continue to explore.

To support these measurement frameworks, we are creating a series of data “Exports” that will help retailers understand the benefit of discovery to the consumer in more detail, whilst also providing ecommerce teams with additional insights on the trends in demand of specific products and categories on a daily, weekly, and monthly basis.


Aligned focus

An added benefit to these measures is that they provide clear goals for our product and engineering team. We'll be working to make Aura as engaging as possible—so customers want to use this as a mode of discovery time and again—and our customers see their upper bound measures increase; and our customers should also see their lower bound measures increase as we improve Aura's ability  to get the most relevant products in front of their customers.

The new Aura reporting interface is available to all Aura customers in January 2018.
Qubit clients can find full documentation here.

Not a Qubit client? Find out more about Aura on our site here, or in our new guide, Rethinking product discovery on mobile.