Stephen Pavlovich is the CEO of conversion.com and took the spot of the second talk in the CRO stream at Measurefest 2019.
Stephen opened his talk by explaining how we are all afraid of failure, and this is engrained with us. As a result of this businesses don’t have an open relationship with failure.
He then went on to use Amazon as an example. A few years ago Amazon attempted to launch two products:
- The Amazon phone
- The Amazon echo
Originally everyone thought the phone would have worked and would have been a lot more successful, this seemed like the product that would have been the most successful at the time. However, as we all know Amazon echo has been a raging success and the phone didn’t lift.
When thinking back to the early pitching stages of the two products, pitching a product that ‘is a black cylinder, that can sit on your side, that you can talk to, and will respond’ definitely sounds like the crazier out of the two. and that the echo would have been quite a crazy idea and then it was the most successful for amazon
So the question is…
How do you make customers/clients not afraid of failure?
An answer to this is to ask the question ‘if you could A/B test anything what would it be?’
Then it is easier to make their A/B tests appear more important and in line with business values.
Stephen has been working with Dominos for a few years now. Every year Dominos releases their annual “survival deal”. Most recently this year they launched it and decided to remove the cookie from the survival deal and reduced the price from £17.99 to £14.99. The share price in the UK then dropped significantly as a result of this.
What does this have to do with experimentation? These are the kind of things you can test. It could have been tested easily what survival deal is most popular and would be the most successful.
The four horsemen of A/B testing
Stephen then goes on to highlight the four key areas of A/B testing that are focused on primarily within the industry.
- Meek tweaks
- Band aid
- Safety check
- Inconclusive
All of the above are safe and do not step outside of the comfort zone at all. People aren’t prepared to invest in the right things due to a fear of failure.
A/B testing allows us to make decisions with informed data
The question to ask before starting to A/B test is: What are your goals, assumptions, risks, investments?
Stephen highlights, however, that as soon as you start running these kind of tests it can be quite hard to do so. He then tells the following story of voice to text technology and uses this as a clear example as to how it can work.
Voice to text example:
In the 1980s IBM wanted to test voice to text technology, this was a relatively new idea so they wanted to see if it would take off and what the reaction of a user would be. But the problem was the processing power of computers weren’t good enough.
IBM got people to speak into a microphone and words then appeared on the screen almost instantly. As the processing power of computers weren’t good enough, a person was typing the words but the person speaking did not know that someone was typing.
IBM could still test the reaction of the user in a realistic way and get an understanding of whether this could be a good concept to invest in.
How can you use this approach to test your biggest hypotheses?
Take something super complex your company would invest in and make a series of experiments to test with. Make the experiment perfect, everything else is optional.
What do we do next?
Get started by asking: ‘if you could A/B test anything what would it be?’
Then ask: ‘what are your goals, assumptions, risks, investments?’
Stephen concludes by talking about how at conversion.com they use an experimentation framework. This framework allows you take the goal you’ve agreed with the stakeholders and take the KPIs and review the audience to see what the engagement is.
Check out the other talks in the CRO stream at Measurefest below:
