Guest article by Guilherme Motta
This evening I received an email asking what A/B Testing was about. After replying to the email, I felt/like writing about my understanding on A/B testing; what is it and what it is not.
Basically, think as if you could achieve the same goal by taking different approaches, measuring its results and outputs to refine how you do things.
Does *it* apply only for software? No!
Take the workflow of buying a product in a website as an example.
Workflow A:
Given that I am the user ->
When I select the product I want to buy ->
And I add to cart ->
And I proceed to checkout ->
Then I’m able to complete my purchase
What if I wanted to have a subset of my users going thru a different workflow to accomplish a purchase?
Lets assume that 50% of my users will follow the 5 step I mentioned above – Workflow A – and the other 50% will not have a cart, you can only buy one product at a time or per transaction. So the second workflow, or Workflow B, would be something like:
Workflow B:
Given that I am the user ->
When I select the product I want to buy ->
And I proceed to checkout ->
Then I’m able to complete my purchase
Can you tell what is A and what is B? Well.. that is why they call A/B Testing or “Canary Releasing” (this term is used on Jez Humble and David Farley’ Continuous Delivery book).
That was easy, two different versions of an application, now what?
I would assume that you are doing an experiment, you are actually trying to verify, with your users, if they will buy more on your website with one version or the other.
Someone might say: “THIS IS A WASTE OF TIME, WHY WOULD I TRY B? IT MAKES NO SENSE, IT IS JUST WRONG!”.
From “Lean Spirit. Summary of Lean Start Up & Lessons Learned from Another Failed Venture”
Get Out of the Building – there is no facts inside the company, only opinions and
assumptions. It is necessary to test those hypotheses through one-to-one interviews in order to understand customer and his or her pain level. It’s an attempt to minimize the risk of total failure by checking your theories against reality before realizing them.
Fine, I will listen to your argument but proofing and having statistics is more valuable then a person opinion. Said that, sometimes trying A/B Testing could be too expensive and not worth the investment depending on the feature or improvement. In the past, I’ve worked on projects that were using this technique and the exact same feature that worked on one specific section of the website to cause a higher number of sales made the number go down for another part of the website.
A critical aspect of A/B Testing is measuring the results. What do you care about? Well, if you have a restaurant you probably care about number of people eating on your facility, number of people taking orders to go, average ticket price per bill, your cost.. and whatever other meaningful numbers that could be impacted by your experiment/hypotheses, for better or worse.
How much time do you want to invest on your experiment? Resources are not infinite and depending on the business the timebox might be critical. If you are trying to innovate, your experiment could lead you to be behind your competitors but instead you are reducing your risk by proving that your idea works before hand.
This could be a good introduction to talk about the “J Curve”, but I will leave it for a further post.
What is the result you expect?
Given that I am the owner of an average restaurant I want to hire a French Chef so that I will increase the number of customers.
With the assumption we will increase 50% the number of customers..
What is the cost of playing this “story”?
What is the ROI of this “story”?
My expectation is that whenever I have my French Chef cooking dinner, my average clientele will go from 40 customers to 60 per night. I probably will spend more to have him cooking, I might spend with marketing to let my customers know when he will be cooking but I need also to understand if those extra 20 customers will be worth the experiment and the costs.
What if you always fail? Well, in my opinion it is better to try something new, something innovative then staying stale. Worst case scenario you will understand more about your business and your customers and this could lead you to a better service or product.
Should I ever try again the same experiment? Yes, I would think so. People change, world changes, people needs and interests change. Maybe something that don’t work today is the solution to tomorrow’s problems.
Startups are using A/B Testing to try to define their business. I’ve heard rumors about some companies releasing the same product, with two different names, two distinct prices for their product/service to help them identifying who is their customers.
Will they be more profitable if they sell to young people with a cheaper price?
Will they be more profitable if they target older people that will pay more for their service?
Is it a combination of both?!
What sells more?
Is my product/service something that scales?
Do I want to sell more?
Can I sell more?
So, A/B Testing won’t ever make the quality of your software go up, right?
Wrong. If you are doing A/B Testing, there is a high probability that you will have statistics about what are the features that the users are using the most, maybe even better, you will be able to understand where your revenue is coming from. Think about a feature that nobody else uses, that button on the menu that is obsolete, why do you still have that on your code base?!
What is the cost of supporting every single feature for every single user to your business?
Bug fixes and enhancements can also be tested by a sub-set of users, you could have multiple versions of your application in production. If managing different versions of your software is not an option, another approach would be using toggles to enable and disable features on your app. Urls, cookies, javascript could be used to enable/disable functionality for a smaller set of users.
