{"id":99,"date":"2012-11-12T13:15:00","date_gmt":"2012-11-12T13:15:00","guid":{"rendered":"https:\/\/caroli.org\/en\/what-is-ab-testing\/"},"modified":"2020-01-10T19:51:01","modified_gmt":"2020-01-10T22:51:01","slug":"what-is-ab-testing","status":"publish","type":"post","link":"https:\/\/caroli.org\/en\/what-is-ab-testing\/","title":{"rendered":"What is A\/B Testing? by Guilherme Motta"},"content":{"rendered":"
\n

Guest article by Guilherme Motta<\/a><\/em><\/p>\n

This evening I received an email asking what A\/B Testing was about. After replying to the email, I felt\/like writing about my understanding on\u00a0 A\/B\u00a0testing;\u00a0 what is it and what it is not.<\/p>\n

Basically, think as if you could achieve the same goal by taking different approaches, measuring its results and outputs to refine how you do things.<\/p>\n

Does *it* apply only for software? No!<\/p>\n

Take the workflow of buying a product in a website as an example.<\/p>\n

Workflow A:
\nGiven that I am the user ->
\nWhen I select the product I want to buy ->
\nAnd I add to cart ->
\nAnd I proceed to checkout ->
\nThen I’m able to complete my purchase<\/p>\n

What if I wanted to have a subset of my users going thru a different workflow to accomplish a purchase?
\nLets assume that 50% of my users will follow the 5 step I mentioned above – Workflow A – and the other 50% will not have a cart, you can only buy one product at a time or per\u00a0transaction. So the second workflow, or Workflow B, would be something like:<\/p>\n

Workflow B:
\nGiven that I am the user ->
\nWhen I select the product I want to buy ->
\nAnd I proceed to checkout ->
\nThen I’m able to complete my purchase<\/p>\n

Can you tell what is A and what is B? Well.. that is why they call A\/B Testing or “Canary Releasing” (this term is used on Jez Humble and David Farley’ Continuous Delivery book<\/a>).<\/p>\n

That was easy, two different versions of an application, now what?
\nI would assume that you are doing an experiment, you are actually trying to verify, with your users, if they will buy more on your website with one version or the\u00a0other.<\/p>\n

Someone might say: “THIS IS A WASTE OF TIME, WHY WOULD I TRY B? IT MAKES NO SENSE, IT IS JUST WRONG!”.<\/p>\n

From “Lean Spirit. Summary of Lean Start Up & Lessons Learned from Another Failed Venture”<\/p>\n

Get Out of the Building \u2013 there is no facts inside the company, only opinions and
\nassumptions. It is necessary to test those hypotheses<\/b> through one-to-one interviews<\/span> in order\u00a0to understand customer and his or her pain level. It\u2019s an attempt to minimize the risk of total\u00a0failure by checking your theories against reality before realizing them.<\/p><\/blockquote>\n

Fine, I will listen to your argument but proofing and having\u00a0statistics is more valuable then a person opinion. Said that, sometimes trying A\/B Testing could be too expensive and not worth the investment depending on the feature\u00a0or improvement. In the past, I’ve worked on projects that were using this technique and the exact same feature that worked on one specific section of the website to\u00a0cause a higher number of sales made the number go down for another part of the website.<\/p>\n

A critical aspect of A\/B Testing is measuring the results. What do you care about? Well, if you have a restaurant you probably care about number of people eating on\u00a0your facility, number of people taking orders to go, average ticket price per bill, your cost.. and whatever other meaningful numbers that could be impacted by your\u00a0experiment\/hypotheses, for better or worse.<\/p>\n

How much time do you want to invest on your experiment? Resources are not infinite and depending on the business the timebox might be critical. If you are trying to\u00a0innovate, your experiment could lead you to be behind your competitors but instead you are reducing your risk by proving that your idea works before hand.<\/p>\n

This could be a good introduction to talk about the “J Curve”, but I will leave it for a further post.<\/p>\n

What is the result you expect?<\/p>\n

Given that I am the owner of an average restaurant I want to hire a French Chef so that I will increase the number of customers.<\/p>\n

With the assumption we will increase 50% the number of customers..
\nWhat is the cost of playing this “story”?
\nWhat is the ROI of this “story”?<\/p>\n

My expectation is that whenever I have my French Chef cooking dinner, my average clientele will go from 40 customers to 60 per night. I\u00a0probably will spend more to have him cooking, I might spend with marketing to let my customers know when he will be cooking but I need also to understand if those\u00a0extra 20 customers will be worth the experiment and the costs.<\/p>\n

What if you always fail? Well, in my opinion it is better to try something new, something innovative then staying stale. Worst case scenario you will understand more\u00a0about your business and your customers and this could lead you to a better service or product.<\/p>\n

Should I ever try again the same experiment? Yes, I would think so. People change, world changes, people needs and interests change. Maybe something that don’t work today is the solution\u00a0to tomorrow’s problems.<\/p>\n

Startups are using A\/B Testing to try to define their business. I’ve heard\u00a0rumors\u00a0about some companies releasing the same product, with two different names, two\u00a0distinct prices for their product\/service to help them identifying who is their customers.<\/p>\n

Will they be more profitable if they sell to young people with a cheaper price?
\nWill they\u00a0be more profitable if they target older people that will pay more for their service?
\nIs it a combination of both?!
\nWhat sells more?
\nIs my product\/service something\u00a0that scales?
\nDo I want to sell more?
\nCan I sell more?<\/p>\n

So, A\/B Testing won’t ever make the quality of your software go up, right?<\/p>\n

Wrong. If you are doing A\/B Testing, there is a high probability that you will have\u00a0statistics about what are the features that the users are using the most, maybe even better, you will be able to understand where your revenue is coming from. Think\u00a0about a feature that nobody else uses, that button on the menu that is obsolete, why do you still have that on your code base?!<\/p>\n

What is the cost of supporting every single feature for every single user to your business?<\/p>\n

Bug fixes and enhancements can also be tested by a sub-set of users, you could have multiple versions of your application in production. If managing different versions\u00a0of your software is not an option, another approach would be using toggles to enable and disable features on your app. Urls, cookies, javascript could be used to\u00a0enable\/disable functionality for a smaller set of users.<\/p>\n

<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

Guest article by Guilherme Motta This evening I received an email asking what A\/B Testing was about. After replying to the email, I felt\/like writing about my understanding on\u00a0 A\/B\u00a0testing;\u00a0 what is it and what it is not. Basically, think as if you could achieve the same goal by taking different approaches, measuring its results […]<\/p>\n","protected":false},"author":1,"featured_media":2389,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[1919],"tags":[750,1563],"inception_activities":[],"colaboradores":[],"class_list":["post-99","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-agile-culture","tag-blog-en","tag-guilherme-motta-en"],"_links":{"self":[{"href":"https:\/\/caroli.org\/en\/wp-json\/wp\/v2\/posts\/99","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/caroli.org\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/caroli.org\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/caroli.org\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/caroli.org\/en\/wp-json\/wp\/v2\/comments?post=99"}],"version-history":[{"count":0,"href":"https:\/\/caroli.org\/en\/wp-json\/wp\/v2\/posts\/99\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/caroli.org\/en\/wp-json\/wp\/v2\/media\/2389"}],"wp:attachment":[{"href":"https:\/\/caroli.org\/en\/wp-json\/wp\/v2\/media?parent=99"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/caroli.org\/en\/wp-json\/wp\/v2\/categories?post=99"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/caroli.org\/en\/wp-json\/wp\/v2\/tags?post=99"},{"taxonomy":"inception_activities","embeddable":true,"href":"https:\/\/caroli.org\/en\/wp-json\/wp\/v2\/inception_activities?post=99"},{"taxonomy":"colaboradores","embeddable":true,"href":"https:\/\/caroli.org\/en\/wp-json\/wp\/v2\/colaboradores?post=99"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}