Jun 21, 2021 | by Budi Tanrim
Side effects of A/B testing
A/B testing. Either design A or design B will come out as a winner. The winner has typically produced better metrics.
It gives a certain comfort and certainty. We know which design is actually “better.” But when the team is not mature and can’t think clearly, they can easily fall into the trap.
The team lack of product vision. The group becomes very focus on small iterations or improvements. Since A/B testing is so certain, they create more experiments, wondering how to increase the metrics or KPI for each quarter. They keep doing this without knowing what the real value they want to deliver to customers.
The team loses its ability to make a decision. They rely on the data too much to make the decisions. I’ve seen designers and product managers who can’t make a simple decision like, “Should we put the entry point here or there?” Then, they set a three weeks A/B testing. Ridiculous.
They are abusing “data” to support the bad decision. For example, a logistic team in an e-commerce app increased the delivery cost to hit the target. Design A has a $20 delivery cost, and design B has a $28. The data? No difference in the transaction. Can that be justified? Perhaps. One month later, the customers raged on Twitter, and the company put it back.
These are just a few side effects.
A/B testing is a good tool. But when the team is not thinking clearly, it becomes a curse.
