"I’ve grown tired of debating such minuscule design decisions. There are more exciting design decisions to tackle.”
The above was written in a very public departure from Google by Douglas Brown, who was lamenting Google processes such as the infamous Forty Seven Shades of Blue. Google was looking for the one ideal shade of blue for links, to be used across all of their products and quite literally tested 47 shades. While it must have felt like insanity, Google later quantified the value of that change: $200 million. The topic is A/B testing. Presented by Paolo Mulabuyo, User Experience Designer for Netflix, it was noted that testing seems to pit subjective vs objective, qualitative vs quantitative, art vs science. Design is actually the spectrum of combining art and science. In surveys, 47% of “non-members” said they’d be more likely to sign up for Netflix if they could browse the catalog beforehand. So with a hypothesis of “allowing non-members to browse the catalog will improve conversion” they A/B tested the current (at the time) design against one with catalog browsing. The original design - with very little information on the homepage - outperformed the catalog browsable version. There was just too much information and it overwhelmed users. They tried it another 4 times in various ways with the same result. Eventually, they enhanced the minimal homepage design with a little more information about what Netflix offered, and finally came to a design that outperformed what they already had. A/B testing separates what people say from what they do. Two other takeaways: A/B testing enables you to make big bets confidently A/B can’t replace qualitative methods. It complements them.