A/B Testing IRL

In marketing, advertising, and software, there is a popular concept called A/B testing. It is quite simple. You run a randomized experiment that tests two scenarios against each other while trying to keep outside variables equal.

For my work in advertising, I have A/B tests running on Facebook all hours of the day. For each active ad, a counter ad is also running that has slightly different copy.

Then, after a week or two, I evaluate which ad brings in a lower cost-per-lead. Next, I’ll drop the losing ad, and run another ad against the winner, this time with slightly different images/copy.

This loop keeps going on indefinitely. It is a great example of an iterated game, which I wrote about two weeks ago.

A/B testing is an infinite loop of improvement.

Web designers will often test out 2 or more types of content, page designs, or buttons to see which gets more clicks and sales.

Here’s a great example from The Daily Egg showing an A/B test run for WallMonkeys, a wall decal store. Visitors see either homepage, and their behavior is tracked for clicks, purchases, etc.

Btw, Homepage B won the test, increasing conversion rates by over 500%.

So, it’s easy to imagine A/B tests on the web, but what about in real life?

Here’s a few A/B tests I’ve seen run IRL.

Food & Drinks

Testing out coffee shops

My ex-girlfriend gets black coffee at every new coffee shop she visits. In her head, she is running an A/B test although she probably would not describe it as such.

She is comparing the absolute base level coffee (no cream, no sugar, no syrup), thus keeping the variables equal. This allows her to test which coffee is better at its root level, and thus choose which coffee shop to visit more frequently.

Salmon-avocado roll anyone?

One of my readers gets a salmon-avocado roll at every sushi restaurant he goes to. Although he works in marketing, I don’t think he’d describe it as an A/B test either. But, that is what it is.

All other variables are relatively equal, which allows him to test the freshness of the salmon and avocado as well as the rice and seaweed paper quality.

Writing

The Tim Ferris Editing Method (Expanded)

When Tim Ferriss is editing a book, he sends a set of chapters to at least 10 friends to get their feedback. Rather than listening to just one editor or reader, he gets feedback from a larger sample size, allowing him to make a more accurate prediction of what the population of Tim Ferriss readers will enjoy.

An A/B test expansion could be sending two versions of the same chapter to two randomly chosen groups of friends. This would allow him to figure out which version of the chapter hits the mark better. Again, the larger the sample size, the more accurate the prediction.

Visual Art

NFTs

If you’ve spent any time scrolling through NFT platforms OpeanSea or Rarible, you’ll notice that there are a lot of variations of the same design. Cryptopunks was the original programmatically generated project of 10,000 designs, but there have been dozens of knockoffs like Cunning Foxes, Lazy Lions, Bored Apes, etc.

When these groups of 10,000 designs are minted and sold for the first time, all designs in the same project go for the same price. From there, they’ll end up going up in price based on market demand.

These NFT projects are great examples of A/B testing to an extreme. The artists create thousands of lookalike designs while keeping the price constant. From there, the forces of supply and demand determines which attributes become the most valuable.

For example, alien Cryptopunks sell for millions more than human Cryptopunks, whereas foxes with melons on their heads sell for a different average price than foxes with no melons on their heads. The NFT economy is strange, and deserves a standalone essay at some point.

Music & Film

Talking to musician friends, I’ve heard that they will often record the same lyric or chorus multiple times. This makes sense. They might not like the first take. But, unless it is horrendous, they never delete the first few takes until the mixing process afterwards.

Why? Just in case it turns out they like the first option better. They are essentially running an A/B test here, keeping all other variables (instrumentation, back-up vocals, etc.) constant.

Directors do the same thing when filming television and movies. They’ll film the same scene multiple times, and wait until afterwards to see which they like better. Often, like musicians, they will seek out friends or trusted advisors to say which version is better.

Takeaway:

A/B testing isa trick typically reserved to advertisers and web designers, but A/B tests are all around us everyday. They’re simple and easy to assess.

Split events into two groups, change one variable, and keep the rest equal. From there, evaluate, improve, iterate, repeat.

What A/B tests are you running?

Jason