If you search the Internet to learn about A/B testing, you’ll find scads of articles bursting with tips for cranking your business performance into the stratosphere.

You’ll get blazing hot secrets like…

BOOST YOUR CONVERSION RATE BY 300% WITH THIS TINY TWEAK

and…

DESTROY YOUR COMPETITION USING A STATISTICALLY SIGNIFICANT SHADE OF BLUE

…and it just keeps going like that, into an overenthusiastic pit of armchair psychology.

As the gurus tell it, A/B testing is like Vegas slots: plunk some crap into a machine, score a handful of 🍒🍒🍒s, and voilà, Easy Bake Revenue!

With a pitch like that, who could resist? It sounds so simple. If you don’t do it, you’re obviously a fool who’s leaving money on the table.

Welp, I have a couple of hard truths for ya:

  1. It’s not as easy as it sounds, and those big gains aren’t such a sure thing.
  2. Setting out to dramatically boost some arbitrary metric (signups, conversions, revenue, whatever) is the wrong way to approach a design problem.

How do I know this?

I spent most of last year testing dozens of conversion-related design ideas, and I found that the jacked-up performance aspect is the least interesting part of the process, by far.

The most interesting part is how it can change your thinking. It energizes your adventurous spirit, introduces you to uncharted territory, and lands you in cool places you never expected to go.

Here’s what I learned, and why I’ve come to love doing experimental design.

Testing turns your designs into trash.

After running tests for a while, you’ll find yourself throwing away mountains of design work for just a handful of meaningful improvements.

Do that enough, and you’ll notice something: Maybe design isn’t such a special endeavor after all!

The truth is…a lot of design is ephemeral, malleable, disposable…garbage.

Anything you make today merely represents one moment in time. Maybe it’s your best idea now, but it’s not necessarily the best idea you’ll have in another two days or two months.

Testing makes this painfully obvious on a shorter time scale. You’ll soon become less emotionally invested in your precious creations, and more focused on the problems you care about.

Testing strengthens your gusto.

I don’t know about you, but it took me years to build up the confidence to make hard design decisions. I still struggle with it sometimes.

You have to succeed and fail a lot. You have to take criticism a lot. And you have to trust your gut and keep at it, day after day.

Experimentation is a great way to build that muscle. It’s an opportunity to try things you aren’t completely sure about, and gives you a sweet little safety net for failures.

Testing makes you thoughtful.

It can be tempting to do experiments like the lottery: throw a bunch of random shit at the wall and then declare victory when one thing performed best by random chance.

You might get lucky a few times that way, but it’s a terrible long-term approach. Without some overarching vision, you’ll be left with a gnarly mess of test results born from guesses, and no clear plan for what to do next.

The better way, of course, is to start with good ideas! Do some research, come up with educated hypotheses and concepts you believe in, then build and test them to verify your thinking instead of defining it.

Testing destroys perfectionism.

It’s so freeing to ship a bare-bones version of an idea because it’s “just a test” that you’ll either improve or throw away when it’s done.

If you thought that same design had to stick around permanently, you’d probably never launch it with a lot of known flaws or incomplete parts. You’d want to fix up every last detail and make everything perfect first.

Amazingly, those rough, imperfect tests often outperform the supposedly perfected version you already had in place. When you see that, you’ll realize your outsized attention to detail might not matter as much as you thought.

A license for imperfection is an extremely useful defense against Fussy Designer disease. We should all be vaccinated early and often.

Testing builds empathy.

This sounds counterintuitive: running an experiment is mostly a pragmatic and statistical kind of thing. How is that related to empathy?

It’s related because you’re forced to learn what happens when real human people interact with your work. Your choices all have directly measurable effects, so you can’t hide behind bullshitty designerspeak or vague justifications when the data shows you’re just flat wrong.

That means you have to get outside your insular designer bubble, stop thinking of people as numbers, and get in their shoes a bit.

When you do that, the business boosts you want will happen as a natural side effect of continually tuning your product to serve your customers’ real-life situations. Making things clearer or more efficient for your customers always pays off.

Testing helps you make space.

One tough challenge in UI design is making physical space for new things you want to do. There’s only so much room on a screen!

You might have ideas that require injecting steps into an existing UI flow, adding more screens, revising a visual hierarchy, or rearranging certain navigational elements.

Doing stuff like that is a gamble. You might be confident that your new version is better in some way, but are you sure your improvements are worth the extra steps or added complexity?

Testing lets you dip your toes in the water. You can run a short experiment and see if you’re busting your business before committing to a direction.

Testing tells the truth.

The truth is weird. Sometimes common sense wins out. Sometimes a wild idea succeeds. Sometimes a version you hated performs the best. Sometimes your favorite design turns out to be a total stinker.

Where else in the design world can you get opinion-free feedback like this? There are no Art Directors or Product Managers or App Store reviewers telling you what they think is right. It’s real human nature telling you what’s right!

It’s a fascinating, powerful, bizarre reality.

This was originally posted on Signal vs. Noise.