In the startup world, experimentation is often seen as a given as entrepreneurs test their assumptions and rapidly pivot based upon the learning those experiments generate.  Does this culture of experimentation actually help startups to succeed though?  That was the question posed by a recent study from Duke’s Fuqua School of Business, which assessed over 35,000 tech startups to understand what makes them thrive.

The results suggest that entrepreneurs who conduct randomized control trials (or A/B testing) of product features seemed to be more successful than those who didn’t take such a data-driven approach to their development.

“To be honest, most startups and most small businesses have a pretty standard framework for making decisions, which is ‘trust your gut,’” the researchers say. “Maybe collect a little bit of data. But usually, entrepreneurs have a belief about something that [their] startup should do, and they pursue it and they follow through.”

Quantitative decision-making

The use of A/B testing allows us to make more rational and quantitative decision-making.  It’s a process that the researchers also argue accelerates the discovery of what works and what doesn’t, which is so important for any startup.

“If an idea is getting traction, you can scale it faster, and if it’s not going anywhere, you can kill it faster,” they say.

The study saw thousands of startups analyzed, in terms of their product innovations, web metrics, and other characteristics.  They also assessed whether they used A/B testing as part of their product development or not.

The results suggest that A/B testing has a clear impact on performance levels, with those who used it growing page views to their app or website by 30 to 100% after a year.  They were also helped to introduce new products at a rate of 18% versus just 9% for those companies who didn’t use it.

A scalable strategy

A common perception around A/B testing is that it’s generally only really useful for incremental changes.  The researchers argue that this isn’t entirely true, however, and it can be equally useful in guiding much more significant decisions.

“The scientific method applied to business decision-making can be a very powerful tool,” the researchers explain. “A/B testing is just one instantiation of actual data-driven decision-making by firms that can have a powerful effect on the quality of a decision that a startup makes.”

Despite these apparent benefits, however, the adoption of A/B testing remained quite low.  Even in Silicon Valley only one in four firms utilized it, with startups in other parts of the country even less so.  Small firms with fewer than ten employees were especially unlikely to utilize the approach.  This is a mistake, as it is often these smaller firms that benefit the most from A/B testing.

“This kind of supports this idea that decision-making may be an important bottleneck for startup success, and that using a more scientific, rigorous approach to start decision-making can really improve quality of a startup’s performance,” the researchers say.

Ideas and potential pivots are especially commonplace in the early days of any new venture, so it’s vital that a way is found to scientifically assess and implement those ideas to ensure they’re acted upon effectively.

“[A/B testing] opens up ideation inside a startup,” the researchers conclude. “With advances in technology that allow startups to potentially test thousands of different options, A/B testing is both a cost-effective and rapid method for discovering startup product viability.”