One might imagine the very act of entrepreneurship is one of rapid experimentation as one’s assumptions are tested in the market.  Indeed, the lean startup methodology popularized by Eric Ries has brought into the popular lexicon the pivot, by which startups change tack after experience from the market encourages a new direction to be taken in some way.

It’s perhaps understandable, therefore, that new research from Harvard Business School finds that startups benefit from using A/B tests, whether in terms of launching new products or attracting more venture capital investment.

Such benefits are not felt across all startups, however, with early-stage ventures suffering higher failure rates if they adopt such experimentation, with the authors arguing that this may be due to managerial inexperience in applying such methods.  Alternatively, the ideas might simply suck.

Going faster

In essence, the experimentation undertaken by the startups seemed to accelerate things, both for good and bad.  So it would help the companies either scale faster or fail faster.

The researchers wanted to test the Silicon Valley doctrine of “failing fast” to see if it actually helped startups to succeed.  They examined over 35,000 tech startups that were founded between 2008 and 2013.

Each of the startups was examined in terms of their technology adoption and general characteristics to try and understand whether the experimentation they undertook helped their overall performance, and indeed how it did so.

Making experimentation easier

The researchers found that A/B testing was such a low cost way of experimenting with the digital platforms of each startup that it made making incremental changes easy and cost effective.

“By making it so much easier to make all these slight improvements, A/B testing also allows the firm to free up the time it would have spent tinkering with the small stuff, which it now delegates to the A/B testing tool,” the researchers say. “The founders can now think about more radical changes, like innovation and new products… You make more changes overall, so [adopting A/B testing] doesn’t come at the cost of big ideas.”

The analysis suggests that firms that were using A/B testing were able to identify the best features for their platforms, which in turn meant that they were visited more often, were stickier for users, and generally had better conversion rates.  It also meant that companies were able to accelerate the innovation process and reach the natural endpoint, whether that be scaling up or failing, much faster.

“Firms that adopt A/B testing are more likely to hit zero page views, which we take as evidence of shutting down, but they’re also more likely to have extreme page views, ending up with well over 50,000 views a week,” the researchers continue.

On average though, firms who had adopted A/B testing had 10% more page views per week, launched up to 18% more products, and were 5% more likely to raise VC investment.

“We think there are more product introductions because when you lower the cost of testing, that makes it more valuable to generate more potential ideas,” the researchers conclude. “If you’re generating more ideas, you’re more likely to generate new products.”