Karim Lakhani is one of the premier researchers into the open innovation landscape, and in particular for his research into the Innocentive platform.

In his famous 2011 paper, he studied 166 challenges posted on InnoCentive to try and discern any patterns in how they were solved.  The study found that as the number of scientific interests amongst the ‘solver’ population increased, so too did the probability that the problem would find a solution.

In other words, the more diverse the community of expertise available via InnoCentive, the more chance there was of finding a solution to the problems posed.

Lakhani has teamed up with Kevin Boudreau from London Business School for a fresh exploration of open innovation, whereby they investigate whether the timing of data release has an impact upon the quality of output from the project.

For instance, is a sequential and regular flow of data more beneficial than the release of the finalized innovation.  In other words, is the working out as valuable as the end product to the wider community?  Does the open source software model provide a better approach to something like the X Prize?

There is rather more evidence of the latter when it comes to innovations, with organizations tending to favour the certainty that comes with this approach over the more dynamic, and open, approach seen in projects such as Linux and the Human Genome project.

The paper sets out to test two main points.  First of all, the authors suggest that the benefit of a steady stream of information in the intermediate approach comes at the cost of diminished incentive for participants, thus potentially limiting the outcome of the project.

They also suggest that when final disclosure occurs, it results in a lot more duplicated effort by participants, with a subsequent increase in collaborative work undertaken when information is disclosed throughout the project.  However, the flip side is that this independent effort also helps to bring about more innovative solutions.

The researchers tested their hypothesis on an online innovation platform akin to Top Coder.

A challenge was set to over 700 contributors to develop and optimize a bioinformatics algorithm.  Some operated under final disclosure conditions, whilst others operated under intermediate disclosure.

Those in this latter group solutions were developed along a trial and error style approach, with each iteration instantly shared with the group.

Under the final disclosure condition, the work of each participant was not shared with the others in the group until the end of the two week challenge period.

Which would work best?

At the end of the two week period, the solutions from each of the group were ranked in order of their overall performance.

As predicted, the regular disclosure group received less effort than the other group, with just 26% fewer participants getting actively involved than in the other group.  These participants then tended to exhibit less effort than their peers in the other group, despite having fewer ‘rivals’ for the final prize.

This manifested itself in a final number of submissions that were 56% lower in the intermediate disclosure group than in the final disclosure one.  What’s more, it also emerged that those in the intermediate group put in a (self-reported) 4 hours less on the project over the two week period.

Ok, so what about re-use?

Now, obviously, the final disclosure group could not re-use any of the work done in the two week period, so how valuable was this?

It emerged that nearly all of the participants in the intermediate group built upon the work of others, with each participant examining 34 other solutions on average.  This was especially potent in the first week of the challenge.

Interestingly, the work was also examined by people who did not eventually choose to enter the challenge, albeit at lower volumes than active participants.  Whether through curiosity or active learning however, it is nonetheless interesting.

Incentives matter as much as re-using

The paper is interesting and useful in that it reminds us that incentives are as important as re-use in ensuring that you get both the number of participants and also the amount of interaction between them.

The experiment found that despite the lower number of participants, the intermediate setting nevertheless provided a better outcome to the challenge for less effort than the other group, due in large part to the high degree of reuse and collaboration amongst participants.

The researchers do declare however that sponsors should beware of the potential for groupthink around a sub-optimal solution under such an approach.

They suggest that a good way around this is to ensure that the participants are drawn from a wide and diverse mixture of people.

“In theory and in our experiment, final disclosure promotes higher levels of entry and effort and independent experimentation. On the one hand, this generated wide diversity of approaches; on the other hand, this led to considerable effort devoted to suboptimal approaches and overall lesser learning and performance achieved,” the authors conclude.

The hope is that with greater awareness of the possible pitfalls of each approach, sponsors can design challenges to ensure those risks are minimized.  If you’re interested in open innovation, then this paper is well worth a read.  Check it out via the link above.