As open innovation has grown as a concept, there have been numerous attempts to better understand what motivates people to participate.  Most of these have focused in on things such as the passion participants have for the topic, and the degree of freedom the challenge affords them in working how they wish to solve the challenge.

A recent paper from researchers at Columbia takes a more extrinsic look at the matter, and explores how the various forms of prize offered to participants impacts motivation.

The paper compares the impact of the winner takes all approach with the equal sharing approach, whereby the prize fund is split between all participants who come up with a solution by the deadline.

It also explores how information is disseminated within the contest, and in particular the open approach, whereby the progress of each participant is known to the others vs the more closed approach whereby no one knows how others are getting on until the deadline is reached.

Which combination motivates participants more?

It’s easy to assume that the first past the post method will attract the heaviest investment of time and energy from participants as it offers them the largest payback.

“There’s truth to that, because if you’re investing your time and effort to develop an innovation, you want to know that you’ll win the full prize if you’re successful,” the researchers say. “And you also want to know if anyone else has already come up with a solution.”

Operating in uncertain conditions

Whilst this may be true in certain conditions, the study found that it should not be considered a catch-all.  The paper reveals that when there is a degree of uncertainty about the chances of success, a more collaborative approach works much better.

“Sometimes the result is impossible to obtain,” the paper says. “Suppose you’re trying to come up with a new pharmaceutical, for example. It might not be feasible given the current technology or within the relevant time frame.”

The paper suggests that this level of uncertainty tends to lead to pessimism over chances of success in time, with the longer the period, the greater the pessimism.  It suggests that the longer the challenge goes without success, the more likely participants are to conclude the investment is not worth it.

To maintain motivation in such a scenario therefore requires a reduction in transparency.  You’re basically hiding the progress of other participants.  Of course, in a winner takes all scenario, that approach doesn’t work as one successful entrant automatically ends the contest, so it also requires an equal sharing approach to prize distribution too.

The paper concludes by recommending an even greater degree of nuance to your contest design, with winner takes all and open information used to begin with, before then switching to equal sharing/closed information at a certain point based upon the likelihood of success.

Collaborative vs competitive

The paper provides an interesting contrast to a recent publication by noted innovation researcher Karim Lakhani.

Lakhani teamed up with Kevin Boudreau from London Business School for a fresh exploration of open innovation, whereby they investigated whether the timing of data release has an impact upon the quality of output from the project.

For instance, is a sequential and regular flow of data more beneficial than the release of the finalized innovation.  In other words, is the working out as valuable as the end product to the wider community?  Does the open source software model provide a better approach to something like the X Prize?

There is rather more evidence of the latter when it comes to innovations, with organizations tending to favour the certainty that comes with this approach over the more dynamic, and open, approach seen in projects such as Linux and the Human Genome project.

The paper sets out to test two main points.  First of all, the authors suggest that the benefit of a steady stream of information in the intermediate approach comes at the cost of diminished incentive for participants, thus potentially limiting the outcome of the project.

They also suggest that when final disclosure occurs, it results in a lot more duplicated effort by participants, with a subsequent increase in collaborative work undertaken when information is disclosed throughout the project.  However, the flip side is that this independent effort also helps to bring about more innovative solutions.

The researchers tested their hypothesis on an online innovation platform akin to Top Coder.

A challenge was set to over 700 contributors to develop and optimize a bioinformatics algorithm.  Some operated under final disclosure conditions, whilst others operated under intermediate disclosure.

Those in this latter group solutions were developed along a trial and error style approach, with each iteration instantly shared with the group.

Under the final disclosure condition, the work of each participant was not shared with the others in the group until the end of the two week challenge period.

Which would work best?

At the end of the two week period, the solutions from each of the group were ranked in order of their overall performance.

As predicted, the regular disclosure group received less effort than the other group, with just 26% fewer participants getting actively involved than in the other group.  These participants then tended to exhibit less effort than their peers in the other group, despite having fewer ‘rivals’ for the final prize.

This manifested itself in a final number of submissions that were 56% lower in the intermediate disclosure group than in the final disclosure one.  What’s more, it also emerged that those in the intermediate group put in a (self-reported) 4 hours less on the project over the two week period.

Ok, so what about re-use?

Now, obviously, the final disclosure group could not re-use any of the work done in the two week period, so how valuable was this?

It emerged that nearly all of the participants in the intermediate group built upon the work of others, with each participant examining 34 other solutions on average.  This was especially potent in the first week of the challenge.

Interestingly, the work was also examined by people who did not eventually choose to enter the challenge, albeit at lower volumes than active participants.  Whether through curiosity or active learning however, it is nonetheless interesting.

Incentives matter as much as re-using

The paper is interesting and useful in that it reminds us that incentives are as important as re-use in ensuring that you get both the number of participants and also the amount of interaction between them.

The experiment found that despite the lower number of participants, the intermediate setting nevertheless provided a better outcome to the challenge for less effort than the other group, due in large part to the high degree of reuse and collaboration amongst participants.

The researchers do declare however that sponsors should beware of the potential for groupthink around a sub-optimal solution under such an approach.

They suggest that a good way around this is to ensure that the participants are drawn from a wide and diverse mixture of people.

“In theory and in our experiment, final disclosure promotes higher levels of entry and effort and independent experimentation. On the one hand, this generated wide diversity of approaches; on the other hand, this led to considerable effort devoted to suboptimal approaches and overall lesser learning and performance achieved,”the authors conclude.

The hope is that with greater awareness of the possible pitfalls of each approach, sponsors can design challenges to ensure those risks are minimized.  If you’re interested in open innovation, then this paper is well worth a read.  Check it out via the link above.