In terms of innovation, there is seldom anything as powerful as the experiment. Whilst there has been no end of cheerleading for ideation and the like, ideas are useless unless they’re actually tried out.
Experimentation therefore is often at the heart of any attempts to innovate. Not only does it beholden organizations to accept that they don’t know the best answer to every challenge, but it also allows one to test out their ideas in the cut and thrust of a live situation, thus transforming something theoretical into something eminently practical.
Experimentation in government
With so many governments around the world struggling to meet both the social and financial challenges of the age, you would think therefore that experimentation would be at the heart of their attempts to find innovative solutions to these challenges.
Alas, a recent survey suggests that is very far from the case. When charitable trust Sense about Science quizzed over 100 Conservative and Labour MPs about their attitudes towards randomized control trials, it emerged that very few were prepared to utilize them, despite most agreeing that they’re a sound idea.
Rather than relying on experimental data to back up their policies, they instead preferred to act on their own principles, and that of ‘experts’ in the field.
The results suggest a rather widespread misunderstanding of just how RCTs work and how they could benefit them in policy making.
For instance, around 35 percent reported that they thought it unfair that some people would be not given a the policy being investigated, despite that being the fundamental strength of the RCT.
This was reflected in apparent support for pilots, despite pilots lacking the validity that a control group brings.
The hunt for evidence
Worryingly, the study also revealed that policy makers are quite keen on evidence, just so long as it isn’t of the practical and experimental sense.
The chart below shows where politicians focus their attention when making decisions.
We see in other walks of life, whether it’s drug development or the layout of an Amazon web page, that RCTs work incredibly well at providing evidence to support a claim.
After all, it’s not as though the state doesn’t have access to help in this regard. Ben Goldacre co-authored a paper, called Test, Learn, Adapt, to provide guidance on setting up policy trials, and there is considerable evidence of their success from the Behavioural Insights Team (aka Nudge Unit).
MIT’s Michael Scrage has identified ten things you need to consider when attempting to sculpt your own culture of experimentation.
- Understand why you’re doing it – before you begin experimenting you should have a firm insight into just why you’re doing it, and your rationale should be strong enough to entice people from across the organization to rally behind you
- Decide what level of support you need – do you need (or even want) support from senior executives? You ideally want your experiments to have the biggest impact in the shortest time, so try experimenting in areas that already have strong executive support
- Recruit a cheerleader – whatever your experiment, it always helps to have someone reasonably high that can support your project and cut through the organizational fluff
- Don’t ask for money – your experiment should be more about human capital than financial capital. By all means ask for people to help you or some facilities to support you, but don’t ask for cash
- Define the scope – as with any new project, it’s crucial that you define what it is you want to achieve, and your experiment is no different
- Select your deadline – you will need focus to succeed, so ensure you select a deadline that is close at hand, and stick to it
- Book time for the final presentation – after your deadline you’ll need to present your findings to the executive team. Make sure you have a time booked in and committed to from the executives concerned
- Secure access to the team – most experiments will involve a team of people, so make sure you have their time and energy committed to your experiment before you start as it will live and die by their efforts
- Kick things off – once you’ve picked and secured your team/s you set the project under way, clarify any questions regarding the project and get moving
- Launch – with minimal resources, it kind of forces you to be a bit rough and ready with your approach, but it also predicates intense discipline in how you behave
The aim here isn’t to provide perfect solutions but to test your thinking out in a way that provides insights that are good enough to act upon. Hopefully these ten steps will help policy makers along the way.
Sadly I'm not at all surprised. I wonder, however, whether the figures are any better in commerce? Do senior leaders have any greater faith in experimentation? I suspect not.
It's a good question Nick. Like you, I suspect the results are not great, but it would be very interesting to see any research into the topic.
Probably comes back to that old William Easterly thing of most being planners rather than searchers.
That's an interesting point Paul, and one I hadn't thought about at the time. Would make sense though.
What a shame it is too. I see so much in this election that seems purely philosophical rather than rooted in any kind of evidence or facts. It's not good at all.
I think it comes back to failure doesn't it? How common is it for a politician to openly admit to failing at anything? I can't think of many times when that happens. If you're not willing to fail then you can't be willing to experiment either.
Excellent point Andrea. Of course, that's not just something to land at politicians, as the media and indeed voters need to take some responsibility for that too.
Yes, exactly. The media are quick to pounce on mistakes, and voters are just as bad. The way the LibDems have been treated is appalling.