Taking the first steps in business policy experimentation
www.nesta.org.uk/blog/taking-the-first-steps-in-business-policy-experimentation/
www.nesta.org.uk/blog/taking-the-first-steps-in-business-policy-experimentation/
IGL has been examining the barriers that prevent a culture of experimentation from being adopted by policymakers
Keen to encourage a culture of experimentation among policymakers, Innovation Growth Lab (IGL) has been examining the barriers that prevent its adoption. We found that these barriers include; a reluctance to disrupt the status quo, fears of a backlash if ‘lotteries’ are used to allocate support, or simply that evaluation is considered too late.
At IGL we believe that these barriers can and should be overcome, but the first step in experimentation doesn’t have to be a giant leap in order to yield valuable insights.
Over the last few years, the UK’s Department for Business Energy and Industrial Strategy has launched a number of large scale RCTs testing the economic impact of business advice programmes (e.g Growth Vouchers). Its first foray in experimentation was much more modest, and it offers useful lessons on how other governments can begin introducing policy experimentation.
As part of a wider effort to promote business mentoring, BEIS (then BIS) launched the ‘Get Mentoring’ campaign, which had the aim of training 15,000 new volunteer mentors by December 2012. Despite lots of people registering to become mentors, relatively few were completing the necessary training.
To rectify this, BEIS decided to test whether behavioural insights could be applied to email reminders sent out to volunteers in order to ‘nudge’ them into completing their training. For the first time in this policy area, RCTs were used to learn what worked best.
The process of testing and improving proved to be instrumental in the programme meeting its goal. By the end of December 2012, 15,305 people had completed their Get Mentoring training. The trials directly generated an additional 778 mentors and perhaps more than double that, once the wider application of lessons learnt are included.
The elements that made the messages most effective were not those that we expected prior to the trial. For example, one of the most successful elements was to prime altruism by including the following at the start of the emails:
Adam Smith, the great British social philosopher once said: “To feel much for others, and little for ourselves; to restrain our selfish, and exercise our benevolent affections, constitutes the perfection of human nature.”
The approach also uncovered other interesting differences that are likely to have otherwise gone unknown. For women, the inclusion of messages on anticipated pride only had the same impact as they did for men when combined with an emotional norm (ie others also feel this pride). For men, the addition of emotional norm made no statistically significant difference.
BEIS later explored other areas where smaller trials might provide insights to improve policy.
Policy design: Research to inform efforts to raise demand for business mentoring
This research found that asking businesses to reflect on their growth ambitions, and then telling them that mentors help to spot growth opportunities, was the most effective approach - increasing both their attitude toward mentoring and motivation to get a mentor.
Raising awareness: Identifying effective ways to raise awareness of policies to support enterprise
A series of trials tested simple tweaks to improve engagement with an email newsletter on available support. For example, opening rates increased by 3.9 per cent just by inviting businesses to reflect on their growth ambitions in the subject line (“Realise your hopes and aspirations”), but highlighting that support was “free” completely undermined this. Possibly due to a lack of perceived value from free support or perhaps simply as the term triggered spam filters (human or automated).
Boost applications: For the Growth Vouchers Programme trials were run to find ways of encouraging applications
A common challenge for business support programmes is making sure that the right businesses are aware of the programme and motivated to apply – not easy, given demands on their attention and time.
As part of the Growth Vouchers Programme (a joint project with the Behavioural Insights Team), trials were conducted to explore how emails could be used to boost applications. Overall, the email led to 9,000 additional applications to the Growth Vouchers Programme, and provided BEIS with valuable information on how we could most effectively communicate with small firms.
As outlined, small rapid experiments can provide quick and actionable insights. They will, however, typically come up short when it comes to providing evidence of more tangible economic benefits.
Returning to the initial mentoring example, are those mentors who are ‘nudged’ into completing their training, as active and effective as those who would always have completed their training? Similarly, once a business is tipped into applying for a programme by a change to marketing messages, will it actually go on and use, implement and benefit from the support to the same extent as other applicants?
This is where fully developed RCTs come into play - but travelling that far (even with the support of IGL) may be easier if you start with a small step.
This blog was originally published on the Innovation Growth Lab website
