Experimentation is very common in the private sector. Companies such as Google and Amazon regularly run online A/B tests to identify what works best on their platforms. It is not as straightforward to do this in government as it is on a website, but it is possible – especially when it comes to testing communications. Local authorities send out lots of letters and other communications. We think that they should be aiming to adopt a culture of regular experimentation, trying out different versions and seeing what works best.

In the first part of this module, we lay out the reasoning for why it is important to evaluate policy initiatives at all. The second section goes through the steps of how to do this in practice.

Randomised controlled trials (RCTs)

The most reliable way to know whether or not a policy initiative is working is to conduct a randomised controlled trial (RCT). They have been common for a long time in medicine and have become more popular in recent years as a way to evaluate government policies.

Historically, the most common way to evaluate a policy was just to implement it and then observe what happened. For example, you might introduce a new programme for unemployed job seekers and then measure how quickly they found work.

This approach has a lot of problems that mean that, at the end of the evaluation, you’re very unlikely to know whether or not the programme worked.

Problem 1: Factors outside your control affect your outcomes. For example, the number of people finding jobs is affected by the state of the wider economy. A sudden recession might make it seem like your policy is ineffective – even if it actually works well.

Problem 2: Selection bias. If you introduce an optional job training programme and compare outcomes for those who choose to use it to those who choose not to, you will get biased data. The people who use the programme are likely to be more motivated and engaged and they might have found jobs more quickly anyway.

The value of RCTs

Randomised controlled trials get around these problems altogether. In a trial, we take a group of people (to stick with the example above, unemployed job seekers) and randomly divide them into groups. One group gets the new job training programme and the other does not. At the end, we can compare outcomes to precisely estimate the impact of the programme. Because the groups are randomly assigned, we can assume any differences between them at the end are because of the programme, and not another factor.