Back in 2011, when we first started making investments from our fund, our starting point was to look for scalable business models that created what we might call direct impact. That is, there is a direct point of contact between the product or service in question (the intervention) and the people it is intended to benefit (the beneficiaries, to use an imperfect but useful term). From our portfolio, Sumdog is a great example of direct impact: Sumdog provides a gamified online maths platform (the intervention), which is used by school children, whose maths skills hopefully improve as a result.

Having started with this approach, one thing that we have learned is that opportunities to make a difference don’t always fall easily into this basket, and that models with direct impact only make up a portion of the range of possible impact investment opportunities. There are plenty of other inspiring and exciting businesses that arguably create a positive impact, but it is indirect impact. We have made some investments that move in this direction. Arbor, one of our investees, provides a management information system (MIS) for schools. The MIS is used by teachers and administrators, which ultimately allows them to do a better job than they were before, which leads to a better education for the students at the school. This is an example of indirect impact – the ultimate beneficiaries, the students, only experience the impact because their teachers, thanks to Arbor, can change aspects of the way they do their jobs.

However, there are plenty of investment opportunities that are appealing to us where the impact is even more indirect or difficult to define. This blog puts forward some ideas for moving this conversation on a step.

For our team and for Nesta, we want to ensure we are making the most of the capital we have. We do not want to be too restrictive in our criteria for impact, such that we miss opportunities to make a huge, but less easily evidenced difference, but we also do not want to open the criteria so wide that anything counts.

The implications go far beyond Nesta: this is about the risk of ‘impact washing’. The broader impact investing industry is very aware of this risk. If we are not clear and precise about what we mean by ‘impact’ then this growing market will simply be bloated by people using the language but not putting anything substantial behind their claims.

We have come up with a simple way of visualising the difference between direct and indirect impact.

The centre of the diagram is the company’s activities. The first ring out represents group A – the people who interact directly with the company’s product or service, and do something differently as a result. The second ring out represents group B – the people who are affected by the change in behaviour of group A.

Sumdog’s impact, therefore, as an example of direct impact, is represented just with the first ring out.

With Arbor, in contrast, we need both rings:

Crucially, this diagram tells us something about impact measurement: the further from the centre a metric is being collected, the more noisy and less reliable it will be.

This is not to say that Sumdog has an easy task generating evidence that its product is effective – this is always challenging. But it is more straightforward for Sumdog to investigate this question than it is for Arbor to draw connections between the use of its MIS and the attainment of the pupils at the schools. While it is conceivable that Arbor could make so much difference to the quality of teaching at a school that it improves student experience, there are so many other factors that feed into student experience that it is incredibly difficult to isolate the effect of improving the MIS. But our reason for investing - the ‘impact return’ we hope to achieve through this investment - is ultimately about improved education for students. It is worth noting that existing research, where it is robust and conducted by reputable organisations, is a valid source of information when thinking about these issues. In the case of Arbor we reviewed evidence around the problem of teacher workload, and the effectiveness of giving teachers access to better quality data, to help us understand the likelihood of their achieving the intended impact.

The target diagram therefore helps us to represent differences in types of impact within our portfolio. But it also helps us to be specific about the challenges that come with understanding indirect impact. In other words, it gives us a way of structuring our thinking about impact when there is no clear line through to a beneficiary group.

Using the target diagram, we can see that the problem of defining indirect impact largely stems from the lack of information about the groups beyond group A.

Challenge 1: putting boundaries on group B

In the Arbor example, it was very clear who is in group B: it is the students in the school that has implemented Arbor’s MIS. In this case, we do at least know who is in this group, and can find out a bit about them.

Sometimes, it is much more difficult to define group B. Consider a company that uses artificial intelligence to improve the quality of content online. Their product is paid for and used by websites, who are better able to monitor the content on their sites. In this case, group A is the people running the websites, who use the product to do their jobs more effectively. Group B are the people who use the website. This group is much harder to define than a list of pupils at a school. This scenario can be visualised like this:

In this case, measurement is made very challenging by the unmanageability of group B. If we do not know who the people in group B are then it is very difficult to find out what impact they are experiencing. If some data is available, it is difficult to know whether it is representative of group B as a wider population.

Challenge 2: the impact is even more indirect

Another possibility we have encountered is where a venture does something that requires several ‘rings’ or layers of people to change their behaviour before the ultimate impact is eventually achieved. Consider, for example, a company that develops a platform which brings together large pharmaceutical companies with the smaller start-ups that specialise in drug discovery. Ultimately, this venture could contribute to the more rapid and efficient development of drugs, which in turn would save people’s lives.

In this case, there is a very long ‘chain’ of behaviour change – the startups use the platform to find the pharmaceutical companies, the drugs get taken through multiple further stages of development before finally being approved and made available to patients, for whom they may or may not work, depending in part on the behaviour of the patients themselves. A situation like this can be represented as follows:

Though we might be excited about the possibility of improving the success rate of drug development, and ultimately saving lives, we have little hope of really being able to connect such changes to the venture we are investing in. Any metrics collected with group D will reflect a situation that has been affected by so many other things they will be virtually meaningless in telling us anything about the effectiveness of the venture’s activities.

These diagrams help to lay out our options more clearly.

As an investor, the first option is to see the feasibility of impact measurement as central to the definition of impact. That is, if impact is so indirect that it is hard to even imagine how it would be measured, then we say that this does not really count as an impact investment, This would be the most rigorous way of defining what counts as impact, and also one of the most restrictive. This would mean sticking to investment opportunities like Sumdog.

The second option is to say that we want to look at indirect impact opportunities, and that we should do what we can to identify metrics, even though they are imperfect. We might frame this in terms of ‘impact risk’ - the difficulty of measuring indirect impact makes it a higher risk option, but we accept this risk because we believe in what the company is trying to do. This would mean expanding our scope to companies like the drug development platform, even though we know it will be very difficult to tell whether the ultimate impact is being achieved.

Both of these options tie the idea of impact together with the idea of measurement.

There is a third option: we can pull these ideas apart and think of impact separately from measurement.

While metrics collected in the outer rings of the target diagram are noisy and less meaningful, metrics collected near to the centre will be clearer and more meaningful. In other words, the closer to the company’s activities we get, the more metrics will tell us. This is because the centre of the target diagram is where the company has control, influence and the ability to change things. Further out, where we are looking at changes in people’s behaviour several steps removed from the company’s activities, they have far less control and influence.

We can therefore turn the question of impact around. Instead of focusing only on the outcomes the company is trying to achieve, how about looking at whether their intended impact is informing the way they are running their business?

For example, the company that provides software that helps clients monitor online content will find it difficult to know whether internet users ultimately have a better experience of using the internet. If we focus only on impact measurement, we might end up trying to collect a few metrics that don’t tell us very much.

If we turn the question of impact around, we start asking: what are the things the company can do to run its business in a way that maximises the chances of creating the biggest impact? This might include things like

• Ensuring they are clear about communicating what their intended impact is to clients, so that their clients are on board with these efforts.
• Asking clients to share data that helps them understand their impact, where possible
• Hiring people who have the skills needed to understand impact, even if there are no external demands for this information
• Bulking up the customer support function to ensure customers are really using the product to its maximum potential - even if this means going beyond what is necessary to keep customers happy

Some of these might be tracked through metrics, but they are more likely to be points of discussion, or changes that we would expect to see in their business plan.

The point here is not to give up on the importance or worth of measuring impact. We will always push our companies to work to generate whatever insight they can. The point is to say that there is much more that can be done beyond focusing on measuring impact among (sometimes very far removed) beneficiaries.

The companies we are working with are balancing commercial and impact imperatives. We work with them to understand how their plans for growth and commercial success tie together with their plans to create impact. We look for alignment between these things, but there is always more that can be done to make sure the company’s decisions are focused on impact as a part of the process, not just as an outcome.