A story of reinventing research and innovation indicators
www.nesta.org.uk/blog/story-reinventing-research-and-innovation-indicators/
www.nesta.org.uk/blog/story-reinventing-research-and-innovation-indicators/
EURITO is a new Horizon 2020 project where we are working with researchers in DTU (Denmark), Cotec (Spain) and Fraunhofer FOKUS (Germany) to transform Research and Innovation (R&I) policy with new data. This post dramatises the rationale for the project and its scoping phase into a fictional story involving a senior R&I policy maker and a Policy Analyst in an unspecified European country. You can find more information about the project and its emerging findings at the bottom of the page.
Mary watched as the television screen flickered with images of demonstrations over rising inequality—images that had become far too familiar in recent years—and closed her eyes to think about how they’d arrived there. In the darkness, she could hear the reporter citing an excerpt from the recently released 2018 Science, Research and Innovation of the EU report:
Over the last decade, developed economies in Europe and elsewhere have faced two major trends with important implications for the well-being of their societies: a slowdown in productivity growth and increasing inequality.
Mary had been working in the federal government for over 25 years, and had been appointed to head up the Ministry of Research and Innovation (R&I) when it was established over 10 years ago. Her career had seen the blurring of lines between industries, the rapid emergence of new fields, and the explosion of sharing and gig economies characterised by companies like Airbnb and Uber. Globalisation had altered the way products are developed, making them difficult to attribute to any one country, and a push toward open borders in Europe had led to unprecedented mobility of researchers. There had also been a profound shift in the actors participating in innovation, with non-governmental bodies, public sector organisations and governments themselves getting involved. The rate at which things were changing was dizzying, even for someone whose job it was to foster change.
Her attention snapped back to the present when her daughter, visiting home on a break from university, pointed at the screen and asked pointedly “what is the government doing about this? In my economics class we’ve been learning about the role of research and innovation in a healthy economy.” Mary thought back to recent meetings where there had been a lot of buzz around concepts like ‘mission-orientation’ and ‘responsible research and innovation’, although it wasn’t always clear how these concepts translated to practice or how they would affect policies moving forward. One thing was clear, however—the goal of research and innovation had moved beyond a focus on metrics like GDP, and people were starting to seriously ask how to build a fairer, more inclusive economy.
Mary looked back at her daughter, and sighed, “it’s complicated”. Knowing that would be an unsatisfactory response, she continued, “so many things have changed in recent years, and part of the challenge is that the data we use doesn’t always give us a clear view of what’s really going on. New industries keep appearing, and they do things in ways we don’t measure. The data we’ve been using for a long time doesn’t always seem to tell us what we need to know.” As if it were a simple question with a simple answer, her daughter continued casually, “so, what are you going to do to get the information you need then?”.
Mary took a moment to reflect on the early efforts in measuring research and innovation by institutions like The Organisation for Economic Co-operation and Development (OECD), with the Frascati and Oslo Manuals having become core methodological pillars over time. Sets of internationally comparable indicators compiled in comprehensive reports like the European Innovation Scoreboard, with data drawn from official offices such as Eurostat, the European Union Intellectual Property Office (EUIPO) and the World Intellectual Property Office (WIPO) had been so central to key policy decisions over the years. But, perhaps her daughter was right—maybe they did need to think about whether new, complementary avenues to get the information they needed exist. “Ok, I’ll see if there’s someone at the office who might have some ideas about this”.
Edgar was nervous. Why had this senior policymaker asked him for a meeting? Hopefully he hadn’t done anything wrong. Edgar had been in his role as a Policy Analyst in the Ministry of Research and Innovation for just under a year. After completing his studies with a double major in data science and economics before going on to complete a master’s in political science, he had rejected several tempting job offers from high-profile tech firms. Edgar’s father had been a committed civil servant and he wanted to carry on the legacy despite feeling a strong pull toward the fast-moving flow of tech and the digital economy. He thought that of the various government ministries, this would be the most likely place to satisfy both sides of the equation.
However, Edgar quickly found that his data science skills were something of a rarity in government and since arriving in his new post he hadn’t had many opportunities to put them to use. He found it curious that the Ministry was using outdated, proprietary software to analyse their data, and that basically all of the data used by the policymakers was bound up in enormous reports that often presented statistics from several years ago. He thought back to a meeting he attended the previous week, where research and development (R&D) expenditure data more than two years old were informing key decisions! Surely things had changed since then, he thought.
Soon after taking up his post, Edgar had started wondering whether this was the case everywhere. Looking around online, it seemed that lots of one-off studies or ad hoc work had been done around the use of big and new data in the policy space, but nothing appeared to have really caught on at a larger scale in R&I policy, despite several encouraging reports on the topic having been published in recent years. Seeing what he thought was an opportunity to contribute, Edgar had spent several days drafting a report on the potential that new data sources and analytics could bring to the R&I policy cycle—from agenda setting to evaluation. In a swift blow to his confidence and motivation, his line manager summarily dismissed the report… Perhaps he should go back and see if one of the tech firms was still hiring, he thought to himself, staring out the window at the bustling city below.
Lost in thought, Edgar’s nervousness immediately kicked back in when Mary walked into the meeting room with a copy of his report in hand. To his surprise, Mary greeted him with a warm smile and asked him to sit down. “I had a look at your report” she started “and I’d be very interested in hearing more about how we might be able to use some of these new data sources and analytics you describe.”
Surprised by this unexpected window of opportunity, Edgar didn’t hesitate to dive directly to the heart of it. “We’re seeing an explosion of new data sources that could be used to tell us something about the research and innovation landscape and its actors”, he started. There are new datasets like CrunchBasethat we can use to look at the startup landscape. We can scrape web data to find out what companies are working on and whether they’re innovating. We can explore open digital innovation using GitHub—a website where programmers collaborate and save their work. We could analyse trends in social media platforms like Twitter to see what people are talking about, and then compare this against what research is being funded in areas like climate change. Techniques like machine learning and text mining are opening up new possibilities for analytics that have tremendous potential.” He stopped to catch his breath. “By providing more granularity and timeliness in our assessments, new data will not only help us to answer questions we’ve already been asking, but may be able to help us answer important questions about the inclusiveness of the economy.”
Mary thought back to the morning’s newscast. Perhaps this was an important piece of the puzzle? She couldn’t help but be encouraged by Edgar’s enthusiasm, and she did see the clear relevance of some of his suggestions—but she also had some reservations. “I can see that there’s a lot going on in the data and analytics space, but are these data going to remain stable enough over time to be helpful for policy decisions? Are some of these data sources even going to exist in the coming years? An important advantage to, say, the data we use to analyse trends in research, is that we can trust it. It’s relatively stable over time. We can’t know how long Twitter will be around for, or whether people will change the way they use it. Plus, I’ve heard people talking about ‘bots’ that could skew any analysis we might do”. She continued, “it’s not just about stability of the data over time, it’s also about comparability. An important part of my job is being able to compare our performance with other countries. Are we going to be able to use these data to make meaningful comparisons at the level of the EU, for example?”.
These were challenges that Edgar had seen hotly debated in the literature he’d reviewed on the use of new data for policy. “I’m not going to pretend to have all of the answers to these difficult and important questions”, he started. “Stability over time and comparability across countries are real challenges—it’s true that some of these new data sources don’t cover all of the countries in Europe, and even fewer cover multiple regions of the world. But some of the challenges that made this sort of work difficult even a couple of years ago have now been overcome, and I’m optimistic that we’ll continue to see rapid progress if we work toward finding solutions. For example, a few years ago if you wanted to compare policies on, say, R&D tax credits from multiple countries by scraping the information from government websites, the fact that the documents are all published in different languages was a real barrier. Over the past couple of years, there have been impressive developments in our ability to automatically translate text in a reliable way using tools such as Google Translate.” He continued “another promising way to validate new data sources and make sure they’re trusted is to link them with, or cross-reference them against, more ‘traditional’ sources, such as existing indicators coming from surveys”. Mary looked interested, but not entirely convinced.
“Let’s take a moment to come back to the opportunities”, Edgar continued. “Tell me, if you could have data on anything, what would you want? How would you use it to make policy decisions?” It was Mary’s turn to share all of her ideas, and she spoke with the same enthusiasm that Edgar had about the data opportunities. “Better data for evaluations”, she started. “We want to understand the impacts of the policies and programmes we’re implementing but existing data make it hard for us to effectively link inputs and activities to ultimate impacts.” She continued, “we also want to better understand the dynamics of knowledge transfer, business innovation, and emerging technologies that might drastically affect people’s jobs and the economy overall.” Digging further into the idea of linkages, she continued “we want to have a better understanding of complex systems and their dynamics—we want to move beyond individual, stand-alone indicators that struggle to convey the whole story”.
Smiling widely, Edgar quickly opened his laptop and pulled up on the wide screen at the front of the conference room a large network diagram full of bright colours, nodes and interconnected lines. “In order to convey complex information, we’ll need to rethink the way we present data”, he said excitedly. “Right now, most of the data we’re using is presented in a static format inside of a long report. What if we could take advantage of new, interactive tools that give the user more control over the types of questions asked, and the way the information is presented back? Some interactive tools have already started to map the innovation landscape in certain places, allowing us to look at a wide variety of dynamics, such as networks of collaborations between universities and private companies. This tool can even recommend collaborations, and it’s completely free and open for anyone to use.”
He looked eagerly at Mary, awaiting a response. Silence, and then “what on earth am I looking at?.” Edgar wasn’t sure how to respond—“it’s… a network diagram.” Mary laughed, “yes, I gathered as much. It looks very nice, but I’m not sure I prefer it to a bar chart.” She paused before continuing “but maybe I just need some time to get used to it, or to see several examples of these side by side in order to make sense of what it’s telling me”. Edgar breathed a sigh of relief and smiled cautiously, wanting to preserve the collaborative and open spirit that had characterised their conversation. Looking at her watch and then back at the screen, Mary closed the meeting, shaking hands with Edward and smiling warmly before saying “that’s a good start for today—if we can get it right, this has a lot of potential. Let’s see where we can take it”.
This blog is part of EURITO’s scoping phase activities, and communicates findings from the “Literature review report: The role of data in the R&I policy cycle” and the first Policy Stakeholder Workshop held on 17 April 2018. To explore in more detail, please refer to:
We encourage policymakers, researchers, industry actors and the public involved in R&I policy and those with a general interest in big data for decision-making to reach out. You can do this by subscribing to our newsletter or following updates on the EURITO page, Medium and social media channels. You can also reach out to the Consortium directly by emailing [email protected].
This project has received funding from the European Union's Horizon 2020 research and innovation programme under Grant Agreement No. 770420 – EURITO.
Disclaimer: The contents of this blog are the sole responsibility of the authors and can in no way be taken to reflect the views of the European Union.
