The pandemic underlined the fact that EdTech is useful to teachers, pupils and parents. We have seen examples of technology making it easier for teachers to adapt their teaching for different abilities, for pupils to remind themselves of key concepts while doing their homework, or for parents to access relevant information to support their children. But too often EdTech fails to live up to expectations.

Lack of evidence is a key part of this problem. Reliable and relevant evidence enables better decision making – by school leaders, EdTech CEOs, investors and government. But evidence is often missing and, when armed with the evidence we do have, it can be difficult to identify clear and specific implications for educational practice in schools (Higgins et al., 2012).

Our partner EdTech organisations cite obstacles that make evidence generation difficult. They struggle to prioritise resources or necessary expertise, see wide variation in implementation, and EdTech tools evolve quickly— meaning interventions are rarely static.

But the overwhelming conclusion from our work is that generating evidence is worth it. In particular, two lessons stand out.

Experimental evaluation: comparing something against a control is often more feasible than it may first appear. Online platforms can facilitate A/B testing relatively easily. For example, working with Hegarty Maths, an online maths platform, and the Behavioural Insights Team, we used LaunchDarkly software to set up A/B experiments to test potential improvements. Some of the improvements had a positive impact, while others had a negative one. For example, introducing streaks that indicated when students answered questions correctly in a row actually led to lower student engagement. This underlines the importance of experimentation to understand whether a piece of technology is actually having the effect we expect it to.

Product use: there’s value in understanding exactly how products are used by teachers and students through process evaluations. For example, Pobble, a platform to support writing in primary schools, increased the number of teachers attending training webinars after learning that teachers wanted more flexibility in the format and timing of webinars.

There is growing recognition that we need more evidence about EdTech – and organisations such as EdTech Impact, The EdTech Evidence Group and EDUCATE are playing an important role – but there is more to be done.

EdTech platforms generate and collect huge amounts of data which usually involves no additional effort on the part of school staff. But these data are often underutilised. Data varies from platform to platform, but can include information about how the platform is used (eg, time spent on the platform), teacher or pupil outcomes (eg, attainment) and pupil characteristics (eg, eligibility for free school meals).

There were two commonly cited reasons among EdTech organisations participating in the EdTech Innovation Programme for not investing more time in making use of platform data. First, EdTech organisations were unsure exactly how their data could benefit them, and so found it difficult to make the case to invest time or resources in data analysis. Second, some organisations were concerned about data privacy and GDPR compliance. As one EdTech provider summarised:

“We routinely delete all our data because we don’t see benefits of keeping it, and we want to avoid potential headaches relating to data privacy and security.” (EdTech CEO)

We worked with EdTech providers to overcome some of these barriers and demonstrate the value of EdTech data. The uses of thisese data fell into one of two categories.

Product improvement and evaluation: Colleagues from Nesta’s Data Analytics Practice worked with MEI Integral, a tool that supports secondary school and college students with maths, to build an aggregated dataset through which it is possible to track progress and usage without the need to store personal data. This enabled MEI Integral to analyse their data to track the impact of platform developments for the first time.

Sector wide insights: During the pandemic Nesta collaborated with SchoolDash to analyse data from several EdTech platforms (which volunteered their data anonymously for free) being used widely by pupils in England. By combining this platform data with publicly available data (eg, Ofsted ratings, Income Deprivation Affecting Children Index, the DfE’s pandemic device rollout scheme, etc.) we were able to better understand what was happening in schools and how trends changed over time. Were more teachers using EdTech platforms? Yes, although there were signs of remote learning fatigue, too; (as can be seen in figure 1 of our study into Edtech useage). Were more children using laptops rather than mobiles to access EdTech platforms? Yes, the device disadvantage gap narrowed; (as seen in figure 3 of our study into EdTech useage). Using EdTech platform data in this way is a potentially rich vein for future research.

Process evaluations of six EdTech tools carried out by ImpactEd through the EdTech R&D Programme revealed that how EdTech tools were used varied hugely from classroom to classroom. For example, where one teacher was using a product as a homework tool, another was using the same tool to introduce students to new concepts in the classroom.

This might suggest a well-designed EdTech tool that can be used flexibly in different ways. But it might also mean that an EdTech tool is effective in one setting but not in another. As the EEF’s 2012 evidence review of digital technologies concluded, “it is not whether technology is used (or not) which makes the difference, but how well the technology is used to support teaching and learning” (Higgins et al, 2012).

More impact evaluation is required to compare EdTech tools being implemented in different ways across different contexts. But, there are three things individual schools/colleges and EdTech companies can do to better understand or improve implementation.

First, participants in the EdTech Innovation Partnership – schools/colleges and EdTech organisations – reported that they benefited from closer collaboration with each other. For EdTech companies, this meant better understanding how their products were being used. For schools/colleges, this meant feeling more able to ask for advice or support. Second, we saw schools/colleges valuing more onboarding support when adopting a new product. Finally, we saw demand from schools and colleges for professional development training. Importantly, this included training on how to use specific tools as well as on how to develop and implement a wider school and college-wide strategy and vision for technology.

School closures and covid-19 thrust EdTech under a spotlight and we’ve seen an acceleration in the adoption of many tools as schools switched to remote learning. While the negative impact of covid-19 on young people has been severe, technology has been a lifeline connecting teachers, children and friends. It’s too early to know the longer-term effects of school closures on the EdTech sector. The increased recognition of its potential as an essential tool for teaching and learning is welcome.

In addition, the last two years have shown us that not all children experience Edtech equally. Access to the hardware required to use EdTech tools is one part of a digital divide, and the government’s distribution of 1.3m devices to those in need is a great achievement. Our analysis of usage data from three EdTech platforms saw a significant shift between November 2020 and February 2021 as children in more disadvantaged areas become more likely over time to use a laptop (rather than a mobile device) to access platforms. But even as children in disadvantaged areas access devices, the same analysis revealed a consistent ‘engagement gap’ between those children and those in areas of less deprivation. This underlines that there are further barriers to fairer distribution of the benefits of EdTech. It’s likely that these are more challenging to overcome, encompassing context-specific improvements to implementation, product design and teacher support.

Nesta’s role as a founding partner of the National Tutoring Programme aimed to support tutoring organisations to deliver the best possible online tutoring experience for schools and pupils, in the programme’s first year. With a specific focus on reaching disadvantaged pupils we delivered capacity building to 33 tuition partners, and provided intensive support and funding to five organisations to improve their virtual offering.

First Pass is an online service for formative assessment developed by Bolton College. It enables teachers to post open-ended questions to students who can then respond to these questions using free-from text. Answers are analysed by the technology, which provides real-time feedback.

Amount awarded: £100,000
Bolton College further developed FirstPass, including the student and teacher facing element of the tool in collaboration with a specialist in Natural Language Processing.

Educake is a formative assessment tool that provides students with thousands of automatically marked written questions, matching the most popular GCSEs, IGCSEs and KS3. Students receive instant on-screen feedback as well as reports on their strengths and weaknesses.

Amount awarded: £89,600
Educake developed features that responded to the urgent needs of remote learning during the covid pandemic.

HegartyMaths is an online maths learning tool with video tutorials and assessments features for pupils and teachers.

Amount awarded: £100,000
Hegartymaths used the funding to hire two new members of staff to deliver training, developed a data tracking system to track teachers’ activity and undertook a correlation study to analyse how teacher activities correlate with students.

Pobble is an educational web-based platform that offers support to teach, improve and assess writing. It provides stories and examples to engage students in writing, as well as tools and resources to practice their writing skills. Teachers can then review and provide feedback to pupils, who can also self-assess their own writing and provide feedback to peers.

Amount awarded: £100,000
Pobble as awarded £100k to develop a variety of individual pupil tools, which were designed to enable pupils to independently access Pobble, self-assess their writing and assess their peers’ writing skills.

Mangahigh is a gamified maths platform for KS1-KS4 maths. Teachers use Mangahigh, in class and for homework, to personalise their instruction based on real-time performance data.

Amount awarded: £100,000
MangaHigh improved the accessibility of its platform by bringing the site to WCAG 2.1 double A standards, by improving the pedagogical content and by introducing text-to-speech for maths content.

The SkillsBuilder Hub is an online learning platform for essential, soft skills. It provides a framework for progress alongside resources and assessment tools.

Amount awarded: £99,765
Skillsbuilder improved their user experience to more effectively guide teachers to formative assessment and teaching materials and demonstrated the accuracy of class-level assessment through regression analysis with individual students.

Seneca is an online teaching and assessment tool that allows students to review their knowledge for KS2, KS3, GCSEs and A Level exams. Seneca uses its algorithm to give each child an individualised learning experience, depending on their current knowledge.

Amount awarded: £100,000
Seneca developed two new features to the Seneca online tool: Smart Homework and Diagnostic Misconceptions.

The University of Oxford/Wollongong have produced the Early Years Toolbox (EYT) numbers app, which is a collection of freely available, brief and playful assessments of children’s developmental progress.

Amount awarded: £99,832
The University of Oxford implemented, tested and evaluated the use of the EYT Numbers App in primary schools and developed a platform that allows teachers to aggregate results for their class or school, giving them an additional layer of analysis beyond individual child data.

Freeflowinfo is a parental engagement platform that enables all parents, regardless of their background and the age of the pupil, to establish and maintain positive attitudes towards learning at home. It also reduces teacher workload.

Amount awarded: £99,400
Freeflowinfo undertook research in secondary schools, developed a online teacher training resources to support workload reduction and delivered school leaders information sessions.

Firefly is a parental engagement and learning hub that brings together files, information and trackers in a single portal for students, parents and teachers.
Amount awarded: £100,000
Firefly improved its parental engagement functionality through a new 360 profile function for each student – Parent Portal and launched a Digital Maturity Diagnostic to help schools assess their digital capabilities.

Studybugs is a parental engagement tool focused on attendance and symptom tracking.

Amount awarded: £99,765
Studybugs built ‘ParentTime’, an integrated video-conferencing app that supports teachers to organise and manage short video-calls with parents efficiently.

Timetabler is a timetabling software used by schools and colleges to create more efficient timetables in a time-efficient way.

Amount awarded: £38,975
October resolution investigated and developed tools and solutions that can assist school timetablers to achieve quality timetabling, taking into consideration the challenges that may be posed by flexible working arrangements.

Edval is a timetabling software tool used by schools and colleges to schedule their timetable, allow students to select subjects, schedule parent-teacher interviews and support daily organisation in schools and colleges.

Amount awarded: £100,000
Edval developed a tool to provide new flexible working features and tested them with pilot schools to gather feedback and further improve the tool.

No More Marking is an online platform that enables schools to use comparative judgement to assess student's written work.

Amount awarded: £63,768
No More Marking ran a research project to validate comparative judgement as a method of marking.

TextHelp created WriQ which provides automated marking of student writing, assessing spelling, grammar and punctuation to provide an accuracy score, with teachers able to give individualised feedback.

Amount awarded: £87,000
Texthelp improved links with the UK curriculum and increased the efficacy of WriQ in UK pilot schools. It also improved the Teacher UI Design through engaging with a teacher advisory board and gathering feedback.

Integral is a product developed by MEI which offers high quality support for AS- and A-level Maths and Further Maths.

Amount awarded: £70,000
Evaluation focus: Key Stage 5 pupils eligible for free school meals

MEI has used funding to improve the mobile device functionality of Integral, to better support home e-learning. The organisation has also explored offline access and improving teacher training and implementation for specific student needs, as well as producing a new set of interactive resources specifically designed to support A-level mathematics.

Learning With Parents aims to narrow the disadvantage gap by supporting all parents and carers to engage in their children’s learning.

Amount awarded: £85,000
Evaluation focus: Key Stage 1 pupils eligible for pupil premium
Learning With Parents worked on making its programme more accessible for all parents by introducing phone number logins, and exploring the use of text message nudges to increase parental engagement.

Pobble provides teachers with everything they need to teach, improve and assess writing.

Amount awarded: £70,000
Evaluation focus: Key Stage 2 pupils with English as an additional language
Pobble focused on improvements to the accessibility of the platform, particularly through translation functionality and provision of multilingual training resources for children and parents.

Seneca is a homework and revision platform covering more than 95 percent of the exams taken in mainstream UK education system for KS2, KS3, GCSE and A-level curriculum.

Amount awarded: £40,000
Evaluation focus: Years 9 and 10 pupils eligible for pupil premium
Seneca developed its Smart Assignment feature and introduced a new reward system to foster independent learning, particularly for pupil premium students.

Sumdog is an online game-based maths and English practice tool.

Amount awarded: £85,000
Evaluation focus: Key Stage 2 pupils eligible for pupil premium
Sumdog has developed an assessment library and a reporting feature to help teachers understand student progress quickly, with the aim of enabling teachers to direct their time and any catch-up support more effectively.

Texthelp have focused on a tool called WriQ, which provides tracking and automated marking of student writing, assessment of spelling, punctuation and grammar and allows teachers to provide personalised feedback along with objective marking through rubrics.

Amount awarded: £70,000
Evaluation focus: Key Stage 2 to Key Stage 5 pupils with English as an additional language/second language
WriQ has a set of new features such as audio-feedback and alternative-word suggestions designed to support EAL students to build their vocabulary.