EdTech usage during the pandemic: A story in four graphs
www.nesta.org.uk/project-updates/edtech-usage-story-four-graphs/
www.nesta.org.uk/project-updates/edtech-usage-story-four-graphs/
As schools have been disrupted by the COVID-19 pandemic over the last year, reliance on education technology (EdTech) has increased. For many, it’s been a lifeline - helping with remote teaching and assessment, or connecting children with friends and support - but not all children’s experience of the pandemic and remote learning have been equal.
As part of a wider partnership with the Department for Education to support more effective use of technology, Nesta and SchoolDash have collaborated to analyse usage data in England from three EdTech industry partners, spanning from January 2020 (before the first national school closures) to March 2021. We set out to understand more about how children’s experience of the pandemic and EdTech have varied, particularly for more disadvantaged children. The analysis tells a story about EdTech during the pandemic in four graphs:
Data from three different products has been explored. In order to respect their confidentiality, they are not named in this report - but we are hugely grateful to them for sharing this data with us to further our understanding of EdTech during the pandemic. This analysis only includes data from England, and so any references to schools closing/reopening refers to lockdown measures in England specifically. All three tools are used mainly in primary schools, but have the following broad characteristics:
Data for Tool 3 is not included within the visualisations below, as we only had access to data from a smaller sample of schools, covering a shorter period. Where analysis of this data does/does not support findings of wider analysis this is indicated in the text.
Use of technology increased significantly as COVID-19 caused partial school closures. Figure 1 shows a pattern where each new national school closure is marked by a spike in activity (teacher sign-ups were over 10 and four times higher at the start of the March 2020 closures than the pre-closures average for tools 1 and 2 respectively). This supports previous analysis of four EdTech platforms which saw increases in usage of between 2 and 7 times pre-closures average.
Figure 1: Relative weekly usage of Tool 1 and 2
Each spike in activity is followed by a slow decline - both in the rate of new users, but also measures of activity such as ‘challenges set’, ‘challenges done’ and ‘lessons saved’. Focus groups carried out with users during the Spring Term 2021 referenced ‘remote learning fatigue’ among pupils and parents, which might account for some of this decline in use. Despite this, usage remains higher than pre-COVID averages for Tool 1 across the year, and higher for Tool 2 apart from during the Autumn Term 2020. This trend is reflected in the data available from Tool 3, where the total number of parent and teacher logins declined during January and February 2021.
We can gain an overview of relative adoption in different categories of schools by measuring teacher-facing activity, such as ‘challenges set’ and ‘lessons saved’. Comparing teacher-facing activity across schools with different proportions of children eligible for free school meals (FSM) reveals changes over time in who is likely to be using Tools 1 and 2 [Figure 2].
Figure 2. : Relative adoption in schools by deprivation level
If teacher activity was evenly distributed across all children, we would expect to see activity in High FSM schools account for around 11 per cent of total teacher activity (the percentage of all children in High FSM schools). Instead, we see teacher activity in High FSM schools move from being under-represented during the first school closures in Spring 2020 (8.2 per cent and 4 per cent of teacher usage in mid-April 2020 for Tools 1 and 2 respectively), to over-represented by the second school closures in Spring 2021 (16.6 per cent and 16.3 per cent in early February 2021 for Tools 1 and 2 respectively). Overall usage of Tool 1 and 2 grew during the period in all school categories, so we can be confident that this trend reflects an increase of use in High FSM schools rather than a decrease elsewhere.
This change in adoption patterns means children attending schools with higher levels of disadvantage were, on average, less likely to use Tools 1 and 2 for remote learning at the beginning of the pandemic. But, by the time the second set of school closures was introduced in Spring 2021, this was reversed.
Access to appropriate hardware devices has been a major aspect of efforts to support disadvantaged children to learn remotely. The Sutton Trust reported that when school closures were first introduced, just 5 per cent of teachers in state schools reported that all their pupils had access to an appropriate device for remote learning, compared to 54 per cent at private schools.
Analysis of device data for Tool 1 tells a story of progress [Figure 3]. During the first school closures in Spring 2020, children in High FSM schools are less likely to access the platform from a desktop or laptop computer than children in Low or Medium FSM schools, and in July we can see a clear ‘device disadvantage gap’ of 9.9 per cent. However, by the second national school closures in Spring 2021, this gap has narrowed significantly to 0.4 per cent in February 2021.
Figure 3: The proportion of usage on a desktop or laptop computer by school deprivation level
Some of this shift may be the results of the Department for Education’s ‘Get Help with Technology during Coronavirus’ effort to dispatch laptops and tablets to children and families without access (rollout indicated by the grey line in Figure 3). As the government’s device distribution accelerates from November 2020 to February 2021, the ‘device disadvantage gap’ among Tool 1 users narrows. We should note that there were many other efforts going on (from individual school fundraising to local campaigns involving community groups and businesses) to help get devices in the hands of children who needed them.
If analysis of the ‘device disadvantage gap’ describes a modest success, investigating measures of engagement with Tools 1 and 2 tell a more complicated story and suggest that there are significant barriers (besides access to devices) that continue to disproportionately impact disadvantaged children. To try and understand relative engagement with platforms, we have measured the amount of pupil or parent (for Tools 1 and 2 respectively) activity per teacher login - ie. the amount of engagement teachers get back for the same amount of platform activity they put in.
Figure 4: Pupil/parents activity per Teacher login by school deprivation level and by Ofsted rating (a proxy for ‘engagement’)
Figure 4: Pupil/parents activity per Teacher login by school deprivation level and by Ofsted rating (a proxy for ‘engagement’)
Figure 4 shows lower numbers of student and parent logins (for Tools 1 and 2 respectively) per teacher login in schools with higher levels of deprivation. Importantly, this gap does not close over time and the ‘engagement gap’ remains persistent. We can also see a similar gap (although slightly narrower) when we compare schools by Ofsted ratings. Data from Tool 3 covering January to March 2021 suggests a similar ‘engagement gap’, with parents or carers of pupil premium pupils less likely to make comments through the platform than those of non-pupil premium pupils.
This analysis provides a glimpse of how children’s experiences of EdTech during the pandemic have varied, revealing clear differences. However, there are reasons for cautious optimism - particularly with changes in the relative adoption among schools with higher deprivation levels, and the narrowing of the device disadvantage gap. We can see the results of progress towards getting the fundamental components of more equitable digital learning in place - such as the availability of EdTech platforms for teachers and children who want them, and the devices on which to access them.
However, consistent gaps in measures of engagement confirm that barriers to fairer distribution of the benefits of EdTech remain. We’re exploring these barriers through the EdTech R&D Programme - a partnership with the Department for Education, ImpactEd, 65 schools, the Teacher Development Trust and six widely used EdTech tools to develop and test improvements designed to support cohorts of children with particular barriers to digital learning. It is likely that these further barriers are more challenging, encompassing context-specific improvements to implementation, product design and teacher support. We look forward to sharing our findings from this work in the Autumn.
